docker devicemapper exception

exception

这里记录早期版本docker （1.11.2）在centos 7（linux3.10）devicemapper存储异常，导致docker容器的空间内文件删除后存储空间不能正常释放。

磁盘40G，早期因为日志异常docker使用了30G，后面删除日志后容器容量一致在30G往上涨，慢慢地主机的磁盘完全被消耗掉，但是实际容器使用7G。

# uname -a
Linux iZu1d6y4hk5Z 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@iZu1d6y4hk5Z:~# lsb_release 
LSB Version:    :core-4.1-amd64:core-4.1-noarch

cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
 
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

# docker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-202:1-1048580-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 32.21 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 37.63 GB
 Data Space Total: 322.1 GB
 Data Space Available: 1.388 GB
 Metadata Space Used: 22.02 MB
 Metadata Space Total: 4.295 GB
 Metadata Space Available: 1.388 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.389 GiB
Name: iZu1d6y4hk5Z
ID: V5QY:4N5B:O3LO:L57J:DCS5:MN4Y:3RZX:L52W:VBME:6YWO:UDTK:UAXQ
Docker Root Dir: /docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

关于如上描述异常，这里有详细分析描述：

• Device-mapper does not release free space from removed images #3182
• Friends Don't Let Friends Run Docker on Loopback in Production
• Clean docker environment: devicemapper

aufs

这里采用aufs,需要说明的是所有更改都需要删除原来容器数据后再次创建。所以这里需要讲当前容器进行备份后恢复。

• 提交容器最新更改，导出镜像文件；

  #docker commit ilink1 export/ilink1 #提交最近更改；
  #docker save export/ilink1 > ~ilink_bak.tar.gz

• 删除当前docker 数据；

  #rm -r /docker/

• 更新非主线的aufs 内核；

  #cd /etc/yum.repo.d
  #wget https://yum.spaceduck.org/kernel-ml-aufs/kernel-ml-aufs.repo
  #yum install kernel-ml-aufs

• 更改grub后重启，并且检测生效；

  #vi /etc/default/grub
  GRUB_DEFAULT=0
  #grub2-mkconfig -o /boot/grub2/grub.cfg
  #reboot

  提示：这里默认GRUB_DEFAULT=saved表示上次成功启动。

• 重启后通过/proc/filesystems 确认是否更改生效；

  ~# cat /proc/filesystems |grep aufs
  nodev    aufs

• 更改docker.service

  ExecStart=/usr/bin/docker daemon --storage-driver=aufs 

• 安装镜像文件；

  docker load < ~/ilink_bak.tar.gz

  docker run xxx -name ilink1