docker devicemapper 存储异常

linux 2020-04-12 245 次浏览 次点赞

docker devicemapper exception

exception

这里记录早期版本docker (1.11.2)在centos 7(linux3.10)devicemapper存储异常,导致docker容器的空间内文件删除后存储空间不能正常释放。

磁盘40G,早期因为日志异常docker使用了30G,后面删除日志后容器容量一致在30G往上涨,慢慢地主机的磁盘完全被消耗掉,但是实际容器使用7G。

# uname -a
Linux iZu1d6y4hk5Z 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@iZu1d6y4hk5Z:~# lsb_release 
LSB Version:    :core-4.1-amd64:core-4.1-noarch
cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
# docker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-202:1-1048580-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 32.21 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 37.63 GB
 Data Space Total: 322.1 GB
 Data Space Available: 1.388 GB
 Metadata Space Used: 22.02 MB
 Metadata Space Total: 4.295 GB
 Metadata Space Available: 1.388 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.389 GiB
Name: iZu1d6y4hk5Z
ID: V5QY:4N5B:O3LO:L57J:DCS5:MN4Y:3RZX:L52W:VBME:6YWO:UDTK:UAXQ
Docker Root Dir: /docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

关于如上描述异常,这里有详细分析描述:

aufs

这里采用aufs,需要说明的是所有更改都需要删除原来容器数据后再次创建。所以这里需要讲当前容器进行备份后恢复。

  • 提交容器最新更改,导出镜像文件;

    #docker commit ilink1 export/ilink1 #提交最近更改;
    #docker save export/ilink1 > ~ilink_bak.tar.gz
  • 删除当前docker 数据;

    #rm -r /docker/
  • 更新非主线的aufs 内核;

    #cd /etc/yum.repo.d
    #wget https://yum.spaceduck.org/kernel-ml-aufs/kernel-ml-aufs.repo
    #yum install kernel-ml-aufs
  • 更改grub后重启,并且检测生效;

    #vi /etc/default/grub
    GRUB_DEFAULT=0
    #grub2-mkconfig -o /boot/grub2/grub.cfg
    #reboot
    提示:这里默认GRUB_DEFAULT=saved表示上次成功启动。
  • 重启后通过/proc/filesystems 确认是否更改生效;

    ~# cat /proc/filesystems |grep aufs
    nodev    aufs
  • 更改docker.service

    ExecStart=/usr/bin/docker daemon --storage-driver=aufs 
  • 安装镜像文件;

    docker load < ~/ilink_bak.tar.gz
    docker run xxx -name ilink1

本文由 lijie 创作,采用 知识共享署名 3.0,可自由转载、引用,但需署名作者且注明文章出处,点赞1

只有地板了

  1. lijie
    lijie

    原来容器从devicemapper迁移到aufs后还存在应用异常。但是看起来过程都没毛病,还需要进一步确认。

添加新评论