问题是这样的:我有一个比较大的docker镜像文件, 在执行 load 时提示“no space left on device”的错误,而实际使用的
/var
目录空间是 60GB 充足的,经过升级内核测试和docker服务端启动参数调整,终于解决了这个问题。下面就是我要分享给大家的处理过程。希望你不要遇到这样的问题哈(镜像文件太大真的不可思议!!!)。
系统运行环境:
主机CPU架构 :X86_64
操作系统版本: RHEL 6.5
Kernel内核版本:2.6.32-431.el6.x86_64
Docker 版本: docker-engine-1.7.1-1.el6.x86_64 graphdriver=devicemapper version=1.7.1
问题说明
- 本地有一个15GB的tensorflow镜像tar文件包, load 到服务端的时候报错导致无法成功加载这个镜像tar包文件,具体错误信息如下:
$ cat /var/log/docker
time="2017-08-16T17:26:19.036130760+08:00" level=info msg="Loading containers: done."
time="2017-08-16T17:26:19.036175169+08:00" level=info msg="Daemon has completed initialization"
time="2017-08-16T17:26:19.036239870+08:00" level=info msg="Docker daemon" commit=786b29d execdriver=native-0.2 graphdriver=devicemapper version=1.7.1
time="2017-08-16T17:26:51.929543993+08:00" level=info msg="POST /v1.19/images/load"
time="2017-08-16T17:31:03.197653849+08:00" level=error msg="Handler for POST /images/load returned error: ApplyLayer exit status 1 stdout: stderr: write /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/execroot/serving/bazel-out/local-fastbuild/bin/tensorflow_serving/servables/tensorflow/simple_servers_test: no space left on device"
time="2017-08-16T17:31:03.197895155+08:00" level=error msg="HTTP Error" err="ApplyLayer exit status 1 stdout: stderr: write /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/execroot/serving/bazel-out/local-fastbuild/bin/tensorflow_serving/servables/tensorflow/simple_servers_test: no space left on device" statusCode=500
time="2017-08-16T17:38:30.359166697+08:00" level=info msg="Processing signal 'terminated'"
\nThu Aug 17 09:40:43 CST 2017\n
解决方案
- 调整docker服务端启动参数(调整含义可以看man docker文档解释),具体操作如下:
vi /etc/sysconfig/docker
## 将 other_args = "" 更改为如下内容
other_args="-s=devicemapper --storage-opt dm.basesize=60G --storage-opt dm.loopmetadatasize=10G"
- 查看调整后的结果
work@host01:/docker/work>
work@host01:/docker/work>ps -ef | grep docker|grep -v grep
/usr/bin/docker -d -s=devicemapper --storage-opt dm.basesize=60G --storage-opt dm.loopmetadatasize=10G
work@host01:/docker/work>
work@host01:/docker/work>docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
tf/master 1.4 db45227e3209 2 weeks ago 15.04 GB