AWS Batch　起動AMIのユーザデータ

確認したAMI

amzn-ami-2018.03.u-amazon-ecs-optimized (ami-0e1aa8c2e9d719f58)

起動AMIから確認。

<br />Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

--==BOUNDARY==
MIME-Version: 1.0
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo ECS_CLUSTER=first-run-compute-environment-minimum_Batch_049f0a53-0a37-37c7-8f7d-9910330bdbec&gt;&gt;/etc/ecs/ecs.config
echo ECS_DISABLE_IMAGE_CLEANUP=false&gt;&gt;/etc/ecs/ecs.config
echo ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=2m&gt;&gt;/etc/ecs/ecs.config
echo ECS_IMAGE_CLEANUP_INTERVAL=10m&gt;&gt;/etc/ecs/ecs.config
echo ECS_IMAGE_MINIMUM_CLEANUP_AGE=10m&gt;&gt;/etc/ecs/ecs.config
echo ECS_NUM_IMAGES_DELETE_PER_CYCLE=5&gt;&gt;/etc/ecs/ecs.config
echo ECS_RESERVED_MEMORY=32&gt;&gt;/etc/ecs/ecs.config

--==BOUNDARY==--

Content-Type: multipart/mixed; boundary="==BOUNDARY=="

MIME-Version: 1.0

--==BOUNDARY==

MIME-Version: 1.0

Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash

echo ECS_CLUSTER=first-run-compute-environment-minimum_Batch_049f0a53-0a37-37c7-8f7d-9910330bdbec>>/etc/ecs/ecs.config

echo ECS_DISABLE_IMAGE_CLEANUP=false>>/etc/ecs/ecs.config

echo ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=2m>>/etc/ecs/ecs.config

echo ECS_IMAGE_CLEANUP_INTERVAL=10m>>/etc/ecs/ecs.config

echo ECS_IMAGE_MINIMUM_CLEANUP_AGE=10m>>/etc/ecs/ecs.config

echo ECS_NUM_IMAGES_DELETE_PER_CYCLE=5>>/etc/ecs/ecs.config

echo ECS_RESERVED_MEMORY=32>>/etc/ecs/ecs.config

--==BOUNDARY==--

AWS Batch：docker No space left on device → docker system prune

コメントを残す

docker利用において、

docker内の以下や

<br />No space left on device

No space left on device

ホストOS側の以下が出る場合がある。

<br />$ docker run -i -t busybox
docker: Error response from daemon: devmapper: Thin Pool has 3975 free data blocks which is less than minimum required 4449 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior.
See 'docker run --help'.

$ docker run -i -t busybox

docker: Error response from daemon: devmapper: Thin Pool has 3975 free data blocks which is less than minimum required 4449 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior.

See 'docker run --help'.

生成したファイルに、rm をしない場合は、ゴミが残る。

<br />$ docker run -i -t busybox
/ # dd if=/dev/zero of=/tmp/6GB bs=1024k count=6000
6000+0 records in
6000+0 records out
6291456000 bytes (5.9GB) copied, 43.371513 seconds, 138.3MB/s
/ # exit

$ docker run -i -t busybox

/ # dd if=/dev/zero of=/tmp/6GB bs=1024k count=6000

6000+0 records in

6000+0 records out

6291456000 bytes (5.9GB) copied, 43.371513 seconds, 138.3MB/s

/ # exit

docker system prune -fで解放される。

<br /># docker system prune -f
Deleted Containers:
9183c3b0b8c20011ce441a23a477becacc44189ea99f60be7018573dae0fea79

Total reclaimed space: 6.291GB

# docker system prune -f

Deleted Containers:

9183c3b0b8c20011ce441a23a477becacc44189ea99f60be7018573dae0fea79

Total reclaimed space: 6.291GB

コンテナサイズ、デフォルト10GBを超えた場合も、

<br />$ docker run -i -t busybox
/ # dd if=/dev/zero of=/tmp/11GB bs=1024k count=11000
dd: error writing '/tmp/11GB': No space left on device
9962+0 records in
9961+0 records out
10445869056 bytes (9.7GB) copied, 72.213538 seconds, 138.0MB/s

$ docker run -i -t busybox

/ # dd if=/dev/zero of=/tmp/11GB bs=1024k count=11000

dd: error writing '/tmp/11GB': No space left on device

9962+0 records in

9961+0 records out

10445869056 bytes (9.7GB) copied, 72.213538 seconds, 138.0MB/s

やはり、docker system prune -fで解放される。

<br /># docker system prune -f
Deleted Containers:
2ec9751723a2c5a11f0a3d5872ad84695945f39a9cb2e4ecce800d9acd6f5ea3

Total reclaimed space: 10.45GB

# docker system prune -f

Deleted Containers:

2ec9751723a2c5a11f0a3d5872ad84695945f39a9cb2e4ecce800d9acd6f5ea3

Total reclaimed space: 10.45GB

上記以外で、rmでファイルを消した場合は、ほとんどゴミデータは残らない。

<br /># docker system prune -f
Deleted Containers:
7d58206669601c966ceb229b0997f1e65d3458e2669e320e810767532d3b89d1

Total reclaimed space: 66B

# docker system prune -f

Deleted Containers:

7d58206669601c966ceb229b0997f1e65d3458e2669e320e810767532d3b89d1

Total reclaimed space: 66B

ゴミデータを残ったままにすると、ブロックデバイスを全て消費して、docker自体起動できない等の状態になる。

<br />$ docker run -i -t busybox
docker: Error response from daemon: devmapper: Thin Pool has 3975 free data blocks which is less than minimum required 4449 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior.
See 'docker run --help'.

$ docker run -i -t busybox

See 'docker run --help'.

ホストOS側からは、次のように動的マウントのマッピングが見える。

<br /># df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 88K 1.9G 1% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/nvme0n1p1 7.9G 835M 6.9G 11% /
/dev/dm-3 9.8G 91M 9.2G 1% /var/lib/docker/devicemapper/mnt/6c3958350431fd2ab3cea269efe31ceaf454ba0acdbccffc23b097f5ec64f512
shm 64M 0 64M 0% /var/lib/docker/containers/2aa149aa5359d9e30ae8582915d8f583e7990a8f3fde69a7426199ca0edc58c5/mounts/shm
/dev/dm-5 9.8G 38M 9.3G 1% /var/lib/docker/devicemapper/mnt/227f02710a4c689085183cab89a28e4b423580ae209be2cc7733d8a63985d3a0
shm 64M 0 64M 0% /var/lib/docker/containers/adf4ab582023913164b3508c90dbb4f820826be45f54881122cf9985c3e81d14/mounts/shm
/dev/dm-4 9.8G 9.8G 0 100% /var/lib/docker/devicemapper/mnt/e79266802dfd19414612e290d8e546d95c31588ddb79fd8c0b1758bc2e514c97
shm 64M 0 64M 0% /var/lib/docker/containers/2ec9751723a2c5a11f0a3d5872ad84695945f39a9cb2e4ecce800d9acd6f5ea3/mounts/shm

# df -h

Filesystem Size Used Avail Use% Mounted on

devtmpfs 1.9G 88K 1.9G 1% /dev

tmpfs 1.9G 0 1.9G 0% /dev/shm

/dev/nvme0n1p1 7.9G 835M 6.9G 11% /

/dev/dm-3 9.8G 91M 9.2G 1% /var/lib/docker/devicemapper/mnt/6c3958350431fd2ab3cea269efe31ceaf454ba0acdbccffc23b097f5ec64f512

shm 64M 0 64M 0% /var/lib/docker/containers/2aa149aa5359d9e30ae8582915d8f583e7990a8f3fde69a7426199ca0edc58c5/mounts/shm

/dev/dm-5 9.8G 38M 9.3G 1% /var/lib/docker/devicemapper/mnt/227f02710a4c689085183cab89a28e4b423580ae209be2cc7733d8a63985d3a0

shm 64M 0 64M 0% /var/lib/docker/containers/adf4ab582023913164b3508c90dbb4f820826be45f54881122cf9985c3e81d14/mounts/shm

/dev/dm-4 9.8G 9.8G 0 100% /var/lib/docker/devicemapper/mnt/e79266802dfd19414612e290d8e546d95c31588ddb79fd8c0b1758bc2e514c97

shm 64M 0 64M 0% /var/lib/docker/containers/2ec9751723a2c5a11f0a3d5872ad84695945f39a9cb2e4ecce800d9acd6f5ea3/mounts/shm

確認したAMI

amzn-ami-2018.03.u-amazon-ecs-optimized (ami-0e1aa8c2e9d719f58)


docker -v
Docker version 18.06.1-ce, build e68fc7a215d7133c34aa18e3b72b4a21fd0c6136

Docker ファイルシステム消費量の調査

コメントを残す

対応前

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 40G 8.5G 31G 22% /
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/
3.5G /var/lib/docker/

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 40G 8.5G 31G 22% /

devtmpfs 236M 56K 236M 1% /dev

tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/

3.5G /var/lib/docker/

Docker上で10GBファイル生成

root@df4efaf1d395:/# dd if=/dev/zero of=/tmp/my_file5 bs=1024k count=10000

1 2	root@df4efaf1d395:/# dd if=/dev/zero of=/tmp/my_file5 bs=1024k count=10000

対応後

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 40G 19G 21G 47% /
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/
24G /var/lib/docker/

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 40G 19G 21G 47% /

devtmpfs 236M 56K 236M 1% /dev

tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/

24G /var/lib/docker/

ホストOS

root@df4efaf1d395:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 40G 19G 21G 47% /
tmpfs 64M 0 64M 0% /dev
tmpfs 246M 0 246M 0% /sys/fs/cgroup
/dev/xvda1 40G 19G 21G 47% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 246M 0 246M 0% /proc/acpi
tmpfs 246M 0 246M 0% /proc/scsi
tmpfs 246M 0 246M 0% /sys/firmware

root@df4efaf1d395:/# df -h

Filesystem Size Used Avail Use% Mounted on

overlay 40G 19G 21G 47% /

tmpfs 64M 0 64M 0% /dev

tmpfs 246M 0 246M 0% /sys/fs/cgroup

/dev/xvda1 40G 19G 21G 47% /etc/hosts

shm 64M 0 64M 0% /dev/shm

tmpfs 246M 0 246M 0% /proc/acpi

tmpfs 246M 0 246M 0% /proc/scsi

tmpfs 246M 0 246M 0% /sys/firmware

結局40GBまではかける

root@df4efaf1d395:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 40G 40G 0 100% /
tmpfs 64M 0 64M 0% /dev
tmpfs 246M 0 246M 0% /sys/fs/cgroup
/dev/xvda1 40G 40G 0 100% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 246M 0 246M 0% /proc/acpi
tmpfs 246M 0 246M 0% /proc/scsi
tmpfs 246M 0 246M 0% /sys/firmware
root@df4efaf1d395:/# dd if=/dev/zero of=/tmp/my_file8 bs=1024k count=2000
dd: error writing '/tmp/my_file8': No space left on device
1+0 records in
0+0 records out
0 bytes copied, 0.000883203 s, 0.0 kB/s

root@df4efaf1d395:/# df -h

Filesystem Size Used Avail Use% Mounted on

overlay 40G 40G 0 100% /

tmpfs 64M 0 64M 0% /dev

tmpfs 246M 0 246M 0% /sys/fs/cgroup

/dev/xvda1 40G 40G 0 100% /etc/hosts

shm 64M 0 64M 0% /dev/shm

tmpfs 246M 0 246M 0% /proc/acpi

tmpfs 246M 0 246M 0% /proc/scsi

tmpfs 246M 0 246M 0% /sys/firmware

root@df4efaf1d395:/# dd if=/dev/zero of=/tmp/my_file8 bs=1024k count=2000

dd: error writing '/tmp/my_file8': No space left on device

1+0 records in

0+0 records out

0 bytes copied, 0.000883203 s, 0.0 kB/s

ホストOS

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 40G 40G 0 100% /
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/
63G /var/lib/docker/

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 40G 40G 0 100% /

devtmpfs 236M 56K 236M 1% /dev

tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/

63G /var/lib/docker/

Docker終了後

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 40G 40G 0 100% /
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/
34G /var/lib/docker/

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 40G 40G 0 100% /

devtmpfs 236M 56K 236M 1% /dev

tmpfs 246M 0 246M 0% /dev/shm

$ sudo du -sh /var/lib/docker/

34G /var/lib/docker/

再起動すると。/tmp配下ファイルが無い？

root@5d4d87a635ca:/# ll /tmp/
total 8
drwxrwxrwt 2 root root 4096 Jun 10 20:41 ./
drwxr-xr-x 1 root root 4096 Jul 2 15:17 ../

root@5d4d87a635ca:/# ll /tmp/

total 8

drwxrwxrwt 2 root root 4096 Jun 10 20:41 ./

drwxr-xr-x 1 root root 4096 Jul 2 15:17 ../

docker rmi してもホストOSの容量が減らない、

$ sudo du -sh /var/lib/docker/
34G /var/lib/docker/

$ sudo du -sh /var/lib/docker/

34G /var/lib/docker/

差分ファイルが全部残っていた

[root@freeserver ec2-user]# du -sh /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file*
9.8G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file4
9.8G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file5
9.8G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file6
1.6G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file7
0 /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file8

[root@freeserver ec2-user]# du -sh /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file*

9.8G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file4

9.8G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file5

9.8G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file6

1.6G /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file7

0 /var/lib/docker/overlay2/e9bcb6bdfdf426e82ddaef8f0a38ecaae359ce2b2f3c73dd529dd019d3b46dff/diff/tmp/my_file8

特定のDockerコンテナに溜まった大量のゴミファイルの掃除

AWS BatchはAMI IDが、Manged by Batch になっていて、docker versionがわからない。

https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/container_agent_versions.html

https://stackoverflow.com/questions/46672001/is-it-safe-to-clean-docker-overlay2

下記で解消された

<br />$ docker system prune -f
Deleted Containers:
d2e678e52f4cf2e9d857240dffcf6c16a60225f59de16151ebb77c37b11d7ba6
f8ced1448f1873b14ede28997aef534f7247612a21936ffdc336eb08d5f16c00
5d4d87a635cab95b10816dd6c52b05ca55ab7f4245ea6e6022ad08d75df1990d
df4efaf1d395ea997bc72a8558f5628581371c3341acbf490025964a2c377972
0baee82fb942bd1a1c9bd46676adae898355e15022e19995ff4e365c829027b2
51c79688447c70de1fe0f21454f9295073d323bdf87df73c1a2cdb27b4e22c34
6dc53777a0839f6a53af98bbc141dcb2d7864c17bd6504f1745885c2ea694c15
02c79a7039038336d9cfef892b72dfad7d587adb85adcb8fd4ce281d0253dc6b
a9a7002c6d1b1411a887550468b090ce080bda9ca62d4671b364b6ba56f62b13
fbe0fddb2d88ac00d45755b0843214631c0b9552a24c09f789f228adac821136
ad1c7c604efcabbea951432b4b27f51d554c2fcb406b549fd89de986a3332922
8833292685be1b5a8041e087a45fbf87841b70633f8347854bbd5f2acc93fcbb
813bfa2f675973ea5cdc17c2157bdab1ae04e85ecbb45733c1b2eee5ff631c7c
950fc90f5feb7bf7c32c767391688eb57171504e3747401a87faf1e11d2b76a5

Total reclaimed space: 35.28GB

$ docker system prune -f

Deleted Containers:

d2e678e52f4cf2e9d857240dffcf6c16a60225f59de16151ebb77c37b11d7ba6

f8ced1448f1873b14ede28997aef534f7247612a21936ffdc336eb08d5f16c00

5d4d87a635cab95b10816dd6c52b05ca55ab7f4245ea6e6022ad08d75df1990d

df4efaf1d395ea997bc72a8558f5628581371c3341acbf490025964a2c377972

0baee82fb942bd1a1c9bd46676adae898355e15022e19995ff4e365c829027b2

51c79688447c70de1fe0f21454f9295073d323bdf87df73c1a2cdb27b4e22c34

6dc53777a0839f6a53af98bbc141dcb2d7864c17bd6504f1745885c2ea694c15

02c79a7039038336d9cfef892b72dfad7d587adb85adcb8fd4ce281d0253dc6b

a9a7002c6d1b1411a887550468b090ce080bda9ca62d4671b364b6ba56f62b13

fbe0fddb2d88ac00d45755b0843214631c0b9552a24c09f789f228adac821136

ad1c7c604efcabbea951432b4b27f51d554c2fcb406b549fd89de986a3332922

8833292685be1b5a8041e087a45fbf87841b70633f8347854bbd5f2acc93fcbb

813bfa2f675973ea5cdc17c2157bdab1ae04e85ecbb45733c1b2eee5ff631c7c

950fc90f5feb7bf7c32c767391688eb57171504e3747401a87faf1e11d2b76a5

Total reclaimed space: 35.28GB

<br />$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
xxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/hello-repository latest 25167261abec 25 hours ago 244MB
hello-world latest 25167261abec 25 hours ago 244MB
ubuntu 16.04 13c9f1285025 13 days ago 119MB
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 40G 5.8G 34G 15% /
devtmpfs 236M 56K 236M 1% /dev
tmpfs 246M 0 246M 0% /dev/shm

$ docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

xxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/hello-repository latest 25167261abec 25 hours ago 244MB

hello-world latest 25167261abec 25 hours ago 244MB

ubuntu 16.04 13c9f1285025 13 days ago 119MB

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 40G 5.8G 34G 15% /

devtmpfs 236M 56K 236M 1% /dev

tmpfs 246M 0 246M 0% /dev/shm

https://github.com/moby/moby/issues/32420

https://github.com/moby/moby/issues/21925

http://docs.docker.jp/engine/reference/commandline/dockerd.html#daemon-configuration-file

furoom

カテゴリーアーカイブ:Docker

AWS Batch　起動AMIのユーザデータ

AWS Batch：docker No space left on device → docker system prune

docker -v

Docker ファイルシステム消費量の調査