Quick article on how to use the atomic storage commands to swap out an underlying
storage device used for docker’s graph storage.
- I am currently using overlay2 for docker storage, and /var/lib/docker is currently on my root partition
- I want to add a 2nd disk just for docker storage.
- I want to keep my images, rather than have to download them again.
I have a few images in my system:
# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/openshift/hello-openshift latest 305f93951299 3 weeks ago 5.635 MB docker.io/centos centos7 3bee3060bfc8 6 weeks ago 192.6 MB docker.io/monitoringartist/grafana-xxl latest 5a73d8e5f278 10 weeks ago 393.4 MB docker.io/fedora latest 4daa661b467f 3 months ago 230.6 MB docker.io/jeremyeder/c7perf latest 3bb51319f973 4 months ago 1.445 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7/rhel-tools latest 264d7d025911 4 months ago 1.488 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7 latest 41a4953dbf95 4 months ago 192.5 MB docker.io/busybox latest 7968321274dc 6 months ago 1.11 MB # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-root 193G 162G 23G 88% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 804K 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/vdc1 100G 33M 100G 1% /var/lib/docker/overlay /dev/vda1 2.0G 549M 1.5G 28% /boot
All of docker’s storage right now consumes about 4GB. It’s important to verify this because the migrate commands we’re about to walk through require this much space to complete the migration:
# du -hs /var/lib/docker 3.9G /var/lib/docker
By default, the atomic migrate commands will write to /var/lib/atomic, so whatever filesystem holds that directory will need at least (in my case) 4GB free.
The migration process has several phases:
- Export any containers and images.
- Allow user to adjust storage on the system.
- Allow user to adjust storage configuration of docker.
- Import containers and images back into the new docker graph storage.
I’m using a VM with spinning disks so this takes a little longer than it otherwise might, but let’s start the export:
# time atomic storage export Exporting image: 5a73d8e5f278 Exporting image: 3bb51319f973 Exporting image: 7968321274dc Exporting image: 3bee3060bfc8 Exporting image: 4daa661b467f Exporting image: 264d7d025911 Exporting image: 41a4953dbf95 Exporting image: 305f93951299 Exporting volumes atomic export completed successfully real 1m57.159s user 0m1.094s sys 0m6.190s
OK that went oddly smoothly, let’s see what it actually did:
# find /var/lib/atomic/migrate /var/lib/atomic/migrate /var/lib/atomic/migrate/info.txt /var/lib/atomic/migrate/containers /var/lib/atomic/migrate/images /var/lib/atomic/migrate/images/4daa661b467f23f983163d75f0b87744cd3d88a2aed11be813d802606e8f13df /var/lib/atomic/migrate/images/3bee3060bfc81c061ce7069df35ce090593bda584d4ef464bc0f38086c11371d /var/lib/atomic/migrate/images/7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768 /var/lib/atomic/migrate/images/264d7d0259119cf980fb95759865938765ccb3f1aa24600cbac49bea6b5b8cfb /var/lib/atomic/migrate/images/305f939512995147aa964bceef36a4a83226fae523c52b011fd69c9a229e3460 /var/lib/atomic/migrate/images/5a73d8e5f27861df210b03ca872530b6ab8b20b6a0d9c815022da3e0812df089 /var/lib/atomic/migrate/images/3bb51319f9734038d7b2d3c67cae6c25bbd9df18163cd7810ffcff952cbe0608 /var/lib/atomic/migrate/images/41a4953dbf957cfc562935239a3153a5da6101f32fa30da7b4a506f23cfcde9d /var/lib/atomic/migrate/volumes /var/lib/atomic/migrate/volumes/volumeData.tar.gz
Seems reasonable…incidentally that info.txt just includes the name of the storage driver used at the time migrate was executed.
# du -hs /var/lib/atomic 3.8G /var/lib/atomic
OK let’s do the deed:
# atomic storage reset Docker daemon must be stopped before resetting storage
Oh, I guess that would make sense.
# systemctl stop docker # atomic storage reset
OK, at this point /etc/sysconfig/docker-storage is reset to it’s default state, and I have nothing in my docker graph storage.
Because I want to continue to use overlay2, I will use the atomic storage modify command to make that so:
# atomic storage modify --driver overlay2 # cat /etc/sysconfig/docker-storage DOCKER_STORAGE_OPTIONS="--storage-driver overlay2 "
Things are looking good so far.
Now about adding more storage.
- I have added a new virtual storage device to my VM called /dev/vdc1
- I have partitioned and formatted it with XFS filesystem.
- I have mounted it at /var/lib/docker and setup an fstab entry.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 200G 0 disk ├─vda1 252:1 0 2G 0 part /boot └─vda2 252:2 0 198G 0 part ├─vg0-swap 253:0 0 2G 0 lvm [SWAP] └─vg0-root 253:1 0 196.1G 0 lvm / vdb 252:16 0 100G 0 disk └─vdb1 252:17 0 100G 0 part vdc 252:32 0 100G 0 disk └─vdc1 252:33 0 100G 0 part /var/lib/docker
At this point we are ready to restart docker and import the images from my previous storage. First let me verify that it’s OK.
# systemctl start docker # docker info|grep -i overlay2 Storage Driver: overlay2
Cool, so docker started up correctly and it has the overlay2 storage driver that I told it to use with the atomic storage modify command (from previous step).
Now for the import…
# time atomic storage import Importing image: 4daa661b467f ae934834014c: Loading layer [==================================================>] 240.3 MB/240.3 MB Loaded image: docker.io/fedora:latest Importing image: 3bee3060bfc8 dc1e2dcdc7b6: Loading layer [==================================================>] 200.2 MB/200.2 MB Loaded image: docker.io/centos:centos7 Importing image: 7968321274dc 38ac8d0f5bb3: Loading layer [==================================================>] 1.312 MB/1.312 MB Loaded image: docker.io/busybox:latest Importing image: 264d7d025911 827264d42df6: Loading layer [==================================================>] 202.3 MB/202.3 MB 9ca8c628d8e7: Loading layer [==================================================>] 10.24 kB/10.24 kB a03f55f719da: Loading layer [==================================================>] 1.336 GB/1.336 GB Loaded image: brew-pulp-docker01.redacted.redhat.com:8888/rhel7/rhel-tools:latest Importing image: 305f93951299 5f70bf18a086: Loading layer [==================================================>] 1.024 kB/1.024 kB c618fb2630cb: Loading layer [==================================================>] 5.637 MB/5.637 MB Loaded image: docker.io/openshift/hello-openshift:latest Importing image: 5a73d8e5f278 8d4d1ab5ff74: Loading layer [==================================================>] 129.4 MB/129.4 MB 405d1c3227e0: Loading layer [==================================================>] 3.072 kB/3.072 kB 048845c41855: Loading layer [==================================================>] 277.2 MB/277.2 MB Loaded image: docker.io/monitoringartist/grafana-xxl:latest Importing image: 3bb51319f973 34e7b85d83e4: Loading layer [==================================================>] 199.9 MB/199.9 MB ab7578fbc6c6: Loading layer [==================================================>] 3.072 kB/3.072 kB 3e89505f5573: Loading layer [==================================================>] 58.92 MB/58.92 MB 753668c55633: Loading layer [==================================================>] 1.169 GB/1.169 GB d778d7335b8f: Loading layer [==================================================>] 11.98 MB/11.98 MB 5cd21edffb34: Loading layer [==================================================>] 45.1 MB/45.1 MB Loaded image: docker.io/jeremyeder/c7perf:latest Importing image: 41a4953dbf95 Loaded image: brew-pulp-docker01.redacted.redhat.com:8888/rhel7:latest Importing volumes atomic import completed successfully Would you like to cleanup (rm -rf /var/lib/atomic/migrate) the temporary directory [y/N]n Please restart docker daemon for the changes to take effect real 1m23.951s user 0m1.391s sys 0m4.095s
Again went smoothly. I opted not to have it automatically clean up /var/lib/atomic/migrate automatically because I want to verify a thing or two first.
Let’s see what’s on my new disk:
# df -h /var/lib/docker Filesystem Size Used Avail Use% Mounted on /dev/vdc1 100G 3.9G 97G 4% /var/lib/docker
OK that looks reasonable. Let’s start docker and see if things imported correctly:
# systemctl restart docker # docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/openshift/hello-openshift latest 305f93951299 3 weeks ago 5.635 MB docker.io/centos centos7 3bee3060bfc8 6 weeks ago 192.6 MB docker.io/monitoringartist/grafana-xxl latest 5a73d8e5f278 10 weeks ago 393.4 MB docker.io/fedora latest 4daa661b467f 3 months ago 230.6 MB docker.io/jeremyeder/c7perf latest 3bb51319f973 4 months ago 1.445 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7/rhel-tools latest 264d7d025911 4 months ago 1.488 GB brew-pulp-docker01.redacted.redhat.com:8888/rhel7 latest 41a4953dbf95 4 months ago 192.5 MB docker.io/busybox latest 7968321274dc 6 months ago 1.11 MB
Images are there. Can I run one?
# docker run --rm fedora pwd /
Indeed I can. All seems well.
This utility is very handy in scenarios where you want to do some surgery on the backend storage, but do not want to throw away/download images and containers. I could envision using this utility when
- Moving from one graph driver to another. Note that we have SELinux support coming to overlay2 in RHEL 7.4.
- Perhaps you have a lot of images or containers and slow internet.
Either way, this process was about as smooth as it could be…and a very clean UX, too.
Jeremy Eder's Blog 