On Docker Storage and NFS

We’re just about to release a new version of Docker Trusted Registry (DTR) at work, and I thought I’d highlight one of the cool features that we did for NFS storage.

In the past, we told people who wanted to use NFS to bind mount their NFS directories directly into a directory which was storing their registry volume. This is really a pain to configure, particularly if you want to automate the process. It requires creating the volume, inspecting it, mounting the NFS directory and then bind mounting that directory in the freshly inspected volume. Oh, and also you need to make certain the NFS client options are correct if you want it to work in your HA cluster. This is less than ideal.

With DTR 2.1, you can just specify a new flag during install or reconfigure called --nfs-storage-url which will automatically take care of everything for you. The best part is when you join a new DTR replica to the cluster, the new node will automatically get the NFS mount point and everything will just work like magic.

So how does this work?

We’re taking use of a little known feature in the local storage volume plugin which allows you to back docker volumes with NFS. Instead of the docker volume being backed with a local file system, the docker daemon instead mounts the configured NFS directory when the volume is attached to a container and that container starts up.

Let’s say we’ve created an NFS server called nfs-srvr and we’ve exported the directory “/exports”. We’ll use this exported directory to back a new volume on a different docker host which we’ll call dckr1. First, on our dckr1 host, let’s make certain that we can access the exported directory

root@dckr1:~# showmount -e nfs-srvr
Export list for nfs-srvr:
/exports \*

Alright, that looks good, so let’s go ahead and create a new volume which is backed by the “/exports” directory.

root@dckr1:~# docker volume create --driver local --opt type=nfs \\
--opt o=addr=<IP ADDRESS>,rw --opt device=:/exports --name=test-vol

And finally, let’s attach it to a new container running alpine

root@dckr1:~# docker run -it -v test-vol:/storage --rm alpine:3.4 sh
/ # ls /storage
...

And that’s pretty much it. There are a couple of caveats to using it right now.  The first is that there is (as of Docker 1.12) a bug which you can’t use the host address of the NFS server, and instead have to use the IP address. Also, make certain that you have the nfs client kernel library installed on the docker host (usually packaged in nfs-common or nfs-utils), otherwise your container will zombie when it starts because it won’t be able to back the volume correctly.