I recently moved a shared file folder to a new NFS server. The change looked trivial. I edited one IP in a stack file, redeployed, and expected the containers to mount the new server. They did not. This post covers what went wrong, how we tested it, and what finally fixed it.
The setup
Our services run on Docker Swarm. Each service mounts an NFS share through a named volume defined in the stack file:
yaml
volumes:
nfs_files:
driver: local
driver_opts:
type: "nfs"
o: "addr=10.0.0.5,nolock,soft,rw"
device: ":/exports/files"
The migration was simple on paper. The share moved to a new server at 10.0.0.9, so I changed addr=10.0.0.5 to addr=10.0.0.9, committed, and let our CI pipeline run
docker stack deploy.
The symptom
The pipeline went green. Then I checked inside a running container:
bash
docker exec -it <container_id> sh
cat /proc/mounts | grep files
The output still showed the old server:
:/exports/files /var/www/html/files nfs rw,vers=3,...,addr=10.0.0.5
The container was less than an hour old, so it had definitely started after my deploy. Yet it was talking to the old IP.
Wrong guesses
My first thoughts were all wrong, and I want to be honest about them:
- The file is wrong. It was not. The new IP was in the repository and in the deployed service spec.
- The dashboard is lagging. We use Portainer, and it showed old values in places. That felt like a cache issue, but Portainer reads live from the Docker API. It was showing the truth.
- The deploy did not apply. It did. Comparing the service spec with the volume told a different story, which leads to the real cause.
The real cause: volume options are frozen at creation
You can see the mismatch with two commands. First, what the service wants:
bash
docker service inspect my_stack_api -f '{{json.Spec.TaskTemplate.ContainerSpec.Mounts}}'
This showed addr=10.0.0.9. Then, what the node actually has:
bash
docker volume inspect my_stack_nfs_files -f '{{json .Options}}'
This showed addr=10.0.0.5.
Here is what happens. A Docker volume is an object identified by its name, and it lives on a single node. The driver_opts are stored when the volume is created and never updated. When a task starts, the node asks for a volume with that name. If one already exists with the same driver, Docker reuses it. It does not compare the options. docker stack deploy updates the service spec, but it cannot rewrite a volume that already exists on a node.
So the stack file was correct, the spec was correct, and the volume kept the old IP.
There is a second catch. Swarm volumes are per node. I fixed the volume on the manager, but the worker nodes still had their own copies with the old options. Every node that runs the service needs the same fix.
How to fix it
There are two clean options.
Option 1: remove the volume on every node. Stop whatever uses it, for example docker service scale my_stack_api=0 or drain the node, then run this on each node:
bash
docker volume rm my_stack_nfs_files
Then redeploy. Swarm recreates the volume from the stack file. This deletes only the Docker volume definition, not the data on the NFS server. The downside is manual work on each node and some downtime.
Option 2: rename the volume in the stack file. A new name means no node has it, so Docker creates it fresh with the new options:
yaml
services:
api:
volumes:
- nfs_files_v2:/var/www/html/files
volumes:
nfs_files_v2:
driver: local
driver_opts:
type: "nfs"
o: "addr=10.0.0.9,soft,rw"
device: ":/exports/files"
A normal rolling deploy works and nobody has to log in to the nodes. The old volume stays behind and you clean it up later. We went with this one.
Rule of thumb: if you change the driver_opts of an existing volume, treat it as a different volume.
Testing before deploying
Because a bad volume means tasks that will not start, we tested on each node before touching production. From the lowest layer up:
- Is the port reachable?
bash
nc -zv -w3 10.0.0.9 2049
nc -zv -w3 10.0.0.9 111
- Can the host mount it, without Docker?
bash
mkdir -p /mnt/t
mount -t nfs -o soft,rw,nfsvers=4.2 10.0.0.9:/exports/files /mnt/t
ls /mnt/t | head
umount /mnt/t
3. Does Docker create and mount the volume with the same options as the stack file?
bash
docker volume create -d local -o type=nfs -o o=addr=10.0.0.9,soft,rw,nfsvers=4.2 -o device=:/exports/files files_test
docker run -v files_test:/mnt alpine sh -c "grep /mnt /proc/mounts; ls /mnt | head"
Clean up afterward with docker rm on the stopped container and docker volume rm files_test.
One trap here bit us. If files_test already exists on that node, docker volume create returns success but keeps the old options. Use a fresh name for every test run, or you will repeat the exact problem you are debugging.
The second problem: a timeout
Our first test, and then our first real deploy, failed with this error:
failed to mount local volume: mount :/exports/files:/var/lib/docker/volumes/.../_data,
data: addr=10.0.0.9,nolock,soft: connection timed out
Port 2049 was open on the firewall, so why a timeout? The answer was in the mount options. We had not set nfsvers, so Docker used NFSv3. NFSv3 needs more than port 2049. It talks to rpcbind on port 111, then to mountd on a dynamic port, and only then to the NFS service. Our firewall allowed 2049 only, so the mount waited and timed out. NFSv4 does everything over 2049.
Another volume on the same server, which already used nfsvers=4.2, worked fine on the same node. That was the clue. Adding nfsvers=4.2 to the new volume fixed the timeout.
I also wondered about nolock, because it was in the failing options. It was not the cause. In NFSv3, file locking uses a separate protocol (NLM), and nolock turns it off, which makes locks local to one client. It does not skip the mount step that was blocked. NFSv4 has locking built in, so there is no reason to keep nolock there.
The stale-name trap also came back. Our first failed deploy had already created the renamed volume on the nodes with the NFSv3 options. Fixing the options afterward changed nothing, because the volume already existed. We had to use another new name.
The final config
yaml
volumes:
nfs_files_v3:
driver: local
driver_opts:
type: "nfs"
o: "addr=10.0.0.9,soft,rw,nfsvers=4.2"
device: ":/exports/files"
After the deploy, this confirmed it on every node:
bash
docker exec <container_id> cat /proc/mounts | grep files
The output showed vers=4.2 and addr=10.0.0.9.
Lessons
- docker stack deploy does not change volumes that already exist. Options are fixed at creation.
- Swarm volumes are per node. Fixing the manager does not fix the workers.
- To change volume options, remove the volume on every node or rename it.
- When you test, never reuse a volume name. Stale options will make a good config look broken.
- If an NFS mount times out while port 2049 is open, check whether you are on NFSv3. Try nfsvers=4.2.
- Compare the service spec (docker service inspect) with the node's volume (docker volume inspect). A mismatch tells you the problem is the volume, not your file.
- Check every service that shares the volume before you call it done. One service left on the old config can recreate the old volume.
I hope this saves someone an evening. If you have hit the same thing, I would like to hear how you handled it.
