Date: Sat, 09 Jul 2016 08:22:25 -0600 From: James Gritton <jamie@gritton.org> To: freebsd-jail@freebsd.org Subject: Re: NFS + nullfs + jail = zombies? Message-ID: <596319b8dce811ea6e332c48d3542451@gritton.org> In-Reply-To: <CAMwYC7ZWH0s3h0AV8KxSLQk_NipvOkaMzkf2jF2Ty9Rx5x1H3w@mail.gmail.com> References: <CAMwYC7ZWH0s3h0AV8KxSLQk_NipvOkaMzkf2jF2Ty9Rx5x1H3w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2016-07-08 12:28, Thomas Johnson wrote:
> I am working on developing a clustered application utilizing jails and
> running into problems that seem to be NFS-related. I'm hoping that
> someone can point out my error.
>
> The jail images and my application data are served via NFS. The host
> mounts NFS at boot, and then uses nullfs mounts to assemble the jail
> tree when the jail is created (fstab files and jail.conf are below).
> This seems to work fine, the jail starts and is usable. The problem
> comes when I remove/restart the jail. Frequently (but not
> consistently), the jail gets stuck in a dying state, causing the
> unmount of the jail root (nullfs) to fail with a "device busy" error.
>
> # jail -f /var/local/jail.conf -r wds1-1a
> Stopping cron.
> Waiting for PIDS: 1361.
> .
> Terminated
> wds1-1a: removed
> umount: unmount of /var/jail/wds1-1a failed: Device busy
> # jls -av
> JID Hostname Path
> Name State
> CPUSetID
> IP Address(es)
> 1 wds1-1a /var/jail/wds1-1a
> wds1-1a DYING
> 2
> 2620:1:1:1:1a::1
>
> Through trial-and-error I have determined that forcing an unmount of
> the root works, but subsequent mounts to that mount point will fail to
> unmount with the same error. Deleting and recreating the mountpoint
> fixes the mounting issue, but the dying jail remains permanently.
>
> I have also found that if I copy the jail root to local storage and
> update the jail's fstab to nullfs mount this, the problem seems to go
> away. This leads me to believe that the issue is related to the NFS
> source for the nullfs mount. statd and lockd are both running on the
> host.
>
> My relevant configurations are below. I can provide any other
> information desired.
>
> # Host fstab line for jail root.
> #
> 10.219.212.1:/vol/dev/wds/jail_base /jail/base nfs ro 0 0
>
>
> # Jail fstab file (mount.fstab)
> #
> /jail/base /var/jail/wds1-1a nullfs ro 0 0
> # writable (UFS-backed) /var
> /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0
>
>
> # jail.conf file
> #
> * {
> devfs_ruleset = "4";
> mount.devfs;
> exec.start = "/bin/sh /etc/rc";
> exec.stop = "/bin/sh /etc/rc.shutdown";
> interface = "vmx1";
> allow.dying = 1;
> exec.prestart = "/usr/local/bin/rsync -avC --delete
> /jail/${image}/var/ /var/jail-vars/${host.hostname}/";
> }
>
> # JMANAGE wds1-1a
> wds1-1a {
> path = "/var/jail/wds1-1a";
> ip6.addr = "2620:1:1:1:1a::1";
> host.hostname = "wds1-1a";
> host.domainname = "dev";
> mount.fstab = "/var/local/fstab.wds1-1a";
> $image = "base";
> }
What happens if you take jails out of the equation? I know this isn't
entirely a non-jail issue, but I wonder if a jail is required for the
mount point to be un-re-mountable. I've heard before of NFS-related
problems where a jail remains dying forever, but this has been more of
an annoyance than a real problem.
It's not so much that I want to absolve jails, as I want to see where
the main fight exists. It's tricky enough fixing an interface between
two systems, but we've got three here.
- Jamie
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?596319b8dce811ea6e332c48d3542451>