From owner-freebsd-jail@freebsd.org Fri Jul 8 21:07:30 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E44D5B858AE for ; Fri, 8 Jul 2016 21:07:30 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x229.google.com (mail-it0-x229.google.com [IPv6:2607:f8b0:4001:c0b::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC58719C3 for ; Fri, 8 Jul 2016 21:07:30 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x229.google.com with SMTP id h190so19286989ith.1 for ; Fri, 08 Jul 2016 14:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=PNA7aC7MBpWej+rTAtpn9mPYY+hWCrZS2ZXk4Oj2gqA=; b=Ycd0hldEqvDt9bOzq9nnrHxZ+EEoI9Y1jO6vPlBxumTZbPJ7BUVKhb2t66ADuC8/Ro mKDD/vqktCtOxDnpGww5je4zJxoxCPEqv9dNnwvXBGlFENdeR/Ej77emfq3TI2PCTyTL R1U6IpzieBBQ983AIzcFM+hAovq4wp+ESk6YqUT9Bi6r98HGNCavHd+6rZAJNPSAk/T2 csTa90sohbODx2a16sZpPND38/X7+uzyy+J3msX5UjkHPJ3KH3BLpgEMN5EXxpCgecBY w2Yck0Ocd3cjNiGH4uvodtalJNNWxiz5/BlVAMrEWuhfIi6G/Hd5rfJ2RNG/NcgpRbue xvPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=PNA7aC7MBpWej+rTAtpn9mPYY+hWCrZS2ZXk4Oj2gqA=; b=jr2bk9+gkXDdEpLU3rO/tT+KcaR3Xe8ovX0M7YY5WMUfnnUDII0zDZ+WuvMFcn5v+j cw8szxhL0oHbatcQ/af18N4zi8xXP78CAHrzfe6rfsBYd0BLIoz3ukFruOtBUxcDI63D wI+Al44l9GLRDUvapm5Bw6+Hua3AQvXBg3/KsN1LLesVcnDlA/IG+p0WOzcS3rER/WEx DaBDijVY330Llo+mEuo6jhIxnVEbvNuCJYtx3qeDbtqyF0xXdWymaejCqWr/MkPmRxdi /o5lrCQ+rPEdbX7Adyv3pnbjiqTJRLh4iVth/OBnQlFKVvcEIgojZf0hK8fgSU488rVV nLzg== X-Gm-Message-State: ALyK8tLBxDwahnbGIv/RjjOu6huDZxCh4MP+pYbhnUv4JfLDtazzUT7J+avmCu+0t/Mkzw== X-Received: by 10.36.117.200 with SMTP id y191mr526187itc.35.1468012050040; Fri, 08 Jul 2016 14:07:30 -0700 (PDT) Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com. [184.56.210.236]) by smtp.googlemail.com with ESMTPSA id v129sm1998920itg.10.2016.07.08.14.07.29 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jul 2016 14:07:29 -0700 (PDT) Message-ID: <57801614.3000205@gmail.com> Date: Fri, 08 Jul 2016 17:07:32 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Thomas Johnson CC: freebsd-jail@freebsd.org Subject: Re: NFS + nullfs + jail = zombies? References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 21:07:31 -0000 Thomas Johnson wrote: > I am working on developing a clustered application utilizing jails and > running into problems that seem to be NFS-related. I'm hoping that > someone can point out my error. > > The jail images and my application data are served via NFS. The host > mounts NFS at boot, and then uses nullfs mounts to assemble the jail > tree when the jail is created (fstab files and jail.conf are below). > This seems to work fine, the jail starts and is usable. The problem > comes when I remove/restart the jail. Frequently (but not > consistently), the jail gets stuck in a dying state, causing the > unmount of the jail root (nullfs) to fail with a "device busy" error. > > # jail -f /var/local/jail.conf -r wds1-1a > Stopping cron. > Waiting for PIDS: 1361. > . > Terminated > wds1-1a: removed > umount: unmount of /var/jail/wds1-1a failed: Device busy > # jls -av > JID Hostname Path > Name State > CPUSetID > IP Address(es) > 1 wds1-1a /var/jail/wds1-1a > wds1-1a DYING > 2 > 2620:1:1:1:1a::1 > > Through trial-and-error I have determined that forcing an unmount of > the root works, but subsequent mounts to that mount point will fail to > unmount with the same error. Deleting and recreating the mountpoint > fixes the mounting issue, but the dying jail remains permanently. > > I have also found that if I copy the jail root to local storage and > update the jail's fstab to nullfs mount this, the problem seems to go > away. This leads me to believe that the issue is related to the NFS > source for the nullfs mount. statd and lockd are both running on the > host. > > My relevant configurations are below. I can provide any other > information desired. > > # Host fstab line for jail root. > # > 10.219.212.1:/vol/dev/wds/jail_base /jail/base nfs ro 0 0 > > > # Jail fstab file (mount.fstab) > # > /jail/base /var/jail/wds1-1a nullfs ro 0 0 > # writable (UFS-backed) /var > /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0 > > > # jail.conf file > # > * { > devfs_ruleset = "4"; > mount.devfs; > exec.start = "/bin/sh /etc/rc"; > exec.stop = "/bin/sh /etc/rc.shutdown"; > interface = "vmx1"; > allow.dying = 1; > exec.prestart = "/usr/local/bin/rsync -avC --delete > /jail/${image}/var/ /var/jail-vars/${host.hostname}/"; > } > > # JMANAGE wds1-1a > wds1-1a { > path = "/var/jail/wds1-1a"; > ip6.addr = "2620:1:1:1:1a::1"; > host.hostname = "wds1-1a"; > host.domainname = "dev"; > mount.fstab = "/var/local/fstab.wds1-1a"; > $image = "base"; > } If I remember correctly from past posts about NFS and jails, NFS does not play well with jails. The jails directory tree has to be on the host running the jail. Remove NFS from the picture and it should just work. Your beating a dead horse.