From owner-freebsd-jail@freebsd.org Fri Jul 8 18:28:55 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 586FFB85B9D for ; Fri, 8 Jul 2016 18:28:55 +0000 (UTC) (envelope-from tommyj27@gmail.com) Received: from mail-vk0-x22a.google.com (mail-vk0-x22a.google.com [IPv6:2607:f8b0:400c:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1508E1744 for ; Fri, 8 Jul 2016 18:28:55 +0000 (UTC) (envelope-from tommyj27@gmail.com) Received: by mail-vk0-x22a.google.com with SMTP id v6so65154868vkb.2 for ; Fri, 08 Jul 2016 11:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=RUaLTzswdADaJlJvtoai3ksxkQWFKe1tXTg1tLJ36a8=; b=shgWxQWCSFP38Djlp11KReuVANggaa9rRT+qro7f8A5A6HbicPpuhyncN8gRUbyjfU 5sGOUwYOqwH/bvRrc/mLgos2IqevN8nueZa9JW6h3Og5rDhhyqurZ+4o5TqYk9fQeoEH onO/5hP7Ynozn2GOjoDFcZMRcjXY6iFo/gsHXKMcUuaBClWXnVLugkyahRwrawJ28TY4 T6eicFFmJg08429vmfUfAC4HaDUp2F61+1bM8fnA3Q32tXv5l3sTU3OGEkjXl3cewo2d u8AKcpSVA1kQv9tnVBAz4KSgN0v5y9cqwXUoSx2RGDv4vUpRSVTbzidaudahtHuFnzoV eNyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=RUaLTzswdADaJlJvtoai3ksxkQWFKe1tXTg1tLJ36a8=; b=jMyNxE78EjScGMSAELfMtEB1yOpoRh+Vsoi5hOfIpLNg/ADLzlOByXFgLY5vqgY0QS 9x9WsDjrnaNuhdm/ve2A6FH73tQWqI2O1wxVqLzm1kCmoR7ic09wo49i2wgkM0QbStCK 7GT0cdnbvHn4cCgaVtCVo95Ecg0W5yzb7jCB35SsyFnbjra3vzuYWLOoqh6e2lUgug3X Y1au1CLferi/kDnBVl6hVEjsWwMetqnGKpLSYkLbDI3m7zNg1RIFutJ7dFkVB++03PXf IFS7H4F/j54of5VzYCG2nD9zOR8Shzd88BVD4a3t02lOy+6YbuutIwrQz8lHM9kshlyk 9dfw== X-Gm-Message-State: ALyK8tLx2iDCWp1L4H3U4mCvfDyQxUvk21R2z9EWlI0pZs+rZu6Ia0q4SRs7b4wU40CfIoL8QOOZ72wALlgyRA== X-Received: by 10.159.41.164 with SMTP id s33mr3324906uas.146.1468002533977; Fri, 08 Jul 2016 11:28:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.33.136 with HTTP; Fri, 8 Jul 2016 11:28:53 -0700 (PDT) From: Thomas Johnson Date: Fri, 8 Jul 2016 13:28:53 -0500 Message-ID: Subject: NFS + nullfs + jail = zombies? To: freebsd-jail@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 18:28:55 -0000 I am working on developing a clustered application utilizing jails and running into problems that seem to be NFS-related. I'm hoping that someone can point out my error. The jail images and my application data are served via NFS. The host mounts NFS at boot, and then uses nullfs mounts to assemble the jail tree when the jail is created (fstab files and jail.conf are below). This seems to work fine, the jail starts and is usable. The problem comes when I remove/restart the jail. Frequently (but not consistently), the jail gets stuck in a dying state, causing the unmount of the jail root (nullfs) to fail with a "device busy" error. # jail -f /var/local/jail.conf -r wds1-1a Stopping cron. Waiting for PIDS: 1361. . Terminated wds1-1a: removed umount: unmount of /var/jail/wds1-1a failed: Device busy # jls -av JID Hostname Path Name State CPUSetID IP Address(es) 1 wds1-1a /var/jail/wds1-1a wds1-1a DYING 2 2620:1:1:1:1a::1 Through trial-and-error I have determined that forcing an unmount of the root works, but subsequent mounts to that mount point will fail to unmount with the same error. Deleting and recreating the mountpoint fixes the mounting issue, but the dying jail remains permanently. I have also found that if I copy the jail root to local storage and update the jail's fstab to nullfs mount this, the problem seems to go away. This leads me to believe that the issue is related to the NFS source for the nullfs mount. statd and lockd are both running on the host. My relevant configurations are below. I can provide any other information desired. # Host fstab line for jail root. # 10.219.212.1:/vol/dev/wds/jail_base /jail/base nfs ro 0 0 # Jail fstab file (mount.fstab) # /jail/base /var/jail/wds1-1a nullfs ro 0 0 # writable (UFS-backed) /var /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0 # jail.conf file # * { devfs_ruleset = "4"; mount.devfs; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; interface = "vmx1"; allow.dying = 1; exec.prestart = "/usr/local/bin/rsync -avC --delete /jail/${image}/var/ /var/jail-vars/${host.hostname}/"; } # JMANAGE wds1-1a wds1-1a { path = "/var/jail/wds1-1a"; ip6.addr = "2620:1:1:1:1a::1"; host.hostname = "wds1-1a"; host.domainname = "dev"; mount.fstab = "/var/local/fstab.wds1-1a"; $image = "base"; }