From owner-freebsd-jail@freebsd.org Sun Mar 5 05:50:52 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 146C7CFAB39 for ; Sun, 5 Mar 2017 05:50:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F0EBE1E8F for ; Sun, 5 Mar 2017 05:50:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v255opur091302 for ; Sun, 5 Mar 2017 05:50:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 217545] jail: exec.poststop not executed, mount.fstab does not umount after removing jail Date: Sun, 05 Mar 2017 05:50:51 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Mar 2017 05:50:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217545 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-jail@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Sun Mar 5 17:39:31 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 150B1CFA261 for ; Sun, 5 Mar 2017 17:39:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE56B15CC for ; Sun, 5 Mar 2017 17:39:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v25HdUON069302 for ; Sun, 5 Mar 2017 17:39:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 217545] jail: exec.poststop not executed, mount.fstab does not umount after removing jail Date: Sun, 05 Mar 2017 17:39:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jamie@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Mar 2017 17:39:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217545 Jamie Gritton changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jamie@FreeBSD.org Resolution|--- |Works As Intended Status|New |Closed --- Comment #1 from Jamie Gritton --- Filesystems mounted with mount.fstab are indeed unmounted when a jail is stopped, and the exec.poststop command(s) are indeed executed. But what yo= u're doing isn't jail removal. Am existing jail is removed by running "jail -r", which will run the prestop and stop commands, actually remove the jail, then run poststop commands and then clean up mounts and IP address assignments. But note that you never r= an "jail -r". When a jail is created, it will (among other things) run the exec.start and= /or "command" commands. Those commands are not expected to all finish - genera= lly the command is something like "sh /etc/rc" which starts daemons which conti= nue to run. If all commands complete, and the jail isn't marked with the "pers= ist" parameter, the jail will naturally die on its own. That has nothing to do = with the jail(8) command, which is likely no longer running at the time, and whi= ch thus cannot running any of the shutdown commands. If you intend on running jails this way, I recommend unmounting the filesys= tems not in exec.poststop, but in exec.poststart which runs as the last step in = the creation process. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Mon Mar 6 17:27:46 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41E86CFBE9A for ; Mon, 6 Mar 2017 17:27:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3083D1762 for ; Mon, 6 Mar 2017 17:27:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v26HRkeU066022 for ; Mon, 6 Mar 2017 17:27:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 217545] jail: exec.poststop not executed, mount.fstab does not umount after removing jail Date: Mon, 06 Mar 2017 17:27:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: la5lbtyi@aon.at X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Mar 2017 17:27:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217545 Martin Birgmeier changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Closed |Open Resolution|Works As Intended |--- --- Comment #2 from Martin Birgmeier --- I also tried this use case: - Executed "jail -c -f " in one terminal - This gives me a shell prompt in the jail - Executed "jail -r " in another terminal - This kills the shell in the jail, prints ": removed", but does = not execute the poststop script (in neither of the two terminals) Expected behavior: The poststop script should be executed. -- Martin --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Mon Mar 6 17:46:56 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31CE9CFA9F6 for ; Mon, 6 Mar 2017 17:46:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 20F2917FE for ; Mon, 6 Mar 2017 17:46:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v26HktqB007091 for ; Mon, 6 Mar 2017 17:46:56 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 217545] jail: exec.poststop not executed, mount.fstab does not umount after removing jail Date: Mon, 06 Mar 2017 17:46:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jamie@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Unable to Reproduce X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Mar 2017 17:46:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D217545 Jamie Gritton changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |Closed Resolution|--- |Unable to Reproduce --- Comment #3 from Jamie Gritton --- I just did a quick test as you outlined, and it worked for me. This seems = more like a Q&A issue than a bug. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Thu Mar 9 02:02:00 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62B06D036BE for ; Thu, 9 Mar 2017 02:02:00 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 14F5E11FB for ; Thu, 9 Mar 2017 02:02:00 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [192.168.34.129] (unknown [153.150.77.170]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 2D99C1E5B4 for ; Thu, 9 Mar 2017 03:01:56 +0100 (CET) From: "Kristof Provost" To: "freebsd-jail@freebsd.org" Subject: Jail stuck in dying Date: Thu, 09 Mar 2017 11:01:48 +0900 Message-ID: MIME-Version: 1.0 X-Mailer: MailMate (2.0BETAr6080) Content-Type: text/plain; charset=utf-8; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 02:02:00 -0000 Hi, On a current box (r314933) I can’t seem to stop my jails. It’s started like this: jail -c name=test0 host.hostname=test vnet persist vnet.interface=epair0b \ path=/usr/jails/jail1 exec.start="/bin/sh /etc/rc" I terminate it with `jail -R test0`, yet the jail stays stuck in dying state: $ jls -a JID IP Address Hostname Path 1 test /usr/jails/jail1 $ jls -na devfs_ruleset=0 dying enforce_statfs=2 host=new ip4=inherit ip6=inherit jid=1 name=test0 osreldate=1200023 osrelease=12.0-CURRENT parent=0 path=/usr/jails/jail1 nopersist securelevel=-1 sysvmsg=disable sysvsem=disable sysvshm=disable vnet=new allow.nochflags allow.nomount allow.mount.nodevfs allow.mount.nofdescfs allow.mount.nolinprocfs allow.mount.nolinsysfs allow.mount.nonullfs allow.mount.noprocfs allow.mount.notmpfs allow.mount.nozfs allow.noquotas allow.noraw_sockets allow.set_hostname allow.nosocket_af allow.nosysvipc children.cur=0 children.max=0 cpuset.id=2 host.domainname="" host.hostid=0 host.hostname=test host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr= ip4.saddrsel ip6.addr= ip6.saddrsel I’ve tried debugging this, but the most I can say is that there appears to be something wrong with the reference counting. prison_deref() returns because pr->pr_ref is 3. There are no more jailed processes running, so I have no idea why this happens. The problem doesn’t appear to be related to vnet. I can reproduce it without setting the vnet flag when creating the jail as well. Regards, Kristof From owner-freebsd-jail@freebsd.org Thu Mar 9 03:22:24 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23E80CD4680 for ; Thu, 9 Mar 2017 03:22:24 +0000 (UTC) (envelope-from dewaynegeraghty@gmail.com) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E26FA12C1; Thu, 9 Mar 2017 03:22:23 +0000 (UTC) (envelope-from dewaynegeraghty@gmail.com) Received: by mail-io0-x232.google.com with SMTP id g6so20567381ioj.1; Wed, 08 Mar 2017 19:22:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=b+e9jAk2MWhG9ieeY9obazc+NQplfRd//cn1fJGi7+o=; b=YE9OBV7vBtMqOCab/s90DxpZt4DDqiwGKaNBxbBe3TbmEoJ6HEadExOw1REJ1Gk2kY 5UxOyiJpcelFcwPtBl1L4BzGf3FxZE+9gDg4+YEgVLnOyJcLwaLstls2hQZCdh/8zZfu 90UpCPEd674HBuyIn0WOqjPlOZHuhX9wyA+Hzh2clzlPN7HrmJB1gMQzNbIUDaRwDYme G9HPo0aCRZUO/j+Ok3crUNQKElpv9BIofN58v+PYr9awRbIAvbuevLR/LT9bKI1hgqeL h9Fuaddosm0fdFFhD1oYIeVvxBK/HZ87bRnnm5ba2MKIlvQTB7xKmR0lASl8+8Uz+TuB kylA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=b+e9jAk2MWhG9ieeY9obazc+NQplfRd//cn1fJGi7+o=; b=dQJ0XmjRv+fugq87v3M+zzHBGY/qAJ4vTQRkvfKYCjYeoTMaDSQOvkp5Dj9QWCDdwW UUVhMzbJkbqqIFBO1i/jdho3hksoAmgAHywHsEorn+3ikgkoN8Dt1DTaMK2Kyh1Sl4GS jKf0FAjwpl9QB5EHpoJPs8ZmrCGw6a+YoEzNf9WdESN17Ri0ZE2gVjR+QPg1EIddCWte oDTgnTzOe5jpFldimwFIE8aVP/4NV7uGAX3cXlYHQ/FpdhsyulBUDRU+wXzBLW8zCL5W LkAsAsDRhTKsiCY3EbiIHSAylrCY/WCK4lVznij9QopfbNLA2eF5+V1da+B8XyMNzefq 6ADA== X-Gm-Message-State: AMke39m6f9hH2LuHSuVpZ2K2AuX2dQx1E1YIUoKJ0gqnNJaSp4iP7Jt2QBFYj8rcAh19JBDE+TwfoZKXELdRfQ== X-Received: by 10.107.33.8 with SMTP id h8mr10334698ioh.144.1489029743133; Wed, 08 Mar 2017 19:22:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.13.142 with HTTP; Wed, 8 Mar 2017 19:21:52 -0800 (PST) In-Reply-To: References: From: Dewayne Geraghty Date: Thu, 9 Mar 2017 14:21:52 +1100 Message-ID: Subject: Re: Jail stuck in dying To: Kristof Provost Cc: "freebsd-jail@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 03:22:24 -0000 Kristof, would you share your devfs.rules settings for the jail that remains in a dying state. I have a similar situation and similar jail.conf settings, in that I use: allow.sysvipc; allow.socket_af; allow.raw_sockets; However I have other jails that stop and can be restarted with the same attributes; but different device rules. So what's different? For the jail that gets stuck in state "dying" devfs.rules: add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path 'bpf' unhide add path 'bpf0' unhide As a side note: in my jail.conf I also have jid=3, ie b3 { jid=3; ip4.addr = "10.0.7.91,10.0.5.91,127.1.5.91"; devfs_ruleset = "7"; allow.sysvipc; allow.socket_af; allow.raw_sockets; } (Yes its an application testing jail) Regards, Dewayne. From owner-freebsd-jail@freebsd.org Thu Mar 9 04:37:43 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A81AD03C26 for ; Thu, 9 Mar 2017 04:37:43 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88EB6F2 for ; Thu, 9 Mar 2017 04:37:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.1.0.36] (unknown [202.238.143.220]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 846011E873; Thu, 9 Mar 2017 05:37:37 +0100 (CET) From: "Kristof Provost" To: "Dewayne Geraghty" Cc: "freebsd-jail@freebsd.org" Subject: Re: Jail stuck in dying Date: Thu, 09 Mar 2017 13:37:33 +0900 Message-ID: In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; markup=markdown X-Mailer: MailMate (2.0BETAr6080) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 04:37:43 -0000 On 9 Mar 2017, at 12:21, Dewayne Geraghty wrote: > Kristof, would you share your devfs.rules settings for the jail that > remains in a dying state. > My devfs.rules is entirely standard. http://people.freebsd.org/~kp/devfs.rules Regards, Kristof From owner-freebsd-jail@freebsd.org Thu Mar 9 15:19:05 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F878D04520 for ; Thu, 9 Mar 2017 15:19:05 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [199.192.165.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gritton.org", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F6FAFE8; Thu, 9 Mar 2017 15:19:04 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [199.192.165.131]) by gritton.org (8.15.2/8.15.2) with ESMTP id v29EpwCL050725; Thu, 9 Mar 2017 07:51:58 -0700 (MST) (envelope-from jamie@freebsd.org) Received: (from www@localhost) by gritton.org (8.15.2/8.15.2/Submit) id v29EpwOs050724; Thu, 9 Mar 2017 07:51:58 -0700 (MST) (envelope-from jamie@freebsd.org) X-Authentication-Warning: gritton.org: www set sender to jamie@freebsd.org using -f To: freebsd-jail@freebsd.org Subject: Re: Jail stuck in dying X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Thu, 09 Mar 2017 07:51:58 -0700 From: James Gritton Cc: Kristof Provost In-Reply-To: References: Message-ID: X-Sender: jamie@freebsd.org User-Agent: Roundcube Webmail/1.2.3 X-Greylist: inspected by milter-greylist-4.6.2 (gritton.org [199.192.165.131]); Thu, 09 Mar 2017 07:51:58 -0700 (MST) for IP:'199.192.165.131' DOMAIN:'gritton.org' HELO:'gritton.org' FROM:'jamie@freebsd.org' RCPT:'' X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (gritton.org [199.192.165.131]); Thu, 09 Mar 2017 07:51:58 -0700 (MST) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 15:19:05 -0000 On 2017-03-08 19:01, Kristof Provost wrote: > Hi, > > On a current box (r314933) I can’t seem to stop my jails. > > It’s started like this: > > jail -c name=test0 host.hostname=test vnet persist > vnet.interface=epair0b \ > path=/usr/jails/jail1 exec.start="/bin/sh /etc/rc" > > I terminate it with `jail -R test0`, yet the jail stays stuck in dying > state: > > $ jls -a > JID IP Address Hostname Path > 1 test /usr/jails/jail1 > > $ jls -na > devfs_ruleset=0 dying enforce_statfs=2 host=new ip4=inherit > ip6=inherit jid=1 name=test0 osreldate=1200023 osrelease=12.0-CURRENT > parent=0 path=/usr/jails/jail1 nopersist securelevel=-1 > sysvmsg=disable sysvsem=disable sysvshm=disable vnet=new > allow.nochflags allow.nomount allow.mount.nodevfs > allow.mount.nofdescfs allow.mount.nolinprocfs allow.mount.nolinsysfs > allow.mount.nonullfs allow.mount.noprocfs allow.mount.notmpfs > allow.mount.nozfs allow.noquotas allow.noraw_sockets > allow.set_hostname allow.nosocket_af allow.nosysvipc children.cur=0 > children.max=0 cpuset.id=2 host.domainname="" host.hostid=0 > host.hostname=test host.hostuuid=00000000-0000-0000-0000-000000000000 > ip4.addr= ip4.saddrsel ip6.addr= ip6.saddrsel > > I’ve tried debugging this, but the most I can say is that there > appears to be something wrong with the reference counting. > prison_deref() returns because pr->pr_ref is 3. There are no more > jailed processes running, so I have no idea why this happens. > > The problem doesn’t appear to be related to vnet. I can reproduce it > without setting the vnet flag when creating the jail as well. It's never about processes - dying jails are those that have no processes but still have other references to them in the kernel. Those "other references" are almost always through the cred system: crcopy() calls prison_hold() to increase the associated prison's pf_ref, and crfree() calls prison_free(). The hard part is tracking down just what might be holding such a credential; there is nothing that points from a prison to its associated creds. But it's almost always associated with the network stack. In normal operation, a jail may stick around for a little while in the dying state until its just-closed TCP connections time out. I've heard report of NFS mounts associated with jails sometimes causing such references that don't go away. There are of course many other places that use creds, but most of them are in some way associated with processes. As long as the problem only manifests as jails in the dying list, the easiest solution is to ignore it. As long as you don't have hard-coded JIDs in jail.conf, you can re-create the jail and it will get a different JID. The need for a particular JID isn't what it used to be, and chances are you can get away with dynamically numbered jails. If you want to track down what's holding the jail half-alive, it becomes a matter of find out exactly what (probably network) resources a jail is using. - Jamie