From owner-freebsd-jail@FreeBSD.ORG Mon Aug 15 11:07:05 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67DAF106566C for ; Mon, 15 Aug 2011 11:07:05 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 56C408FC23 for ; Mon, 15 Aug 2011 11:07:05 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p7FB75lJ014776 for ; Mon, 15 Aug 2011 11:07:05 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p7FB74dG014772 for freebsd-jail@FreeBSD.org; Mon, 15 Aug 2011 11:07:04 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 15 Aug 2011 11:07:04 GMT Message-Id: <201108151107.p7FB74dG014772@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-jail@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-jail@FreeBSD.org X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Aug 2011 11:07:05 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/156111 jail [jail] procstat -b not supported in jail o misc/155765 jail [patch] `buildworld' does not honors WITHOUT_JAIL o conf/154246 jail [jail] [patch] Bad symlink created if devfs mount poin o conf/149050 jail [jail] rcorder ``nojail'' too coarse for Jail+VNET s conf/142972 jail [jail] [patch] Support JAILv2 and vnet in rc.d/jail o conf/141317 jail [patch] uncorrect jail stop in /etc/rc.d/jail o kern/133265 jail [jail] is there a solution how to run nfs client in ja o kern/119842 jail [smbfs] [jail] "Bad address" with smbfs inside a jail o bin/99566 jail [jail] [patch] fstat(1) according to specified jid o bin/32828 jail [jail] w(1) incorrectly handles stale utmp slots with 10 problems total. From owner-freebsd-jail@FreeBSD.ORG Wed Aug 17 20:36:48 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D3DD106566B for ; Wed, 17 Aug 2011 20:36:48 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 066568FC08 for ; Wed, 17 Aug 2011 20:36:46 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA00935; Wed, 17 Aug 2011 23:21:43 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QtmcR-000Du9-7E; Wed, 17 Aug 2011 23:21:43 +0300 Message-ID: <4E4C22D6.6070407@FreeBSD.org> Date: Wed, 17 Aug 2011 23:21:42 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110817 Thunderbird/6.0 MIME-Version: 1.0 To: freebsd-jail@FreeBSD.org, freebsd-hackers@FreeBSD.org References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org> <4E43E272.1060204@FreeBSD.org> <62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk> <4E440865.1040500@FreeBSD.org> <6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk> <4E441314.6060606@FreeBSD.org> <2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk> <4E48D967.9060804@FreeBSD.org> <9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk> <4E490DAF.1080009@FreeBSD.org> <796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk> <4E491D01.1090902@FreeBSD.org> <570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk> <4E4AD35C.7020504@FreeBSD.org> <6A7238AED44542A880B082A40304D940@multiplay.co.uk> <4E4BA21F.6010805@FreeBSD.org> <581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk> <4E4BBA7F.30907@FreeBSD.org> <88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> In-Reply-To: <88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> X-Enigmail-Version: 1.2.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 20:36:48 -0000 Thanks to the debug that Steven provided and to the help that I received from Kostik, I think that now I understand the basic mechanics of this panic, but, unfortunately, not the details of its root cause. It seems like everything starts with some kind of a race between terminating processes in a jail and termination of the jail itself. This is where the details are very thin so far. What we see is that a process (http) is in exit(2) syscall, in exit1() function actually, and past the place where P_WEXIT flag is set and even past the place where p_limit is freed and reset to NULL. At that place the thread calls prison_proc_free(), which calls prison_deref(). Then, we see that in prison_deref() the thread gets a page fault because of what seems like a NULL pointer dereference. That's just the start of the problem and its root cause. Then, trap_pfault() gets invoked and, because addresses close to NULL look like userspace addresses, vm_fault/vm_fault_hold gets called, which in its turn goes on to call vm_map_growstack. First thing that vm_map_growstack does is a call to lim_cur(), but because p_limit is already NULL, that call results in a NULL pointer dereference and a page fault. Goto the beginning of this paragraph. So we get this recursion of sorts, which only ends when a stack is exhausted and a CPU generates a double-fault. So, of course, Steven is interested in finding and fixing the root cause. I hope we will get to that with some help from the "prison guards" :-) But I also would like to use this opportunity to discuss how we can make it easier to debug such issue as this. I think that this problem demonstrates that when we treat certain junk in kernel address value as a userland address value, we throw additional heaps of irrelevant stuff on top of an actual problem. One solution could be to use a special flag that would mark all actual attempts to access userland address (e.g. setting the flag on entrance to copyin and clearing it upon return), so that in the page fault handler we could distinguish actual faults on userland addresses from faults on garbage kernel addresses. I am sure that there could be other clever techniques to catch such garbage addresses early. -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Wed Aug 17 21:29:51 2011 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48700106564A for ; Wed, 17 Aug 2011 21:29:51 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id DC9C08FC14 for ; Wed, 17 Aug 2011 21:29:50 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p7HLAnic075382 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 18 Aug 2011 00:10:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p7HLAm2E007269; Thu, 18 Aug 2011 00:10:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p7HLAmDw007268; Thu, 18 Aug 2011 00:10:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 18 Aug 2011 00:10:48 +0300 From: Kostik Belousov To: Andriy Gapon Message-ID: <20110817211048.GZ17489@deviant.kiev.zoral.com.ua> References: <796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk> <4E491D01.1090902@FreeBSD.org> <570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk> <4E4AD35C.7020504@FreeBSD.org> <6A7238AED44542A880B082A40304D940@multiplay.co.uk> <4E4BA21F.6010805@FreeBSD.org> <581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk> <4E4BBA7F.30907@FreeBSD.org> <88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> <4E4C22D6.6070407@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MPHowW9WJBu+8Ajw" Content-Disposition: inline In-Reply-To: <4E4C22D6.6070407@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-hackers@freebsd.org, freebsd-jail@freebsd.org, freebsd-stable@freebsd.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 21:29:51 -0000 --MPHowW9WJBu+8Ajw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 17, 2011 at 11:21:42PM +0300, Andriy Gapon wrote: [skip] > But I also would like to use this opportunity to discuss how we can > make it easier to debug such issue as this. I think that this problem > demonstrates that when we treat certain junk in kernel address value > as a userland address value, we throw additional heaps of irrelevant > stuff on top of an actual problem. One solution could be to use a > special flag that would mark all actual attempts to access userland > address (e.g. setting the flag on entrance to copyin and clearing it > upon return), so that in the page fault handler we could distinguish > actual faults on userland addresses from faults on garbage kernel > addresses. I am sure that there could be other clever techniques to > catch such garbage addresses early. We already have such mechanism, the kernel code aware of the usermode page access sets pcb_onfault. See the end of trap_pfault() handler. In fact, we can catch it earlier, before even calling vm_fault(). BTW, I think this is esp. useful in the combination with the support for the SMEP in recent Intel CPUs. commit 2e1b36fa93f9499e37acf04a66ff0646d4f13536 Author: Konstantin Belousov Date: Thu Aug 18 00:08:50 2011 +0300 Assert that the exiting process does not return to usermode. On x86, do not call vm_fault() when the kernel is not prepared to handle unsuccessful page fault. diff --git a/sys/amd64/amd64/trap.c b/sys/amd64/amd64/trap.c index 4e5f8b8..55e1e5a 100644 --- a/sys/amd64/amd64/trap.c +++ b/sys/amd64/amd64/trap.c @@ -674,6 +674,19 @@ trap_pfault(frame, usermode) goto nogo; =20 map =3D &vm->vm_map; + + /* + * When accessing a usermode address, kernel must be + * ready to accept the page fault, and provide a + * handling routine. Since accessing the address + * without the handler is a bug, do not try to handle + * it normally, and panic immediately. + */ + if (!usermode && (td->td_intr_nesting_level !=3D 0 || + PCPU_GET(curpcb)->pcb_onfault =3D=3D NULL)) { + trap_fatal(frame, eva); + return (-1); + } } =20 /* diff --git a/sys/i386/i386/trap.c b/sys/i386/i386/trap.c index 5a8016c..e6d2b5a 100644 --- a/sys/i386/i386/trap.c +++ b/sys/i386/i386/trap.c @@ -831,6 +831,11 @@ trap_pfault(frame, usermode, eva) goto nogo; =20 map =3D &vm->vm_map; + if (!usermode && (td->td_intr_nesting_level !=3D 0 || + PCPU_GET(curpcb)->pcb_onfault =3D=3D NULL)) { + trap_fatal(frame, eva); + return (-1); + } } =20 /* diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c index 3527ed1..a69b7b8 100644 --- a/sys/kern/subr_trap.c +++ b/sys/kern/subr_trap.c @@ -99,6 +99,8 @@ userret(struct thread *td, struct trapframe *frame) =20 CTR3(KTR_SYSC, "userret: thread %p (pid %d, %s)", td, p->p_pid, td->td_name); + KASSERT((p->p_flag & P_WEXIT) =3D=3D 0, + ("Exiting process returns to usermode")); #if 0 #ifdef DIAGNOSTIC /* Check that we called signotify() enough. */ --MPHowW9WJBu+8Ajw Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk5MLlgACgkQC3+MBN1Mb4hyewCgpKYy+yhG+S3bXm5A324n/C8+ 6lIAoPRTszmVWdyBQqw5vhJUnpNbhluY =i6E1 -----END PGP SIGNATURE----- --MPHowW9WJBu+8Ajw-- From owner-freebsd-jail@FreeBSD.ORG Wed Aug 17 23:26:40 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 58841106566C; Wed, 17 Aug 2011 23:26:40 +0000 (UTC) (envelope-from prvs=1210f20b9f=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 9B4928FC17; Wed, 17 Aug 2011 23:26:39 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 18 Aug 2011 00:15:17 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 18 Aug 2011 00:15:17 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014640704.msg; Thu, 18 Aug 2011 00:15:16 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1210f20b9f=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <4019027648B5493AAC4B654BD821DE88@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" , , References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> <4E4C22D6.6070407@FreeBSD.org> Date: Thu, 18 Aug 2011 00:15:56 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 23:26:40 -0000 ----- Original Message ----- From: "Andriy Gapon" > Thanks to the debug that Steven provided and to the help that I received from > Kostik, I think that now I understand the basic mechanics of this panic, but, > unfortunately, not the details of its root cause. > > It seems like everything starts with some kind of a race between terminating > processes in a jail and termination of the jail itself. This is where the > details are very thin so far. What we see is that a process (http) is in > exit(2) syscall, in exit1() function actually, and past the place where P_WEXIT > flag is set and even past the place where p_limit is freed and reset to NULL. > At that place the thread calls prison_proc_free(), which calls prison_deref(). > Then, we see that in prison_deref() the thread gets a page fault because of what > seems like a NULL pointer dereference. That's just the start of the problem and > its root cause. Thats interesting, are you using http as an example or is that something thats been gleaned from the debugging of our output? I ask as there's only one process running in each of our jails and thats a single java process. Now given your description there may be something I can add that may help clarify what the cause could be. In a nutshell the jail manager we're using will attempt to resurrect the jail from a dieing state in a few specific scenarios. Here's an exmaple:- 1. jail restart requested 2. jail is stopped, so the java processes is killed off, but active tcp sessions may prevent the timely full shutdown of the jail. 3. if an existing jail is detected, i.e. a dieing jail from #2, instead of starting a new jail we attach to the old one and exec the new java process. 4. if an existing jail isnt detected, i.e. where there where not hanging tcp sessions and #2 cleanly shutdown the jail, a new jail is created, attached to and the java exec'ed. The system uses static jailid's so its possible to determine if an existing jail for this "service" exists or not. This prevents duplicate services as well as making services easy to identify by their jailid. So what we could be seeing is a race between the jail shutdown and the attach of the new process? Now man 2 jail seems to indicate this is a valid use case for jail_set, as it documents its support for JAIL_DYING as a valid option for flags, but I suspect its something quite out of the ordinary to actually do, which may be why this panic hasnt been seen before now. As some background the reason we use static jailid's is to ensure only one instance of the jailed service is running, and the reason we re-attach to the dieing jail is so that jails can be restarted in a timely manor. Without using the re-attach we would need to wait of all tcp sessions which have been aborted to timeout. > So, of course, Steven is interested in finding and fixing the root cause. I > hope we will get to that with some help from the "prison guards" :-) Does the above potentially explain how we're getting to the situation which generates the panic? If so we can certainly look at using alternatives to the current design to workaround this issue. Flagging the jail as permanent and using manual process management and additional external locking to prevent duplicates, is what instantly springs to mind. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Thu Aug 18 09:21:21 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CABC9106566C; Thu, 18 Aug 2011 09:21:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id AB12E8FC22; Thu, 18 Aug 2011 09:21:20 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA10730; Thu, 18 Aug 2011 12:21:17 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E4CD98C.1000301@FreeBSD.org> Date: Thu, 18 Aug 2011 12:21:16 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> <4E4C22D6.6070407@FreeBSD.org> <4019027648B5493AAC4B654BD821DE88@multiplay.co.! uk> In-Reply-To: <4019027648B5493AAC4B654BD821DE88@multiplay.co.uk> X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org, freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 09:21:21 -0000 on 18/08/2011 02:15 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" > >> Thanks to the debug that Steven provided and to the help that I received from >> Kostik, I think that now I understand the basic mechanics of this panic, but, >> unfortunately, not the details of its root cause. >> >> It seems like everything starts with some kind of a race between terminating >> processes in a jail and termination of the jail itself. This is where the >> details are very thin so far. What we see is that a process (http) is in >> exit(2) syscall, in exit1() function actually, and past the place where P_WEXIT >> flag is set and even past the place where p_limit is freed and reset to NULL. >> At that place the thread calls prison_proc_free(), which calls prison_deref(). >> Then, we see that in prison_deref() the thread gets a page fault because of what >> seems like a NULL pointer dereference. That's just the start of the problem and >> its root cause. > > Thats interesting, are you using http as an example or is that something thats > been gleaned from the debugging of our output? I ask as there's only one process > running in each of our jails and thats a single java process. It's from the debug data: p_comm = "httpd" I also would like to ask you to revert the last patch that I sent you (with tf_rip comparisons) and try the patch from Kostik instead. Given what we suspect about the problem, can please also try to provoke the problem by e.g. doing frequent jail restarts or something else that supposedly should hit the bug. -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Thu Aug 18 10:47:04 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AC10106566C; Thu, 18 Aug 2011 10:47:04 +0000 (UTC) (envelope-from prvs=12111cb08a=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 312988FC18; Thu, 18 Aug 2011 10:47:02 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 18 Aug 2011 11:35:21 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 18 Aug 2011 11:35:21 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014645776.msg; Thu, 18 Aug 2011 11:35:20 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12111cb08a=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Andriy Gapon" References: uk> <4E4CD98C.1000301@FreeBSD.org> Date: Thu, 18 Aug 2011 11:35:58 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-hackers@FreeBSD.org, freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 10:47:04 -0000 ----- Original Message ----- From: "Andriy Gapon" >> Thats interesting, are you using http as an example or is that something thats >> been gleaned from the debugging of our output? I ask as there's only one process >> running in each of our jails and thats a single java process. > > > It's from the debug data: p_comm = "httpd" Hmm, there's only one httpd thats ever run on the machine and thats not in the jail its on the raw machine. > I also would like to ask you to revert the last patch that I sent you (with tf_rip > comparisons) and try the patch from Kostik instead. Sure. > Given what we suspect about the problem, can please also try to provoke the > problem by e.g. doing frequent jail restarts or something else that supposedly > should hit the bug. I've tried doing this for quite some days on the test machine, but I've been unable to provoke it, will continue to try. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Thu Aug 18 11:11:08 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32E58106566C; Thu, 18 Aug 2011 11:11:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 21EA88FC1B; Thu, 18 Aug 2011 11:11:06 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA12584; Thu, 18 Aug 2011 14:11:04 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E4CF347.6030908@FreeBSD.org> Date: Thu, 18 Aug 2011 14:11:03 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland References: uk> <4E4CD98C.1000301@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org, freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 11:11:08 -0000 on 18/08/2011 13:35 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" >>> Thats interesting, are you using http as an example or is that something thats >>> been gleaned from the debugging of our output? I ask as there's only one process >>> running in each of our jails and thats a single java process. >> >> >> It's from the debug data: p_comm = "httpd" > > Hmm, there's only one httpd thats ever run on the machine and thats not in the jail > its on the raw machine. Probably I have mistakenly assumed that the 'prison' in prison_derefer() has something to do with an actual jail, while it could have been just prison0 where all non-jailed processes belong. -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Thu Aug 18 11:26:06 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17692106566C; Thu, 18 Aug 2011 11:26:06 +0000 (UTC) (envelope-from prvs=12111cb08a=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 08B508FC16; Thu, 18 Aug 2011 11:26:04 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 18 Aug 2011 12:24:30 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 18 Aug 2011 12:24:30 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014646198.msg; Thu, 18 Aug 2011 12:24:29 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12111cb08a=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Andriy Gapon" References: uk> <4E4CD98C.1000301@FreeBSD.org> <4E4CF347.6030908@FreeBSD.org> Date: Thu, 18 Aug 2011 12:25:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-hackers@FreeBSD.org, freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 11:26:06 -0000 ----- Original Message ----- From: "Andriy Gapon" > Probably I have mistakenly assumed that the 'prison' in prison_derefer() has > something to do with an actual jail, while it could have been just prison0 where > all non-jailed processes belong. That makes sense as this particular panic was caused by a machine reboot, which is slightly different from the more common jail panic we're seeing. Doesn't help with our reproduction scenario though unfortunately. If we don't have any joy reproducing on our single test machine I'll have this kernel rolled out across a portion of the farm, which should mean we see the panic results in a few days time. I understand there's a risk involved in this but, its important for us to determine the cause and get a confirmed fix, as well as being able to prove that the panic fix works which will help everyone in the long run. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Thu Aug 18 14:31:28 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2C4F106566B; Thu, 18 Aug 2011 14:31:28 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B50648FC1F; Thu, 18 Aug 2011 14:31:27 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA15396; Thu, 18 Aug 2011 17:31:23 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E4D222E.2090802@FreeBSD.org> Date: Thu, 18 Aug 2011 17:31:10 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Steven Hartland , freebsd-jail@FreeBSD.org References: uk> <4E4CD98C.1000301@FreeBSD.org> <4E4CF347.6030908@FreeBSD.org> In-Reply-To: <4E4CF347.6030908@FreeBSD.org> X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers , freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 14:31:29 -0000 on 18/08/2011 14:11 Andriy Gapon said the following: > Probably I have mistakenly assumed that the 'prison' in prison_derefer() has > something to do with an actual jail, while it could have been just prison0 where > all non-jailed processes belong. So, indeed: (kgdb) p $2->p_ucred->cr_prison $10 = (struct prison *) 0xffffffff807d5080 (kgdb) p &prison0 $11 = (struct prison *) 0xffffffff807d5080 (kgdb) p *$2->p_ucred->cr_prison $12 = {pr_list = {tqe_next = 0x0, tqe_prev = 0x0}, pr_id = 0, pr_ref = 398, pr_uref = 0, pr_flags = 386, pr_children = {lh_first = 0x0}, pr_sibling = {le_next = 0x0, le_prev = 0x0}, pr_parent = 0x0, pr_mtx = {lock_object = {lo_name = 0xffffffff8063007c "jail mutex", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, pr_task = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0, ta_context = 0x0}, pr_osd = {osd_nslots = 0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, pr_cpuset = 0xffffff0012d65dc8, pr_vnet = 0x0, pr_root = 0xffffff00166ebce8, pr_ip4s = 0, pr_ip6s = 0, pr_ip4 = 0x0, pr_ip6 = 0x0, pr_sparep = {0x0, 0x0, 0x0, 0x0}, pr_childcount = 0, pr_childmax = 999999, pr_allow = 127, pr_securelevel = -1, pr_enforce_statfs = 0, pr_spare = {0, 0, 0, 0, 0}, pr_hostid = 3251597242, pr_name = "0", '\0' , pr_path = "/", '\0' , pr_hostname = "censored", '\0' , pr_domainname = '\0' , pr_hostuuid = "54443842-0054-2500-902c-0025902c3cb0", '\0' } Also, let's consider this code: if (flags & PD_DEUREF) { for (tpr = pr;; tpr = tpr->pr_parent) { if (tpr != pr) mtx_lock(&tpr->pr_mtx); if (--tpr->pr_uref > 0) break; KASSERT(tpr != &prison0, ("prison0 pr_uref=0")); mtx_unlock(&tpr->pr_mtx); } /* Done if there were only user references to remove. */ if (!(flags & PD_DEREF)) { mtx_unlock(&tpr->pr_mtx); if (flags & PD_LIST_SLOCKED) sx_sunlock(&allprison_lock); else if (flags & PD_LIST_XLOCKED) sx_xunlock(&allprison_lock); return; } if (tpr != pr) { mtx_unlock(&tpr->pr_mtx); mtx_lock(&pr->pr_mtx); } } The most suspicious thing is that pr_uref is zero in the debug data. With INVARIANTS we would hit the "prison0 pr_uref=0" KASSERT. Then, because this is prison0 and because pr_uref reached zero, tpr gets assigned to NULL. And then because tpr != pr we try to execute mtx_unlock(&tpr->pr_mtx). That's where the NULL pointer deref happens. So, now the big question is how/why we reached pr_uref == 0. -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 04:10:25 2011 Return-Path: Delivered-To: freebsd-jail@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F66E106566B; Sat, 20 Aug 2011 04:10:25 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3E5BB8FC08; Sat, 20 Aug 2011 04:10:25 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p7K4AP2t036415; Sat, 20 Aug 2011 04:10:25 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p7K4APM5036406; Sat, 20 Aug 2011 04:10:25 GMT (envelope-from linimon) Date: Sat, 20 Aug 2011 04:10:25 GMT Message-Id: <201108200410.p7K4APM5036406@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-jail@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/159918: [jail] inter-jail communication failure X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 04:10:25 -0000 Old Synopsis: inter-jail communication failure New Synopsis: [jail] inter-jail communication failure Responsible-Changed-From-To: freebsd-bugs->freebsd-jail Responsible-Changed-By: linimon Responsible-Changed-When: Sat Aug 20 04:09:27 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=159918 From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 10:02:32 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03FB9106564A; Sat, 20 Aug 2011 10:02:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 157338FC08; Sat, 20 Aug 2011 10:02:30 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA10374; Sat, 20 Aug 2011 13:02:27 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QuiNn-000NrY-IE; Sat, 20 Aug 2011 13:02:27 +0300 Message-ID: <4E4F8631.1070300@FreeBSD.org> Date: Sat, 20 Aug 2011 13:02:25 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110819 Thunderbird/6.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> <4E4C22D6.6070407@FreeBSD.org> <4019027648B5493AAC4B654BD821DE88@multiplay.co.! uk> In-Reply-To: <4019027648B5493AAC4B654BD821DE88@multiplay.co.uk> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 10:02:32 -0000 on 18/08/2011 02:15 Steven Hartland said the following: > In a nutshell the jail manager we're using will attempt to resurrect the jail > from a dieing state in a few specific scenarios. > > Here's an exmaple:- > 1. jail restart requested > 2. jail is stopped, so the java processes is killed off, but active tcp sessions > may prevent the timely full shutdown of the jail. > 3. if an existing jail is detected, i.e. a dieing jail from #2, instead of > starting a new jail we attach to the old one and exec the new java process. > 4. if an existing jail isnt detected, i.e. where there where not hanging tcp > sessions and #2 cleanly shutdown the jail, a new jail is created, attached to > and the java exec'ed. > > The system uses static jailid's so its possible to determine if an existing > jail for this "service" exists or not. This prevents duplicate services as > well as making services easy to identify by their jailid. > > So what we could be seeing is a race between the jail shutdown and the attach > of the new process? Not a jail expert at all, but a few suggestions... First, wouldn't the 'persist' jail option simplify your life a little bit? Second, you may want to try to monitor value of prison0.pr_uref variable (e.g. via kgdb) while executing various scenarios of what you do now. If after finishing a certain scenario you end up with a value lower than at the start of scenario, then this is the troublesome one. Please note that prison0.pr_uref is composed from a number of non-jailed processes plus a number of top-level jails. So take this into account when comparing prison0.pr_uref values - it's better to record the initial value when no jails are started and it's important to keep the number of non-jailed processes the same (or to account for its changes). -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 10:10:55 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C3F0106564A; Sat, 20 Aug 2011 10:10:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 91EF98FC17; Sat, 20 Aug 2011 10:10:44 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA10423; Sat, 20 Aug 2011 13:10:42 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QuiVl-000Nrj-VF; Sat, 20 Aug 2011 13:10:41 +0300 Message-ID: <4E4F8821.80108@FreeBSD.org> Date: Sat, 20 Aug 2011 13:10:41 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110819 Thunderbird/6.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> <4E4C22D6.6070407@FreeBSD.org> <4019027648B5493AAC4B654BD821DE88@multiplay.co.uk> <4E4F8631.1070300@FreeBSD.org> In-Reply-To: <4E4F8631.1070300@FreeBSD.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 10:10:55 -0000 on 20/08/2011 13:02 Andriy Gapon said the following: > on 18/08/2011 02:15 Steven Hartland said the following: >> In a nutshell the jail manager we're using will attempt to resurrect the jail >> from a dieing state in a few specific scenarios. >> >> Here's an exmaple:- >> 1. jail restart requested >> 2. jail is stopped, so the java processes is killed off, but active tcp sessions >> may prevent the timely full shutdown of the jail. >> 3. if an existing jail is detected, i.e. a dieing jail from #2, instead of >> starting a new jail we attach to the old one and exec the new java process. >> 4. if an existing jail isnt detected, i.e. where there where not hanging tcp >> sessions and #2 cleanly shutdown the jail, a new jail is created, attached to >> and the java exec'ed. >> >> The system uses static jailid's so its possible to determine if an existing >> jail for this "service" exists or not. This prevents duplicate services as >> well as making services easy to identify by their jailid. >> >> So what we could be seeing is a race between the jail shutdown and the attach >> of the new process? > > Not a jail expert at all, but a few suggestions... > > First, wouldn't the 'persist' jail option simplify your life a little bit? > > Second, you may want to try to monitor value of prison0.pr_uref variable (e.g. > via kgdb) while executing various scenarios of what you do now. If after > finishing a certain scenario you end up with a value lower than at the start of > scenario, then this is the troublesome one. > Please note that prison0.pr_uref is composed from a number of non-jailed > processes plus a number of top-level jails. So take this into account when > comparing prison0.pr_uref values - it's better to record the initial value when > no jails are started and it's important to keep the number of non-jailed > processes the same (or to account for its changes). BTW, I suspect the following scenario, but I am not able to verify it either via testing or in the code: - last process in a dying jail exits - pr_uref of the jail reaches zero - pr_uref of prison0 gets decremented - you attach to the jail and resurrect it - but pr_uref of prison0 stays decremented Repeat this enough times and prison0.pr_uref reaches zero. To reach zero even sooner just kill enough of non-jailed processes. -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 13:24:51 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05984106566B; Sat, 20 Aug 2011 13:24:51 +0000 (UTC) (envelope-from prvs=12137168ef=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 1125A8FC0C; Sat, 20 Aug 2011 13:24:49 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 14:13:34 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 14:13:34 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014672793.msg; Sat, 20 Aug 2011 14:13:34 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12137168ef=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <4E55CB4A4F694A7997FEBDF9EADF87F5@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk> <4E4C22D6.6070407@FreeBSD.org> <4019027648B5493AAC4B654BD821DE88@multiplay.co.uk> <4E4F8631.1070300@FreeBSD.org> <4E4F8821.80108@ FreeBSD.org> Date: Sat, 20 Aug 2011 14:14:15 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 13:24:51 -0000 ----- Original Message ----- From: "Andriy Gapon" > BTW, I suspect the following scenario, but I am not able to > verify it either via testing or in the code: > - last process in a dying jail exits > - pr_uref of the jail reaches zero > - pr_uref of prison0 gets decremented > - you attach to the jail and resurrect it > - but pr_uref of prison0 stays decremented > > Repeat this enough times and prison0.pr_uref reaches zero. > To reach zero even sooner just kill enough of non-jailed processes. Ahh now that explains all of our experienced panic scenarios:- 1. jail stop / start causing the panic but only after at least a few days worth of uptime. Here what we're seeing is enough "leak" of pr_uref from the restarted jails to decrement prison0.pr_uref to 0 even with all the standard unjailed processes still running. 2. A machine reboot, after all jails have been stopped but after less time than #2. In this case we haven't seen enough leakage to decrement prison0.pr_uref to 0 given the number or prison0 process but it has been incorrectly decremented, so as soon as the reboot kicks in and prison0 processes start exiting, prison0.pr_uref gets further decremented and again hits 0 when it shouldn't Now if this is the case, we should be able to confirm it with a little more info. 1. What exactly does pr_uref represent? 2. Can what its value should be, be calculated from examining other details of the system i.e. number of running processes, number of running jails? If we can calculate the value that prison0.pr_uref should be, then by examining the machines we have which have been up for a while, we should be able to confirm if an incorrect value is present on them and hence prove this is the case. Ideally a little script to run in kgdb to test this would be the best way to go. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 15:51:03 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10E9E106566C; Sat, 20 Aug 2011 15:51:03 +0000 (UTC) (envelope-from prvs=12137168ef=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 2364E8FC15; Sat, 20 Aug 2011 15:51:01 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 16:50:27 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 16:50:27 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014673864.msg; Sat, 20 Aug 2011 16:50:26 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12137168ef=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E4380C0.7070908@FreeBSD.org><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk><4E4C22D6.6070407@FreeBSD.org><4019027648B5493AAC4B654BD821DE88@multiplay.co.uk><4E4F8631.1070300@FreeBSD.org> <4E4F8821.80108@Fre eBSD.org> Date: Sat, 20 Aug 2011 16:51:50 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 15:51:03 -0000 ----- Original Message ----- From: "Andriy Gapon" > BTW, I suspect the following scenario, but I am not able to verify it either via > testing or in the code: > - last process in a dying jail exits > - pr_uref of the jail reaches zero > - pr_uref of prison0 gets decremented > - you attach to the jail and resurrect it > - but pr_uref of prison0 stays decremented > > Repeat this enough times and prison0.pr_uref reaches zero. > To reach zero even sooner just kill enough of non-jailed processes. I've just checked across a number of the panic dumps from the past few days and they all have prison0.pr_uref = 0 which confirms the cause of the panic. I've tried scripting continuous jail start stops, but even after 1000's of iterations have been unable to trigger this on my test machine, so I'm going to dig into the jail code to see if I can find out how its incorrectly decrementing prison0 via inspection. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 16:48:30 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E35821065677; Sat, 20 Aug 2011 16:48:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id EE0C48FC1C; Sat, 20 Aug 2011 16:48:29 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA12940; Sat, 20 Aug 2011 19:48:27 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Quoig-000O3V-LU; Sat, 20 Aug 2011 19:48:26 +0300 Message-ID: <4E4FE55A.9000101@FreeBSD.org> Date: Sat, 20 Aug 2011 19:48:26 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110819 Thunderbird/6.0 MIME-Version: 1.0 To: Steven Hartland References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk><4E4C22D6.6070407@FreeBSD.org><4019027648B5493AAC4B654BD821DE88@multiplay.co.uk><4E4F8631.1070300@FreeBSD.org> <4E4F8821.80108@Fre eBSD.org> <82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk> In-Reply-To: <82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 16:48:31 -0000 on 20/08/2011 18:51 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" > >> BTW, I suspect the following scenario, but I am not able to verify it either via >> testing or in the code: >> - last process in a dying jail exits >> - pr_uref of the jail reaches zero >> - pr_uref of prison0 gets decremented >> - you attach to the jail and resurrect it >> - but pr_uref of prison0 stays decremented >> >> Repeat this enough times and prison0.pr_uref reaches zero. >> To reach zero even sooner just kill enough of non-jailed processes. > > I've just checked across a number of the panic dumps from the > past few days and they all have prison0.pr_uref = 0 which confirms > the cause of the panic. > > I've tried scripting continuous jail start stops, but even after 1000's > of iterations have been unable to trigger this on my test machine, so > I'm going to dig into the jail code to see if I can find out how its > incorrectly decrementing prison0 via inspection. Steve, thanks for doing this! I'll reiterate my suspicion just in case - I think that you should look for the cases where you stop a jail, but then re-attach and resurrect the jail before it's completely dead. -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 18:23:30 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6852106566B; Sat, 20 Aug 2011 18:23:30 +0000 (UTC) (envelope-from marquis@roble.com) Received: from mx5.roble.com (mx5.roble.com [206.40.34.5]) by mx1.freebsd.org (Postfix) with ESMTP id B7EB08FC08; Sat, 20 Aug 2011 18:23:30 +0000 (UTC) Received: from mx5.roble.com (mx5.roble.com [206.40.34.5]) by mx5.roble.com (Postfix) with ESMTP id 2FCA867899; Sat, 20 Aug 2011 11:10:31 -0700 (PDT) Date: Sat, 20 Aug 2011 11:10:31 -0700 (PDT) From: Roger Marquis To: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org In-Reply-To: <82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk> References: <82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Message-Id: <20110820182330.C6852106566B@hub.freebsd.org> Cc: Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 18:23:30 -0000 >> Repeat this enough times and prison0.pr_uref reaches zero. >> To reach zero even sooner just kill enough of non-jailed processes. Interesting. We've been getting kernel panics in -stable but with only one jail started at boot without being restarted. Are you using SAS drives by any chance? Setting ethernet polling and HZ? How about softupdates, gmirror, and/or anything in sysctl.conf? Roger Marquis From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 18:30:46 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECA10106567A; Sat, 20 Aug 2011 18:30:46 +0000 (UTC) (envelope-from prvs=12137168ef=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 431938FC22; Sat, 20 Aug 2011 18:30:45 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 19:30:11 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 19:30:11 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014675318.msg; Sat, 20 Aug 2011 19:30:09 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12137168ef=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Roger Marquis" , , References: <82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk> <20110820182330.C6852106566B@hub.freebsd.org> Date: Sat, 20 Aug 2011 19:31:11 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 18:30:47 -0000 ----- Original Message ----- From: "Roger Marquis" To: ; Sent: Saturday, August 20, 2011 7:10 PM Subject: Re: debugging frequent kernel panics on 8.2-RELEASE >>> Repeat this enough times and prison0.pr_uref reaches zero. >>> To reach zero even sooner just kill enough of non-jailed processes. > > Interesting. We've been getting kernel panics in -stable but with only > one jail started at boot without being restarted. > > Are you using SAS drives by any chance? Setting ethernet polling and HZ? > How about softupdates, gmirror, and/or anything in sysctl.conf? If your not restarting things it may be unrelated. No SAS, polling is compiled in but no devices have it active and using ZFS only. Are you seeing a double fault panic? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 20:00:19 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B79E2106566B; Sat, 20 Aug 2011 20:00:17 +0000 (UTC) (envelope-from prvs=12137168ef=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id C2F038FC0C; Sat, 20 Aug 2011 20:00:16 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 20:59:42 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 20:59:41 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014676027.msg; Sat, 20 Aug 2011 20:59:41 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12137168ef=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Andriy Gapon" References: <47F0D04ADF034695BC8B0AC166553371@multiplay.co.uk><4E43E272.1060204@FreeBSD.org><62BF25D0ED914876BEE75E2ADF28DDF7@multiplay.co.uk><4E440865.1040500@FreeBSD.org><6F08A8DE780545ADB9FA93B0A8AA4DA1@multiplay.co.uk><4E441314.6060606@FreeBSD.org><2C4B0D05C8924F24A73B56EA652FA4B0@multiplay.co.uk><4E48D967.9060804@FreeBSD.org><9D034F992B064E8092E5D1D249B3E959@multiplay.co.uk><4E490DAF.1080009@FreeBSD.org><796FD5A096DE4558B57338A8FA1E125B@multiplay.co.uk><4E491D01.1090902@FreeBSD.org><570C5495A5E242F7946E806CA7AC5D68@multiplay.co.uk><4E4AD35C.7020504@FreeBSD.org><6A7238AED44542A880B082A40304D940@multiplay.co.uk><4E4BA21F.6010805@FreeBSD.org><581C95046B0948FC82D6F2E86948F87B@multiplay.co.uk><4E4BBA7F.30907@FreeBSD.org><88A6CE3E8B174E0694A3A9A5283479B4@multiplay.co.uk><4E4C22D6.6070407@FreeBSD.org><4019027648B5493AAC4B654BD821DE88@multiplay.co.uk><4E4F8631.1070300@FreeBSD.org> <4E4F8821.80108@Fre eBSD.org> <82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk> <4E4FE55A.9000101@ FreeBSD.org> Date: Sat, 20 Aug 2011 21:01:00 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 20:00:20 -0000 ----- Original Message ----- From: "Andriy Gapon" > thanks for doing this! I'll reiterate my suspicion just in case - I think that > you should look for the cases where you stop a jail, but then re-attach and > resurrect the jail before it's completely dead. Yer that's where I think its happening too, but I also suspect its not just dieing jail that's needed, I think its a dieing jail in the final stages of cleanup. Looking through the code I believe I may have noticed a scenario which could trigger the problem. Given the following code:- static void prison_deref(struct prison *pr, int flags) { struct prison *ppr, *tpr; int vfslocked; if (!(flags & PD_LOCKED)) mtx_lock(&pr->pr_mtx); /* Decrement the user references in a separate loop. */ if (flags & PD_DEUREF) { for (tpr = pr;; tpr = tpr->pr_parent) { if (tpr != pr) mtx_lock(&tpr->pr_mtx); if (--tpr->pr_uref > 0) break; KASSERT(tpr != &prison0, ("prison0 pr_uref=0")); mtx_unlock(&tpr->pr_mtx); } /* Done if there were only user references to remove. */ if (!(flags & PD_DEREF)) { mtx_unlock(&tpr->pr_mtx); if (flags & PD_LIST_SLOCKED) sx_sunlock(&allprison_lock); else if (flags & PD_LIST_XLOCKED) sx_xunlock(&allprison_lock); return; } if (tpr != pr) { mtx_unlock(&tpr->pr_mtx); mtx_lock(&pr->pr_mtx); } } If you take a scenario of a simple one level prison setup running a single process where a prison has just been stopped. In the above code pr_uref of the processes prison is decremented. As this is the last process then pr_uref will hit 0 and the loop continues instead of breaking early. Now at the end of the loop iteration the mtx is unlocked so other process can now manipulate the jail, this is where I think the problem may be. If we now have another process come in and attach to the jail but then instantly exit, this process may allow another kernel thread to hit this same bit of code and so two process for the same prison get into the section which decrements prison0's pr_uref, instead of only one. In essence I think we can get the following flow where 1# = process1 and 2# = process2 1#1. prison1.pr_uref = 1 (single process jail) 1#2. prison_deref( prison1,... 1#3. prison1.pr_uref-- (prison1.pr_uref = 0) 1#3. prison1.mtx_unlock <-- this now allows others to change prison1.pr_uref 1#3. prison0.pr_uref-- 2#1. process1.attach( prison1 ) (prison1.pr_uref = 1) 2#2. process1.exit 2#3. prison_deref( prison1,... 2#4. prison1.pr_uref-- (prison1.pr_uref = 0) 2#5. prison1.mtx_unlock <-- this now allows others to change prison1.pr_uref 2#5. prison0.pr_uref-- (prison1.pr_ref has now been decremented twice by prison1) It seems like the action on the parent prison to decrement the pr_uref is happening too early, while the jail can still be used and without the lock on the child jails mtx, so causing a race condition. I think the fix is to the move the decrement of parent prison pr_uref's down so it only takes place if the jail is "really" being removed. Either that or to change the locking semantics so that once the lock is aquired in this prison_deref its not unlocked until the function completes. What do people think? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 20:23:41 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC564106567A; Sat, 20 Aug 2011 20:23:41 +0000 (UTC) (envelope-from prvs=12137168ef=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 076718FC2A; Sat, 20 Aug 2011 20:23:39 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 21:23:06 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 21:23:06 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014676211.msg; Sat, 20 Aug 2011 21:23:05 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12137168ef=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Andriy Gapon" References: eBSD.org><82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk><4E4FE55A.9000101@ FreeBSD.org> Date: Sat, 20 Aug 2011 21:24:30 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 20:23:42 -0000 ----- Original Message ----- From: "Steven Hartland" > Looking through the code I believe I may have noticed a scenario which could > trigger the problem. > > Given the following code:- > > static void > prison_deref(struct prison *pr, int flags) > { > struct prison *ppr, *tpr; > int vfslocked; > > if (!(flags & PD_LOCKED)) > mtx_lock(&pr->pr_mtx); > /* Decrement the user references in a separate loop. */ > if (flags & PD_DEUREF) { > for (tpr = pr;; tpr = tpr->pr_parent) { > if (tpr != pr) > mtx_lock(&tpr->pr_mtx); > if (--tpr->pr_uref > 0) > break; > KASSERT(tpr != &prison0, ("prison0 pr_uref=0")); > mtx_unlock(&tpr->pr_mtx); > } > /* Done if there were only user references to remove. */ > if (!(flags & PD_DEREF)) { > mtx_unlock(&tpr->pr_mtx); > if (flags & PD_LIST_SLOCKED) > sx_sunlock(&allprison_lock); > else if (flags & PD_LIST_XLOCKED) > sx_xunlock(&allprison_lock); > return; > } > if (tpr != pr) { > mtx_unlock(&tpr->pr_mtx); > mtx_lock(&pr->pr_mtx); > } > } > > If you take a scenario of a simple one level prison setup running a single process > where a prison has just been stopped. > > In the above code pr_uref of the processes prison is decremented. As this is the > last process then pr_uref will hit 0 and the loop continues instead of breaking > early. > > Now at the end of the loop iteration the mtx is unlocked so other process can > now manipulate the jail, this is where I think the problem may be. > > If we now have another process come in and attach to the jail but then instantly > exit, this process may allow another kernel thread to hit this same bit of code > and so two process for the same prison get into the section which decrements > prison0's pr_uref, instead of only one. > > In essence I think we can get the following flow where 1# = process1 > and 2# = process2 > 1#1. prison1.pr_uref = 1 (single process jail) > 1#2. prison_deref( prison1,... > 1#3. prison1.pr_uref-- (prison1.pr_uref = 0) > 1#3. prison1.mtx_unlock <-- this now allows others to change prison1.pr_uref > 1#3. prison0.pr_uref-- > 2#1. process1.attach( prison1 ) (prison1.pr_uref = 1) > 2#2. process1.exit > 2#3. prison_deref( prison1,... > 2#4. prison1.pr_uref-- (prison1.pr_uref = 0) > 2#5. prison1.mtx_unlock <-- this now allows others to change prison1.pr_uref > 2#5. prison0.pr_uref-- (prison1.pr_ref has now been decremented twice by prison1) > > It seems like the action on the parent prison to decrement the pr_uref is > happening too early, while the jail can still be used and without the lock on > the child jails mtx, so causing a race condition. > > I think the fix is to the move the decrement of parent prison pr_uref's down > so it only takes place if the jail is "really" being removed. Either that or > to change the locking semantics so that once the lock is aquired in this > prison_deref its not unlocked until the function completes. > > What do people think? After reviewing the changes to prison_deref in commit which added hierarchical jails, the removal of the lock by the inital loop on the passed in prison may be unintentional. http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_jail.c.diff?r1=1.101;r2=1.102;f=h If so the following may be all that's needed to fix this issue:- diff -u sys/kern/kern_jail.c.orig sys/kern/kern_jail.c --- sys/kern/kern_jail.c.orig 2011-08-20 21:17:14.856618854 +0100 +++ sys/kern/kern_jail.c 2011-08-20 21:18:35.307201425 +0100 @@ -2455,7 +2455,8 @@ if (--tpr->pr_uref > 0) break; KASSERT(tpr != &prison0, ("prison0 pr_uref=0")); - mtx_unlock(&tpr->pr_mtx); + if (tpr != pr) + mtx_unlock(&tpr->pr_mtx); } /* Done if there were only user references to remove. */ if (!(flags & PD_DEREF)) { Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 20:34:56 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42F871065672; Sat, 20 Aug 2011 20:34:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 6FC888FC0C; Sat, 20 Aug 2011 20:34:55 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA14332; Sat, 20 Aug 2011 23:34:52 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QusFo-000O9N-7B; Sat, 20 Aug 2011 23:34:52 +0300 Message-ID: <4E501A6A.3030801@FreeBSD.org> Date: Sat, 20 Aug 2011 23:34:50 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110819 Thunderbird/6.0 MIME-Version: 1.0 To: Steven Hartland References: eBSD.org><82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk><4E4FE55A.9000101@ FreeBSD.org> In-Reply-To: X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 20:34:56 -0000 on 20/08/2011 23:24 Steven Hartland said the following: > ----- Original Message ----- From: "Steven Hartland" >> Looking through the code I believe I may have noticed a scenario which could >> trigger the problem. >> >> Given the following code:- >> >> static void >> prison_deref(struct prison *pr, int flags) >> { >> struct prison *ppr, *tpr; >> int vfslocked; >> >> if (!(flags & PD_LOCKED)) >> mtx_lock(&pr->pr_mtx); >> /* Decrement the user references in a separate loop. */ >> if (flags & PD_DEUREF) { >> for (tpr = pr;; tpr = tpr->pr_parent) { >> if (tpr != pr) >> mtx_lock(&tpr->pr_mtx); >> if (--tpr->pr_uref > 0) >> break; >> KASSERT(tpr != &prison0, ("prison0 pr_uref=0")); >> mtx_unlock(&tpr->pr_mtx); >> } >> /* Done if there were only user references to remove. */ >> if (!(flags & PD_DEREF)) { >> mtx_unlock(&tpr->pr_mtx); >> if (flags & PD_LIST_SLOCKED) >> sx_sunlock(&allprison_lock); >> else if (flags & PD_LIST_XLOCKED) >> sx_xunlock(&allprison_lock); >> return; >> } >> if (tpr != pr) { >> mtx_unlock(&tpr->pr_mtx); >> mtx_lock(&pr->pr_mtx); >> } >> } >> >> If you take a scenario of a simple one level prison setup running a single >> process >> where a prison has just been stopped. >> >> In the above code pr_uref of the processes prison is decremented. As this is the >> last process then pr_uref will hit 0 and the loop continues instead of breaking >> early. >> >> Now at the end of the loop iteration the mtx is unlocked so other process can >> now manipulate the jail, this is where I think the problem may be. >> >> If we now have another process come in and attach to the jail but then instantly >> exit, this process may allow another kernel thread to hit this same bit of code >> and so two process for the same prison get into the section which decrements >> prison0's pr_uref, instead of only one. >> >> In essence I think we can get the following flow where 1# = process1 >> and 2# = process2 >> 1#1. prison1.pr_uref = 1 (single process jail) >> 1#2. prison_deref( prison1,... >> 1#3. prison1.pr_uref-- (prison1.pr_uref = 0) >> 1#3. prison1.mtx_unlock <-- this now allows others to change prison1.pr_uref >> 1#3. prison0.pr_uref-- >> 2#1. process1.attach( prison1 ) (prison1.pr_uref = 1) >> 2#2. process1.exit >> 2#3. prison_deref( prison1,... >> 2#4. prison1.pr_uref-- (prison1.pr_uref = 0) >> 2#5. prison1.mtx_unlock <-- this now allows others to change prison1.pr_uref >> 2#5. prison0.pr_uref-- (prison1.pr_ref has now been decremented twice by prison1) >> >> It seems like the action on the parent prison to decrement the pr_uref is >> happening too early, while the jail can still be used and without the lock on >> the child jails mtx, so causing a race condition. >> >> I think the fix is to the move the decrement of parent prison pr_uref's down >> so it only takes place if the jail is "really" being removed. Either that or >> to change the locking semantics so that once the lock is aquired in this >> prison_deref its not unlocked until the function completes. >> >> What do people think? > > After reviewing the changes to prison_deref in commit which added hierarchical > jails, the removal of the lock by the inital loop on the passed in prison may > be unintentional. > http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_jail.c.diff?r1=1.101;r2=1.102;f=h > > > If so the following may be all that's needed to fix this issue:- > > diff -u sys/kern/kern_jail.c.orig sys/kern/kern_jail.c > --- sys/kern/kern_jail.c.orig 2011-08-20 21:17:14.856618854 +0100 > +++ sys/kern/kern_jail.c 2011-08-20 21:18:35.307201425 +0100 > @@ -2455,7 +2455,8 @@ > if (--tpr->pr_uref > 0) > break; > KASSERT(tpr != &prison0, ("prison0 pr_uref=0")); > - mtx_unlock(&tpr->pr_mtx); > + if (tpr != pr) > + mtx_unlock(&tpr->pr_mtx); > } > /* Done if there were only user references to remove. */ > if (!(flags & PD_DEREF)) { Not sure if this would fly as is - please double check the later block where pr->pr_mtx is re-locked. -- Andriy Gapon From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 20:49:58 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B68F106566C; Sat, 20 Aug 2011 20:49:58 +0000 (UTC) (envelope-from prvs=12137168ef=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 9306A8FC15; Sat, 20 Aug 2011 20:49:57 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 21:49:23 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 21:49:23 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014676446.msg; Sat, 20 Aug 2011 21:49:23 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12137168ef=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <7585E1DAE11E47488CD5A7F038957F4D@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: eBSD.org><82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk><4E4FE55A.9000101@FreeBSD.org> <4E501A6A.3030801@FreeBSD.org> Date: Sat, 20 Aug 2011 21:50:51 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 20:49:58 -0000 ----- Original Message ----- From: "Andriy Gapon" >> diff -u sys/kern/kern_jail.c.orig sys/kern/kern_jail.c >> --- sys/kern/kern_jail.c.orig 2011-08-20 21:17:14.856618854 +0100 >> +++ sys/kern/kern_jail.c 2011-08-20 21:18:35.307201425 +0100 >> @@ -2455,7 +2455,8 @@ >> if (--tpr->pr_uref > 0) >> break; >> KASSERT(tpr != &prison0, ("prison0 pr_uref=0")); >> - mtx_unlock(&tpr->pr_mtx); >> + if (tpr != pr) >> + mtx_unlock(&tpr->pr_mtx); >> } >> /* Done if there were only user references to remove. */ >> if (!(flags & PD_DEREF)) { > > Not sure if this would fly as is - please double check the later block where > pr->pr_mtx is re-locked. Will do, I'm now 99.9% sure this is the problem and even better I now have a reproducible scenario :) Something else you many be more interested in Andriy:- I added in debugging options DDB & INVARIANTS to see if I can get a more useful info and the panic results in a looping panic constantly scrolling up the console. Not sure if this is a side effect of the patches we've been trying. Going to see if I can confirm that, lmk if there's something you want me to try? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-jail@FreeBSD.ORG Sat Aug 20 21:38:56 2011 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0C5A106564A; Sat, 20 Aug 2011 21:38:56 +0000 (UTC) (envelope-from prvs=12137168ef=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 1A5868FC0A; Sat, 20 Aug 2011 21:38:55 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 22:38:21 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Aug 2011 22:38:21 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50014676893.msg; Sat, 20 Aug 2011 22:38:20 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=12137168ef=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <75D250E28B9A424EAF387E07CA223213@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "Andriy Gapon" References: eBSD.org><82E865FBA30747078AF6EE3C1701F973@multiplay.co.uk><4E4FE55A.9000101@FreeBSD.org><4E501A6A.3030801@FreeBSD.org> <7585E1DAE11E47488CD5A7F038957F4D@multiplay.co.uk> Date: Sat, 20 Aug 2011 22:38:49 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-jail@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: debugging frequent kernel panics on 8.2-RELEASE X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2011 21:38:57 -0000 ----- Original Message ----- From: "Steven Hartland" > Something else you many be more interested in Andriy:- > I added in debugging options DDB & INVARIANTS to see if I can get a more > useful info and the panic results in a looping panic constantly scrolling up > the console. Not sure if this is a side effect of the patches we've been > trying. > > Going to see if I can confirm that, lmk if there's something you want me > to try? Seems the stop_scheduler_on_panic.8.x.patch is the cause of this. Removing it allows me to drop to ddb when the panic due to the KASSERT happens. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.