From owner-freebsd-current@FreeBSD.ORG Tue Dec 28 21:50:51 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AA0D916A507 for ; Tue, 28 Dec 2004 21:50:51 +0000 (GMT) Received: from mail3.speakeasy.net (mail3.speakeasy.net [216.254.0.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id 29ABF43D39 for ; Tue, 28 Dec 2004 21:50:51 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 10499 invoked from network); 28 Dec 2004 21:50:51 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 28 Dec 2004 21:50:50 -0000 Received: from [10.50.41.243] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id iBSLoQg9095856; Tue, 28 Dec 2004 16:50:46 -0500 (EST) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-current@FreeBSD.org Date: Tue, 28 Dec 2004 15:38:15 -0500 User-Agent: KMail/1.6.2 References: <20041209144233.GA46928@peter.osted.lan> <20041226225651.GA87178@peter.osted.lan> <20041227013745.GA5267@technokratis.com> In-Reply-To: <20041227013745.GA5267@technokratis.com> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200412281538.15251.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Bosko Milekic cc: current@FreeBSD.org Subject: Re: panic: uma_zone_slab is looping X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Dec 2004 21:50:51 -0000 On Sunday 26 December 2004 08:37 pm, Bosko Milekic wrote: > On Sun, Dec 26, 2004 at 11:56:51PM +0100, Peter Holm wrote: > > On Sun, Dec 26, 2004 at 01:17:38PM -0500, Bosko Milekic wrote: > > > On Sun, Dec 26, 2004 at 05:11:53PM +0100, Peter Holm wrote: > > > > Yes, I think that I have verified your exelent analysis of the > > > > problem: http://www.holm.cc/stress/log/freeze04.html > > > > > > > > So, do have any fix suggenstons? :-) > > > > > > Not yet, because the problem is non-obvious from the trace. > > > > > > I need to know exactly when the UMA RCntSlabs zone recurses _first_, > > > and I need to confirm that it is an actual recursion. I've looked at > > > the VM code and I don't see how/why recursion on the RCntSlabs zone > > > would happen. > > > > > > Please modify the printf code to look exactly like this: > > > > > > if (keg->uk_flags & UMA_ZFLAG_INTERNAL && keg->uk_recurse != 0) { > > > if ((zone == slabzone) || (zone == slabrefzone)) > > > panic("Zone %s forced to fail due to recurse non-null: %d\n", > > > zone->uz_name, keg->uk_recurse); > > > return (NULL); > > > } > > > > > > (You don't need to check any global counter -- the counter is > > > imperfect anyway -- because even a single recursion on slabzone or > > > slabrefzone should be illegal). > > > > > > I'd like to see the trace from the above panic, if possible. > > > > Here it is: http://www.holm.cc/stress/log/freeze05.html > > I have checked the code here and looked at possible code paths and > have unfortunately resorted to reguessing, and now I believe I have > identified a problematic scenario. > > Consider this particular timeline (time moves downward): > [I hope you can handle ASCII art] > > By the way, the stack trace you show would correspond to that of > thread 2. I refer to a frame number below. > > thread 1 (t1) thread 2 (t2) > ------------------------------------------------------------------------- > > t1.a) Allocating from a zone, > needs slab header from one of > the slab header zones (either > slabzone or slabrefzone). Let's > assume it is slabzone, as in > your trace above. The allocation > is performed with M_WAITOK. > > t2.a) Needs to allocate from > a zone, and it needs a > slab header too. The allocation will > be performed with M_WAITOK. Let's > assume that the slab header zone > we're allocating is also slabzone. > > t1.b) in uma_zone_slab(), has > slabzone's keg lock, increments > keg's uk_recurse. > Enters slab_zalloc(). > > t2.b) Blocks on zone lock. > > t1.c) Drops zone lock to > allocate from VM, uk_recurse > for the slabzone is currently > 1 (we incremented it in t1.b). > > t2.c) Takes zone lock for slabzone, > now in uma_zone_slab() (Frame 11), > and since uk_recurse is 1, it > decides recursion happened. Wants > to return NULL even though > allocation was done with M_WAITOK. > Our panic is triggered. > > I'll have to reserve some more time to think about this. One way I > think it might be solvable would be to change that check that > triggers the NULL return explicitly check for the bucketzone, and not > for all UMA_ZONE_INTERNAL zones; I need to think this through a little > more. > > Does the scenario seem likely to you? This is what I wondered about earlier in the thread. The problem is that recursion needs to be a per-allocation (or per-thread) state, not per-zone, since a zone may be used by multiple threads at the same time. Is it really desired behavior of the bucket zone that if two threads alloc at the same time one gets NULL because it sees the other's use and thinks it is recursing? -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org