From owner-freebsd-current@FreeBSD.ORG Sat Jul 28 18:06:19 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B063216A501 for ; Sat, 28 Jul 2007 18:06:19 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 97A7913C48E for ; Sat, 28 Jul 2007 18:06:19 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from rot26.obsecurity.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id A907A1A4D7C; Sat, 28 Jul 2007 11:06:09 -0700 (PDT) Received: by rot26.obsecurity.org (Postfix, from userid 1001) id E401BBEC3; Sat, 28 Jul 2007 14:06:18 -0400 (EDT) Date: Sat, 28 Jul 2007 14:06:18 -0400 From: Kris Kennaway To: Thierry Herbelot Message-ID: <20070728180618.GA66481@rot26.obsecurity.org> References: <200707210657.11159.thierry@herbelot.com> <20070727200826.GA53337@dan.emsphone.com> <20070727205634.GA49495@rot26.obsecurity.org> <200707280831.34518.thierry@herbelot.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200707280831.34518.thierry@herbelot.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-current@freebsd.org, Dan Nelson , Kris Kennaway Subject: Re: ZFS panic: System call unlink returning with 1 locks held X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jul 2007 18:06:19 -0000 On Sat, Jul 28, 2007 at 08:31:32AM +0200, Thierry Herbelot wrote: > Le Friday 27 July 2007, Kris Kennaway a ?crit : > > On Fri, Jul 27, 2007 at 03:08:26PM -0500, Dan Nelson wrote: > > > In the last episode (Jul 21), Thierry Herbelot said: > > > > with a recent -current -built yesterday), I just got a panic while > > > > rebuilding -j4 the world and portupgrading firefox. > > > > > > > > the machine is pretty much memory limited (only 320 MB of RAM), with > > > > two CPUs, running a straight GENERIC kernel, including WITNESS and > > > > INVARIANTS. > > > > > > [..] > > > > > > > the panic message is : > > > > > > > > panic: System call unlink returning with 1 locks held > > > > cpuid = 0 > > > > KDB: enter: panic > > > > [thread pid 42789 tid 100102 ] > > > > Stopped at kdb_enter+0x32: leave > > > > db> where > > > > Tracing pid 42789 tid 100102 td 0xc2ce3200 > > > > kdb_enter(c0a92bc5,0,c0ac0a31,d5457c8c,0,...) at kdb_enter+0x32 > > > > panic(c0ac0a31,c0a98f5c,1,c0a98f5c,c0b3f030,...) at panic+0x124 > > > > syscall(d5457d38) at syscall+0x46e > > > > Xint0x80_syscall() at Xint0x80_syscall+0x20 > > > > > > I've been seeing this, as late as on a Jul 24 kernel. Happened once > > > during the cleandir stage of a buildworld, and few more times when the > > > system was relatively idle (although it is an mrtg server so lots of > > > files are constantly created and rm'd). My system is i386 with 1GB of > > > RAM, has a ZFS root, and is SMP. I've also gotten a similar "System > > > call rename returning with 1 locks held" panic. Is there any way to > > > find out what lock is being held? I've got a couple crashdumps. > > > > It appears to be a leak in the lock counters somewhere, perhaps > > related to recursively acquired rwlocks (e.g. double increment, single > > free). I eventually disabled the check because even adding extensive > > extra debugging there was no evidence of an actual lock being leaked > > anywhere. > > > > Kris > > Hello, > > I saw the same panic once or twice since the first report (same trace, > different phases of make buildworld). > > I had vfs.zfs.zil_disable="1" in /boot/loader.conf, which I just removed. > we'll see if the stability improves. AFAIK, disabling zil remains necessary to avoid low memory deadlocks. Just remove the test or turn off INVARIANTS to disable it. Kris