From owner-freebsd-current@FreeBSD.ORG Wed May 4 09:07:34 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4E10106567A for ; Wed, 4 May 2011 09:07:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 792688FC20 for ; Wed, 4 May 2011 09:07:33 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p4497ImV074959 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 4 May 2011 12:07:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p4497IVj026752; Wed, 4 May 2011 12:07:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p4497Itp026751; Wed, 4 May 2011 12:07:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 4 May 2011 12:07:18 +0300 From: Kostik Belousov To: Garrett Cooper Message-ID: <20110504090718.GN48734@deviant.kiev.zoral.com.ua> References: <201105040559.p445xEJ5024585@chez.mckusick.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZaNUQUWeUBsaJYg9" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS,SUBJECT_FUZZY_TION autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Kirk McKusick , FreeBSD Current Subject: Re: Nasty non-recursive lockmgr panic on softdep only enabled UFS partition when filesystem full X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2011 09:07:34 -0000 --ZaNUQUWeUBsaJYg9 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 03, 2011 at 11:58:49PM -0700, Garrett Cooper wrote: > On Tue, May 3, 2011 at 11:42 PM, Garrett Cooper wrot= e: > > On Tue, May 3, 2011 at 10:59 PM, Kirk McKusick = wrote: > >>> Date: Tue, 3 May 2011 22:40:26 -0700 > >>> Subject: Nasty non-recursive lockmgr panic on softdep only enabled UFS > >>> =9Apartition when filesystem full > >>> From: Garrett Cooper > >>> To: Jeff Roberson , > >>> =9A =9A =9A =9A Marshall Kirk McKusick > >>> Cc: FreeBSD Current > >>> > >>> Hi Jeff and Dr. McKusick, > >>> =9A =9A Ran into this panic when /usr ran out of space doing a make > >>> universe on amd64/r221219 (it took ~15 minutes for the panic to occur > >>> after the filesystem ran out of space -- wasn't quite sure what it was > >>> doing at the time): > >>> > >>> ... > >>> > >>> =9A =9A Let me know what other commands you would like for me to run = in kgdb. > >>> Thanks, > >>> -Garrett > >> > >> You did not indicate whether you are running an 8.X system or a 9-curr= ent > >> system. It would be helpful to know that. > > > > I've actually been running CURRENT for a few years now, but you're righ= t -- > > I didn't mention that part. > > > >> Jeff thinks that there may be a potential race in the locking code for > >> softdep_request_cleanup. If so, this patch for 9-current should fix it: > >> > >> Index: ffs_softdep.c > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> --- ffs_softdep.c =9A =9A =9A (revision 221385) > >> +++ ffs_softdep.c =9A =9A =9A (working copy) > >> @@ -11380,7 +11380,8 @@ > >> =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9Acontinu= e; > >> =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A} > >> =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9AMNT_IUNLOCK(mp); > >> - =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A if (vget(lvp, LK_EXCLUSI= VE | LK_INTERLOCK, curthread)) { > >> + =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A if (vget(lvp, LK_EXCLUSI= VE | LK_NOWAIT | LK_INTERLOCK, > >> + =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A curthread)) { > >> =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9AMNT_ILO= CK(mp); > >> =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9Acontinu= e; > >> =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A} > >> > >> If you are running an 8.X system, hopefully you will be able to apply = it. > > > > =9A =9AI've applied it, rebuilt and installed the kernel, and trying to > > repro the case again. Will let you know how things go! >=20 > Happened again with the change. It's really easy to repro: >=20 > 1. Get a filesystem with UFS+SU > 2. Execute something that does a large number of small writes to a partit= ion. > 3. 'dd if=3D/dev/zero of=3DFOO bs=3D10m' on the same partition >=20 > The kernel will panic with the issue I discussed above. > Thanks! Jeff' change is required to avoid LORs, but it is not sufficient to prevent recursion. We must skip the vnode supplied as a parameter to softdep_request_cleanup(). Theoretically, other vnodes might be also locked by curthread, thus I think the change below is needed. Try this. diff --git a/sys/ufs/ffs/ffs_softdep.c b/sys/ufs/ffs/ffs_softdep.c index a6d4441..25fa5d6 100644 --- a/sys/ufs/ffs/ffs_softdep.c +++ b/sys/ufs/ffs/ffs_softdep.c @@ -11380,7 +11380,9 @@ retry: continue; } MNT_IUNLOCK(mp); - if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK, curthread)) { + if (VOP_ISLOCKED(lvp) || + vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK | LK_NOWAIT, + curthread)) { MNT_ILOCK(mp); continue; } --ZaNUQUWeUBsaJYg9 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk3BF0YACgkQC3+MBN1Mb4i3iwCgz7uiG4c0n6uwFrvwpleaYTxO jCkAoLNhIi1EzRnMf7XANzcTxW71VY8d =As9g -----END PGP SIGNATURE----- --ZaNUQUWeUBsaJYg9--