From owner-freebsd-fs@FreeBSD.ORG Thu Jul 30 13:52:12 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B900106567A; Thu, 30 Jul 2009 13:52:12 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id BFE988FC15; Thu, 30 Jul 2009 13:52:11 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 741F546B37; Thu, 30 Jul 2009 09:52:11 -0400 (EDT) Received: from jhbbsd.hudson-trading.com (unknown [209.249.190.8]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 7AB9D8A0A5; Thu, 30 Jul 2009 09:52:10 -0400 (EDT) From: John Baldwin To: Rene Ladan Date: Thu, 30 Jul 2009 09:51:43 -0400 User-Agent: KMail/1.9.7 References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <20090730092507.GF1884@deviant.kiev.zoral.com.ua> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200907300951.44364.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Thu, 30 Jul 2009 09:52:10 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,RDNS_NONE autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: freebsd-fs@freebsd.org Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2009 13:52:12 -0000 On Thursday 30 July 2009 8:55:48 am Rene Ladan wrote: > 2009/7/30 Kostik Belousov : > > On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote: > >> 2009/7/29 John Baldwin : > >> > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: > >> >> 2009/7/29 John Baldwin : > >> >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: > >> >> >> 2009/7/28 John Baldwin : > >> >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > >> >> >> >> 2009/7/28 John Baldwin : > >> >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > >> >> >> >> >> The following reply was made to PR kern/136945; it has bee= n=20 noted > >> > by > >> >> >> > GNATS. > >> >> >> >> >> > >> >> >> >> >> From: Rene Ladan > >> >> >> >> >> To: John Baldwin > >> >> >> >> >> Cc: bug-followup@freebsd.org > >> >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/u= fs=20 (poll) > >> >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > >> >> >> >> >> > >> >> >> >> >> =A02009/7/27 John Baldwin : > >> >> >> >> >> =A0> I would actually expect this to be the correct order = for=20 these > >> > two > >> >> >> >> > locks.=3D > >> >> >> >> >> =A0 =3DA0Can > >> >> >> >> >> =A0> you capture the output of the 'debug.witness.fullgrap= h'=20 sysctl > >> > to a > >> >> >> > file? > >> >> >> >> >> =A0> > >> >> >> >> >> =A0Yes, see attachment. =A0I'm still running the same 8.0-= BETA2. > >> >> >> >> > > >> >> >> >> > Hmm, the attachment was eaten by a grue, can you post the f= ile > >> >> > somewhere? > >> >> >> >> > > >> >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > >> >> >> > > >> >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at= =20 some > >> >> > point. =A0Can > >> >> >> > you patch sys/kern/subr_witness.c to add a section to the=20 order_lists[] > >> >> > array > >> >> >> > after the 'ZFS locking list' and before the spin locks list th= at=20 looks > >> >> > like > >> >> >> > this: > >> >> >> > > >> >> >> > =A0 =A0 =A0 =A0{ "filedesc structure", &lock_class_sx }, > >> >> >> > =A0 =A0 =A0 =A0{ "ufs", &lock_class_lockmgr}, > >> >> >> > =A0 =A0 =A0 =A0{ NULL, NULL }, > >> >> >> > > >> >> >> The LOR seems to be gone, previously it showed up only once right > >> >> >> after booting the system. > >> >> >> > >> >> >> But now a new LOR (according to the LOR page) seems pop up: > >> >> >> Trying to mount root from ufs:/dev/ad0s1a > >> >> >> lock order reversal: > >> >> >> =A01st 0xffffff0002a4ad80 ufs (ufs) > >> > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 > >> >> >> =A02nd 0xffffff0002b29a48 filedesc structure (filedesc structure= ) @ > >> >> >> /usr/src/sys/kern/kern_descrip.c:2478 > >> >> >> KDB: stack backtrace: > >> >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > >> >> >> _witness_debugger() at _witness_debugger+0x49 > >> >> >> witness_checkorder() at witness_checkorder+0x7ea > >> >> >> _sx_xlock() at _sx_xlock+0x44 > >> >> >> mountcheckdirs() at mountcheckdirs+0x80 > >> >> >> vfs_donmount() at vfs_donmount+0xfbf > >> >> >> kernel_mount() at kernel_mount+0xa1 > >> >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 > >> >> >> vfs_mountroot() at vfs_mountroot+0x47d > >> >> >> start_init() at start_init+0x62 > >> >> >> fork_exit() at fork_exit+0x12a > >> >> >> fork_trampoline() at fork_trampoline+0xe > >> >> >> --- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 --- > >> >> >> > >> >> >> The output of `df' and `mount' looks ok. > >> >> > > >> >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll()= =20 case > >> > should > >> >> > be the normal order. =A0I believe this should fix=20 it. =A0mountcheckdirs() > >> > doesn't > >> >> > need the vnodes locked, it just needs the caller to hold referenc= es=20 on > >> > them > >> >> > so they aren't recycled: > >> >> > > >> >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 > >> >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > >> >> > @@ -1069,9 +1069,10 @@ > >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_event_signal(NULL, VQ_MOUNT, 0= ); > >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (VFS_ROOT(mp, LK_EXCLUSIVE, &ne= wdp)) > >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("mount: lost= mount"); > >> >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(newdp, 0); > >> >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0); > >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mountcheckdirs(vp, newdp); > >> >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 vput(newdp); > >> >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0); > >> >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 vrele(newdp); > >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if ((mp->mnt_flag & MNT_RDONLY) = =3D=3D 0) > >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0error =3D vfs_allo= cate_syncvnode(mp); > >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_unbusy(mp); > >> >> > > >> >> The LOR is still present, but at a different place without the > >> >> mountcheckdirs() call (not on the LOR page either) : > >> > > >> > Ok, try this patch as well: > >> > > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 > >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > >> > @@ -1481,6 +1481,8 @@ > >> > =A0 =A0 =A0 =A0if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &= rootvnode)) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("Cannot find root vnode"); > >> > > >> > + =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0); > >> > + > >> > =A0 =A0 =A0 =A0p =3D curthread->td_proc; > >> > =A0 =A0 =A0 =A0FILEDESC_XLOCK(p->p_fd); > >> > > >> > @@ -1496,8 +1498,6 @@ > >> > > >> > =A0 =A0 =A0 =A0FILEDESC_XUNLOCK(p->p_fd); > >> > > >> > - =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0); > >> > - > >> > =A0 =A0 =A0 =A0EVENTHANDLER_INVOKE(mountroot); > >> > =A0} > >> > > >> > >> Still no luck, I now get a LOR that is similar to LOR 281 right after= =20 booting: > >> > >> lock order reversal: > >> =A01st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 > >> =A02nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ > >> /usr/src/sys/kern/vfs_syscalls.c:3776 > >> KDB: stack backtrace: > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > >> _witness_debugger() at _witness_debugger+0x49 > >> witness_checkorder() at witness_checkorder+0x7ea > >> _sx_slock() at _sx_slock+0x44 > >> kern_mkdirat() at kern_mkdirat+0x201 > >> syscall() at syscall+0x1af > >> Xfast_syscall() at Xfast_syscall+0xe1 > >> --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x800729dac, rsp =3D > >> 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- > > > > Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(= ). > > > I removed the two lines at sys/kern/vfs_syscalls.c (3776 and 3778), > but there still seem to be some LORs > (attached). The two LORs about the reboot call are from before Kostiks=20 patch. Kostik has already suggested a fix for the second one. The first one is a = bit=20 harder to fix. :-/ The third one is a false LOR that you can ignore. =2D-=20 John Baldwin