Date: Thu, 30 Jul 2009 08:45:27 -0400 From: John Baldwin <jhb@freebsd.org> To: Kostik Belousov <kostikbel@gmail.com> Cc: freebsd-fs@freebsd.org, Rene Ladan <rene@freebsd.org> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) Message-ID: <200907300845.27663.jhb@freebsd.org> In-Reply-To: <20090730092507.GF1884@deviant.kiev.zoral.com.ua> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <e890cae60907300205v5d3d5586qe86969bd28fe8621@mail.gmail.com> <20090730092507.GF1884@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday 30 July 2009 5:25:07 am Kostik Belousov wrote: > On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote: > > 2009/7/29 John Baldwin <jhb@freebsd.org>: > > > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: > > >> 2009/7/29 John Baldwin <jhb@freebsd.org>: > > >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: > > >> >> 2009/7/28 John Baldwin <jhb@freebsd.org>: > > >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: > > >> >> >> 2009/7/28 John Baldwin <jhb@freebsd.org>: > > >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: > > >> >> >> >> The following reply was made to PR kern/136945; it has been= =20 noted > > > by > > >> >> > GNATS. > > >> >> >> >> > > >> >> >> >> From: Rene Ladan <rene@freebsd.org> > > >> >> >> >> To: John Baldwin <jhb@freebsd.org> > > >> >> >> >> Cc: bug-followup@freebsd.org > > >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/uf= s=20 (poll) > > >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 > > >> >> >> >> > > >> >> >> >> =A02009/7/27 John Baldwin <jhb@freebsd.org>: > > >> >> >> >> =A0> I would actually expect this to be the correct order f= or=20 these > > > two > > >> >> >> > locks.=3D > > >> >> >> >> =A0 =3DA0Can > > >> >> >> >> =A0> you capture the output of the 'debug.witness.fullgraph= '=20 sysctl > > > to a > > >> >> > file? > > >> >> >> >> =A0> > > >> >> >> >> =A0Yes, see attachment. =A0I'm still running the same 8.0-B= ETA2. > > >> >> >> > > > >> >> >> > Hmm, the attachment was eaten by a grue, can you post the fi= le > > >> > somewhere? > > >> >> >> > > > >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt > > >> >> > > > >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at s= ome > > >> > point. =A0Can > > >> >> > you patch sys/kern/subr_witness.c to add a section to the=20 order_lists[] > > >> > array > > >> >> > after the 'ZFS locking list' and before the spin locks list tha= t=20 looks > > >> > like > > >> >> > this: > > >> >> > > > >> >> > =A0 =A0 =A0 =A0{ "filedesc structure", &lock_class_sx }, > > >> >> > =A0 =A0 =A0 =A0{ "ufs", &lock_class_lockmgr}, > > >> >> > =A0 =A0 =A0 =A0{ NULL, NULL }, > > >> >> > > > >> >> The LOR seems to be gone, previously it showed up only once right > > >> >> after booting the system. > > >> >> > > >> >> But now a new LOR (according to the LOR page) seems pop up: > > >> >> Trying to mount root from ufs:/dev/ad0s1a > > >> >> lock order reversal: > > >> >> =A01st 0xffffff0002a4ad80 ufs (ufs) > > > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 > > >> >> =A02nd 0xffffff0002b29a48 filedesc structure (filedesc structure)= @ > > >> >> /usr/src/sys/kern/kern_descrip.c:2478 > > >> >> KDB: stack backtrace: > > >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > >> >> _witness_debugger() at _witness_debugger+0x49 > > >> >> witness_checkorder() at witness_checkorder+0x7ea > > >> >> _sx_xlock() at _sx_xlock+0x44 > > >> >> mountcheckdirs() at mountcheckdirs+0x80 > > >> >> vfs_donmount() at vfs_donmount+0xfbf > > >> >> kernel_mount() at kernel_mount+0xa1 > > >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 > > >> >> vfs_mountroot() at vfs_mountroot+0x47d > > >> >> start_init() at start_init+0x62 > > >> >> fork_exit() at fork_exit+0x12a > > >> >> fork_trampoline() at fork_trampoline+0xe > > >> >> --- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 --- > > >> >> > > >> >> The output of `df' and `mount' looks ok. > > >> > > > >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() c= ase > > > should > > >> > be the normal order. =A0I believe this should fix it. =A0mountchec= kdirs() > > > doesn't > > >> > need the vnodes locked, it just needs the caller to hold reference= s=20 on > > > them > > >> > so they aren't recycled: > > >> > > > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 > > >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > > >> > @@ -1069,9 +1069,10 @@ > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_event_signal(NULL, VQ_MOUNT, 0); > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (VFS_ROOT(mp, LK_EXCLUSIVE, &new= dp)) > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("mount: lost = mount"); > > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(newdp, 0); > > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0); > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mountcheckdirs(vp, newdp); > > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 vput(newdp); > > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0); > > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 vrele(newdp); > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if ((mp->mnt_flag & MNT_RDONLY) =3D= =3D 0) > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0error =3D vfs_alloc= ate_syncvnode(mp); > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_unbusy(mp); > > >> > > > >> The LOR is still present, but at a different place without the > > >> mountcheckdirs() call (not on the LOR page either) : > > > > > > Ok, try this patch as well: > > > > > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 > > > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > > > @@ -1481,6 +1481,8 @@ > > > =A0 =A0 =A0 =A0if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &r= ootvnode)) > > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("Cannot find root vnode"); > > > > > > + =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0); > > > + > > > =A0 =A0 =A0 =A0p =3D curthread->td_proc; > > > =A0 =A0 =A0 =A0FILEDESC_XLOCK(p->p_fd); > > > > > > @@ -1496,8 +1498,6 @@ > > > > > > =A0 =A0 =A0 =A0FILEDESC_XUNLOCK(p->p_fd); > > > > > > - =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0); > > > - > > > =A0 =A0 =A0 =A0EVENTHANDLER_INVOKE(mountroot); > > > =A0} > > > > >=20 > > Still no luck, I now get a LOR that is similar to LOR 281 right after=20 booting: > >=20 > > lock order reversal: > > 1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 > > 2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ > > /usr/src/sys/kern/vfs_syscalls.c:3776 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > > _witness_debugger() at _witness_debugger+0x49 > > witness_checkorder() at witness_checkorder+0x7ea > > _sx_slock() at _sx_slock+0x44 > > kern_mkdirat() at kern_mkdirat+0x201 > > syscall() at syscall+0x1af > > Xfast_syscall() at Xfast_syscall+0xe1 > > --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x800729dac, rsp =3D > > 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- >=20 > Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(). Several other system calls have the same LOR and need the same fix. I've=20 consolidated all the fixes so far into=20 http://www.FreeBSD.org/~jhb/patches/vnode_filedesc.patch =2D-=20 John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200907300845.27663.jhb>