Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 30 Jul 2009 09:51:43 -0400
From:      John Baldwin <jhb@freebsd.org>
To:        Rene Ladan <rene@freebsd.org>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll)
Message-ID:  <200907300951.44364.jhb@freebsd.org>
In-Reply-To: <e890cae60907300555x63de4a0dva503171d8fe2d3e6@mail.gmail.com>
References:  <200907271400.n6RE05Rv056472@freefall.freebsd.org> <20090730092507.GF1884@deviant.kiev.zoral.com.ua> <e890cae60907300555x63de4a0dva503171d8fe2d3e6@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday 30 July 2009 8:55:48 am Rene Ladan wrote:
> 2009/7/30 Kostik Belousov <kostikbel@gmail.com>:
> > On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote:
> >> 2009/7/29 John Baldwin <jhb@freebsd.org>:
> >> > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote:
> >> >> 2009/7/29 John Baldwin <jhb@freebsd.org>:
> >> >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote:
> >> >> >> 2009/7/28 John Baldwin <jhb@freebsd.org>:
> >> >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote:
> >> >> >> >> 2009/7/28 John Baldwin <jhb@freebsd.org>:
> >> >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote:
> >> >> >> >> >> The following reply was made to PR kern/136945; it has bee=
n=20
noted
> >> > by
> >> >> >> > GNATS.
> >> >> >> >> >>
> >> >> >> >> >> From: Rene Ladan <rene@freebsd.org>
> >> >> >> >> >> To: John Baldwin <jhb@freebsd.org>
> >> >> >> >> >> Cc: bug-followup@freebsd.org
> >> >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/u=
fs=20
(poll)
> >> >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200
> >> >> >> >> >>
> >> >> >> >> >> =A02009/7/27 John Baldwin <jhb@freebsd.org>:
> >> >> >> >> >> =A0> I would actually expect this to be the correct order =
for=20
these
> >> > two
> >> >> >> >> > locks.=3D
> >> >> >> >> >> =A0 =3DA0Can
> >> >> >> >> >> =A0> you capture the output of the 'debug.witness.fullgrap=
h'=20
sysctl
> >> > to a
> >> >> >> > file?
> >> >> >> >> >> =A0>
> >> >> >> >> >> =A0Yes, see attachment. =A0I'm still running the same 8.0-=
BETA2.
> >> >> >> >> >
> >> >> >> >> > Hmm, the attachment was eaten by a grue, can you post the f=
ile
> >> >> > somewhere?
> >> >> >> >> >
> >> >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt
> >> >> >> >
> >> >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at=
=20
some
> >> >> > point. =A0Can
> >> >> >> > you patch sys/kern/subr_witness.c to add a section to the=20
order_lists[]
> >> >> > array
> >> >> >> > after the 'ZFS locking list' and before the spin locks list th=
at=20
looks
> >> >> > like
> >> >> >> > this:
> >> >> >> >
> >> >> >> > =A0 =A0 =A0 =A0{ "filedesc structure", &lock_class_sx },
> >> >> >> > =A0 =A0 =A0 =A0{ "ufs", &lock_class_lockmgr},
> >> >> >> > =A0 =A0 =A0 =A0{ NULL, NULL },
> >> >> >> >
> >> >> >> The LOR seems to be gone, previously it showed up only once right
> >> >> >> after booting the system.
> >> >> >>
> >> >> >> But now a new LOR (according to the LOR page) seems pop up:
> >> >> >> Trying to mount root from ufs:/dev/ad0s1a
> >> >> >> lock order reversal:
> >> >> >> =A01st 0xffffff0002a4ad80 ufs (ufs)
> >> > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465
> >> >> >> =A02nd 0xffffff0002b29a48 filedesc structure (filedesc structure=
) @
> >> >> >> /usr/src/sys/kern/kern_descrip.c:2478
> >> >> >> KDB: stack backtrace:
> >> >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> >> >> >> _witness_debugger() at _witness_debugger+0x49
> >> >> >> witness_checkorder() at witness_checkorder+0x7ea
> >> >> >> _sx_xlock() at _sx_xlock+0x44
> >> >> >> mountcheckdirs() at mountcheckdirs+0x80
> >> >> >> vfs_donmount() at vfs_donmount+0xfbf
> >> >> >> kernel_mount() at kernel_mount+0xa1
> >> >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177
> >> >> >> vfs_mountroot() at vfs_mountroot+0x47d
> >> >> >> start_init() at start_init+0x62
> >> >> >> fork_exit() at fork_exit+0x12a
> >> >> >> fork_trampoline() at fork_trampoline+0xe
> >> >> >> --- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 ---
> >> >> >>
> >> >> >> The output of `df' and `mount' looks ok.
> >> >> >
> >> >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll()=
=20
case
> >> > should
> >> >> > be the normal order. =A0I believe this should fix=20
it. =A0mountcheckdirs()
> >> > doesn't
> >> >> > need the vnodes locked, it just needs the caller to hold referenc=
es=20
on
> >> > them
> >> >> > so they aren't recycled:
> >> >> >
> >> >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96
> >> >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c
> >> >> > @@ -1069,9 +1069,10 @@
> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_event_signal(NULL, VQ_MOUNT, 0=
);
> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (VFS_ROOT(mp, LK_EXCLUSIVE, &ne=
wdp))
> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("mount: lost=
 mount");
> >> >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(newdp, 0);
> >> >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0);
> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mountcheckdirs(vp, newdp);
> >> >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 vput(newdp);
> >> >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0);
> >> >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 vrele(newdp);
> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if ((mp->mnt_flag & MNT_RDONLY) =
=3D=3D 0)
> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0error =3D vfs_allo=
cate_syncvnode(mp);
> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_unbusy(mp);
> >> >> >
> >> >> The LOR is still present, but at a different place without the
> >> >> mountcheckdirs() call (not on the LOR page either) :
> >> >
> >> > Ok, try this patch as well:
> >> >
> >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97
> >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c
> >> > @@ -1481,6 +1481,8 @@
> >> > =A0 =A0 =A0 =A0if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &=
rootvnode))
> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("Cannot find root vnode");
> >> >
> >> > + =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0);
> >> > +
> >> > =A0 =A0 =A0 =A0p =3D curthread->td_proc;
> >> > =A0 =A0 =A0 =A0FILEDESC_XLOCK(p->p_fd);
> >> >
> >> > @@ -1496,8 +1498,6 @@
> >> >
> >> > =A0 =A0 =A0 =A0FILEDESC_XUNLOCK(p->p_fd);
> >> >
> >> > - =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0);
> >> > -
> >> > =A0 =A0 =A0 =A0EVENTHANDLER_INVOKE(mountroot);
> >> > =A0}
> >> >
> >>
> >> Still no luck, I now get a LOR that is similar to LOR 281 right after=
=20
booting:
> >>
> >> lock order reversal:
> >> =A01st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083
> >> =A02nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @
> >> /usr/src/sys/kern/vfs_syscalls.c:3776
> >> KDB: stack backtrace:
> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> >> _witness_debugger() at _witness_debugger+0x49
> >> witness_checkorder() at witness_checkorder+0x7ea
> >> _sx_slock() at _sx_slock+0x44
> >> kern_mkdirat() at kern_mkdirat+0x201
> >> syscall() at syscall+0x1af
> >> Xfast_syscall() at Xfast_syscall+0xe1
> >> --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x800729dac, rsp =3D
> >> 0x7fffffffec88, rbp =3D 0x7fffffffef66 ---
> >
> > Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(=
).
> >
> I removed the two lines at sys/kern/vfs_syscalls.c (3776 and 3778),
> but there still seem to be some LORs
> (attached).  The two LORs about the reboot call are from before Kostiks=20
patch.

Kostik has already suggested a fix for the second one.  The first one is a =
bit=20
harder to fix. :-/  The third one is a false LOR that you can ignore.

=2D-=20
John Baldwin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200907300951.44364.jhb>