Date: Thu, 30 Jul 2009 14:55:48 +0200 From: Rene Ladan <rene@freebsd.org> To: Kostik Belousov <kostikbel@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) Message-ID: <e890cae60907300555x63de4a0dva503171d8fe2d3e6@mail.gmail.com> In-Reply-To: <20090730092507.GF1884@deviant.kiev.zoral.com.ua> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> <e890cae60907290820i65abae2fracbc5ab935465089@mail.gmail.com> <200907291135.17569.jhb@freebsd.org> <e890cae60907300205v5d3d5586qe86969bd28fe8621@mail.gmail.com> <20090730092507.GF1884@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
--0016367f9694a45687046febd226 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable 2009/7/30 Kostik Belousov <kostikbel@gmail.com>: > On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote: >> 2009/7/29 John Baldwin <jhb@freebsd.org>: >> > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: >> >> 2009/7/29 John Baldwin <jhb@freebsd.org>: >> >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: >> >> >> 2009/7/28 John Baldwin <jhb@freebsd.org>: >> >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: >> >> >> >> 2009/7/28 John Baldwin <jhb@freebsd.org>: >> >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: >> >> >> >> >> The following reply was made to PR kern/136945; it has been = noted >> > by >> >> >> > GNATS. >> >> >> >> >> >> >> >> >> >> From: Rene Ladan <rene@freebsd.org> >> >> >> >> >> To: John Baldwin <jhb@freebsd.org> >> >> >> >> >> Cc: bug-followup@freebsd.org >> >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs= (poll) >> >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 >> >> >> >> >> >> >> >> >> >> =9A2009/7/27 John Baldwin <jhb@freebsd.org>: >> >> >> >> >> =9A> I would actually expect this to be the correct order fo= r these >> > two >> >> >> >> > locks.=3D >> >> >> >> >> =9A =3DA0Can >> >> >> >> >> =9A> you capture the output of the 'debug.witness.fullgraph'= sysctl >> > to a >> >> >> > file? >> >> >> >> >> =9A> >> >> >> >> >> =9AYes, see attachment. =9AI'm still running the same 8.0-BE= TA2. >> >> >> >> > >> >> >> >> > Hmm, the attachment was eaten by a grue, can you post the fil= e >> >> > somewhere? >> >> >> >> > >> >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt >> >> >> > >> >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at so= me >> >> > point. =9ACan >> >> >> > you patch sys/kern/subr_witness.c to add a section to the order_= lists[] >> >> > array >> >> >> > after the 'ZFS locking list' and before the spin locks list that= looks >> >> > like >> >> >> > this: >> >> >> > >> >> >> > =9A =9A =9A =9A{ "filedesc structure", &lock_class_sx }, >> >> >> > =9A =9A =9A =9A{ "ufs", &lock_class_lockmgr}, >> >> >> > =9A =9A =9A =9A{ NULL, NULL }, >> >> >> > >> >> >> The LOR seems to be gone, previously it showed up only once right >> >> >> after booting the system. >> >> >> >> >> >> But now a new LOR (according to the LOR page) seems pop up: >> >> >> Trying to mount root from ufs:/dev/ad0s1a >> >> >> lock order reversal: >> >> >> =9A1st 0xffffff0002a4ad80 ufs (ufs) >> > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 >> >> >> =9A2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) = @ >> >> >> /usr/src/sys/kern/kern_descrip.c:2478 >> >> >> KDB: stack backtrace: >> >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> >> >> _witness_debugger() at _witness_debugger+0x49 >> >> >> witness_checkorder() at witness_checkorder+0x7ea >> >> >> _sx_xlock() at _sx_xlock+0x44 >> >> >> mountcheckdirs() at mountcheckdirs+0x80 >> >> >> vfs_donmount() at vfs_donmount+0xfbf >> >> >> kernel_mount() at kernel_mount+0xa1 >> >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 >> >> >> vfs_mountroot() at vfs_mountroot+0x47d >> >> >> start_init() at start_init+0x62 >> >> >> fork_exit() at fork_exit+0x12a >> >> >> fork_trampoline() at fork_trampoline+0xe >> >> >> --- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 --- >> >> >> >> >> >> The output of `df' and `mount' looks ok. >> >> > >> >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() ca= se >> > should >> >> > be the normal order. =9AI believe this should fix it. =9Amountcheck= dirs() >> > doesn't >> >> > need the vnodes locked, it just needs the caller to hold references= on >> > them >> >> > so they aren't recycled: >> >> > >> >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 >> >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c >> >> > @@ -1069,9 +1069,10 @@ >> >> > =9A =9A =9A =9A =9A =9A =9A =9Avfs_event_signal(NULL, VQ_MOUNT, 0); >> >> > =9A =9A =9A =9A =9A =9A =9A =9Aif (VFS_ROOT(mp, LK_EXCLUSIVE, &newd= p)) >> >> > =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9Apanic("mount: lost m= ount"); >> >> > + =9A =9A =9A =9A =9A =9A =9A VOP_UNLOCK(newdp, 0); >> >> > + =9A =9A =9A =9A =9A =9A =9A VOP_UNLOCK(vp, 0); >> >> > =9A =9A =9A =9A =9A =9A =9A =9Amountcheckdirs(vp, newdp); >> >> > - =9A =9A =9A =9A =9A =9A =9A vput(newdp); >> >> > - =9A =9A =9A =9A =9A =9A =9A VOP_UNLOCK(vp, 0); >> >> > + =9A =9A =9A =9A =9A =9A =9A vrele(newdp); >> >> > =9A =9A =9A =9A =9A =9A =9A =9Aif ((mp->mnt_flag & MNT_RDONLY) =3D= =3D 0) >> >> > =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9Aerror =3D vfs_alloca= te_syncvnode(mp); >> >> > =9A =9A =9A =9A =9A =9A =9A =9Avfs_unbusy(mp); >> >> > >> >> The LOR is still present, but at a different place without the >> >> mountcheckdirs() call (not on the LOR page either) : >> > >> > Ok, try this patch as well: >> > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c >> > @@ -1481,6 +1481,8 @@ >> > =9A =9A =9A =9Aif (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &ro= otvnode)) >> > =9A =9A =9A =9A =9A =9A =9A =9Apanic("Cannot find root vnode"); >> > >> > + =9A =9A =9A VOP_UNLOCK(rootvnode, 0); >> > + >> > =9A =9A =9A =9Ap =3D curthread->td_proc; >> > =9A =9A =9A =9AFILEDESC_XLOCK(p->p_fd); >> > >> > @@ -1496,8 +1498,6 @@ >> > >> > =9A =9A =9A =9AFILEDESC_XUNLOCK(p->p_fd); >> > >> > - =9A =9A =9A VOP_UNLOCK(rootvnode, 0); >> > - >> > =9A =9A =9A =9AEVENTHANDLER_INVOKE(mountroot); >> > =9A} >> > >> >> Still no luck, I now get a LOR that is similar to LOR 281 right after bo= oting: >> >> lock order reversal: >> =9A1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 >> =9A2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ >> /usr/src/sys/kern/vfs_syscalls.c:3776 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x49 >> witness_checkorder() at witness_checkorder+0x7ea >> _sx_slock() at _sx_slock+0x44 >> kern_mkdirat() at kern_mkdirat+0x201 >> syscall() at syscall+0x1af >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x800729dac, rsp =3D >> 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- > > Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat(). > I removed the two lines at sys/kern/vfs_syscalls.c (3776 and 3778), but there still seem to be some LORs (attached). The two LORs about the reboot call are from before Kostiks pat= ch. --0016367f9694a45687046febd226 Content-Type: text/plain; charset=US-ASCII; name="dmesg-80b2-kib.txt" Content-Disposition: attachment; filename="dmesg-80b2-kib.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_fxrho36l1 RnJlZUJTRCA4LjAtQkVUQTIgIzI6IFRodSBKdWwgMzAgMDk6eHg6eHggQ0VTVCAyMDA5DQoNCmxv Y2sgb3JkZXIgcmV2ZXJzYWw6ICgjMjc2KQ0KDQpsb2NrIG9yZGVyIHJldmVyc2FsOg0KIDFzdCAw eGZmZmZmZjAwMDJiNTAyNzAgdWZzICh1ZnMpIEAgL3Vzci9zcmMvc3lzL2tlcm4vdmZzX21vdW50 LmM6MTIwMA0KIDJuZCAweGZmZmZmZmZmODBiZGNjYTAgYWxscHJvYyAoYWxscHJvYykgQCAvdXNy L3NyYy9zeXMva2Vybi9rZXJuX2Rlc2NyaXAuYzoyNDczDQpLREI6IHN0YWNrIGJhY2t0cmFjZToN CmRiX3RyYWNlX3NlbGZfd3JhcHBlcigpIGF0IGRiX3RyYWNlX3NlbGZfd3JhcHBlcisweDJhDQpf d2l0bmVzc19kZWJ1Z2dlcigpIGF0IF93aXRuZXNzX2RlYnVnZ2VyKzB4NDkNCndpdG5lc3NfY2hl Y2tvcmRlcigpIGF0IHdpdG5lc3NfY2hlY2tvcmRlcisweDdlYQ0KX3N4X3Nsb2NrKCkgYXQgX3N4 X3Nsb2NrKzB4NDQNCm1vdW50Y2hlY2tkaXJzKCkgYXQgbW91bnRjaGVja2RpcnMrMHgzZg0KZG91 bm1vdW50KCkgYXQgZG91bm1vdW50KzB4NDc3DQp2ZnNfdW5tb3VudGFsbCgpIGF0IHZmc191bm1v dW50YWxsKzB4NTQNCmJvb3QoKSBhdCBib290KzB4ODE4DQpta2R1bXBoZWFkZXIoKSBhdCBta2R1 bXBoZWFkZXINCnN5c2NhbGwoKSBhdCBzeXNjYWxsKzB4MWFmDQpYZmFzdF9zeXNjYWxsKCkgYXQg WGZhc3Rfc3lzY2FsbCsweGUxDQotLS0gc3lzY2FsbCAoNTUsIEZyZWVCU0QgRUxGNjQsIHJlYm9v dCksIHJpcCA9IDB4ODAwNzhmODVjLCByc3AgPSAweDdmZmZmZmZmZWFiOCwgcmJwID0gMCAtLS0N Cg0KDQpGcmVlQlNEIDguMC1CRVRBMiAjMzogVGh1IEp1bCAzMCAxMzoyOTo0NiBDRVNUIDIwMDkN Cg0KbG9jayBvcmRlciByZXZlcnNhbDoNCiAxc3QgMHhmZmZmZmYwMDUxMGE1ZDgwIHVmcyAodWZz KSBAIC91c3Ivc3JjL3N5cy9rZXJuL2tlcm5fZXhlYy5jOjU3MA0KIDJuZCAweGZmZmZmZjAwMDJk ZmUyNDggZmlsZWRlc2Mgc3RydWN0dXJlIChmaWxlZGVzYyBzdHJ1Y3R1cmUpIEAgL3Vzci9zcmMv c3lzL2tlcm4va2Vybl9kZXNjcmlwLmM6MTg2NA0KS0RCOiBzdGFjayBiYWNrdHJhY2U6DQpkYl90 cmFjZV9zZWxmX3dyYXBwZXIoKSBhdCBkYl90cmFjZV9zZWxmX3dyYXBwZXIrMHgyYQ0KX3dpdG5l c3NfZGVidWdnZXIoKSBhdCBfd2l0bmVzc19kZWJ1Z2dlcisweDQ5DQp3aXRuZXNzX2NoZWNrb3Jk ZXIoKSBhdCB3aXRuZXNzX2NoZWNrb3JkZXIrMHg3ZWENCl9zeF94bG9jaygpIGF0IF9zeF94bG9j aysweDQ0DQpzZXR1Z2lkc2FmZXR5KCkgYXQgc2V0dWdpZHNhZmV0eSsweDQwDQprZXJuX2V4ZWN2 ZSgpIGF0IGtlcm5fZXhlY3ZlKzB4ZjIyDQpleGVjdmUoKSBhdCBleGVjdmUrMHgzOA0Kc3lzY2Fs bCgpIGF0IHN5c2NhbGwrMHgxYWYNClhmYXN0X3N5c2NhbGwoKSBhdCBYZmFzdF9zeXNjYWxsKzB4 ZTENCi0tLSBzeXNjYWxsICg1OSwgRnJlZUJTRCBFTEY2NCwgZXhlY3ZlKSwgcmlwID0gMHg4MDA3 YzNkMGMsIHJzcCA9IDB4N2ZmZmZmZmZlYzQ4LCByYnAgPSAweDdmZmZmZmZmZWQ1MCAtLS0NCg0K bG9jayBvcmRlciByZXZlcnNhbDogKGxpa2UgIzI2MSkNCiAxc3QgMHhmZmZmZmY4MDI5NTEyNDM4 IGJ1ZndhaXQgKGJ1ZndhaXQpIEAgL3Vzci9zcmMvc3lzL2tlcm4vdmZzX2Jpby5jOjI1NTgNCiAy bmQgMHhmZmZmZmYwMDAyYzQ0NDAwIGRpcmhhc2ggKGRpcmhhc2gpIEAgL3Vzci9zcmMvc3lzL3Vm cy91ZnMvdWZzX2Rpcmhhc2guYzoyODUNCktEQjogc3RhY2sgYmFja3RyYWNlOg0KZGJfdHJhY2Vf c2VsZl93cmFwcGVyKCkgYXQgZGJfdHJhY2Vfc2VsZl93cmFwcGVyKzB4MmENCl93aXRuZXNzX2Rl YnVnZ2VyKCkgYXQgX3dpdG5lc3NfZGVidWdnZXIrMHg0OQ0Kd2l0bmVzc19jaGVja29yZGVyKCkg YXQgd2l0bmVzc19jaGVja29yZGVyKzB4N2VhDQpfc3hfeGxvY2soKSBhdCBfc3hfeGxvY2srMHg0 NA0KdWZzZGlyaGFzaF9hY3F1aXJlKCkgYXQgdWZzZGlyaGFzaF9hY3F1aXJlKzB4MjkNCnVmc2Rp cmhhc2hfbW92ZSgpIGF0IHVmc2Rpcmhhc2hfbW92ZSsweDE5DQp1ZnNfZGlyZW50ZXIoKSBhdCB1 ZnNfZGlyZW50ZXIrMHg0YTkNCnVmc19tYWtlaW5vZGUoKSBhdCB1ZnNfbWFrZWlub2RlKzB4MmE3 DQpWT1BfQ1JFQVRFX0FQVigpIGF0IFZPUF9DUkVBVEVfQVBWKzB4OGQNCnZuX29wZW5fY3JlZCgp IGF0IHZuX29wZW5fY3JlZCsweDQwNg0Ka2Vybl9vcGVuYXQoKSBhdCBrZXJuX29wZW5hdCsweDE2 Mw0Kc3lzY2FsbCgpIGF0IHN5c2NhbGwrMHgxYWYNClhmYXN0X3N5c2NhbGwoKSBhdCBYZmFzdF9z eXNjYWxsKzB4ZTENCi0tLSBzeXNjYWxsICg1LCBGcmVlQlNEIEVMRjY0LCBvcGVuKSwgcmlwID0g MHg4MDA5ZGFkZWMsIHJzcCA9IDB4N2ZmZmZmZmZlNWM4LCByYnAgPSAweDFiNiAtLS0NCg== --0016367f9694a45687046febd226--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e890cae60907300555x63de4a0dva503171d8fe2d3e6>