From owner-freebsd-fs@FreeBSD.ORG Thu Jul 30 09:05:35 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DFBA106566B; Thu, 30 Jul 2009 09:05:35 +0000 (UTC) (envelope-from r.c.ladan@gmail.com) Received: from mail-ew0-f206.google.com (mail-ew0-f206.google.com [209.85.219.206]) by mx1.freebsd.org (Postfix) with ESMTP id 7008C8FC1C; Thu, 30 Jul 2009 09:05:33 +0000 (UTC) (envelope-from r.c.ladan@gmail.com) Received: by ewy2 with SMTP id 2so552419ewy.43 for ; Thu, 30 Jul 2009 02:05:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=9k17aCIyvHGz1ioyJyPM7Lb8RxDwi/SgxqV1EGBT+Ao=; b=KLaEsbuSvvTZGCvXLb9zcPN4Vurlz43aa6ftTBcQ8hdemez5r4Kf3/Mrz64RjN/AWJ jNS/8AD6tF9dZnhjnmHwM2ktb4sxNkqo5u26kGQdS2EDrwB41BxOSfUsRemRKXgudZH2 9xFNpFWRxKsILCBpp+3CqXfhiDRIFlVq2rYbg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=I3DJSgjfgPlKhYXneazQHVmwg5AKEQXbIcxm0qO0Oa/PwuUTnayRdEVvHKldrQ3B0S u9OzI6rRnUngfTiHKQ2+CezVLpvnv0U4cgcS3mJQzmbwlmCa6xdPqeczEPeff/XXPCuf HTP7soPS7prYKtWsPnteaR9Wl7IUyGMUQx5RY= MIME-Version: 1.0 Sender: r.c.ladan@gmail.com Received: by 10.216.20.210 with SMTP id p60mr186046wep.172.1248944732741; Thu, 30 Jul 2009 02:05:32 -0700 (PDT) In-Reply-To: <200907291135.17569.jhb@freebsd.org> References: <200907271400.n6RE05Rv056472@freefall.freebsd.org> <200907290742.20838.jhb@freebsd.org> <200907291135.17569.jhb@freebsd.org> Date: Thu, 30 Jul 2009 11:05:32 +0200 X-Google-Sender-Auth: 7880b114fe81c463 Message-ID: From: Rene Ladan To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (poll) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2009 09:05:35 -0000 2009/7/29 John Baldwin : > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote: >> 2009/7/29 John Baldwin : >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote: >> >> 2009/7/28 John Baldwin : >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote: >> >> >> 2009/7/28 John Baldwin : >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote: >> >> >> >> The following reply was made to PR kern/136945; it has been not= ed > by >> >> > GNATS. >> >> >> >> >> >> >> >> From: Rene Ladan >> >> >> >> To: John Baldwin >> >> >> >> Cc: bug-followup@freebsd.org >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs (p= oll) >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200 >> >> >> >> >> >> >> >> =A02009/7/27 John Baldwin : >> >> >> >> =A0> I would actually expect this to be the correct order for t= hese > two >> >> >> > locks.=3D >> >> >> >> =A0 =3DA0Can >> >> >> >> =A0> you capture the output of the 'debug.witness.fullgraph' sy= sctl > to a >> >> > file? >> >> >> >> =A0> >> >> >> >> =A0Yes, see attachment. =A0I'm still running the same 8.0-BETA2= . >> >> >> > >> >> >> > Hmm, the attachment was eaten by a grue, can you post the file >> > somewhere? >> >> >> > >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt >> >> > >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at some >> > point. =A0Can >> >> > you patch sys/kern/subr_witness.c to add a section to the order_lis= ts[] >> > array >> >> > after the 'ZFS locking list' and before the spin locks list that lo= oks >> > like >> >> > this: >> >> > >> >> > =A0 =A0 =A0 =A0{ "filedesc structure", &lock_class_sx }, >> >> > =A0 =A0 =A0 =A0{ "ufs", &lock_class_lockmgr}, >> >> > =A0 =A0 =A0 =A0{ NULL, NULL }, >> >> > >> >> The LOR seems to be gone, previously it showed up only once right >> >> after booting the system. >> >> >> >> But now a new LOR (according to the LOR page) seems pop up: >> >> Trying to mount root from ufs:/dev/ad0s1a >> >> lock order reversal: >> >> =A01st 0xffffff0002a4ad80 ufs (ufs) > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 >> >> =A02nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @ >> >> /usr/src/sys/kern/kern_descrip.c:2478 >> >> KDB: stack backtrace: >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> >> _witness_debugger() at _witness_debugger+0x49 >> >> witness_checkorder() at witness_checkorder+0x7ea >> >> _sx_xlock() at _sx_xlock+0x44 >> >> mountcheckdirs() at mountcheckdirs+0x80 >> >> vfs_donmount() at vfs_donmount+0xfbf >> >> kernel_mount() at kernel_mount+0xa1 >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177 >> >> vfs_mountroot() at vfs_mountroot+0x47d >> >> start_init() at start_init+0x62 >> >> fork_exit() at fork_exit+0x12a >> >> fork_trampoline() at fork_trampoline+0xe >> >> --- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 --- >> >> >> >> The output of `df' and `mount' looks ok. >> > >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case > should >> > be the normal order. =A0I believe this should fix it. =A0mountcheckdir= s() > doesn't >> > need the vnodes locked, it just needs the caller to hold references on > them >> > so they aren't recycled: >> > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96 >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c >> > @@ -1069,9 +1069,10 @@ >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_event_signal(NULL, VQ_MOUNT, 0); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp)) >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("mount: lost moun= t"); >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(newdp, 0); >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mountcheckdirs(vp, newdp); >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 vput(newdp); >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 VOP_UNLOCK(vp, 0); >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 vrele(newdp); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if ((mp->mnt_flag & MNT_RDONLY) =3D=3D = 0) >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0error =3D vfs_allocate_= syncvnode(mp); >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0vfs_unbusy(mp); >> > >> The LOR is still present, but at a different place without the >> mountcheckdirs() call (not on the LOR page either) : > > Ok, try this patch as well: > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97 > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c > @@ -1481,6 +1481,8 @@ > =A0 =A0 =A0 =A0if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootv= node)) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0panic("Cannot find root vnode"); > > + =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0); > + > =A0 =A0 =A0 =A0p =3D curthread->td_proc; > =A0 =A0 =A0 =A0FILEDESC_XLOCK(p->p_fd); > > @@ -1496,8 +1498,6 @@ > > =A0 =A0 =A0 =A0FILEDESC_XUNLOCK(p->p_fd); > > - =A0 =A0 =A0 VOP_UNLOCK(rootvnode, 0); > - > =A0 =A0 =A0 =A0EVENTHANDLER_INVOKE(mountroot); > =A0} > Still no luck, I now get a LOR that is similar to LOR 281 right after booti= ng: lock order reversal: 1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083 2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @ /usr/src/sys/kern/vfs_syscalls.c:3776 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7ea _sx_slock() at _sx_slock+0x44 kern_mkdirat() at kern_mkdirat+0x201 syscall() at syscall+0x1af Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x800729dac, rsp =3D 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- Ren=E9