Date: Thu, 17 Jun 2010 11:32:39 +0300 From: Kostik Belousov <kostikbel@gmail.com> To: Dmitry Pryanishnikov <lynx.ripe@gmail.com> Cc: freebsd-fs@FreeBSD.org, trasz@FreeBSD.org Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010) Message-ID: <20100617083239.GZ13238@deviant.kiev.zoral.com.ua> In-Reply-To: <20100617072841.GY13238@deviant.kiev.zoral.com.ua> References: <201006162220.o5GMK3uM002900@freefall.freebsd.org> <20100617072841.GY13238@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
--pr2uf1USn+1lHhL+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jun 17, 2010 at 10:28:41AM +0300, Kostik Belousov wrote: > On Wed, Jun 16, 2010 at 10:20:03PM +0000, Dmitry Pryanishnikov wrote: > > The following reply was made to PR kern/147890; it has been noted by GN= ATS. > >=20 > > From: Dmitry Pryanishnikov <lynx.ripe@gmail.com> > > To: Bruce Cran <bruce@cran.org.uk> > > Cc: bug-followup@freebsd.org > > Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem i= n=20 > > RELENG_8 (18.04.2010 -> 20.04.2010) > > Date: Thu, 17 Jun 2010 01:12:17 +0300 > >=20 > > Hello! > > =20 > > 2010/6/16 Bruce Cran <bruce@cran.org.uk>: > > > Could you run the commands shown in > > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook= /kerneldebug-deadlocks.html > > > to get more information to help debug the problem please? > > =20 > > Done - I've rebuilt the kernel from the sources as of 20.04.2010 > > with the missing debugging options, repeated the problem, turned > I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS > and WITNESS seems to be missed. >=20 > > ddb(4) capture buffer on, issued recommended ddb commands, then called > > 'doadump'; then rebooted and extracted capture buffer contents from > > the crashdump using ddb(8). > Due to missed options, we cannot trace the lock chains and see why the > issue happens. >=20 > > =20 > > BTW, is there a way to get this information _without repeating the > > problem_ - using only a crashdump from it's previous occurrence? All > > data required for analysis should already be there. I suspect not > > every deadlock is easy repeatable (like this one)... > It is, but you have to understand how locks work, and what you are > looking for. >=20 > > =20 > > db> ps > > =20 > > pid ppid pgrp uid state wmesg wchan cmd > > 2143 1846 2143 0 S+ ufs 0xcd466dc8 sync > > 2131 1 1911 1000 S ufs 0xcd466dc8 initial thread > ... > > 18 0 0 0 SL ufs 0xcd466dc8 [syncer] > .... > > Tracing command syncer pid 18 tid 100067 td 0xc7383000 > > sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 =3D sched_= switch+0x330 > > mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 =3D mi_switch+0x200 > > sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =3D > > sleepq_switch+0x15f > > sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 =3D sleepq_wai= t+0x63 > > __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =3D > > __lockmgr_args+0xb1f > > ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1 > > =3D ffs_lock+0xa1 > > VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at > > 0xc070c945 =3D VOP_LOCK1_APV+0xb5 > > _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 =3D _vn_lock= +0x78 > > vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db =3D vget+= 0xbb > > qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 =3D qsync+0x197 > > ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 =3D ffs_s= ync+0x349 > > sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at > > 0xc059812f =3D sync_fsync+0x18f > > VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at > > 0xc070a545 =3D VOP_FSYNC_APV+0xc5 > > sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b =3D sync_= vnode+0x16b > > sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =3D > > sched_sync+0x273 > > fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 =3D fork_exit+0xb8 > > fork_trampoline() at 0xc06d6b80 =3D fork_trampoline+0x8 > > --- trap 0, eip =3D 0, esp =3D 0xe732cd70, ebp =3D 0 --- > Do you have custoom kernel build, in particular, are quotas compiled in ? > What are the drives ? >=20 > Having to pull the generic information about the problematic system > does not make debugging faster. What _could_ happen in your case is the relock in ufs_accessx for QUOTA. ufs_delete_denied() calls VOP_ACCESSX for both parent directory and target vnode while parent directory is locked. If source directory is not exclusively locked, LOR can happen, since ufs_accessx has to unlock and lock directory vnode. I am not sure how many similar cases of VOP_ACCESS[X] is places in the problematic locations. Try the (untested) patch below. diff --git a/sys/ufs/ufs/ufs_lookup.c b/sys/ufs/ufs/ufs_lookup.c index 0030c52..c779491 100644 --- a/sys/ufs/ufs/ufs_lookup.c +++ b/sys/ufs/ufs/ufs_lookup.c @@ -77,6 +77,32 @@ SYSCTL_INT(_debug, OID_AUTO, dircheck, CTLFLAG_RW, &dirc= hk, 0, ""); /* true if old FS format...*/ #define OFSFMT(vp) ((vp)->v_mount->mnt_maxsymlinklen <=3D 0) =20 +#ifdef QUOTA +static int +ufs_lookup_upgrade_lock(struct vnode *vp) +{ + int error; + + ASSERT_VOP_LOCKED(vp, __FUNCTION__); + if (VOP_ISLOCKED(vp) =3D=3D LK_EXCLUSIVE) + return (0); + + error =3D 0; + + /* + * Upgrade vnode lock, since getinoquota() + * requires exclusive lock to modify inode. + */ + vhold(vp); + vn_lock(vp, LK_UPGRADE | LK_RETRY); + VI_LOCK(vp); + if (vp->v_iflag & VI_DOOMED) + error =3D ENOENT; + vdropl(vp); + return (error); +} +#endif + static int ufs_delete_denied(struct vnode *vdp, struct vnode *tdp, struct ucred *cred, struct thread *td) @@ -232,6 +258,13 @@ ufs_lookup_ino(struct vnode *vdp, struct vnode **vpp, = struct componentname *cnp, vnode_create_vobject(vdp, DIP(dp, i_size), cnp->cn_thread); =20 bmask =3D VFSTOUFS(vdp->v_mount)->um_mountp->mnt_stat.f_iosize - 1; +#ifdef QUOTA + if ((nameiop =3D=3D DELETE || nameiop =3D=3D RENAME) && (flags & ISLASTCN= )) { + error =3D ufs_lookup_upgrade_lock(vdp); + if (error !=3D 0) + return (error); + } +#endif =20 restart: bp =3D NULL; --pr2uf1USn+1lHhL+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkwZ3acACgkQC3+MBN1Mb4hShQCeMwojotSEFN+ENxpWsRTP39uV SycAoI7g34eITBqfnVZ1tFhUqzDVtHro =lAI2 -----END PGP SIGNATURE----- --pr2uf1USn+1lHhL+--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100617083239.GZ13238>