From owner-freebsd-fs@FreeBSD.ORG Tue Aug 2 16:14:34 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E35E416A41F for ; Tue, 2 Aug 2005 16:14:34 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from fileserver.fields.utoronto.ca (fileserver.fields.utoronto.ca [128.100.216.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C75A43D48 for ; Tue, 2 Aug 2005 16:14:34 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from fields.fields.utoronto.ca (fields.localdomain [192.168.216.11]) by fileserver.fields.utoronto.ca (8.12.8/8.12.8/Fields 6.0) with ESMTP id j72GEXNV006328 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 2 Aug 2005 12:14:33 -0400 Received: from obsecurity.dyndns.org (fields.fields.utoronto.ca [128.100.216.11]) by fields.fields.utoronto.ca (8.12.8/8.12.8/Fields WS 6.0) with ESMTP id j72GEX6P012025; Tue, 2 Aug 2005 12:14:33 -0400 Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id A094B512BD; Tue, 2 Aug 2005 12:14:24 -0400 (EDT) Date: Tue, 2 Aug 2005 12:14:24 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20050802161424.GA90200@xor.obsecurity.org> References: <200508021527.LAA45116@snowhite.cis.uoguelph.ca> <20050802155936.GA74261@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5" Content-Disposition: inline In-Reply-To: <20050802155936.GA74261@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: openbsd-nfsv4@sfobug.org, fs@freebsd.org, rick@snowhite.cis.uoguelph.ca Subject: Re: Re: FreeBSD6.0-BETA1 panics X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2005 16:14:35 -0000 --bg08WKrSYDhXBjb5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 02, 2005 at 11:59:36AM -0400, Kris Kennaway wrote: > On Tue, Aug 02, 2005 at 11:27:35AM -0400, rick@snowhite.cis.uoguelph.ca w= rote: > > > I suspect the server is returning bogus data, perhaps because of the > > > locking problems. > >=20 > > My server doesn't support the lockd/statd protocol, so v3 mounts won't > > have any advisory locking support. Is that likely to be the cause of th= is? > > (I don't know anything about the inner workings of cvs.) >=20 > I meant vnode locking in the kernel (i.e. race between the two nfsd > processes reading files on the server). Everything is read-only here > (and static on the server), so there is no write collision from the > client. >=20 > > > OK, I triggered another deadlock while running a simultaneous cvs > > > checkout on the server (via ufs) and over nfs3 from a remote machine. > >=20 > > Ok, thanks for the info. I'll try and figure this one out. > > (Given your original panic, I assume you're running a kernel with > > DEBUG_LOCKS and DEBUG_VFS_LOCKS options? That would have caught the > > obvious "forgot to unlock the vnode" type problems, which would suggest > > a race between the local ufs syscall and newnfsd for vnode locking. > > Could be a fun one to find:-) >=20 > Only DEBUG_LOCKS, I think..I'll add the other. Didn't help. Running two cvs updates over NFS and one locally over ufs (all against the same repo) again caused a deadlock in a few seconds: 721 c5aedc48 0 718 721 0004002 [SLPQ ufs 0xc799a528][SLP] cvs 703 c5d4a000 0 702 702 0000000 [SLPQ ufs 0xc797b18c][SLP] newnfsd Client and server are SMP machines, btw. db> show lockedvnods Locked vnodes 0xc797b134: tag ufs, type VDIR usecount 3, writecount 0, refcount 6 mountedhere 0 flags () v_object 0xc799118c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc5ae5480 (pid 721) with 1 pe= nding ino 259303, on dev da0s1e 0xc799a4d0: tag ufs, type VREG usecount 2, writecount 0, refcount 2 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xc5b0e000 (pid 703) with 1 pe= nding ino 265591, on dev da0s1e db> wh 721 Tracing pid 721 tid 100195 td 0xc5ae5480 sched_switch(c5ae5480,0,1,11e,4418d673) at sched_switch+0x18f mi_switch(1,0,c0703742,1a3,0) at mi_switch+0x2bf sleepq_switch(c799a528,c06ffec7,18a,0,f7c7196c) at sleepq_switch+0x135 sleepq_wait(c799a528,0,c0701247,da,0) at sleepq_wait+0x42 msleep(c799a528,c076163c,50,c0706087,0) at msleep+0x374 acquire(f7c719c8,40,60000,b5,c5ae5480) at acquire+0x88 debuglockmgr(c799a528,2002,c799a568,c5ae5480,c070775e) at debuglockmgr+0x4af vop_stdlock(f7c71a54,0,0,c074bca0,f7c71a54) at vop_stdlock+0x4e VOP_LOCK_APV(c074c3a0,f7c71a54,f7c71a2c,c06d797b,f7c71a54) at VOP_LOCK_APV+= 0xa6 ffs_lock(f7c71a54,0,f7c71a3c,2002,c799a4d0) at ffs_lock+0x19 VOP_LOCK_APV(c074bca0,f7c71a54,c07615ac,1,c06ffec7) at VOP_LOCK_APV+0xa6 debug_vn_lock(c799a4d0,2002,c5ae5480,c0708045,765) at debug_vn_lock+0xe5 vget(c799a4d0,2002,c5ae5480,1ca,0) at vget+0xc7 cache_lookup(c797b134,f7c71c74,f7c71c88,c5ae5480,c610e100) at cache_lookup+= 0x3d0 vfs_cache_lookup(f7c71bb0,f7c71bb0,0,c797b134,0) at vfs_cache_lookup+0xa4 VOP_LOOKUP_APV(c074bca0,f7c71bb0,f7c71b9c,c5ae5480,17c) at VOP_LOOKUP_APV+0= xa6 lookup(f7c71c60,0,c0707948,b4,c06ffec7) at lookup+0x484 namei(f7c71c60,c07617c8,0,c0701108,c610e100) at namei+0x40a kern_access(c5ae5480,80cd200,0,4,f7c71d30) at kern_access+0x72 access(c5ae5480,f7c71d04,8,421,2) at access+0x29 syscall(80a003b,80d003b,bfbf003b,80cd200,80cd180) at syscall+0x295 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (33, FreeBSD ELF32, access), eip =3D 0x28309d83, esp =3D 0xbfbf= de6c, ebp =3D 0xbfbfde78 --- db> wh 703 Tracing pid 703 tid 100215 td 0xc5b0e000 sched_switch(c5b0e000,0,1,11e,1506f98b) at sched_switch+0x18f mi_switch(1,0,c0703742,1a3,1) at mi_switch+0x2bf sleepq_switch(c797b18c,c06ffec7,18a,1,f7cb18f0) at sleepq_switch+0x135 sleepq_wait(c797b18c,0,c0701247,da,0) at sleepq_wait+0x42 msleep(c797b18c,c07615ac,50,c0706087,0) at msleep+0x374 acquire(f7cb194c,40,60000,b5,c5b0e000) at acquire+0x88 debuglockmgr(c797b18c,3002,c797b1cc,c5b0e000,c070775e) at debuglockmgr+0x4af vop_stdlock(f7cb19d8,c797b134,c07615ac,c074bca0,f7cb19d8) at vop_stdlock+0x= 4e VOP_LOCK_APV(c074c3a0,f7cb19d8,f7cb19b0,c06d797b,f7cb19d8) at VOP_LOCK_APV+= 0xa6 ffs_lock(f7cb19d8,0,c5b0e000,1002,c797b134) at ffs_lock+0x19 VOP_LOCK_APV(c074bca0,f7cb19d8,c0708ba1,31f,c5b0e000) at VOP_LOCK_APV+0xa6 debug_vn_lock(c797b134,1002,c5b0e000,c070f6a6,48d) at debug_vn_lock+0xe5 nfsrvn_getattr(c797b134,f7cb1a9c,c5d47580,c5b0e000,180) at nfsrvn_getattr+0= x6b nfsrvd_lookup(c5d47500,c5d46600,c797b134,0,f7cb1c1c) at nfsrvd_lookup+0x133 nfsrvd_dorpc(c5d47500,c5d46600,c5b0e000,6e5,0) at nfsrvd_dorpc+0x2f3 nfsrvd_nfsd(c5b0e000,0,c070f6a6,33d,c5d4a068) at nfsrvd_nfsd+0x404 nfssvc(c5b0e000,f7cb1d04,8,bfbff000,2) at nfssvc+0x3a8 syscall(3b,3b,3b,2804f40b,bfbfe884) at syscall+0x295 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip =3D 0x280ba523, esp =3D 0xbfb= fe64c, ebp =3D 0xbfbfe848 --- db> --bg08WKrSYDhXBjb5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFC75vgWry0BWjoQKURAjsZAJ9ucFoYX2HykXrSle81o5qcgrW6fQCg7GF9 LQWMjKNeZZJ7FvAPfwsXq0I= =8cQc -----END PGP SIGNATURE----- --bg08WKrSYDhXBjb5--