Date: Sat, 29 Oct 2005 00:53:14 +0800 From: Xin LI <delphij@frontfree.net> To: freebsd-current@FreeBSD.org Subject: [RELENG_6] NFS panic on locking against myself Message-ID: <20051028165314.GA89706@frontfree.net>
next in thread | raw e-mail | index | archive | help
--zhXaljGHf11kAtnf
Content-Type: text/plain; charset=unknown-8bit
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hi,
On a production CVS server of ours we got panics when there is some wrong
data was injected to the NFS TCP connection. This may indicate some error
in our error handling code of NFS client.
However, the issue happens only when the gateway between the CVS server and
the NFS server is heavily loaded, therefore reproducing the issue is somewh=
at
hard. I have enabled DEBUG_VFS_LOCK to see if I can catch something.
The backtrace goes here:
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
<3>impossible packet length (745074944) from nfs server 10.88.15.238:/data0=
/vhost/wiki/vol/APPLE/matrixdata/docroot
panic: lockmgr: locking against myself
KDB: enter: panic
Dumping 1022 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 1022MB (261600 pages) 1006 990 974 958 942 926 910 894 878 862 8=
46 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 =
542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254=
238 222 206 190 174 158 142 126 110 94 78 62 46 30 14
#0 doadump () at pcpu.h:165
in pcpu.h
(kgdb) bt full
#0 doadump () at pcpu.h:165
No locals.
#1 0xc047f373 in db_fncall (dummy1=3D-1066385920, dummy2=3D0, dummy3=3D-10=
67193049,=20
dummy4=3D0xe775d7a0 "=CC=D7u=E7\224=D4c=C0=B8=D7u=E7=BC=D7u=E7\220\a") =
at /usr/src/sys/ddb/db_command.c:492
fn_addr =3D -1068348316
args =3D {1, 0, 544593784, -1067199340, -1066463456, -1066463680, 0, -4117=
07512, 2, -1066737952}
nargs =3D 0
retval =3D 0
t =3D 0
#2 0xc047f178 in db_command (last_cmdp=3D0xc06dc4c4, cmd_table=3D0x0, aux_=
cmd_tablep=3D0xc06a83f4,=20
aux_cmd_tablep_end=3D0xc06a8410) at /usr/src/sys/ddb/db_command.c:350
cmd =3D (struct command *) 0xc06ae080
t =3D 0
modif =3D "=CC=D7u=E7\224=D4c=C0=B8=D7u=E7=BC=D7u=E7\220\a\000\000\220\a\0=
00\000=CF\a\000\000\000\000\000\000\000>p=C0\r\000\000\000\000>p=C0\000>p=
=C0\r\000\000\000\001\000\000\000=F8=D7u=E7O=CEc=C0=F8=D7u=E7h=CEc=C0@\016o=
=C0`rn=C0x\000\000\000=C0=CDm=C0\000\000\000\000\030=D8u=E7=F0\021H=C0\000$=
i=C0=E0\016H=C0\000\000\000\000=C0=CDm=C0\222\006H?
addr =3D -1066385920
count =3D -1067193049
have_addr =3D 0
result =3D 0
#3 0xc047f240 in db_command_loop () at /usr/src/sys/ddb/db_command.c:458
No locals.
#4 0xc0480e4d in db_trap (type=3D3, code=3D0) at /usr/src/sys/ddb/db_main.=
c:221
jb =3D {{_jb =3D {-411707304, -411707324, -411707252, -1006365520, 0, -106=
9019674, -1068274507, -1066851157,=20
-1066845781, -1066851596, -411707248, -1068273655}}}
prev_jb =3D (void *) 0x0
bkpt =3D 0
#5 0xc053e2af in kdb_trap (type=3D3, code=3D0, tf=3D0xe775d8e0) at /usr/sr=
c/sys/kern/subr_kdb.c:473
handled =3D -411707168
#6 0xc0659578 in trap (frame=3D
{tf_fs =3D -411762680, tf_es =3D -1068302296, tf_ds =3D -1066860504, =
tf_edi =3D 1, tf_esi =3D -1066857605, tf_ebp =3D -411707104, tf_isp =3D -41=
1707124, tf_ebx =3D -411707060, tf_edx =3D 0, tf_ecx =3D -1061072896, tf_ea=
x =3D 18, tf_trapno =3D 3, tf_err =3D 0, tf_eip =3D -1068244941, tf_cs =3D =
32, tf_eflags =3D 658, tf_esp =3D -411707072, tf_ss =3D -1068346465})
at /usr/src/sys/i386/i386/trap.c:591
td =3D (struct thread *) 0xc40414b0
p =3D (struct proc *) 0xc4044418
sticks =3D 17104896
i =3D 0
ucode =3D 0
type =3D 3
code =3D 0
eva =3D 0
#7 0xc06498aa in calltrap () at /usr/src/sys/i386/i386/exception.s:139
No locals.
#8 0xc053e033 in kdb_enter (msg=3D0x12 <Address 0x12 out of bounds>) at cp=
ufunc.h:60
No locals.
#9 0xc052539f in panic (fmt=3D0xc0690b7b "lockmgr: locking against myself"=
) at /usr/src/sys/kern/kern_shutdown.c:539
td =3D (struct thread *) 0xc40414b0
bootopt =3D 256
newpanic =3D 1
ap =3D 0xe775d94c "=B0\024\004?
buf =3D "lockmgr: locking against myself", '\0' <repeats 224 times>
#10 0xc0518966 in lockmgr (lkp=3D0xc2d109e8, flags=3D8194, interlkp=3D0x80,=
td=3D0xc40414b0) at /usr/src/sys/kern/kern_lock.c:330
error =3D 0
thr =3D (struct thread *) 0xc40414b0
extflags =3D 128
lockflags =3D 18
#11 0xc0573246 in vop_stdlock (ap=3D0x0) at /usr/src/sys/kern/vfs_default.c=
:258
vp =3D (struct vnode *) 0xc0c15000
#12 0xc0669583 in VOP_LOCK_APV (vop=3D0xc06c2c60, a=3D0xe775d9b0) at vnode_=
if.c:1642
rc =3D -1066652576
#13 0xc0587e78 in vn_lock (vp=3D0xc2d10990, flags=3D8194, td=3D0xc40414b0) =
at vnode_if.h:844
error =3D 18
#14 0xc057be9a in vrele (vp=3D0xc2d10990) at /usr/src/sys/kern/vfs_subr.c:2=
050
td =3D (struct thread *) 0xc40414b0
#15 0xc05cbe2c in nfs_lookup (ap=3D0x12) at /usr/src/sys/nfsclient/nfs_vnop=
s.c:893
cnp =3D (struct componentname *) 0xe775dc90
dvp =3D (struct vnode *) 0xc43ab110
vpp =3D (struct vnode **) 0xe775dc7c
flags =3D 16814096
newvp =3D (struct vnode *) 0xc2d10990
bpos =3D 0xc511d150 "h"
dpos =3D 0xc44e0ab0 ""
mreq =3D (struct mbuf *) 0xc511d100
mrep =3D (struct mbuf *) 0x0
md =3D (struct mbuf *) 0xc44e0a00
mb =3D (struct mbuf *) 0xc511d100
len =3D 72
fhp =3D (nfsfh_t *) 0xc44e0a38
np =3D (struct nfsnode *) 0xc44ee564
error =3D 72
attrflag =3D 0
fhsize =3D 28
v3 =3D 512
td =3D (struct thread *) 0xc40414b0
#16 0xc06689a7 in VOP_LOOKUP_APV (vop=3D0xc06c8820, a=3D0xe775db3c) at vnod=
e_if.c:99
rc =3D -1066629088
#17 0xc0575389 in lookup (ndp=3D0xe775dc68) at vnode_if.h:56
cp =3D 0xc2a2805b ""
dp =3D (struct vnode *) 0xc43ab110
tdp =3D (struct vnode *) 0xc2290bb0
mp =3D (struct mount *) 0xc2a2805b
docache =3D 0
wantparent =3D 16
rdonly =3D 0
trailing_slash =3D 0
error =3D 0
dpunlocked =3D 0
cnp =3D (struct componentname *) 0xe775dc90
td =3D (struct thread *) 0xc40414b0
vfslocked =3D 1
tvfslocked =3D 1
#18 0xc0574cca in namei (ndp=3D0xe775dc68) at /usr/src/sys/kern/vfs_lookup.=
c:203
fdp =3D (struct filedesc *) 0xc2ba2000
cp =3D 0xc2ba2000 "d =BA=C2?=BA=C2 =C2i=C2Pe\035=C2Pe\035=C2\024"
dp =3D (struct vnode *) 0xc21d6550
aiov =3D {iov_base =3D 0xc0582a7b, iov_len =3D 8194}
auio =3D {uio_iov =3D 0xe775dbb4, uio_iovcnt =3D 128, uio_offset =3D -4322=
306996204929024, uio_resid =3D 0,=20
uio_segflg =3D 3228314720, uio_rw =3D 3883260924, uio_td =3D 0x4}
error =3D -1038260912
linklen =3D -1038260912
cnp =3D (struct componentname *) 0xe775dc90
td =3D (struct thread *) 0xc40414b0
p =3D (struct proc *) 0x0
vfslocked =3D 0
#19 0xc0583d90 in kern_rename (td=3D0xc40414b0, from=3D0x12 <Address 0x12 o=
ut of bounds>,=20
to=3D0x12 <Address 0x12 out of bounds>, pathseg=3DUIO_USERSPACE) at /us=
r/src/sys/kern/vfs_syscalls.c:3188
mp =3D (struct mount *) 0x0
tvp =3D (struct vnode *) 0x2002
fvp =3D (struct vnode *) 0x0
tdvp =3D (struct vnode *) 0x0
fromnd =3D {ni_dirp =3D 0x82435dc <Address 0x82435dc out of bounds>, ni_se=
gflg =3D UIO_USERSPACE, ni_startdir =3D 0x0,=20
ni_rootdir =3D 0xc21d6550, ni_topdir =3D 0xc21d6550, ni_vp =3D 0x0, ni_dv=
p =3D 0xc43ab110, ni_pathlen =3D 1,=20
ni_next =3D 0xc2a2805b "", ni_loopcnt =3D 0, ni_cnd =3D {cn_nameiop =3D 2=
, cn_flags =3D 16814096, cn_thread =3D 0xc40414b0,=20
cn_cred =3D 0xc2757680, cn_lkflags =3D 2,=20
cn_pnbuf =3D 0xc2a28000 "/usr/local/share/docroot/bkup/cvs/mailtech/Foo=
App1/myapp10/stuff/nconf/#cvs.cvsup-2172.6595",=20
cn_nameptr =3D 0xc2a28047 "#cvs.cvsup-2172.6595", cn_namelen =3D 20, cn=
_consume =3D 0}}
tond =3D {ni_dirp =3D 0xc057c1f2 "\203=C4\004d\213\025", ni_segflg =3D 326=
8479376, ni_startdir =3D 0xc2d10990,=20
ni_rootdir =3D 0xe775dc48, ni_topdir =3D 0xc057bf36, ni_vp =3D 0xc2d10990=
, ni_dvp =3D 0xc06d7940, ni_pathlen =3D 3268479376,=20
ni_next =3D 0x0, ni_loopcnt =3D 3288601776, ni_cnd =3D {cn_nameiop =3D 1,=
cn_flags =3D 0, cn_thread =3D 0xe775dcc4,=20
cn_cred =3D 0xc0582b05, cn_lkflags =3D -1026487920, cn_pnbuf =3D 0xc404=
14b0 "\030D\004=C4=E0\004N?,=20
cn_nameptr =3D 0xc2d10990 "\001", cn_namelen =3D 493, cn_consume =3D -4=
11706264}}
tvfslocked =3D -411706372
fvfslocked =3D -1067018852
error =3D -1006353384
#20 0xc0583d49 in rename (td=3D0xc40414b0, uap=3D0x12) at /usr/src/sys/kern=
/vfs_syscalls.c:3167
No locals.
#21 0xc0659dcb in syscall (frame=3D
{tf_fs =3D 1858994235, tf_es =3D -1078001605, tf_ds =3D 136249403, tf=
_edi =3D 1859007112, tf_esi =3D -1077940604, tf_ebp =3D 136256060, tf_isp =
=3D -411706012, tf_ebx =3D 3, tf_edx =3D 32768, tf_ecx =3D 0, tf_eax =3D 12=
8, tf_trapno =3D 22, tf_err =3D 2, tf_eip =3D 1859694163, tf_cs =3D 51, tf_=
eflags =3D 530, tf_esp =3D 136255664, tf_ss =3D 59}) at /usr/src/sys/i386/i=
386/trap.c:976
params =3D 0x81f18b4 <Address 0x81f18b4 out of bounds>
callp =3D (struct sysent *) 0xc06b74c0
td =3D (struct thread *) 0xc40414b0
p =3D (struct proc *) 0xc4044418
orig_tf_eflags =3D 530
sticks =3D 688
error =3D 0
narg =3D 2
args =3D {136590812, 136590216, 80, 0, 0, 0, 688, -1006353384}
code =3D 128
#22 0xc06498ff in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s=
:200
No locals.
#23 0x00000033 in ?? ()
No symbol table info available.
(kgdb)=20
Cheers,
--=20
Xin LI <delphij frontfree net> http://www.delphij.net/
See complete headers for GPG key and other information.
--zhXaljGHf11kAtnf
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (FreeBSD)
iD8DBQFDYld6/cVsHxFZiIoRAnlwAJ9kYgCVrvCFjKGSgP7TVwtKq36SuACaArz8
370ypzz45+ZF1TMePnlsy2s=
=oEVa
-----END PGP SIGNATURE-----
--zhXaljGHf11kAtnf--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051028165314.GA89706>