From owner-freebsd-current@FreeBSD.ORG Fri Oct 28 16:53:41 2005 Return-Path: X-Original-To: freebsd-current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2332416A423 for ; Fri, 28 Oct 2005 16:53:41 +0000 (GMT) (envelope-from delphij@frontfree.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [210.51.165.229]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3DE2043D48 for ; Fri, 28 Oct 2005 16:53:39 +0000 (GMT) (envelope-from delphij@frontfree.net) Received: from localhost (tarsier.geekcn.org [210.51.165.229]) by tarsier.geekcn.org (Postfix) with ESMTP id AD50CEB144C for ; Sat, 29 Oct 2005 00:53:33 +0800 (CST) Received: from tarsier.geekcn.org ([210.51.165.229]) by localhost (mail.geekcn.org [210.51.165.229]) (amavisd-new, port 10024) with ESMTP id 11948-17 for ; Sat, 29 Oct 2005 00:53:29 +0800 (CST) Received: from beastie.frontfree.net (unknown [211.71.95.7]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTP id B43F3EB09E7 for ; Sat, 29 Oct 2005 00:53:26 +0800 (CST) Received: from localhost (localhost.frontfree.net [127.0.0.1]) by beastie.frontfree.net (Postfix) with ESMTP id DECA313676C for ; Sat, 29 Oct 2005 00:53:18 +0800 (CST) Received: from beastie.frontfree.net ([127.0.0.1]) by localhost (beastie.frontfree.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 88711-06 for ; Sat, 29 Oct 2005 00:53:18 +0800 (CST) Received: by beastie.frontfree.net (Postfix, from userid 1001) id 0B189136765; Sat, 29 Oct 2005 00:53:15 +0800 (CST) Date: Sat, 29 Oct 2005 00:53:14 +0800 From: Xin LI To: freebsd-current@FreeBSD.org Message-ID: <20051028165314.GA89706@frontfree.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zhXaljGHf11kAtnf" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-GPG-key-ID/Fingerprint: 0xCAEEB8C0 / 43B8 B703 B8DD 0231 B333 DC28 39FB 93A0 CAEE B8C0 X-GPG-Public-Key: http://www.delphij.net/delphij.asc X-Operating-System: FreeBSD beastie.frontfree.net 5.4-RELEASE-p6 FreeBSD 5.4-RELEASE-p6 #4: Thu Jul 28 10:59:26 CST 2005 delphij@beastie.frontfree.net:/usr/obj/usr/src/sys/BEASTIE i386 X-URL: http://www.delphij.net X-By: delphij@beastie.frontfree.net X-Location: Beijing, China X-Virus-Scanned: amavisd-new at frontfree.net X-Virus-Scanned: amavisd-new at geekcn.org Cc: Subject: [RELENG_6] NFS panic on locking against myself X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2005 16:53:41 -0000 --zhXaljGHf11kAtnf Content-Type: text/plain; charset=unknown-8bit Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, On a production CVS server of ours we got panics when there is some wrong data was injected to the NFS TCP connection. This may indicate some error in our error handling code of NFS client. However, the issue happens only when the gateway between the CVS server and the NFS server is heavily loaded, therefore reproducing the issue is somewh= at hard. I have enabled DEBUG_VFS_LOCK to see if I can catch something. The backtrace goes here: GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: <3>impossible packet length (745074944) from nfs server 10.88.15.238:/data0= /vhost/wiki/vol/APPLE/matrixdata/docroot panic: lockmgr: locking against myself KDB: enter: panic Dumping 1022 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1022MB (261600 pages) 1006 990 974 958 942 926 910 894 878 862 8= 46 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 = 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254= 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:165 in pcpu.h (kgdb) bt full #0 doadump () at pcpu.h:165 No locals. #1 0xc047f373 in db_fncall (dummy1=3D-1066385920, dummy2=3D0, dummy3=3D-10= 67193049,=20 dummy4=3D0xe775d7a0 "=CC=D7u=E7\224=D4c=C0=B8=D7u=E7=BC=D7u=E7\220\a") = at /usr/src/sys/ddb/db_command.c:492 fn_addr =3D -1068348316 args =3D {1, 0, 544593784, -1067199340, -1066463456, -1066463680, 0, -4117= 07512, 2, -1066737952} nargs =3D 0 retval =3D 0 t =3D 0 #2 0xc047f178 in db_command (last_cmdp=3D0xc06dc4c4, cmd_table=3D0x0, aux_= cmd_tablep=3D0xc06a83f4,=20 aux_cmd_tablep_end=3D0xc06a8410) at /usr/src/sys/ddb/db_command.c:350 cmd =3D (struct command *) 0xc06ae080 t =3D 0 modif =3D "=CC=D7u=E7\224=D4c=C0=B8=D7u=E7=BC=D7u=E7\220\a\000\000\220\a\0= 00\000=CF\a\000\000\000\000\000\000\000>p=C0\r\000\000\000\000>p=C0\000>p= =C0\r\000\000\000\001\000\000\000=F8=D7u=E7O=CEc=C0=F8=D7u=E7h=CEc=C0@\016o= =C0`rn=C0x\000\000\000=C0=CDm=C0\000\000\000\000\030=D8u=E7=F0\021H=C0\000$= i=C0=E0\016H=C0\000\000\000\000=C0=CDm=C0\222\006H? addr =3D -1066385920 count =3D -1067193049 have_addr =3D 0 result =3D 0 #3 0xc047f240 in db_command_loop () at /usr/src/sys/ddb/db_command.c:458 No locals. #4 0xc0480e4d in db_trap (type=3D3, code=3D0) at /usr/src/sys/ddb/db_main.= c:221 jb =3D {{_jb =3D {-411707304, -411707324, -411707252, -1006365520, 0, -106= 9019674, -1068274507, -1066851157,=20 -1066845781, -1066851596, -411707248, -1068273655}}} prev_jb =3D (void *) 0x0 bkpt =3D 0 #5 0xc053e2af in kdb_trap (type=3D3, code=3D0, tf=3D0xe775d8e0) at /usr/sr= c/sys/kern/subr_kdb.c:473 handled =3D -411707168 #6 0xc0659578 in trap (frame=3D {tf_fs =3D -411762680, tf_es =3D -1068302296, tf_ds =3D -1066860504, = tf_edi =3D 1, tf_esi =3D -1066857605, tf_ebp =3D -411707104, tf_isp =3D -41= 1707124, tf_ebx =3D -411707060, tf_edx =3D 0, tf_ecx =3D -1061072896, tf_ea= x =3D 18, tf_trapno =3D 3, tf_err =3D 0, tf_eip =3D -1068244941, tf_cs =3D = 32, tf_eflags =3D 658, tf_esp =3D -411707072, tf_ss =3D -1068346465}) at /usr/src/sys/i386/i386/trap.c:591 td =3D (struct thread *) 0xc40414b0 p =3D (struct proc *) 0xc4044418 sticks =3D 17104896 i =3D 0 ucode =3D 0 type =3D 3 code =3D 0 eva =3D 0 #7 0xc06498aa in calltrap () at /usr/src/sys/i386/i386/exception.s:139 No locals. #8 0xc053e033 in kdb_enter (msg=3D0x12
) at cp= ufunc.h:60 No locals. #9 0xc052539f in panic (fmt=3D0xc0690b7b "lockmgr: locking against myself"= ) at /usr/src/sys/kern/kern_shutdown.c:539 td =3D (struct thread *) 0xc40414b0 bootopt =3D 256 newpanic =3D 1 ap =3D 0xe775d94c "=B0\024\004? buf =3D "lockmgr: locking against myself", '\0' #10 0xc0518966 in lockmgr (lkp=3D0xc2d109e8, flags=3D8194, interlkp=3D0x80,= td=3D0xc40414b0) at /usr/src/sys/kern/kern_lock.c:330 error =3D 0 thr =3D (struct thread *) 0xc40414b0 extflags =3D 128 lockflags =3D 18 #11 0xc0573246 in vop_stdlock (ap=3D0x0) at /usr/src/sys/kern/vfs_default.c= :258 vp =3D (struct vnode *) 0xc0c15000 #12 0xc0669583 in VOP_LOCK_APV (vop=3D0xc06c2c60, a=3D0xe775d9b0) at vnode_= if.c:1642 rc =3D -1066652576 #13 0xc0587e78 in vn_lock (vp=3D0xc2d10990, flags=3D8194, td=3D0xc40414b0) = at vnode_if.h:844 error =3D 18 #14 0xc057be9a in vrele (vp=3D0xc2d10990) at /usr/src/sys/kern/vfs_subr.c:2= 050 td =3D (struct thread *) 0xc40414b0 #15 0xc05cbe2c in nfs_lookup (ap=3D0x12) at /usr/src/sys/nfsclient/nfs_vnop= s.c:893 cnp =3D (struct componentname *) 0xe775dc90 dvp =3D (struct vnode *) 0xc43ab110 vpp =3D (struct vnode **) 0xe775dc7c flags =3D 16814096 newvp =3D (struct vnode *) 0xc2d10990 bpos =3D 0xc511d150 "h" dpos =3D 0xc44e0ab0 "" mreq =3D (struct mbuf *) 0xc511d100 mrep =3D (struct mbuf *) 0x0 md =3D (struct mbuf *) 0xc44e0a00 mb =3D (struct mbuf *) 0xc511d100 len =3D 72 fhp =3D (nfsfh_t *) 0xc44e0a38 np =3D (struct nfsnode *) 0xc44ee564 error =3D 72 attrflag =3D 0 fhsize =3D 28 v3 =3D 512 td =3D (struct thread *) 0xc40414b0 #16 0xc06689a7 in VOP_LOOKUP_APV (vop=3D0xc06c8820, a=3D0xe775db3c) at vnod= e_if.c:99 rc =3D -1066629088 #17 0xc0575389 in lookup (ndp=3D0xe775dc68) at vnode_if.h:56 cp =3D 0xc2a2805b "" dp =3D (struct vnode *) 0xc43ab110 tdp =3D (struct vnode *) 0xc2290bb0 mp =3D (struct mount *) 0xc2a2805b docache =3D 0 wantparent =3D 16 rdonly =3D 0 trailing_slash =3D 0 error =3D 0 dpunlocked =3D 0 cnp =3D (struct componentname *) 0xe775dc90 td =3D (struct thread *) 0xc40414b0 vfslocked =3D 1 tvfslocked =3D 1 #18 0xc0574cca in namei (ndp=3D0xe775dc68) at /usr/src/sys/kern/vfs_lookup.= c:203 fdp =3D (struct filedesc *) 0xc2ba2000 cp =3D 0xc2ba2000 "d =BA=C2?=BA=C2 =C2i=C2Pe\035=C2Pe\035=C2\024" dp =3D (struct vnode *) 0xc21d6550 aiov =3D {iov_base =3D 0xc0582a7b, iov_len =3D 8194} auio =3D {uio_iov =3D 0xe775dbb4, uio_iovcnt =3D 128, uio_offset =3D -4322= 306996204929024, uio_resid =3D 0,=20 uio_segflg =3D 3228314720, uio_rw =3D 3883260924, uio_td =3D 0x4} error =3D -1038260912 linklen =3D -1038260912 cnp =3D (struct componentname *) 0xe775dc90 td =3D (struct thread *) 0xc40414b0 p =3D (struct proc *) 0x0 vfslocked =3D 0 #19 0xc0583d90 in kern_rename (td=3D0xc40414b0, from=3D0x12
,=20 to=3D0x12
, pathseg=3DUIO_USERSPACE) at /us= r/src/sys/kern/vfs_syscalls.c:3188 mp =3D (struct mount *) 0x0 tvp =3D (struct vnode *) 0x2002 fvp =3D (struct vnode *) 0x0 tdvp =3D (struct vnode *) 0x0 fromnd =3D {ni_dirp =3D 0x82435dc
, ni_se= gflg =3D UIO_USERSPACE, ni_startdir =3D 0x0,=20 ni_rootdir =3D 0xc21d6550, ni_topdir =3D 0xc21d6550, ni_vp =3D 0x0, ni_dv= p =3D 0xc43ab110, ni_pathlen =3D 1,=20 ni_next =3D 0xc2a2805b "", ni_loopcnt =3D 0, ni_cnd =3D {cn_nameiop =3D 2= , cn_flags =3D 16814096, cn_thread =3D 0xc40414b0,=20 cn_cred =3D 0xc2757680, cn_lkflags =3D 2,=20 cn_pnbuf =3D 0xc2a28000 "/usr/local/share/docroot/bkup/cvs/mailtech/Foo= App1/myapp10/stuff/nconf/#cvs.cvsup-2172.6595",=20 cn_nameptr =3D 0xc2a28047 "#cvs.cvsup-2172.6595", cn_namelen =3D 20, cn= _consume =3D 0}} tond =3D {ni_dirp =3D 0xc057c1f2 "\203=C4\004d\213\025", ni_segflg =3D 326= 8479376, ni_startdir =3D 0xc2d10990,=20 ni_rootdir =3D 0xe775dc48, ni_topdir =3D 0xc057bf36, ni_vp =3D 0xc2d10990= , ni_dvp =3D 0xc06d7940, ni_pathlen =3D 3268479376,=20 ni_next =3D 0x0, ni_loopcnt =3D 3288601776, ni_cnd =3D {cn_nameiop =3D 1,= cn_flags =3D 0, cn_thread =3D 0xe775dcc4,=20 cn_cred =3D 0xc0582b05, cn_lkflags =3D -1026487920, cn_pnbuf =3D 0xc404= 14b0 "\030D\004=C4=E0\004N?,=20 cn_nameptr =3D 0xc2d10990 "\001", cn_namelen =3D 493, cn_consume =3D -4= 11706264}} tvfslocked =3D -411706372 fvfslocked =3D -1067018852 error =3D -1006353384 #20 0xc0583d49 in rename (td=3D0xc40414b0, uap=3D0x12) at /usr/src/sys/kern= /vfs_syscalls.c:3167 No locals. #21 0xc0659dcb in syscall (frame=3D {tf_fs =3D 1858994235, tf_es =3D -1078001605, tf_ds =3D 136249403, tf= _edi =3D 1859007112, tf_esi =3D -1077940604, tf_ebp =3D 136256060, tf_isp = =3D -411706012, tf_ebx =3D 3, tf_edx =3D 32768, tf_ecx =3D 0, tf_eax =3D 12= 8, tf_trapno =3D 22, tf_err =3D 2, tf_eip =3D 1859694163, tf_cs =3D 51, tf_= eflags =3D 530, tf_esp =3D 136255664, tf_ss =3D 59}) at /usr/src/sys/i386/i= 386/trap.c:976 params =3D 0x81f18b4
callp =3D (struct sysent *) 0xc06b74c0 td =3D (struct thread *) 0xc40414b0 p =3D (struct proc *) 0xc4044418 orig_tf_eflags =3D 530 sticks =3D 688 error =3D 0 narg =3D 2 args =3D {136590812, 136590216, 80, 0, 0, 0, 688, -1006353384} code =3D 128 #22 0xc06498ff in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s= :200 No locals. #23 0x00000033 in ?? () No symbol table info available. (kgdb)=20 Cheers, --=20 Xin LI http://www.delphij.net/ See complete headers for GPG key and other information. --zhXaljGHf11kAtnf Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQFDYld6/cVsHxFZiIoRAnlwAJ9kYgCVrvCFjKGSgP7TVwtKq36SuACaArz8 370ypzz45+ZF1TMePnlsy2s= =oEVa -----END PGP SIGNATURE----- --zhXaljGHf11kAtnf--