From owner-freebsd-current@FreeBSD.ORG Mon Mar 9 18:58:24 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B3DD106564A for ; Mon, 9 Mar 2009 18:58:24 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id E5CAE8FC1B for ; Mon, 9 Mar 2009 18:58:23 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so728911fgb.35 for ; Mon, 09 Mar 2009 11:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=HD2A1Y2OERgHJYeVh6+ljfi+d1vInrMcHePvuOuEoWM=; b=SggK9vc5FO8TN4EsbdfNxvrzXzjDbMsc624FdPr+4I11NHyHOAS8Pa7sEVg/ISwb57 YBePEaxIC5okEFAf8Pt7N9pIhHtkZLGoDxczjaexiVCx4DgRZGwmalmvNktFIRbEGuOB vUqND3BqGYo40IUsCMoJVfvzaBpj3OLRImKNY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=YYvlB1IqzYNV11WXSErcnW6c7YsxDV5l6rbmehFiKikR6a5luBSMXlPFZNuMSG/VUn RW/c3QZCmVKJAFf7pk+GlpJ1v6Y1L7Ki0Q1at4jqd5HPO0nAgOilV5fiqG3m49IrXIVp xEL+eSTzueqGIKKQbwYxjjjywiXcBGDqOz8bI= MIME-Version: 1.0 Received: by 10.86.98.18 with SMTP id v18mr4307861fgb.46.1236625102833; Mon, 09 Mar 2009 11:58:22 -0700 (PDT) In-Reply-To: <3a142e750903090615t63a594e8rbadd7f7231049180@mail.gmail.com> References: <3a142e750903090615t63a594e8rbadd7f7231049180@mail.gmail.com> Date: Mon, 9 Mar 2009 21:58:22 +0300 Message-ID: From: pluknet To: "Paul B. Mahol" , marcus@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current Subject: Re: nullfs: panic: vrele: negative ref cnt X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2009 18:58:25 -0000 2009/3/9 Paul B. Mahol : > Hi, > here is part of textdump: > > db:1:lockinfo> show locks > db:1:locks> show alllocks > Process 722 (sh) thread 0xc4134000 (100065) > db:1:alllocks> show lockedvnods > Locked vnodes > db:0:kdb.enter.panic> show pcpu > cpuid =3D 1 > curthread =3D 0xc4134000: pid 722 "sh" > curpcb =3D 0xc3ba8d90 > fpcurthread =3D 0xc4134000: pid 722 "sh" > idlethread =3D 0xc3d08d80: pid 10 "idle: cpu1" > APIC ID =3D 1 > currentldt =3D 0x50 > spin locks held: > db:0:kdb.enter.panic> bt > Tracing pid 722 tid 100065 td 0xc4134000 > kdb_enter(c0610bfe,c0610bfe,c061db9e,c3ba8aa8,1,...) at kdb_enter+0x3a > panic(c061db9e,0,c061cee6,88e,4,...) at panic+0x136 > vrele(c4326d9c,0,c433fb5f,82,c060f3c7,...) at vrele+0x111 > null_nodeget(c3fd4780,c4326d9c,c3ba8bf4,0,c3ba8be4,...) at null_nodeget+0= xa0 > null_bypass(c3ba8be0,c3f35a78,c3ba8c28) at null_bypass+0x141 > VOP_VPTOCNP_APV(c4340240,c3ba8be0,c061bf01,387,c3cecc00,...) at > VOP_VPTOCNP_APV+0xb3 > vn_vptocnp(c3ba8c4c,c061bf01,3af,c3ba8c3c,c3f35a78,...) at vn_vptocnp+0xd= 9 > vn_fullpath1(c3cecc00,c3ba8c5c,3fc,315,c3cecc00,...) at vn_fullpath1+0x29= a > kern___getcwd(c4134000,bfbfe884,0,400,c3ba8d2c,...) at kern___getcwd+0xd0 > __getcwd(c4134000,c3ba8cf8,8,c0617c09,c0640290,...) at __getcwd+0x29 > syscall(c3ba8d38) at syscall+0x261 > Xint0x80_syscall() at Xint0x80_syscall+0x20 > --- syscall (326, FreeBSD ELF32, __getcwd), eip =3D 0x2814edfb, esp =3D > 0xbfbfe6cc, ebp =3D 0xbfbfe788 --- > > > panic: vrele: negative ref cnt > cpuid =3D 1 > KDB: enter: panic > shared lockmgr ufs (ufs) r =3D 0 (0xc43c36a0) locked @ > /usr/src/sys/kern/vfs_default.c:790 > shared lockmgr ufs (ufs) r =3D 0 (0xc43c36a0) locked @ > /usr/src/sys/kern/vfs_default.c:790 > > 0xc43c3648: tag ufs, type VDIR > usecount 1, writecount 0, refcount 1 mountedhere 0 > flags () > lock type ufs: SHARED (count 1) > ino 3583142, on dev ad0s1h > > 0xc43c3430: tag null, type VDIR > usecount 4, writecount 0, refcount 5 mountedhere 0 > flags () > lock type ufs: SHARED (count 1) > vp=3D0xc43c3430, lowervp=3D0xc43c3648 > > > How to reproduce it: > > # mount_nullfs /usr/local/src /usr/src > # cd /sys > # whereis something > > > It appears panic have been introduced after r189488 > [marcus@ cc'ed] Futher debugging shows: Script started on Mon Mar 9 21:43:17 2009 c8-vb# kgdb /boot/ke=07rnel/ke=07rnel /var/crash/vm=07core.1 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you ar= e welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: , FreeBSD ELF32, __getcwd), eip =3D 0x28154b2b, esp =3D 0xbfbfe71c, ebp =3D 0xbfbfe7d8 --- null_hashget: 0xc4451560 is not locked but should be KDB: enter: lock violation panic: mutex Giant not owned at /usr/src/sys/kern/tty_ttydisc.c:1127 cpuid =3D 0 Uptime: 43m7s Physical memory: 1015 MB Dumping 51 MB: 36 20 4 Reading symbols from /boot/kernel/if_em.ko...Reading symbols from /boot/kernel/if_em.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_em.ko Reading symbols from /boot/kernel/nfsclient.ko...Reading symbols from /boot/kernel/nfsclient.ko.symbols...done. done. Loaded symbols for /boot/kernel/nfsclient.ko #0 doadump () at pcpu.h:246 246 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:246 #1 0xc05c458e in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:4= 20 #2 0xc05c4862 in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:576 #3 0xc05b4c77 in _mtx_assert (m=3D0xc08b6170, what=3D4, file=3D0xc0829baf "/usr/src/sys/kern/tty_ttydisc.c", line=3D1127) at /usr/src/sys/kern/kern_mutex.c:639 #4 0xc06197dd in ttydisc_getc (tp=3D0xc3dbf000, buf=3D0xe66bc7c0, len=3D12= 8) at /usr/src/sys/kern/tty_ttydisc.c:1127 #5 0xc050daf4 in sctty_outwakeup (tp=3D0xc3dbf000) at /usr/src/sys/dev/syscons/syscons.c:323 #6 0xc050e05c in scgetc (sc=3D0xc0a73000, flags=3D3) at /usr/src/sys/dev/syscons/syscons.c:3281 #7 0xc050e680 in sc_cngetc (cd=3D0xc0863d20) at /usr/src/sys/dev/syscons/syscons.c:1607 #8 0xc058cd68 in cncheckc () at /usr/src/sys/kern/kern_cons.c:377 #9 0xc058cda6 in cngetc () at /usr/src/sys/kern/kern_cons.c:355 #10 0xc0480435 in db_readline (lstart=3D0xc08a0100 "", lsize=3D120) at /usr/src/sys/ddb/db_input.c:326 #11 0xc0480e0a in db_read_line () at /usr/src/sys/ddb/db_lex.c:56 #12 0xc047f0b6 in db_command_loop () at /usr/src/sys/ddb/db_command.c:496 #13 0xc0480f2d in db_trap (type=3D3, code=3D0) at /usr/src/sys/ddb/db_main.= c:229 #14 0xc05f1d86 in kdb_trap (type=3D3, code=3D0, tf=3D0xe66bca34) at /usr/src/sys/kern/subr_kdb.c:534 #15 0xc07bc856 in trap (frame=3D0xe66bca34) at /usr/src/sys/i386/i386/trap.= c:678 ---Type to continue, or q to quit--- #16 0xc07a15bb in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #17 0xc05f1f0a in kdb_enter (why=3D0xc082deeb "vfslock", msg=3D0xc082dedc "lock violation") at cpufunc.h:71 #18 0xc064a541 in vfs_badlock (msg=3D0xc082df39 "is not locked but should b= e", str=3D0xc08173c4 "null_hashget", vp=3D0xc4451560) at /usr/src/sys/kern/vfs_subr.c:3699 #19 0xc064ddf2 in assert_vop_locked (vp=3D0xc4451560, str=3D0xc08173c4 "null_hashget") at /usr/src/sys/kern/vfs_subr.c:3723 #20 0xc055c5df in null_nodeget (mp=3D0xc40152cc, lowervp=3D0xc4451560, vpp=3D0xe66bcbf0) at /usr/src/sys/fs/nullfs/null_subr.c:110 #21 0xc055d75a in null_bypass (ap=3D0xe66bcbdc) at /usr/src/sys/fs/nullfs/null_vnops.c:325 #22 0xc07c86a3 in VOP_VPTOCNP_APV (vop=3D0xc08688a0, a=3D0xe66bcbdc) at vnode_if.c:2871 #23 0xc063a749 in vn_vptocnp (vp=3D0xe66bcc24, bp=3D0xe66bcc28, buf=3D0xc3e15c00 'p' ..., buflen=3D0xe66bcc48) at vnode_if.h:1541 #24 0xc063aaca in vn_fullpath1 (td=3DVariable "td" is not available. ) at /usr/src/sys/kern/vfs_cache.c:1001 #25 0xc063ae60 in kern___getcwd (td=3D0xc4435000, buf=3D0xbfbfe8d4
, bufseg=3DUIO_USERS= PACE, buflen=3D1024) at /usr/src/sys/kern/vfs_cache.c:795 #26 0xc063b009 in __getcwd (td=3D0xc4435000, uap=3D0xe66bccf8) at /usr/src/sys/kern/vfs_cache.c:769 #27 0xc07bbfd4 in syscall (frame=3D0xe66bcd38) ---Type to continue, or q to quit--- at /usr/src/sys/i386/i386/trap.c:1066 #28 0xc07a1620 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261 #29 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) c8-vb# c8-vb# ^D=08=08exit Script done on Mon Mar 9 21:43:35 2009 --=20 wbr, pluknet