From owner-freebsd-i386@FreeBSD.ORG Sun Jun 1 11:20:03 2008 Return-Path: Delivered-To: freebsd-i386@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A679C106566C for ; Sun, 1 Jun 2008 11:20:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id ABBAA8FC0C for ; Sun, 1 Jun 2008 11:20:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m51BK3rq044379 for ; Sun, 1 Jun 2008 11:20:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m51BK3EC044378; Sun, 1 Jun 2008 11:20:03 GMT (envelope-from gnats) Date: Sun, 1 Jun 2008 11:20:03 GMT Message-Id: <200806011120.m51BK3EC044378@freefall.freebsd.org> To: freebsd-i386@FreeBSD.org From: Alexander Cc: Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris net4801 (sys/vm/vm_pageout.c) X-BeenThere: freebsd-i386@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Alexander List-Id: I386-specific issues for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Jun 2008 11:20:03 -0000 The following reply was made to PR i386/123768; it has been noted by GNATS. From: Alexander To: bug-followup@freebsd.org Cc: Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris net4801 (sys/vm/vm_pageout.c) Date: Sun, 01 Jun 2008 13:08:42 +0200 This appears to be related to geli+gjournal, I managed to crash the =20 system just by deleting a couple of huge files on the geli+gjournal =20 volume. A few more crashes: Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 55967744B (53 MB) Blocksize: 512 Dumptime: Mon May 19 08:38:28 2008 Hostname: cakebox.tis Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008 root@cpqak.tis:/usr/obj/usr/src/sys/net4801 Panic String: lockmgr: thread 0xc0fdf440, not exclusive lock holder =20 0xc1425000 unlocking Dump Parity: 322179330 Bounds: 4 Dump Status: good root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.4 kgdb: core file: /var/crash/vmcore.4 kgdb: kernel image: /boot/kernel/kernel [GDB will not be able to debug user-mode threads: =20 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions= . Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". There is no member named pathname. Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20 /boot/kernel/ng_ether.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ether.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20 /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20 from /boot/kernel/geom_journal.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_journal.ko Unread portion of the kernel message buffer: panic: lockmgr: thread 0xc0fdf440, not exclusive lock holder =20 0xc1425000 unlocking Uptime: 5h33m42s Physical memory: 123 MB Dumping 53 MB: 38 22 6 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41= 8 #2 0xc0504820 in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc04f5a67 in _lockmgr (lkp=3D0xc129baf8, flags=3D6, interlkp=3D0xc129bb= 28, td=3D0xc0fdf440, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_lock.c= :414 #4 0xc055e519 in vop_stdunlock (ap=3D0xc7b6ac1c) at /usr/src/sys/kern/vfs_default.c:280 #5 0xc065fecb in VOP_UNLOCK_APV (vop=3D0xc06b5dc0, a=3D0xc7b6ac1c) at vnode_if.c:1667 #6 0xc056a350 in vput (vp=3D0xc129baa0) at vnode_if.h:877 #7 0xc06377cd in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1025 #8 0xc04e8d7b in fork_exit (callout=3D0xc0636a30 , arg=3D0x0, frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783 #9 0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 05 root@cakebox /var/crash > cat info.5 Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 50630656B (48 MB) Blocksize: 512 Dumptime: Mon May 19 20:03:08 2008 Hostname: cakebox.tis Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008 root@cpqak.tis:/usr/obj/usr/src/sys/net4801 Panic String: vn_finished_write: neg cnt Dump Parity: 2553044824 Bounds: 5 Dump Status: good root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.5 kgdb: core file: /var/crash/vmcore.5 kgdb: kernel image: /boot/kernel/kernel [GDB will not be able to debug user-mode threads: =20 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions= . Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". There is no member named pathname. Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20 /boot/kernel/ng_ether.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ether.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20 /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20 from /boot/kernel/geom_journal.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_journal.ko Unread portion of the kernel message buffer: panic: vn_finished_write: neg cnt Uptime: 5h36m3s Physical memory: 123 MB Dumping 48 MB: 33 17 1 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41= 8 #2 0xc0504820 in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc0574115 in vn_finished_write (mp=3D0xc0ff6538) at /usr/src/sys/kern/vfs_vnops.c:1049 #4 0xc0637816 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1028 #5 0xc04e8d7b in fork_exit (callout=3D0xc0636a30 , arg=3D0x0, frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783 #6 0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 05 root@cakebox /var/crash > cat info.6 Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 54063104B (51 MB) Blocksize: 512 Dumptime: Tue May 20 18:44:08 2008 Hostname: cakebox.tis Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008 root@cpqak.tis:/usr/obj/usr/src/sys/net4801 Panic String: vn_finished_write: neg cnt Dump Parity: 2624807000 Bounds: 6 Dump Status: good root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.6 kgdb: core file: /var/crash/vmcore.6 kgdb: kernel image: /boot/kernel/kernel [GDB will not be able to debug user-mode threads: =20 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions= . Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". There is no member named pathname. Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20 /boot/kernel/ng_ether.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ether.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20 /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20 from /boot/kernel/geom_journal.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_journal.ko Unread portion of the kernel message buffer: lockmgr: thread 0xc0fe0440 unlocking unheld lock panic: vn_finished_write: neg cnt Uptime: 22h39m57s Physical memory: 123 MB Dumping 51 MB: 36 20 4 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41= 8 #2 0xc0504820 in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc0574115 in vn_finished_write (mp=3D0xc0ff6538) at /usr/src/sys/kern/vfs_vnops.c:1049 #4 0xc0637816 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1028 #5 0xc04e8d7b in fork_exit (callout=3D0xc0636a30 , arg=3D0x0, frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783 #6 0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 05 Dump7 is pretty much the same, so I'll continue with dump 8: root@cakebox /var/crash > cat info.8 Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 55463936B (52 MB) Blocksize: 512 Dumptime: Fri May 23 10:31:47 2008 Hostname: cakebox.tis Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008 root@cpqak.tis:/usr/obj/usr/src/sys/net4801 Panic String: lockmgr: thread 0xc0fe0440, not exclusive lock holder =20 0xc0fdfaa0 unlocking Dump Parity: 3532742481 Bounds: 8 Dump Status: good root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.8 kgdb: core file: /var/crash/vmcore.8 kgdb: kernel image: /boot/kernel/kernel [GDB will not be able to debug user-mode threads: =20 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions= . Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". There is no member named pathname. Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20 /boot/kernel/ng_ether.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ether.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20 /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20 from /boot/kernel/geom_journal.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_journal.ko Unread portion of the kernel message buffer: panic: lockmgr: thread 0xc0fe0440, not exclusive lock holder =20 0xc0fdfaa0 unlocking Uptime: 17h33m3s Physical memory: 123 MB Dumping 52 MB: 37 21 5 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41= 8 #2 0xc0504820 in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc04f5a67 in _lockmgr (lkp=3D0xc18ff6b8, flags=3D6, interlkp=3D0xc18ff6= e8, td=3D0xc0fe0440, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_lock.c= :414 #4 0xc055e519 in vop_stdunlock (ap=3D0xc7b6ac1c) at /usr/src/sys/kern/vfs_default.c:280 #5 0xc065fecb in VOP_UNLOCK_APV (vop=3D0xc06b5dc0, a=3D0xc7b6ac1c) at vnode_if.c:1667 #6 0xc056a350 in vput (vp=3D0xc18ff660) at vnode_if.h:877 #7 0xc06377cd in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1025 #8 0xc04e8d7b in fork_exit (callout=3D0xc0636a30 , arg=3D0x0, frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783 #9 0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2= 05 (kgdb) list *0xc055e519 0xc055e519 is in vop_stdunlock (/usr/src/sys/kern/vfs_default.c:282). 277 { 278 struct vnode *vp =3D ap->a_vp; 279 280 return (lockmgr(vp->v_vnlock, ap->a_flags | =20 LK_RELEASE, VI_MTX(vp), 281 ap->a_td)); 282 } 283 284 /* See above. */ 285 int 286 vop_stdislocked(ap) We are now at Dump15 but it's just more of the same... ---------------------------------------------------------------- cakebox.homeunix.net - all the machine one needs..