From owner-freebsd-current@FreeBSD.ORG Fri May 9 05:34:34 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4B1ED37B40F for ; Fri, 9 May 2003 05:34:34 -0700 (PDT) Received: from mail.acquirer.com (mail.acquirer.com [213.190.156.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 65D7B43FAF for ; Fri, 9 May 2003 05:34:30 -0700 (PDT) (envelope-from nick-list@netability.ie) X-Envelope-To: Received: from [192.168.100.44] (pancake.netability.ie [192.168.100.44]) by mail.acquirer.com (8.12.9/8.12.3) with ESMTP id h49CYSiC097850 for ; Fri, 9 May 2003 13:34:29 +0100 (IST) (envelope-from nick-list@netability.ie) From: Nick Hilliard To: current@freebsd.org Content-Type: text/plain Organization: Message-Id: <1052483667.62283.10.camel@pancake.netability.ie> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.1 Date: 09 May 2003 13:34:28 +0100 Content-Transfer-Encoding: 7bit Subject: panic: bremfree: removing a buffer not on a queue (again) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 May 2003 12:34:34 -0000 I'm getting fully repeatable crashes on an nfs server running -current, with a 4.6-release nfs client. This has been happening for some while, and the last crash was with kernel/world built from sources cvsupped yesterday. The client was running a buildworld with both obj and src mounted over nfs. The server was otherwise quiet, and had recently undergone a full, clean fsck. The server runs an smp kernel with 2 cpu's. Crash details below. If anyone is interested in having a look at this, please let me know. Nick pancake# gdb -k kernel.debug.8 vmcore.8 GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-undermydesk-freebsd"... panic: bremfree: removing a buffer not on a queue panic messages: --- panic: softdep_disk_io_initiation: read cpuid = 0; lapic.id = 00000000 boot() called on cpu#0 syncing disks, buffers remaining... panic: bremfree: removing a buffer not on a queue cpuid = 0; lapic.id = 00000000 boot() called on cpu#0 Uptime: 7m37s Dumping 255 MB ata0: resetting devices .. done 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 --- Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/vesa/vesa.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/vesa/vesa.ko.debug Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/linux/linux.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/linux/linux.ko.debug Reading symbols from /boot/kernel/sysvshm.ko...done. Loaded symbols for /boot/kernel/sysvshm.ko Reading symbols from /boot/kernel/sysvsem.ko...done. Loaded symbols for /boot/kernel/sysvsem.ko Reading symbols from /boot/kernel/sysvmsg.ko...done. Loaded symbols for /boot/kernel/sysvmsg.ko Reading symbols from /boot/kernel/if_xl.ko...done. Loaded symbols for /boot/kernel/if_xl.ko Reading symbols from /boot/kernel/miibus.ko...done. Loaded symbols for /boot/kernel/miibus.ko Reading symbols from /boot/kernel/snd_pcm.ko...done. Loaded symbols for /boot/kernel/snd_pcm.ko Reading symbols from /boot/kernel/snd_ich.ko...done. Loaded symbols for /boot/kernel/snd_ich.ko Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/random/random.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/random/random.ko.debug Reading symbols from /boot/kernel/ahc.ko...done. Loaded symbols for /boot/kernel/ahc.ko Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/fdc/fdc.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/fdc/fdc.ko.debug Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/acpi/acpi.ko.debug Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/nfsclient/nfsclient.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/nfsclient/nfsclient.ko.debug Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/nfsserver/nfsserver.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/nfsserver/nfsserver.ko.debug Reading symbols from /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/usb/usb.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/PANCAKE/modules/usr/src/sys/modules/usb/usb.ko.debug #0 doadump () at ../../../kern/kern_shutdown.c:239 239 dumping++; (kgdb) bt full #0 doadump () at ../../../kern/kern_shutdown.c:239 No locals. #1 0xc0191b81 in boot (howto=260) at ../../../kern/kern_shutdown.c:371 No locals. #2 0xc0191ed0 in poweroff_wait (junk=0xc02e351b, howto=1) at ../../../kern/kern_shutdown.c:542 td = (struct thread *) 0xc0eca0f0 bootopt = 260 buf = "bremfree: removing a buffer not on a queue", '\0' #3 0xc01d1600 in bremfreel (bp=0xc02e351b) at ../../../kern/vfs_bio.c:636 old_qindex = 260 #4 0xc01d1513 in bremfree (bp=0x0) at ../../../kern/vfs_bio.c:618 No locals. #5 0xc01d35cd in vfs_bio_awrite (bp=0x104) at ../../../kern/vfs_bio.c:1687 i = 1 j = 0 lblkno = 26 vp = (struct vnode *) 0xc0eca0f0 ncl = -1070713573 nwritten = -1070713573 size = 8192 maxcl = 16 #6 0xc025b622 in ffs_fsync (ap=0xcd2999cc) at ../../../ufs/ffs/ffs_vnops.c:255 vp = (struct vnode *) 0xc2cbe36c ip = (struct inode *) 0xc77c9400 bp = (struct buf *) 0xc77c9400 nbp = (struct buf *) 0xc779ad80 error = 0 wait = 0 passes = 4 skipmeta = 0 lbn = 38 #7 0xc025a709 in ffs_sync (mp=0xc269b600, waitfor=2, cred=0xc0eb5e80, td=0xc02fd5c0) at vnode_if.h:612 nvp = (struct vnode *) 0xc2cbe248 vp = (struct vnode *) 0xc2cbe36c devvp = (struct vnode *) 0xc2cbe36c ip = (struct inode *) 0x0 ump = (struct ufsmount *) 0xc2667c00 fs = (struct fs *) 0xc2727800 error = 0 count = 0 wait = 0 lockreq = 18 allerror = 0 #8 0xc01e866b in sync (td=0xc02fd5c0, uap=0x0) at ../../../kern/vfs_syscalls.c:138 mp = (struct mount *) 0xc269b600 nmp = (struct mount *) 0x0 asyncflag = 0 #9 0xc0191697 in boot (howto=256) at ../../../kern/kern_shutdown.c:280 bp = (struct buf *) 0xc0ec7000 iter = 4 nbusy = 0 pbusy = 1 subiter = 0 #10 0xc0191ed0 in poweroff_wait (junk=0xc02ea540, howto=-1071835885) at ../../../kern/kern_shutdown.c:542 td = (struct thread *) 0xc0eca0f0 bootopt = 256 buf = "bremfree: removing a buffer not on a queue", '\0' #11 0xc0253a2c in softdep_disk_io_initiation (bp=0xc0eca0f0) at ../../../ufs/ffs/ffs_softdep.c:3465 wk = (struct worklist *) 0xc02ea540 nextwk = (struct worklist *) 0xc773acf0 indirdep = (struct indirdep *) 0x100 inodedep = (struct inodedep *) 0x0 #12 0xc01da24d in cluster_wbuild (vp=0xc2cbe36c, size=8192, start_lbn=26, len=6) at buf.h:422 bp = (struct buf *) 0xc7720468 tbp = (struct buf *) 0xc77c9400 i = 5 j = 2 totalwritten = 0 dbsize = 16 ---Type to continue, or q to quit--- #13 0xc01d35c3 in vfs_bio_awrite (bp=0xc77683f8) at ../../../kern/vfs_bio.c:1681 i = 6 j = 0 lblkno = 21 vp = (struct vnode *) 0xc2cbe36c ncl = 6 nwritten = 6 size = 8192 maxcl = 16 #14 0xc025b622 in ffs_fsync (ap=0xcd299c18) at ../../../ufs/ffs/ffs_vnops.c:255 vp = (struct vnode *) 0xc2cbe36c ip = (struct inode *) 0xc77683f8 bp = (struct buf *) 0xc77683f8 nbp = (struct buf *) 0xc77388d8 error = 0 wait = 0 passes = 4 skipmeta = 0 lbn = 38 #15 0xc025a709 in ffs_sync (mp=0xc269b600, waitfor=3, cred=0xc0eb5e80, td=0xc0eca0f0) at vnode_if.h:612 nvp = (struct vnode *) 0xc2cbe248 vp = (struct vnode *) 0xc2cbe36c devvp = (struct vnode *) 0xc2cbe36c ip = (struct inode *) 0x0 ump = (struct ufsmount *) 0xc2667c00 fs = (struct fs *) 0xc2727800 error = 0 count = -1072152611 wait = 0 lockreq = 18 allerror = 0 #16 0xc01e7dea in sync_fsync (ap=0xcd299cc4) at ../../../kern/vfs_subr.c:3493 syncvp = (struct vnode *) 0x0 mp = (struct mount *) 0xc269b600 td = (struct thread *) 0xc0eca0f0 error = -852910908 asyncflag = 0 #17 0xc01e40ab in sched_sync () at vnode_if.h:612 slp = (struct synclist *) 0xcd299cc4 vp = (struct vnode *) 0x0 mp = (struct mount *) 0x0 starttime = 1052481081 td = (struct thread *) 0xc0eca0f0 #18 0xc017c793 in fork_exit (callout=0xc01e3ed0 , arg=0x0, frame=0x0) at ../../../kern/kern_fork.c:875 td = (struct thread *) 0x0 p = (struct proc *) 0xc25eb400 (kgdb)