From owner-freebsd-stable@FreeBSD.ORG Fri Jul 30 00:32:00 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1D82D16A4CE for ; Fri, 30 Jul 2004 00:32:00 +0000 (GMT) Received: from services.cse.ucsc.edu (services.cse.ucsc.edu [128.114.48.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id E93CE43D5E for ; Fri, 30 Jul 2004 00:31:59 +0000 (GMT) (envelope-from paul@moondance.cse.ucsc.edu) Received: from moondance (moondance.cse.ucsc.edu [128.114.49.1]) i6U0VnCT002390; Thu, 29 Jul 2004 17:31:49 -0700 (PDT) Message-Id: <200407300031.i6U0VnCT002390@services.cse.ucsc.edu> To: freebsd-stable@freebsd.org Date: Thu, 29 Jul 2004 17:31:50 -0700 From: Paul Tatarsky cc: paul@services.cse.ucsc.edu Subject: Panic in ufs_dirhash.c 5.2.1-RELEASE-p9 help? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2004 00:32:00 -0000 I'm not sure if this is the right place to post this. I hope 5.2.1-RELEASE is considered part of this list. If not, I apologize. I don't know much about debuging kernel crashes in FreeBSD so here goes trying to provide enough information about the problem: I am running 5.2.1-RELEASE-p9. FreeBSD hgdownload 5.2.1-RELEASE-p9 FreeBSD 5.2.1-RELEASE-p9 #1: Thu Jul 29 08:29:56 PDT 2004 root@hgdownload:/usr/src/sys/i386/compile/HGDOWNLOAD i386 On a Dell 2650 with a PERC3/QC. Both data area and boot/swap drive is on the PERC3/QC drives. Separate logical drives. Recently, under heavy load (rsync download server for genome data) system has repeatably crashed with the following gdb -k output every time: (NOTE, the nullfs module is because we do a few loopback mounts) panic: mtx_lock() of spin mutex (null) @ ../../../ufs/ufs/ufs_dirhash.c:282 panic messages: --- panic: mtx_lock() of spin mutex (null) @ ../../../ufs/ufs/ufs_dirhash.c:282 Stack backtrace: Uptime: 7h24m50s Dumping 2047 MB 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960 976 992 1008 1024 1040 1056 1072 1088 1104 1120 1136 1152 1168 1184 1200 1216 1232 1248 1264 1280 1296 1312 1328 1344 1360 1376 1392 1408 1424 1440 1456 1472 1488 1504 1520 1536 1552 1568 1584 1600 1616 1632 1648 1664 1680 1696 1712 1728 1744 1760 1776 1792 1808 1824 1840 1856 1872 1888 1904 1920 1936 1952 1968 1984 2000 2016 2032 --- Reading symbols from /usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/acpi/acpi.ko.debug Reading symbols from /usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/nullfs/nullfs.ko.debug...done. Loaded symbols for /usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/nullfs/nullfs.ko.debug #0 doadump () at ../../../kern/kern_shutdown.c:240 240 dumping++; (kgdb) where #0 doadump () at ../../../kern/kern_shutdown.c:240 #1 0xc04c8158 in boot (howto=260) at ../../../kern/kern_shutdown.c:372 #2 0xc04c84e7 in panic () at ../../../kern/kern_shutdown.c:550 #3 0xc04be773 in _mtx_lock_flags (m=0x0, opts=0, file=0xc0674938 "../../../ufs/ufs/ufs_dirhash.c", line=282) at ../../../kern/kern_mutex.c:215 #4 0xc05d9eab in ufsdirhash_free (ip=0xc8c85578) at ../../../ufs/ufs/ufs_dirhash.c:282 #5 0xc05dbf44 in ufs_reclaim (ap=0x0) at ../../../ufs/ufs/ufs_inode.c:182 #6 0xc05e3eb8 in ufs_vnoperate (ap=0x0) at ../../../ufs/ufs/ufs_vnops.c:2793 #7 0xc0521c08 in vclean (vp=0xc8c9f514, flags=8, td=0xc8248640) at vnode_if.h:981 #8 0xc0522171 in vgonel (vp=0xc8c9f514, td=0x0) at ../../../kern/vfs_subr.c:2788 #9 0xc051e15a in vtryrecycle (vp=0xc8c9f514) at ../../../kern/vfs_subr.c:898 #10 0xc051e3e9 in getnewvnode (tag=0xc066da2c "ufs", mp=0x0, vops=0x0, vpp=0x0) at ../../../kern/vfs_subr.c:960 #11 0xc05d4be5 in ffs_vget (mp=0xc8235c00, ino=187006288, flags=2, vpp=0xee5aea4c) at ../../../ufs/ffs/ffs_vfsops.c:1268 #12 0xc05dcd12 in ufs_lookup (ap=0xee5aeb10) at ../../../ufs/ufs/ufs_lookup.c:603 #13 0xc05e3eb8 in ufs_vnoperate (ap=0x0) at ../../../ufs/ufs/ufs_vnops.c:2793 #14 0xc0514ca1 in vfs_cache_lookup (ap=0x0) at vnode_if.h:82 #15 0xc05e3eb8 in ufs_vnoperate (ap=0x0) at ../../../ufs/ufs/ufs_vnops.c:2793 #16 0xc0519ed2 in lookup (ndp=0xee5aec28) at vnode_if.h:52 #17 0xc05198be in namei (ndp=0xee5aec28) at ../../../kern/vfs_lookup.c:183 #18 0xc0527262 in lstat (td=0xc8248640, uap=0xee5aed14) at ../../../kern/vfs_syscalls.c:2030 #19 0xc06339e0 in syscall (frame= {tf_fs = 134742063, tf_es = 134742063, tf_ds = -1078001617, tf_edi = 134563840, tf_esi = 134563912, tf_ebp = -1077940936, tf_isp = -296030860, tf_ebx = 672370976, tf_edx = 134524928, tf_ecx = 0, tf_eax = 190, tf_trapno = 12, tf_err = 2, tf_eip = 671863887, tf_cs = 31, tf_eflags = 582, tf_esp = -1077941092, tf_ss = 47}) at ../../../i386/i386/trap.c:1010 #20 0xc062483d in Xint0x80_syscall () at {standard input}:136 ---Can't read userspace from dump, or kernel process--- Kernel Config is as follows, note we are running it uniprocessor to try to rule out SMP problems. We have also turned off logical processors (hyperthreading). machine i386 cpu I686_CPU ident HGDOWNLOAD options SCHED_4BSD #4BSD scheduler options INET #InterNETworking options INET6 #IPv6 communications protocols options FFS #Berkeley Fast Filesystem options SOFTUPDATES #Enable FFS soft updates support options UFS_ACL #Support for access control lists options UFS_DIRHASH #Improve performance on big directories options MD_ROOT #MD is a potential root device options NFSCLIENT #Network Filesystem Client options NFSSERVER #Network Filesystem Server options NFS_ROOT #NFS usable as /, requires NFSCLIENT options MSDOSFS #MSDOS Filesystem options CD9660 #ISO 9660 Filesystem options PROCFS #Process filesystem (requires PSEUDOFS) options PSEUDOFS #Pseudo-filesystem framework options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_FREEBSD4 #Compatible with FreeBSD4 options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI options KTRACE #ktrace(1) support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. options PFIL_HOOKS # pfil(9) framework makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols options DDB options DDB_TRACE options DDB_UNATTENDED ptions INVARIANTS #Enable calls of extra sanity checking options INVARIANT_SUPPORT #Extra sanity checks of internal structur # To make an SMP kernel, the next two are needed #options SMP # Symmetric MultiProcessor Kernel #device apic # I/O APIC device isa device pci device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) device amr # AMI MegaRAID device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc device agp # support several AGP chipsets # Floating point support - do not disable. device npx # Add suspend/resume support for the i8254. device pmtimer # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer device ppi # Parallel port interface device # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device bfe # Broadcom BCM440x 10/100 ethernet device bge # Broadcom BCM570xx Gigabit Ethernet device random # Entropy device device loop # Network loopback device ether # Ethernet support device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) Dmesg output follows: Copyright (c) 1992-2004 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.2.1-RELEASE-p9 #1: Thu Jul 29 08:29:56 PDT 2004 root@hgdownload:/usr/src/sys/i386/compile/HGDOWNLOAD Preloaded elf kernel "/boot/kernel/kernel" at 0xc07a8000. Preloaded elf module "/boot/kernel/acpi.ko" at 0xc07a821c. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2784.39-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff real memory = 2147352576 (2047 MB) avail memory = 2084798464 (1988 MB) Pentium Pro MTRR support enabled npx0: [FAST] npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 pcibios: BIOS version 2.10 Using $PIR table, 9 entries at 0xc00fc410 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 acpi_cpu0: on acpi0 acpi_cpu1: on acpi0 device_probe_and_attach: acpi_cpu1 attach returned 6 acpi_cpu1: on acpi0 device_probe_and_attach: acpi_cpu1 attach returned 6 acpi_cpu1: on acpi0 device_probe_and_attach: acpi_cpu1 attach returned 6 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib0: slot 4 INTA is routed to irq 11 pcib0: slot 4 INTB is routed to irq 10 pcib0: slot 4 INTC is routed to irq 7 pcib0: slot 15 INTA is routed to irq 5 pci0: at device 4.0 (no driver attached) pci0: at device 4.1 (no driver attached) pci0: at device 4.2 (no driver attached) pci0: at device 14.0 (no driver attached) pci0: at device 15.1 (no driver attached) pci0: at device 15.2 (no driver attached) isab0: at device 15.3 on pci0 isa0: on isab0 pcib1: on acpi0 pci6: on pcib1 pcib2: at device 8.0 on pci6 pci7: on pcib2 pcib2: slot 6 INTA is routed to irq 7 pcib2: slot 6 INTB is routed to irq 11 pci7: at device 6.0 (no driver attached) pci7: at device 6.1 (no driver attached) pcib3: on acpi0 pci5: on pcib3 pcib3: slot 6 INTA is routed to irq 11 pcib3: slot 8 INTA is routed to irq 10 bge0: mem 0xeff10000-0xeff 1ffff irq 11 at device 6.0 on pci5 bge0: Ethernet address: 00:0d:56:ba:cd:45 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX -FDX, auto bge1: mem 0xeff00000-0xeff 0ffff irq 10 at device 8.0 on pci5 bge1: Ethernet address: 00:0d:56:ba:cd:46 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX -FDX, auto pcib4: on acpi0 pci4: on pcib4 pcib5: on acpi0 pci1: on pcib5 pcib6: at device 8.0 on pci1 pci2: on pcib6 pcib5: slot 8 INTB is routed to irq 11 pcib6: slot 1 INTA is routed to irq 11 pcib5: slot 8 INTC is routed to irq 10 pcib6: slot 2 INTA is routed to irq 10 pcib7: at device 0.0 on pci2 pci3: on pcib7 pcib5: slot 8 INTA is routed to irq 7 pcib6: slot 0 INTA is routed to irq 7 pcib7: slot 0 INTA is routed to irq 7 amr0: mem 0xf0000000-0xf7ffffff irq 7 at device 0.0 on pci3 amr0: Firmware 196T, BIOS 3.33, 128MB RAM pci2: at device 1.0 (no driver attached) pci2: at device 2.0 (no driver attached) fdc0: port 0x3f7,0x3f0-0 x3f5 irq 6 drq 2 on acpi0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0 port 0x3f8-0x3ff irq 4 on acpi0 sio0: type 16550A sio1 port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A acpi_cpu1: on acpi0 device_probe_and_attach: acpi_cpu1 attach returned 6 acpi_cpu1: on acpi0 device_probe_and_attach: acpi_cpu1 attach returned 6 acpi_cpu1: on acpi0 device_probe_and_attach: acpi_cpu1 attach returned 6 pmtimer0 on isa0 orm0: