Date: Mon, 21 Nov 2005 01:31:29 GMT From: Rory Arms <rorya+FreeBSD@TrueStep.com> To: freebsd-gnats-submit@FreeBSD.org Subject: i386/89340: 6.0-STABLE (2005-11-07) panic Message-ID: <200511210131.jAL1VTKm048957@www.freebsd.org> Resent-Message-ID: <200511210140.jAL1ePYV098309@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 89340
>Category: i386
>Synopsis: 6.0-STABLE (2005-11-07) panic
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: freebsd-i386
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Nov 21 01:40:25 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator: Rory Arms
>Release: FreeBSD 6.0-STABLE, using 2005-11-07 RELENG_6 sources
>Organization:
>Environment:
FreeBSD Tserver 6.0-STABLE FreeBSD 6.0-STABLE #4: Mon Nov 7 22:16:59 EST 2005 root@Tserver:/usr/obj/usr/src/sys/TSERVER i386
>Description:
This is a dual processor Pentium II 366Mhz with all SCSI drives (ahc(4)) server had been up for about 12 days when it spontaneously crashed. It crashed when it was mostly idle, so I don't think it would have been doing much. I had the dumpdev directive on, in rc.conf(5), so it saved a core dump of the panic, that I could inspect.
I used kgdb(1) to get a trackback, however I noticed when I first invoked it, it printed a bunch of garbage, after printing "Unread portion of the kernel message buffer:." But, after several screens of "garbage" I got a (kgdb) prompt. So, what follows is the text presented from that point forward:
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb)
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc0535912 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xc0535d04 in panic (
fmt=0xc07b2e33 "initiate_write_inodeblock_ufs1: already started")
at /usr/src/sys/kern/kern_shutdown.c:555
#3 0xc06c4e9c in initiate_write_inodeblock_ufs1 (inodedep=0xc3ac5680, bp=0x0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3722
#4 0xc06c4a3c in softdep_disk_io_initiation (bp=0xcbeb6788)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3598
#5 0xc06ce60d in ffs_geom_strategy (bo=0xc1cf00c0, bp=0xcbeb6788) at buf.h:422
#6 0xc058dba9 in bufwrite (bp=0xcbeb6788) at buf.h:415
#7 0xc06ce5b8 in ffs_bufwrite (bp=0xcbeb6788)
at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1663
#8 0xc058fcad in vfs_bio_awrite (bp=0xcbeb6788) at buf.h:399
#9 0xc059897c in vop_stdfsync (ap=0xd570fca0)
at /usr/src/sys/kern/vfs_default.c:417
#10 0xc04da1ae in devfs_fsync (ap=0xd570fca0)
at /usr/src/sys/fs/devfs/devfs_vnops.c:307
#11 0xc077006e in VOP_FSYNC_APV (vop=0x0, a=0x0) at vnode_if.c:1020
#12 0xc05a2a84 in sync_vnode (bo=0xc1cf00c0, td=0xc1a97000) at vnode_if.h:537
#13 0xc05a2e6f in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1665
#14 0xc0519800 in fork_exit (callout=0xc05a2b80 <sched_sync>, arg=0x0,
frame=0x0) at /usr/src/sys/kern/kern_fork.c:789
---Type <return> to continue, or q <return> to quit---
#15 0xc0746aec in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
(kgdb)
The "panic line" as reported by savecore(8) is: "initiate_write_inodeblock_ufs1: already started"
Also worth noting, I had some of my UFS2 filesystems mounted with options "sync", since I wanted to avoid data loss because I was experiencing some problems with panics after upgrading from 6.0-BETA4 to 6.0-RELEASE (due to a problem in if_bridge(4) as it turns out, I see that there is a an outstanding PR about this; switching to bridge(4) stopped the panics) So, I wonder if that, perhaps, might have something to do with the panics. Actually I think only one filesystem was synchronous.
After a few days, I re-mounted some of them to "nosync" (mount -u -o nosync), after it seemed like this last update to -STABLE seemed, well, "stable". I think only one of them was still set to synchronous, But that was over a week ago, so I don't know if that is related to the panic. Also, AFAIK, all the volumes are formatted as UFS2.
Here is the output of mount(8) after the server has restarted from the panic:
> mount
/dev/da0s1a on / (ufs, local, synchronous)
devfs on /dev (devfs, local)
/dev/da0s1e on /usr (ufs, NFS exported, local, synchronous, soft-updates)
/dev/da0s1d on /var (ufs, local, synchronous, soft-updates)
/dev/ad0s1a on /Network (ufs, NFS exported, local, synchronous, soft-updates)
/dev/da1d on /mnt (ufs, local, synchronous)
devfs on /var/named/dev (devfs, local)
So, as you can tell, in my fstab(5), I have them defaulted to be mounted with options "sync", since they've all initially mounted with that option set.
Last, but not least I'll list the KLDs loaded as well as my kernel's config file (with some of comments stripped for brevity):
> kldstat
Id Refs Address Size Name
1 6 0xc0400000 4d9830 kernel
2 1 0xc08da000 51e8 if_tap.ko
3 1 0xc08e0000 62530 acpi.ko
4 1 0xc1eea000 2000 green_saver.ko
And the config:
> grep -v ^# /sys/i386/conf/TSERVER
machine i386
cpu I686_CPU
ident TSERVER_6R
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFS_ROOT # NFS usable as /, requires NFSCLIENT
options MSDOSFS # MSDOS Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_GPT # GUID Partition Tables.
options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
options ADAPTIVE_GIANT # Giant mutex is adaptive.
device apic # I/O APIC
options SMP # Symmetric Multi Processing
options IPDIVERT # divert(4)
options IPFIREWALL # ipfirewall(4)
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
options DUMMYNET # dummynet(4)
options IPSEC
options IPSEC_ESP
options BRIDGE
device pci
device fdc
device ata
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
options ATA_STATIC_ID # Static device numbering
device ahc # AHA2940 and onboard AIC7xxx devices
device scbus # SCSI bus (required for SCSI)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
device ses # SCSI Environmental Services (and SAF-TE)
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device vga # VGA video card driver
device splash # Splash screen and screen saver support
device sc
device agp # support several AGP chipsets
device pmtimer
device sio # 8250, 16[45]50 based serial ports
device miibus # MII bus support
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device wlan # 802.11 support
device ath
device ath_hal # Atheros HAL (includes binary component)
device ath_rate_onoe # Onoe rate control for ath driver
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device bpf # Berkeley packet filter
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device usb # USB Bus (required)
device ugen # Generic
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device ulpt # Printer
device ums # Mouse
>How-To-Repeat:
I don't know, it seems to be random, and this is the first time I've seen a panic of this type. As far as I can tell, at the time of the panic, the server wasn't doing anything out of the ordinary.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200511210131.jAL1VTKm048957>