Date: Wed, 1 Mar 2000 22:45:06 -0800 (PST) From: dhesi@rahul.net To: FreeBSD-gnats-submit@freebsd.org Subject: kern/17122: crash due to softdep_disk_write_complete: lock is held Message-ID: <20000302064506.88BC37C32@yellow.rahul.net>
next in thread | raw e-mail | index | archive | help
>Number: 17122
>Category: kern
>Synopsis: crash due to: softdep_disk_write_complete: lock is held
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Mar 1 22:50:00 PST 2000
>Closed-Date:
>Last-Modified:
>Originator: bolero rdroot
>Release: FreeBSD 3.4-STABLE i386
>Organization:
a2i communications
>Environment:
FreeBSD 3.4-STABLE i386, updated with cvsup Mon Feb 14 17:04:55 PST
2000. 300 MHz AMD-K6 CPU, FIC PA2013 (I think ) m/b, 128 M ECC memory,
Tulip NIC, ncr 53c875-based SCSI. SOFTUPDATES enabled in kernel. One
IBM DGHS09Z 9-gig SCSI disk.
>Description:
The machine crashes and reboots once every 1-3 days. Traceback of
crash dump is always similar and indicates softupdates as the likely
point of trouble. A debugging kernel is now in use and a full
traceback with symbols is at the end of this problem report. The kernel
config file is also included.
Please let me know if more information is needed.
>How-To-Repeat:
Wait for crash.
>Fix:
None known.
HERE IS THE TRACEBACK:
Script started on Wed Mar 1 22:22:29 2000
...
# gdb -k kernel.debug vmcore.4
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
IdlePTD 3538944
initial pcb at 2de888
panicstr: softdep_disk_write_complete: lock is held
panic messages:
---
panic: softdep_fsync: pending ops
syncing disks... panic: softdep_lock: locking against myself
panic: softdep_disk_write_complete: lock is held
panic: softdep_disk_write_complete: lock is held
panic: softdep_disk_write_complete: lock is held
panic: softdep_disk_write_complete: lock is held
panic: softdep_disk_write_complete: lock is held
panic: softdep_disk_write_complete: lock is held
dumping to dev 20401, offset 786432
dump 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
---
#0 boot (howto=260) at ../../kern/kern_shutdown.c:285
285 dumppcb.pcb_cr3 = rcr3();
(kgdb) where
#0 boot (howto=260) at ../../kern/kern_shutdown.c:285
#1 0xc0164440 in at_shutdown (
function=0xc02b0bd2 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2002>, arg=0xc4503f08, queue=-1044665344) at ../../kern/kern_shutdown.c:446
#2 0xc0221a40 in softdep_disk_write_complete (bp=0xc4503f08)
at ../../ufs/ffs/ffs_softdep.c:2898
#3 0xc0182ac2 in biodone (bp=0xc4503f08) at ../../kern/vfs_bio.c:1925
#4 0xc012377a in dadone (periph=0xc1bb6180, done_ccb=0xc1bf2e00)
at ../../cam/scsi/scsi_da.c:1312
#5 0xc011f367 in camisr (queue=0xc02ddb84) at ../../cam/cam_xpt.c:6144
#6 0xc011f179 in swi_cambio () at ../../cam/cam_xpt.c:6051
#7 0xc011c808 in xpt_polled_action (start_ccb=0xc9833324)
at ../../cam/cam_xpt.c:3336
#8 0xc0123c49 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1604
#9 0xc016418b in boot (howto=260) at ../../kern/kern_shutdown.c:280
#10 0xc0164440 in at_shutdown (
function=0xc02b0bd2 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2002>, arg=0xc44f7388, queue=-1044665344) at ../../kern/kern_shutdown.c:446
#11 0xc0221a40 in softdep_disk_write_complete (bp=0xc44f7388)
at ../../ufs/ffs/ffs_softdep.c:2898
#12 0xc0182ac2 in biodone (bp=0xc44f7388) at ../../kern/vfs_bio.c:1925
#13 0xc012377a in dadone (periph=0xc1bb6180, done_ccb=0xc1cae200)
---Type <return> to continue, or q <return> to quit---
at ../../cam/scsi/scsi_da.c:1312
#14 0xc011f367 in camisr (queue=0xc02ddb84) at ../../cam/cam_xpt.c:6144
#15 0xc011f179 in swi_cambio () at ../../cam/cam_xpt.c:6051
#16 0xc011c808 in xpt_polled_action (start_ccb=0xc9833808)
at ../../cam/cam_xpt.c:3336
#17 0xc0123c49 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1604
#18 0xc016418b in boot (howto=260) at ../../kern/kern_shutdown.c:280
#19 0xc0164440 in at_shutdown (
function=0xc02b0bd2 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2002>, arg=0xc455db18, queue=-1044665344) at ../../kern/kern_shutdown.c:446
#20 0xc0221a40 in softdep_disk_write_complete (bp=0xc455db18)
at ../../ufs/ffs/ffs_softdep.c:2898
#21 0xc0182ac2 in biodone (bp=0xc455db18) at ../../kern/vfs_bio.c:1925
#22 0xc012377a in dadone (periph=0xc1bb6180, done_ccb=0xc1bf2a00)
at ../../cam/scsi/scsi_da.c:1312
#23 0xc011f367 in camisr (queue=0xc02ddb84) at ../../cam/cam_xpt.c:6144
#24 0xc011f179 in swi_cambio () at ../../cam/cam_xpt.c:6051
#25 0xc011c808 in xpt_polled_action (start_ccb=0xc9833cec)
at ../../cam/cam_xpt.c:3336
#26 0xc0123c49 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1604
#27 0xc016418b in boot (howto=260) at ../../kern/kern_shutdown.c:280
---Type <return> to continue, or q <return> to quit---
#28 0xc0164440 in at_shutdown (
function=0xc02b0bd2 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2002>, arg=0xc4538678, queue=-1044665344) at ../../kern/kern_shutdown.c:446
#29 0xc0221a40 in softdep_disk_write_complete (bp=0xc4538678)
at ../../ufs/ffs/ffs_softdep.c:2898
#30 0xc0182ac2 in biodone (bp=0xc4538678) at ../../kern/vfs_bio.c:1925
#31 0xc012377a in dadone (periph=0xc1bb6180, done_ccb=0xc1bbee00)
at ../../cam/scsi/scsi_da.c:1312
#32 0xc011f367 in camisr (queue=0xc02ddb84) at ../../cam/cam_xpt.c:6144
#33 0xc011f179 in swi_cambio () at ../../cam/cam_xpt.c:6051
#34 0xc011c7cd in xpt_polled_action (start_ccb=0xc98341d4)
at ../../cam/cam_xpt.c:3325
#35 0xc0123c49 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1604
#36 0xc016418b in boot (howto=260) at ../../kern/kern_shutdown.c:280
#37 0xc0164440 in at_shutdown (
function=0xc02b0bd2 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2002>, arg=0xc450f930, queue=-1044665344) at ../../kern/kern_shutdown.c:446
#38 0xc0221a40 in softdep_disk_write_complete (bp=0xc450f930)
at ../../ufs/ffs/ffs_softdep.c:2898
#39 0xc0182ac2 in biodone (bp=0xc450f930) at ../../kern/vfs_bio.c:1925
#40 0xc012377a in dadone (periph=0xc1bb6180, done_ccb=0xc1b69200)
at ../../cam/scsi/scsi_da.c:1312
---Type <return> to continue, or q <return> to quit---
#41 0xc011f367 in camisr (queue=0xc02ddb84) at ../../cam/cam_xpt.c:6144
#42 0xc011f179 in swi_cambio () at ../../cam/cam_xpt.c:6051
#43 0xc011c7cd in xpt_polled_action (start_ccb=0xc98346bc)
at ../../cam/cam_xpt.c:3325
#44 0xc0123c49 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1604
#45 0xc016418b in boot (howto=260) at ../../kern/kern_shutdown.c:280
#46 0xc0164440 in at_shutdown (
function=0xc02b0bd2 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2002>, arg=0xc4544540, queue=-1044665344) at ../../kern/kern_shutdown.c:446
#47 0xc0221a40 in softdep_disk_write_complete (bp=0xc4544540)
at ../../ufs/ffs/ffs_softdep.c:2898
#48 0xc0182ac2 in biodone (bp=0xc4544540) at ../../kern/vfs_bio.c:1925
#49 0xc012377a in dadone (periph=0xc1bb6180, done_ccb=0xc1bbe600)
at ../../cam/scsi/scsi_da.c:1312
#50 0xc011f367 in camisr (queue=0xc02ddb84) at ../../cam/cam_xpt.c:6144
#51 0xc011f179 in swi_cambio () at ../../cam/cam_xpt.c:6051
#52 0xc011c7cd in xpt_polled_action (start_ccb=0xc9834ba4)
at ../../cam/cam_xpt.c:3325
#53 0xc0123c49 in dashutdown (howto=260, arg=0x0)
at ../../cam/scsi/scsi_da.c:1604
#54 0xc016418b in boot (howto=260) at ../../kern/kern_shutdown.c:280
#55 0xc0164440 in at_shutdown (
---Type <return> to continue, or q <return> to quit---
function=0xc02b01cc <__set_sysuninit_set_sym_M_DIRREM_uninit_sys_uninit+4>, arg=0xc1c97200, queue=-914141780) at ../../kern/kern_shutdown.c:446
#56 0xc021ea9d in acquire_lock (lk=0xc02d271c)
at ../../ufs/ffs/ffs_softdep.c:266
#57 0xc0222451 in softdep_update_inodeblock (ip=0xc1c97200, bp=0xc455c520,
waitfor=0) at ../../ufs/ffs/ffs_softdep.c:3447
#58 0xc021db24 in ffs_update (vp=0xc96dcd80, waitfor=0)
at ../../ufs/ffs/ffs_inode.c:105
#59 0xc0226f86 in ffs_fsync (ap=0xc9834e64) at ../../ufs/ffs/ffs_vnops.c:258
#60 0xc022535f in ffs_sync (mp=0xc1bcbe00, waitfor=2, cred=0xc0756580,
p=0xc02fa078) at vnode_if.h:499
#61 0xc0189c3b in sync (p=0xc02fa078, uap=0x0) at ../../kern/vfs_syscalls.c:549
#62 0xc0164001 in boot (howto=256) at ../../kern/kern_shutdown.c:203
#63 0xc0164440 in at_shutdown (
function=0xc02b0f42 <__set_sysctl__debug_sym_sysctl___debug_rush_requests+2882>, arg=0xc98a8080, queue=0) at ../../kern/kern_shutdown.c:446
#64 0xc02226f2 in softdep_fsync (vp=0xc98a8080)
at ../../ufs/ffs/ffs_softdep.c:3567
#65 0xc018c52a in fsync (p=0xc97e04c0, uap=0xc9834f94)
at ../../kern/vfs_syscalls.c:2451
#66 0xc025bb8b in syscall (frame={tf_es = 39, tf_ds = 39,
tf_edi = -1077945576, tf_esi = 0, tf_ebp = -1077945816,
tf_isp = -914141212, tf_ebx = 134664136, tf_edx = 8, tf_ecx = 8,
---Type <return> to continue, or q <return> to quit---
tf_eax = 95, tf_trapno = 7, tf_err = 2, tf_eip = 671803168, tf_cs = 31,
tf_eflags = 582, tf_esp = -1077945832, tf_ss = 39})
at ../../i386/i386/trap.c:1100
#67 0xc024ecfc in Xint0x80_syscall ()
#68 0x804e728 in ?? ()
#69 0x8049bf1 in ?? ()
#70 0x804baf5 in ?? ()
#71 0x804bc06 in ?? ()
#72 0x80529fd in ?? ()
#73 0x804c377 in ?? ()
#74 0x8049f02 in ?? ()
#75 0x80499f1 in ?? ()
(kgdb) quit
# exit
exit
Script done on Wed Mar 1 22:23:17 2000
HERE IS THE KERNEL CONFIG
#
# GENERIC -- Generic machine with WD/AHx/NCR/BTx family disks
#
# For more information read the handbook part System Administration ->
# Configuring the FreeBSD Kernel -> The Configuration File.
# The handbook is available in /usr/share/doc/handbook or online as
# latest version from the FreeBSD World Wide Web server
# <URL:http://www.FreeBSD.ORG/>;
#
# An exhaustive list of options and more detailed explanations of the
# device lines is present in the ./LINT configuration file. If you are
# in doubt as to the purpose or necessity of a line, check first in LINT.
#
# $Id: i2,v 1.14 2000/03/01 11:01:31 rdroot Exp $
machine "i386"
#cpu "I386_CPU"
#cpu "I486_CPU"
cpu "I586_CPU"
cpu "I686_CPU"
ident GENERIC
maxusers 256
options MATH_EMULATE #Support for x87 emulation
options INET #InterNETworking
options FFS #Berkeley Fast Filesystem
options FFS_ROOT #FFS usable as root device [keep this!]
options MFS #Memory Filesystem
options MFS_ROOT #MFS usable as root device, "MFS" req'ed
options NFS #Network Filesystem
options NFS_ROOT #NFS usable as root device, "NFS" req'ed
options MSDOSFS #MSDOS Filesystem
options "CD9660" #ISO 9660 Filesystem
options "CD9660_ROOT" #CD-ROM usable as root. "CD9660" req'ed
options PROCFS #Process filesystem
options "COMPAT_43" #Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=15000 #Be pessimistic about Joe SCSI device
options UCONSOLE #Allow users to grab the console
options FAILSAFE #Be conservative
options USERCONFIG #boot -c editor
options VISUAL_USERCONFIG #visual boot -c editor
options NMBCLUSTERS=8192 #add more mbufs (default is 1024)
# Soft updates is technique for improving file system speed and
# making abrupt shutdown less risky. It is not enabled by default due
# to copyright restraints on the code that implement it.
#
# Read ../../ufs/ffs/README.softupdates to learn what you need to
# do to enable this. ../../contrib/softupdates/README gives
# more details on how they actually work.
#
options SOFTUPDATES
# do optimization (not done by default)
# .. disabled to see if kernel crashes go away
##makeoptions COPTFLAGS="-O2 -pipe"
config kernel root on wd0
# To make an SMP kernel, the next two are needed
#options SMP # Symmetric MultiProcessor Kernel
#options APIC_IO # Symmetric (APIC) I/O
# Optionally these may need tweaked, (defaults shown):
#options NCPU=2 # number of CPUs
#options NBUS=4 # number of busses
#options NAPIC=1 # number of IO APICs
#options NINTR=24 # number of INTs
controller isa0
controller eisa0
controller pci0
controller fdc0 at isa? port "IO_FD1" bio irq 6 drq 2
disk fd0 at fdc0 drive 0
disk fd1 at fdc0 drive 1
options "CMD640" # work around CMD640 chip deficiency
controller wdc0 at isa? port "IO_WD1" bio irq 14
disk wd0 at wdc0 drive 0
disk wd1 at wdc0 drive 1
controller wdc1 at isa? port "IO_WD2" bio irq 15
disk wd2 at wdc1 drive 0
disk wd3 at wdc1 drive 1
options ATAPI #Enable ATAPI support for IDE bus
options ATAPI_STATIC #Don't do it as an LKM
device acd0 #IDE CD-ROM
device wfd0 #IDE Floppy (e.g. LS-120)
# A single entry for any of these controllers (ncr, ahb, ahc) is
# sufficient for any number of installed devices.
controller ncr0
controller ahb0
controller ahc0
controller isp0
# This controller offers a number of configuration options, too many to
# document here - see the LINT file in this directory and look up the
# dpt0 entry there for much fuller documentation on this.
controller dpt0
controller adv0 at isa? port ? cam irq ?
controller adw0
controller bt0 at isa? port ? cam irq ?
controller aha0 at isa? port ? cam irq ?
#controller aic0 at isa? port 0x340 bio irq 11
controller scbus0
# including device ch0
# .. causes boot to hang
#device ch0 #SCSI media changers
device da0
device sa0
device pass0
device cd0 #Only need one of these, the code dynamically grows
device wt0 at isa? port 0x300 bio irq 5 drq 1
device mcd0 at isa? port 0x300 bio irq 10
controller matcd0 at isa? port 0x230 bio
device scd0 at isa? port 0x230 bio
# atkbdc0 controlls both the keyboard and the PS/2 mouse
controller atkbdc0 at isa? port IO_KBD tty
device atkbd0 at isa? tty irq 1
device psm0 at isa? tty irq 12
device vga0 at isa? port ? conflicts
# splash screen/screen saver
pseudo-device splash
# syscons is the default console driver, resembling an SCO console
device sc0 at isa? tty
# Enable this and PCVT_FREEBSD for pcvt vt220 compatible console driver
#device vt0 at isa? tty
#options XSERVER # support for X server
#options FAT_CURSOR # start with block cursor
# If you have a ThinkPAD, uncomment this along with the rest of the PCVT lines
#options PCVT_SCANSET=2 # IBM keyboards are non-std
device npx0 at isa? port IO_NPX irq 13
#
# Laptop support (see LINT for more options)
#
device apm0 at isa? disable flags 0x31 # Advanced Power Management
# PCCARD (PCMCIA) support
#controller card0
#device pcic0 at card?
#device pcic1 at card?
# Flags 0x30 = sum of 0x10 and 0x20.
# Flags 0x10 means this device is a potential system console.
# Flags 0x20 means this device is forced to be the system console.
# See 'man sio' for other flags.
device sio0 at isa? port "IO_COM1" flags 0x30 tty irq 4
device sio1 at isa? port "IO_COM2" tty irq 3
device sio2 at isa? disable port "IO_COM3" tty irq 5
device sio3 at isa? disable port "IO_COM4" tty irq 9
# Parallel port
device ppc0 at isa? port? net irq 7
controller ppbus0
device nlpt0 at ppbus?
device plip0 at ppbus?
device ppi0 at ppbus?
#controller vpo0 at ppbus?
# Order is important here due to intrusive probes, do *not* alphabetize
# this list of network interfaces until the probes have been fixed.
# Right now it appears that the ie0 must be probed before ep0. See
# revision 1.20 of this file.
device ax0
device de0
device fxp0
device mx0
device pn0
device rl0
device tl0
device tx0
device vr0
device vx0
device wb0
device xl0
device ed0 at isa? port 0x280 net irq 10 iomem 0xd8000
device ie0 at isa? port 0x300 net irq 10 iomem 0xd0000
device ep0 at isa? port 0x300 net irq 10
device ex0 at isa? port? net irq?
device fe0 at isa? port 0x300 net irq ?
device le0 at isa? port 0x300 net irq 5 iomem 0xd0000
device lnc0 at isa? port 0x280 net irq 10 drq 0
device ze0 at isa? port 0x300 net irq 10 iomem 0xd8000
device zp0 at isa? port 0x300 net irq 10 iomem 0xd8000
device cs0 at isa? port 0x300 net irq ?
pseudo-device loop
pseudo-device ether
pseudo-device sl 1
pseudo-device ppp 1
pseudo-device tun 1
pseudo-device pty 64
pseudo-device gzip # Exec gzipped a.out's
pseudo-device vn #Vnode driver (turns a file into a device)
pseudo-device ccd 4 #Concatenated disk driver
# KTRACE enables the system-call tracing facility ktrace(2).
# This adds 4 KB bloat to your kernel, and slightly increases
# the costs of each syscall.
options KTRACE #kernel tracing
# This provides support for System V shared memory and message queues.
#
options SYSVSHM
options SYSVMSG
# The `bpfilter' pseudo-device enables the Berkeley Packet Filter. Be
# aware of the legal and administrative consequences of enabling this
# option. The number of devices determines the maximum number of
# simultaneous BPF clients programs runnable.
pseudo-device bpfilter 4 #Berkeley packet filter
##options DUMMYNET # for dummynet(4) bandwidth limiter
options IPDIVERT # to use divert(4) sockets
options IPFIREWALL # needed by dummynet
options IPFIREWALL_DEFAULT_TO_ACCEPT # change default of above
# USB support
#controller uhci0
#controller ohci0
#controller usb0
#
# for the moment we have to specify the priorities of the device
# drivers explicitly by the ordering in the list below. This will
# be changed in the future.
#
#device ums0
#device ukbd0
#device ulpt0
#device uhub0
#device ucom0
#device umodem0
#device hid0
#device ugen0
#
#options USB_DEBUG
#options USBVERBOSE
== END ==
>Release-Note:
>Audit-Trail:
>Unformatted:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20000302064506.88BC37C32>