Date: Sun, 1 Sep 2002 11:17:22 +0700 (OMSST) From: El Vampiro <vampiro@rootshell.ru> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed. Message-ID: <20020901041722.7C3385430@vampiro.rootshell.ru>
next in thread | raw e-mail | index | archive | help
>Number: 42277
>Category: kern
>Synopsis: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sat Aug 31 21:20:01 PDT 2002
>Closed-Date:
>Last-Modified:
>Originator: El Vampiro
>Release: FreeBSD 4.6-STABLE i386
>Organization:
>Environment:
System: FreeBSD rshb.com.ru 4.6-STABLE FreeBSD 4.6-STABLE #0: Sat Aug 31 13:17:44 OMSST 2002 vampiro@vampiro.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER i386
Hardware: new Intel brand server
Kernel:
machine i386
cpu I586_CPU
cpu I686_CPU
ident NEWMONSTER
maxusers 128
makeoptions DEBUG=-g
options CPU_ENABLE_SSE
options INET
options FFS
options FFS_ROOT
options NFS
options SOFTUPDATES
options PROCFS
options CD9660
options COMPAT_43
options UCONSOLE
options USERCONFIG
options VISUAL_USERCONFIG
options P1003_1B
options _KPOSIX_PRIORITY_SCHEDULING
options _KPOSIX_VERSION=199309L
options KTRACE
options PERFMON
options SYSVSHM
options SYSVMSG
options SYSVSEM
options SHMMAXPGS=8192 # max amount of shared memory pages (4k on i386)
options SHMMNI=512 # max shared mem id's per system
options SHMSEG=256 # max shared mem id's per process
options MSGMNB=8192 # max # of bytes in a queue
options MSGMNI=256 # number of message queue identifiers
options MSGSEG=256 # number of message segments per queue
options MSGSSZ=64 # size of a message segment
options MSGTQL=8192 # max messages in system
options SEMMAP=256
options SEMMNI=256
options SEMMNS=512
options SEMMNU=256
options INCLUDE_CONFIG_FILE
options IPFILTER
options IPFILTER_LOG
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFW2
options RANDOM_IP_ID
options ICMP_BANDLIM
options ACCEPT_FILTER_HTTP
options VESA
options PANIC_REBOOT_WAIT_TIME=20
options SMBFS
options LIBMCHAIN
options LIBICONV
options NETSMB
options NETSMBCRYPTO
options UFS_DIRHASH
options SHOW_BUSYBUFS
options HZ=1000
device isa
device pci
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device fd1 at fdc0 drive 1
device ata0 at isa? port IO_WD1 irq 14
device ata1 at isa? port IO_WD2 irq 15
device ata
device atadisk
device atapicd
options ATA_STATIC_ID
device ahc
device aic0 at isa?
device scbus
device da
device sa
device cd
device pass
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1
device vga0 at isa?
device sc0 at isa? flags 0x100
device npx0 at nexus? port IO_NPX irq 13
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
device miibus
device dc
device fxp
pseudo-device loop
pseudo-device ether
pseudo-device tun
pseudo-device pty
pseudo-device bpf 8
pseudo-device vn 2
pseudo-device gzip
pseudo-device splash
device smbus
device intpm
device alpm
device ichsmb
device viapm
device smb
device iicbus
device iicbb
device ic
device iic
device iicsmb
device apm0 at nexus?
Copyright (c) 1992-2002 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.6-STABLE #0: Sat Aug 31 13:17:44 OMSST 2002
vampiro@vampiro.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER
Timecounter "i8254" frequency 1193182 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (999.72-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x68a Stepping = 10
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory = 268369920 (262080K bytes)
avail memory = 257720320 (251680K bytes)
Preloaded elf kernel "kernel" at 0xc0360000.
VESA: v2.0, 4096k memory, flags:0x0, mode table:0xc02fe6c2 (1000022)
VESA: ATI MACH64
netsmb_dev: loaded
Pentium Pro MTRR support enabled
Using $PIR table, 268435454 entries at 0xc00fdf10
apm: protected mode connections are not supported
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GV graphics accelerator> at 2.0 irq 11
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0x5800-0x583f mem 0xfb000000-0xfb0fffff,0xfb101000-0xfb101fff irq 10 at device 3.0 on pci0
fxp0: Ethernet address 00:d0:b7:b8:ab:64
inphy0: <i82555 10/100 media interface> on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
dc0: <Davicom DM9102A 10/100BaseTX> port 0x5400-0x54ff mem 0xfb102000-0xfb1020ff irq 7 at device 7.0 on pci0
dc0: Ethernet address: 00:80:ad:08:12:2b
miibus1: <MII bus> on dc0
ukphy0: <Generic IEEE 802.3u media interface> on miibus1
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks ROSB4 ATA33 controller> port 0x5840-0x584f,0x374-0x377,0x170-0x177 at device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pcib1: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6000-0x60ff mem 0xfd000000-0xfd000fff irq 9 at device 4.0 on pci1
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6400-0x64ff mem 0xfd001000-0xfd001fff irq 5 at device 4.1 on pci1
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcf7ff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ipfw2 initialized, divert disabled, rule-based forwarding enabled, default to accept, logging unlimited
IP Filter: v3.4.27 initialized. Default = pass all, Logging = enabled
acd0: CDROM <SONY CD-ROM CDU5221> at ata0-master PIO4
Waiting 2 seconds for SCSI devices to settle
sa0 at ahc1 bus 0 target 0 lun 0
sa0: <ARCHIVE Python 04106-XXX 7550> Removable Sequential Access SCSI-2 device
sa0: 10.000MB/s transfers (10.000MHz, offset 15)
da0 at ahc0 bus 0 target 0 lun 0
da0: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
da0: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
da1 at ahc0 bus 0 target 1 lun 0
da1: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device
da1: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
da1: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
Mounting root from ufs:/dev/da0s1a
>Description:
Several kernel panics per day with panicstr: softdep_lock: locki ng against myself
Upon reboot fsck discovers many filesystem errors. File damage occurs often.
$ gdb -k kernel.debug vmcore.0
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
/big1/vampiro/crashes/4-NEW/vmcore.0: Permission denied.
(kgdb) quit
$ gdb -k kernel.debug vmcore.0
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
IdlePTD at phsyical address 0x0037f000
initial pcb at physical address 0x002ea800
panicstr: softdep_lock: locking against myself
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xffff000a
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc022988c
stack pointer = 0x10:0xcd0f9d10
frame pointer = 0x10:0xcd0f9d10
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 6 (syncer)
interrupt mask = bio
trap number = 12
panic: page fault
syncing disks... panic: softdep_lock: locking against myself
Uptime: 3h29m7s
dumping to dev #da/0x20009, offset 128
dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) where
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc01888d3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0188cf8 in poweroff_wait (junk=0xc02ae0c0, howto=-1053066752)
at /usr/src/sys/kern/kern_shutdown.c:595
#3 0xc02295be in acquire_lock (lk=0xc02d8c7c)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
#4 0xc022d6d8 in softdep_update_inodeblock (ip=0xc13b7a00, bp=0xc64ec464,
waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
#5 0xc022870d in ffs_update (vp=0xcdbcefc0, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#6 0xc0232105 in ffs_fsync (ap=0xcd0f9bb0)
at /usr/src/sys/ufs/ffs/ffs_vnops.c:273
#7 0xc02309e3 in ffs_sync (mp=0xc12a1a00, waitfor=2, cred=0xc0a3c880,
p=0xc02ff300) at vnode_if.h:558
#8 0xc01b8f97 in sync (p=0xc02ff300, uap=0x0)
at /usr/src/sys/kern/vfs_syscalls.c:576
#9 0xc0188646 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#10 0xc0188cf8 in poweroff_wait (junk=0xc02b868c, howto=-1070890577)
at /usr/src/sys/kern/kern_shutdown.c:595
#11 0xc0274866 in trap_fatal (frame=0xcd0f9cd0, eva=4294901770)
at /usr/src/sys/i386/i386/trap.c:974
#12 0xc0274539 in trap_pfault (frame=0xcd0f9cd0, usermode=0, eva=4294901770)
at /usr/src/sys/i386/i386/trap.c:867
#13 0xc02740f7 in trap (frame={tf_fs = -854654960, tf_es = -1053884400,
tf_ds = 16, tf_edi = 0, tf_esi = -1053851648, tf_ebp = -854614768,
tf_isp = -854614788, tf_ebx = -65536, tf_edx = 0, tf_ecx = -65536,
tf_eax = -1051831424, tf_trapno = 12, tf_err = 0, tf_eip = -1071474548,
tf_cs = 8, tf_eflags = 66067, tf_esp = -854614736, tf_ss = -1071458418})
at /usr/src/sys/i386/i386/trap.c:466
#14 0xc022988c in worklist_remove (item=0xffff0000)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
#15 0xc022d78e in softdep_update_inodeblock (ip=0xc12f8000, bp=0xc651ab80,
waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
#16 0xc022870d in ffs_update (vp=0xcdd565c0, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#17 0xc02289f1 in ffs_truncate (vp=0xcdd565c0, length=0, flags=0, cred=0x0,
p=0xcc00a5e0) at /usr/src/sys/ufs/ffs/ffs_inode.c:201
#18 0xc0232e08 in ufs_inactive (ap=0xcd0f9ed8)
at /usr/src/sys/ufs/ufs/ufs_inode.c:89
#19 0xc0238301 in ufs_vnoperate (ap=0xcd0f9ed8)
at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
#20 0xc01b70e8 in vput (vp=0xcdd565c0) at vnode_if.h:815
#21 0xc022c594 in handle_workitem_remove (dirrem=0xc14034e0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
#22 0xc0229c0d in process_worklist_item (matchmnt=0x0, flags=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
#23 0xc0229ab2 in softdep_process_worklist (matchmnt=0x0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
#24 0xc01b6a0f in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177
(kgdb) up 14
#14 0xc022988c in worklist_remove (item=0xffff0000)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
467 panic("worklist_remove: lock not held");
(kgdb) l
462 worklist_remove(item)
463 struct worklist *item;
464 {
465
466 if (lk.lkt_held == -1)
467 panic("worklist_remove: lock not held");
468 if ((item->wk_state & ONWORKLIST) == 0) {
469 FREE_LOCK(&lk);
470 panic("worklist_remove: not on list");
471 }
(kgdb) p item
$1 = (struct worklist *) 0x0
(kgdb) p lk
$2 = {lkt_spl = 0, lkt_held = -1}
(kgdb) up
#15 0xc022d78e in softdep_update_inodeblock (ip=0xc12f8000, bp=0xc651ab80,
waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
3847 WORKLIST_REMOVE(wk);
(kgdb) l
3842 * operations dependent on the inode being written to disk
3843 * can be moved to the id_bufwait so that they will be
3844 * processed when the buffer I/O completes.
3845 */
3846 while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
3847 WORKLIST_REMOVE(wk);
3848 WORKLIST_INSERT(&inodedep->id_bufwait, wk);
3849 }
3850 /*
3851 * Newly allocated inodes cannot be written until the bitmap
(kgdb) p wk
$3 = (struct worklist *) 0x68c040
(kgdb) p inodedep
$4 = (struct inodedep *) 0xc14e5380
(kgdb) p inodedep->id_inowait
$5 = {lh_first = 0xffff0000}
(kgdb) p inodedep->id_bufwait
$6 = {lh_first = 0x0}
I can provide more info upon request.
>How-To-Repeat:
I found no way to repeat this. System can panic even upon minimal activity and can work upon hard load.
>Fix:
The only way I found is turn softupdates off
>Release-Note:
>Audit-Trail:
>Unformatted:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020901041722.7C3385430>