Date: Tue, 10 Sep 2002 19:10:04 -0700 (PDT) From: El Vampiro <vampiro@rootshell.ru> To: freebsd-bugs@FreeBSD.org Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed. Message-ID: <200209110210.g8B2A4CZ045816@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/42277; it has been noted by GNATS.
From: El Vampiro <vampiro@rootshell.ru>
To: freebsd-gnats-submit@FreeBSD.org
Cc:
Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
Date: Wed, 11 Sep 2002 09:06:16 +0700
One more panic
# gdb -k kernel.debug.4 vmcore.4
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs
Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf
IdlePTD at phsyical address 0x003ee000
initial pcb at physical address 0x0035cf60
panicstr: softdep_lock: locking against myself
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xffff000a
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc022f4ac
stack pointer = 0x10:0xcd774ce8
frame pointer = 0x10:0xcd774ce8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 85221 (nmbd)
interrupt mask = bio
trap number = 12
panic: page fault
syncing disks... panic: softdep_lock: locking against myself
Uptime: 1d1h33m56s
dumping to dev #da/0x20009, offset 128
dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) where
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc01918af in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0191cd4 in poweroff_wait (junk=0xc03213c0, howto=-968037120)
at /usr/src/sys/kern/kern_shutdown.c:595
#3 0xc022f1de in acquire_lock (lk=0xc034b57c) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
#4 0xc02348dc in softdep_count_dependencies (bp=0xc64ced00, wantcount=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792
#5 0xc0237b18 in ffs_fsync (ap=0xcd774b88) at /usr/src/sys/ufs/ffs/ffs_vnops.c:168
#6 0xc0236603 in ffs_sync (mp=0xc12a6400, waitfor=2, cred=0xc0a42680, p=0xc03717e0) at vnode_if.h:558
#7 0xc01c1a53 in sync (p=0xc03717e0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
#8 0xc019164a in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#9 0xc0191cd4 in poweroff_wait (junk=0xc032b72c, howto=-1070419377)
at /usr/src/sys/kern/kern_shutdown.c:595
#10 0xc0279816 in trap_fatal (frame=0xcd774ca8, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
#11 0xc02794e9 in trap_pfault (frame=0xcd774ca8, usermode=0, eva=4294901770)
at /usr/src/sys/i386/i386/trap.c:867
#12 0xc02790a7 in trap (frame={tf_fs = -847839216, tf_es = -1048182768, tf_ds = 16, tf_edi = 0,
tf_esi = -1048123136, tf_ebp = -847819544, tf_isp = -847819564, tf_ebx = -65536, tf_edx = 0,
tf_ecx = -65536, tf_eax = -1046726272, tf_trapno = 12, tf_err = 0, tf_eip = -1071450964,
tf_cs = 8, tf_eflags = 66067, tf_esp = -847819512, tf_ss = -1071434834})
at /usr/src/sys/i386/i386/trap.c:466
#13 0xc022f4ac in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
#14 0xc02333ae in softdep_update_inodeblock (ip=0xc186e900, bp=0xc64c5fa4, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
#15 0xc022e32d in ffs_update (vp=0xce00c880, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#16 0xc022e611 in ffs_truncate (vp=0xce00c880, length=0, flags=0, cred=0x0, p=0xcd693560)
at /usr/src/sys/ufs/ffs/ffs_inode.c:201
#17 0xc0239390 in ufs_inactive (ap=0xcd774eb0) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
#18 0xc023e889 in ufs_vnoperate (ap=0xcd774eb0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
#19 0xc01bfba4 in vput (vp=0xce00c880) at vnode_if.h:815
#20 0xc01c2e8d in unlink (p=0xcd693560, uap=0xcd774f80) at /usr/src/sys/kern/vfs_syscalls.c:1504
#21 0xc0279a3a in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = -1078001617, tf_edi = 672627448,
tf_esi = 0, tf_ebp = -1077937936, tf_isp = -847818796, tf_ebx = 1, tf_edx = 135532544,
tf_ecx = 135548928, tf_eax = 10, tf_trapno = 12, tf_err = 2, tf_eip = 672239144, tf_cs = 31,
tf_eflags = 647, tf_esp = -1077940028, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
#22 0xc026d1e5 in Xint0x80_syscall ()
#23 0x805d23c in ?? ()
#24 0x804b714 in ?? ()
#25 0x804c40d in ?? ()
#26 0x804a871 in ?? ()
(kgdb) up 13
#13 0xc022f4ac in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
467 panic("worklist_remove: lock not held");
(kgdb) l
462 worklist_remove(item)
463 struct worklist *item;
464 {
465
466 if (lk.lkt_held == -1)
467 panic("worklist_remove: lock not held");
468 if ((item->wk_state & ONWORKLIST) == 0) {
469 FREE_LOCK(&lk);
470 panic("worklist_remove: not on list");
471 }
(kgdb) p lk
$1 = {lkt_spl = 0, lkt_held = -1}
(kgdb) up
#14 0xc02333ae in softdep_update_inodeblock (ip=0xc186e900, bp=0xc64c5fa4, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
3847 WORKLIST_REMOVE(wk);
(kgdb) l
3842 * operations dependent on the inode being written to disk
3843 * can be moved to the id_bufwait so that they will be
3844 * processed when the buffer I/O completes.
3845 */
3846 while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
3847 WORKLIST_REMOVE(wk);
3848 WORKLIST_INSERT(&inodedep->id_bufwait, wk);
3849 }
3850 /*
3851 * Newly allocated inodes cannot be written until the bitmap
(kgdb) p wk
$2 = (struct worklist *) 0x68c040
(kgdb) p *wk
Cannot access memory at address 0x68c040.
(kgdb) p *inodedep
$3 = {id_list = {wk_list = {le_next = 0x0, le_prev = 0xc19c1400}, wk_type = 1, wk_state = 32777},
id_hash = {le_next = 0x0, le_prev = 0xc1230f58}, id_fs = 0xc12d0800, id_ino = 22126,
id_nlinkdelta = 0, id_savedino = 0x0, id_deps = {le_next = 0x0, le_prev = 0xc1411118}, id_buf = 0x0,
id_savedsize = -1, id_pendinghd = {lh_first = 0x0}, id_bufwait = {lh_first = 0xc14aa380},
id_inowait = {lh_first = 0xffff0000}, id_inoupdt = {tqh_first = 0x0, tqh_last = 0xc19c39c4},
id_newinoupdt = {tqh_first = 0x0, tqh_last = 0xc19c39cc}}
Here is the kernel:
machine i386
cpu I586_CPU
cpu I686_CPU
ident NEWMONSTER
maxusers 128
makeoptions DEBUG=-g
options CPU_ENABLE_SSE
options INET
options FFS
options FFS_ROOT
options MFS
options NFS
options PROCFS
options CD9660
options COMPAT_43
options UCONSOLE
options USERCONFIG
options VISUAL_USERCONFIG
options PERFMON
options SYSVSHM
options SYSVMSG
options SYSVSEM
options SHMMAXPGS=8192 # max amount of shared memory pages (4k on i386)
options SHMALL=33554432 # max amount of shared memory (bytes)
options MSGMNB=8192 # max # of bytes in a queue
options MSGMNI=256 # number of message queue identifiers per system
options MSGSEG=256 # number of message segments per queue
options MSGSSZ=64 # size of a message segment
options MSGTQL=8192 # max messages in system per system
options SEMMAP=512
options SEMMNI=512
options SEMMNS=1024
options SEMMNU=512
options P1003_1B
options _KPOSIX_PRIORITY_SCHEDULING
options _KPOSIX_VERSION=199309L
options INCLUDE_CONFIG_FILE
options IPFILTER
options IPFILTER_LOG
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFW2
options RANDOM_IP_ID
options ICMP_BANDLIM
options PANIC_REBOOT_WAIT_TIME=20
options SMBFS
options LIBMCHAIN
options LIBICONV
options NETSMB
options NETSMBCRYPTO
options UFS_DIRHASH
options SOFTUPDATES
device isa
device pci
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device fd1 at fdc0 drive 1
device ata0 at isa? port IO_WD1 irq 14
device ata1 at isa? port IO_WD2 irq 15
device ata
device atadisk
device atapicd
options ATA_STATIC_ID
device ahc
device aic0 at isa?
device isp # Qlogic family
device ispfw # Firmware for QLogic HBAs
device scbus
device da
device sa
device cd
device pass
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1
device vga0 at isa?
device sc0 at isa? flags 0x100
options SC_HISTORY_SIZE=256
device npx0 at nexus? port IO_NPX irq 13
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
# device sio2 at isa? disable port IO_COM3 irq 5
# device sio3 at isa? disable port IO_COM4 irq 9
device miibus
device dc
device fxp
pseudo-device loop
pseudo-device ether
pseudo-device tun
pseudo-device pty
pseudo-device bpf 6
pseudo-device vn 2
pseudo-device gzip
pseudo-device splash
pseudo-device snp
And more info from crashdumps:
# vmstat -M vmcore.4 -N kernel.debug.4
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id
2 2 0 180728 26468 1058 0 0 0 1222 1 0 0 207 4001 150 4 5 91
# iostat -M vmcore.4 -N kernel.debug.4
tty da0 da1 acd0 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 373 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 4 0 5 0 91
# netstat -m -M vmcore.4 -N kernel.debug.4
65/1152/10240 mbufs in use (current/peak/max):
65 mbufs allocated to data
64/526/2560 mbuf clusters in use (current/peak/max)
1340 Kbytes allocated to network (17% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
# vmstat -m -M vmcore.4 -N kernel.debug.4 > vm4
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 984 296 936609 1280 0
32 554 342 4261294 640 0
64 15908 4956 30941725 320 22344
128 1344 448 6128690 160 43544
256 15654 2378 4975665 80 3876
512 470 154 91754 40 54
1K 118 22 3868477 20 6607
2K 57 113 2138 10 1322
4K 30 2 602308 5 0
8K 2 1 51 5 0
16K 6 0 250 5 0
32K 8 0 20 5 0
64K 10 0 10 5 0
128K 3 0 4 5 0
Memory usage type by bucket size
Size Type(s)
16 uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl,
ether_multi, BPF, vnodes, mount, pcb, soname, shm, rman, bus,
sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld
32 atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag,
indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl,
ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname,
taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl, uidinfo,
subproc, pgrp, temp, devbuf, proc-args, sigio, kld
64 isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon,
NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes,
vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc,
session, temp, devbuf, lockf, proc-args, file
128 ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF,
vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred,
temp, devbuf, zombie, proc-args, dev_t, timecounter, kld
256 UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon,
NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes,
ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue,
file desc
512 UFS dirhash, UFS mount, NFSV3 diroff, NFS daemon, BPF, mount,
BIO buffer, ptys, msg, ioctlops, bus, uidinfo, ATA generic, temp,
devbuf, prison, file desc
1K uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, shm, ioctlops,
bus, temp, devbuf, kqueue, file desc
2K UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver,
ioctlops, bus, proc, devbuf, file desc
4K memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus,
temp, devbuf
8K pagedep, syncache, bus, temp
16K VM pgdata, indirdep, shm, sem, msg, devbuf
32K BPF, sem, msg, temp, devbuf
64K ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf
128K vfscache, msg, SWAP
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
atkbddev 1 1K 1K 42379K 1 0 0 32
uc_devlist 14 2K 2K 42379K 14 0 0 16,1K
memdesc 1 4K 4K 42379K 1 0 0 4K
mbuf 1 4K 4K 42379K 1 0 0 4K
isadev 10 1K 1K 42379K 10 0 0 64
ZONE 16 2K 2K 42379K 16 0 0 128
VM pgdata 1 16K 16K 42379K 1 0 0 16K
ISOFS mount 1 64K 64K 42379K 1 0 0 64K
UFS dirhash 848 164K 253K 42379K 2445 0 0 16,32,64,128,256,512,1K,2K,4K
UFS mount 21 46K 46K 42379K 21 0 0 512,2K,4K
UFS ihash 1 64K 64K 42379K 1 0 0 64K
FFS node 15321 3831K 4424K 42379K 531382 0 0 256
dirrem 5 1K 5K 42379K 24227 0 0 32
mkdir 0 0K 1K 42379K 126 0 0 32
diradd 1 1K 3K 42379K 24703 0 0 32
freefile 1 1K 3K 42379K 13050 0 0 32
freeblks 2 1K 10K 42379K 11708 0 0 128
freefrag 0 0K 3K 42379K 5387 0 0 32
allocindir 0 0K 154K 42379K 24140 0 0 64
indirdep 0 0K 33K 42379K 1103 0 0 32,16K
allocdirect 2 1K 5K 42379K 21386 0 0 64
bmsafemap 2 1K 1K 42379K 10251 0 0 32
newblk 1 1K 1K 42379K 45527 0 0 32,256
inodedep 9 65K 77K 42379K 22480 0 0 128,64K
pagedep 10 9K 10K 42379K 11889 0 0 64,8K
p1003.1b 1 1K 1K 42379K 1 0 0 16
NFS hash 1 64K 64K 42379K 1 0 0 64K
NQNFS Lease 1 1K 1K 42379K 1 0 0 1K
NFSV3 srvdesc 0 0K 1K 42379K 63902 0 0 16,256
NFSV3 diroff 1 1K 1K 42379K 1 0 0 512
NFS daemon 71 8K 8K 42379K 71 0 0 64,256,512
NFS req 0 0K 1K 42379K 1950920 0 0 64
NFS srvsock 1 1K 1K 42379K 1 0 0 256
syncache 1 8K 8K 42379K 1 0 0 8K
tseg_qent 0 0K 3K 42379K 19470 0 0 32
IpFw/IpAcct 88 13K 59K 42379K 12062 0 0 64,128,256,1K
in_multi 3 1K 1K 42379K 3 0 0 32
routetbl 152 21K 135K 42379K 425559 0 0 16,32,64,128,256
ether_multi 12 1K 1K 42379K 12 0 0 16,32,64
ifaddr 27 6K 6K 42379K 27 0 0 32,64,256,2K
BPF 15 137K 266K 42379K 39 0 0 16,32,128,512,4K,32K
Export Host 6 2K 2K 42379K 6 0 0 256
vnodes 27 7K 7K 42379K 359 0 0 16,32,64,128,256
mount 16 8K 8K 42379K 18 0 0 16,128,512
cluster_save buffer 0 0K 1K 42379K 1595 0 0 32
vfscache 14754 1058K 1299K 42379K 559532 0 0 64,128,128K
BIO buffer 34 68K 290K 42379K 2145 0 0 512,2K
pcb 64 6K 8K 42379K 15315 0 0 16,32,64,2K
soname 38 4K 5K 42379K 900210 0 0 16,32,64,128
ACD driver 2 3K 3K 42379K 2 0 0 256,2K
ptys 6 3K 3K 42379K 6 0 0 512
ttys 210 27K 52K 42379K 2862 0 0 128,256
shm 12 23K 31K 42379K 1068 0 0 16,1K,16K
sem 3 88K 88K 42379K 3 0 0 16K,32K,64K
msg 4 137K 137K 42379K 4 0 0 512,16K,32K,128K
rman 58 4K 4K 42379K 401 0 0 16,64
iov 0 0K 1K 42379K 366 0 0 128
ioctlops 0 0K 4K 42379K 14 0 0 512,1K,2K,4K
taskqueue 1 1K 1K 42379K 1 0 0 32
SWAP 2 73K 145K 42379K 4 0 0 32,128K
ATAPI generic 1 1K 1K 42379K 2 0 0 32,128
kobj 1 4K 4K 42379K 1 0 0 4K
eventhandler 15 1K 1K 42379K 15 0 0 32,64
bus 375 34K 38K 42379K 690 0 0 16,32,64,128,256,512,1K,2K,4K,8K
sysctloid 10 1K 1K 42379K 10 0 0 16,64
sysctl 0 0K 1K 42379K 218127 0 0 16,32
uidinfo 11 1K 1K 42379K 1060 0 0 32,512
cred 76 10K 15K 42379K 712896 0 0 128
subproc 232 17K 27K 42379K 7763310 0 0 32,64,256
proc 2 4K 4K 42379K 2 0 0 2K
session 47 3K 4K 42379K 5890 0 0 64
pgrp 47 2K 2K 42379K 6119 0 0 32
ATA generic 2 1K 1K 42379K 2 0 0 16,512
temp 311 136K 151K 42379K 8534373 0 0 16,32,64,128,256,512,1K,4K,8K,32K,64K
devbuf 522 419K 419K 42379K 1185 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K,64K
lockf 43 3K 6K 42379K 1517843 0 0 64
prison 3 2K 2K 42379K 3 0 0 512
atexit 1 1K 1K 42379K 1 0 0 16
zombie 0 0K 2K 42379K 3877840 0 0 128
proc-args 78 4K 7K 42379K 3943027 0 0 16,32,64,128,256
kqueue 7 7K 24K 42379K 7347 0 0 256,1K
sigio 1 1K 1K 42379K 127 0 0 32
file 675 43K 72K 42379K 16631410 0 0 64
file desc 131 36K 66K 42379K 3881154 0 0 256,512,1K,2K
dev_t 663 83K 83K 42379K 663 0 0 128
timecounter 10 2K 2K 42379K 10 0 0 128
kld 4 1K 1K 42379K 35 0 0 16,32,128
Memory Totals: In Use Free Requests
6836K 1317K 51808995
--
VAMPIRO-RIPN
http://vampiro.rootshell.ru
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200209110210.g8B2A4CZ045816>