Date: Sun, 8 Sep 2002 19:30:04 -0700 (PDT) From: El Vampiro <vampiro@rootshell.ru> To: freebsd-bugs@FreeBSD.org Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed. Message-ID: <200209090230.g892U4Cl067557@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/42277; it has been noted by GNATS.
From: El Vampiro <vampiro@rootshell.ru>
To: freebsd-gnats-submit@FreeBSD.org
Cc:
Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
Date: Mon, 9 Sep 2002 09:25:46 +0700
Two more panisc during holidays.
First:
IdlePTD at phsyical address 0x003f0000
initial pcb at physical address 0x0035e080
panicstr: softdep_lock: locking against myself
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xffff000a
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc022f84c
stack pointer = 0x10:0xcd0c6d10
frame pointer = 0x10:0xcd0c6d10
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 6 (syncer)
interrupt mask = bio
trap number = 12
panic: page fault
syncing disks... panic: softdep_lock: locking against myself
Uptime: 1d2h45m48s
dumping to dev #da/0x20009, offset 128
dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) where
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc0191837 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0191c5c in poweroff_wait (junk=0xc03224c0, howto=-1052082688) at /usr/src/sys/kern/kern_shutdown.c:595
#3 0xc022f57e in acquire_lock (lk=0xc034c6bc) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
#4 0xc0233698 in softdep_update_inodeblock (ip=0xc14a7e00, bp=0xc64dce38, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
#5 0xc022e6cd in ffs_update (vp=0xcd869980, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#6 0xc02380c5 in ffs_fsync (ap=0xcd0c6bb0) at /usr/src/sys/ufs/ffs/ffs_vnops.c:273
#7 0xc02369a3 in ffs_sync (mp=0xc12a7200, waitfor=2, cred=0xc0a42680, p=0xc0372b60) at vnode_if.h:558
#8 0xc01c19db in sync (p=0xc0372b60, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
#9 0xc01915aa in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#10 0xc0191c5c in poweroff_wait (junk=0xc032c60c, howto=-1070415569) at /usr/src/sys/kern/kern_shutdown.c:595
#11 0xc0278df6 in trap_fatal (frame=0xcd0c6cd0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
#12 0xc0278ac9 in trap_pfault (frame=0xcd0c6cd0, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867
#13 0xc02786b3 in trap (frame={tf_fs = -854851568, tf_es = -1052377072, tf_ds = 16, tf_edi = 0, tf_esi = -1052351744,
tf_ebp = -854823664, tf_isp = -854823684, tf_ebx = -65536, tf_edx = -1051899520, tf_ecx = -65536,
tf_eax = -1051899520, tf_trapno = 12, tf_err = 0, tf_eip = -1071450036, tf_cs = 8, tf_eflags = 66067,
tf_esp = -854823632, tf_ss = -1071433906}) at /usr/src/sys/i386/i386/trap.c:466
#14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
#15 0xc023374e in softdep_update_inodeblock (ip=0xc1466300, bp=0xc64cde4c, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
#16 0xc022e6cd in ffs_update (vp=0xcdde9200, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#17 0xc022e9b1 in ffs_truncate (vp=0xcdde9200, length=0, flags=0, cred=0x0, p=0xcbfd75e0)
at /usr/src/sys/ufs/ffs/ffs_inode.c:201
#18 0xc0238dc8 in ufs_inactive (ap=0xcd0c6ed8) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
#19 0xc023e2c1 in ufs_vnoperate (ap=0xcd0c6ed8) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
#20 0xc01bfb2c in vput (vp=0xcdde9200) at vnode_if.h:815
#21 0xc0232554 in handle_workitem_remove (dirrem=0xc151a960) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
#22 0xc022fbcd in process_worklist_item (matchmnt=0x0, flags=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
#23 0xc022fa72 in softdep_process_worklist (matchmnt=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
#24 0xc01bf453 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177
(kgdb) up 14
#14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
467 panic("worklist_remove: lock not held");
(kgdb) l
462 worklist_remove(item)
463 struct worklist *item;
464 {
465
466 if (lk.lkt_held == -1)
467 panic("worklist_remove: lock not held");
468 if ((item->wk_state & ONWORKLIST) == 0) {
469 FREE_LOCK(&lk);
470 panic("worklist_remove: not on list");
471 }
(kgdb) p item
$1 = (struct worklist *) 0x0
(kgdb) p lk
$2 = {lkt_spl = 0, lkt_held = -1}
(kgdb) up
#15 0xc023374e in softdep_update_inodeblock (ip=0xc1466300, bp=0xc64cde4c, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
3847 WORKLIST_REMOVE(wk);
(kgdb) l
3842 * operations dependent on the inode being written to disk
3843 * can be moved to the id_bufwait so that they will be
3844 * processed when the buffer I/O completes.
3845 */
3846 while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
3847 WORKLIST_REMOVE(wk);
3848 WORKLIST_INSERT(&inodedep->id_bufwait, wk);
3849 }
3850 /*
3851 * Newly allocated inodes cannot be written until the bitmap
(kgdb) p wk
$3 = (struct worklist *) 0x68c040
(kgdb) p *wk
Cannot access memory at address 0x68c040.
(kgdb) p *inodedep
$4 = {id_list = {wk_list = {le_next = 0xc1943000, le_prev = 0xc64cdf8c}, wk_type = 1, wk_state = 32777}, id_hash = {
le_next = 0x0, le_prev = 0xc1222e94}, id_fs = 0xc12c7000, id_ino = 22082, id_nlinkdelta = 0, id_savedino = 0x0,
id_deps = {le_next = 0x0, le_prev = 0x0}, id_buf = 0x0, id_savedsize = -1, id_pendinghd = {lh_first = 0x0}, id_bufwait = {
lh_first = 0xc14d4c00}, id_inowait = {lh_first = 0xffff0000}, id_inoupdt = {tqh_first = 0x0, tqh_last = 0xc14d49c4},
id_newinoupdt = {tqh_first = 0x0, tqh_last = 0xc14d49cc}}
# vmstat -m -M vmcore.2
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 952 328 874701 1280 0
32 536 360 4176708 640 0
64 16355 5917 31821882 320 9082
128 1247 513 6295455 160 59721
256 17477 779 4662394 80 635
512 443 213 18611 40 0
1K 102 18 3986474 20 19
2K 42 138 2378 10 1463
4K 30 2 658022 5 0
8K 3 1 40 5 0
16K 5 0 245 5 0
32K 8 0 16 5 0
64K 10 0 10 5 0
128K 3 0 4 5 0
Memory usage type by bucket size
Size Type(s)
16 uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl,
ether_multi, BPF, vnodes, mount, pcb, soname, accf, shm, rman, bus,
sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld
32 atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag,
indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl,
ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname,
accf, taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl,
uidinfo, subproc, pgrp, temp, devbuf, proc-args, sigio, kld
64 isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon,
NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes,
vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc,
session, temp, devbuf, lockf, proc-args, file
128 ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF,
vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred,
temp, devbuf, zombie, proc-args, dev_t, timecounter, kld
256 UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon,
NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes,
ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue,
file desc
512 UFS dirhash, UFS mount, NFS daemon, BPF, mount, BIO buffer, ptys,
msg, bus, uidinfo, ATA generic, temp, devbuf, prison, file desc
1K uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, ioctlops, bus,
temp, devbuf, kqueue, file desc
2K UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver, shm,
ioctlops, bus, proc, devbuf, file desc
4K memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus,
temp, devbuf
8K pagedep, syncache, shm, bus, temp
16K VM pgdata, indirdep, sem, msg, temp, devbuf
32K BPF, sem, msg, temp, devbuf
64K ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf
128K vfscache, msg, SWAP
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
atkbddev 1 1K 1K 42378K 1 0 0 32
uc_devlist 14 2K 2K 42378K 14 0 0 16,1K
memdesc 1 4K 4K 42378K 1 0 0 4K
mbuf 1 4K 4K 42378K 1 0 0 4K
isadev 10 1K 1K 42378K 10 0 0 64
ZONE 16 2K 2K 42378K 16 0 0 128
VM pgdata 1 16K 16K 42378K 1 0 0 16K
ISOFS mount 1 64K 64K 42378K 1 0 0 64K
UFS dirhash 776 153K 264K 42378K 2226 0 0 16,32,64,128,256,512,1K,2K,4K
UFS mount 21 46K 46K 42378K 39 0 0 512,2K,4K
UFS ihash 1 64K 64K 42378K 1 0 0 64K
FFS node 17135 4284K 4485K 42378K 520818 0 0 256
dirrem 2 1K 3K 42378K 25912 0 0 32
mkdir 0 0K 1K 42378K 124 0 0 32
diradd 0 0K 3K 42378K 26553 0 0 32
freefile 0 0K 2K 42378K 13804 0 0 32
freeblks 1 1K 5K 42378K 12336 0 0 128
freefrag 0 0K 3K 42378K 5170 0 0 32
allocindir 1 1K 152K 42378K 20648 0 0 64
indirdep 1 1K 33K 42378K 1158 0 0 32,16K
allocdirect 1 1K 6K 42378K 22316 0 0 64
bmsafemap 3 1K 1K 42378K 11096 0 0 32
newblk 1 1K 1K 42378K 42965 0 0 32,256
inodedep 4 65K 77K 42378K 23254 0 0 128,64K
pagedep 3 9K 11K 42378K 12282 0 0 64,8K
p1003.1b 1 1K 1K 42378K 1 0 0 16
NFS hash 1 64K 64K 42378K 1 0 0 64K
NQNFS Lease 1 1K 1K 42378K 1 0 0 1K
NFSV3 srvdesc 0 0K 1K 42378K 62136 0 0 16,256
NFS daemon 71 8K 8K 42378K 71 0 0 64,256,512
NFS req 0 0K 1K 42378K 2015016 0 0 64
NFS srvsock 1 1K 1K 42378K 1 0 0 256
syncache 1 8K 8K 42378K 1 0 0 8K
tseg_qent 0 0K 2K 42378K 30284 0 0 32
IpFw/IpAcct 79 11K 53K 42378K 14699 0 0 64,128,256,1K
in_multi 3 1K 1K 42378K 3 0 0 32
routetbl 163 23K 126K 42378K 3651 0 0 16,32,64,128,256
ether_multi 12 1K 1K 42378K 12 0 0 16,32,64
ifaddr 27 6K 6K 42378K 27 0 0 32,64,256,2K
BPF 15 137K 266K 42378K 31 0 0 16,32,128,512,4K,32K
Export Host 6 2K 2K 42378K 6 0 0 256
vnodes 26 6K 6K 42378K 337 0 0 16,32,64,128,256
mount 16 8K 8K 42378K 24 0 0 16,128,512
cluster_save buffer 0 0K 1K 42378K 1521 0 0 32
vfscache 15065 1073K 1332K 42378K 549206 0 0 64,128,128K
BIO buffer 4 8K 286K 42378K 1184 0 0 512,2K
pcb 69 6K 8K 42378K 14966 0 0 16,32,64,2K
soname 38 4K 6K 42378K 839397 0 0 16,32,64,128
ACD driver 2 3K 3K 42378K 2 0 0 256,2K
accf 3 1K 1K 42378K 3 0 0 16,32
ptys 5 3K 3K 42378K 5 0 0 512
ttys 210 27K 47K 42378K 2355 0 0 128,256
shm 16 37K 51K 42378K 1085 0 0 16,2K,8K
sem 3 88K 88K 42378K 3 0 0 16K,32K,64K
msg 4 137K 137K 42378K 4 0 0 512,16K,32K,128K
rman 58 4K 4K 42378K 401 0 0 16,64
iov 0 0K 1K 42378K 406 0 0 128
ioctlops 0 0K 4K 42378K 7 0 0 1K,2K,4K
taskqueue 1 1K 1K 42378K 1 0 0 32
SWAP 2 73K 145K 42378K 4 0 0 32,128K
ATAPI generic 1 1K 1K 42378K 2 0 0 32,128
kobj 1 4K 4K 42378K 1 0 0 4K
eventhandler 15 1K 1K 42378K 15 0 0 32,64
bus 375 34K 37K 42378K 696 0 0 16,32,64,128,256,512,1K,2K,4K,8K
sysctloid 10 1K 1K 42378K 10 0 0 16,64
sysctl 0 0K 1K 42378K 4771 0 0 16,32
uidinfo 11 1K 1K 42378K 1271 0 0 32,512
cred 67 9K 13K 42378K 715419 0 0 128
subproc 241 17K 26K 42378K 8001154 0 0 32,64,256
proc 2 4K 4K 42378K 2 0 0 2K
session 47 3K 4K 42378K 4512 0 0 64
pgrp 47 2K 2K 42378K 4620 0 0 32
ATA generic 2 1K 1K 42378K 2 0 0 16,512
temp 305 138K 153K 42378K 8754546 0 0 16,32,64,128,256,512,1K,4K,8K,16K,32K,64K
devbuf 506 412K 413K 42378K 1141 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K,64K
lockf 42 3K 6K 42378K 1564032 0 0 64
prison 3 2K 2K 42378K 3 0 0 512
atexit 1 1K 1K 42378K 1 0 0 16
zombie 0 0K 1K 42378K 3997472 0 0 128
proc-args 81 5K 7K 42378K 4062493 0 0 16,32,64,128,256
kqueue 5 5K 22K 42378K 7104 0 0 256,1K
sigio 1 1K 1K 42378K 133 0 0 32
file 728 46K 67K 42378K 17098401 0 0 64
file desc 147 40K 62K 42378K 4000832 0 0 256,512,1K,2K
dev_t 663 83K 83K 42378K 663 0 0 128
timecounter 10 2K 2K 42378K 10 0 0 128
kld 4 1K 1K 42378K 35 0 0 16,32,128
Memory Totals: In Use Free Requests
7243K 1062K 52496940
And second panic:
IdlePTD at phsyical address 0x003f0000
initial pcb at physical address 0x0035e080
panicstr: softdep_lock: locking against myself
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xffff000a
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc022f84c
stack pointer = 0x10:0xcda0ace8
frame pointer = 0x10:0xcda0ace8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 28039 (nmbd)
interrupt mask = bio
trap number = 12
panic: page fault
syncing disks... panic: softdep_lock: locking against myself
Uptime: 1d5h27m47s
dumping to dev #da/0x20009, offset 128
dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) where
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc0191837 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0191c5c in poweroff_wait (junk=0xc03224c0, howto=-1052660736) at /usr/src/sys/kern/kern_shutdown.c:595
#3 0xc022f57e in acquire_lock (lk=0xc034c6bc) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
#4 0xc0233698 in softdep_update_inodeblock (ip=0xc141ac00, bp=0xc65508dc, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
#5 0xc022e6cd in ffs_update (vp=0xcd67e780, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#6 0xc02380c5 in ffs_fsync (ap=0xcda0ab88) at /usr/src/sys/ufs/ffs/ffs_vnops.c:273
#7 0xc02369a3 in ffs_sync (mp=0xc12a7400, waitfor=2, cred=0xc0a42680, p=0xc0372b60) at vnode_if.h:558
#8 0xc01c19db in sync (p=0xc0372b60, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
#9 0xc01915aa in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#10 0xc0191c5c in poweroff_wait (junk=0xc032c60c, howto=-1070415569) at /usr/src/sys/kern/kern_shutdown.c:595
#11 0xc0278df6 in trap_fatal (frame=0xcda0aca8, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
#12 0xc0278ac9 in trap_pfault (frame=0xcda0aca8, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867
#13 0xc02786b3 in trap (frame={tf_fs = -845152240, tf_es = -1049231344, tf_ds = 16, tf_edi = 0, tf_esi = -1049228288,
tf_ebp = -845107992, tf_isp = -845108012, tf_ebx = -65536, tf_edx = -1045960576, tf_ecx = -65536,
tf_eax = -1045960576, tf_trapno = 12, tf_err = 0, tf_eip = -1071450036, tf_cs = 8, tf_eflags = 66071,
tf_esp = -845107960, tf_ss = -1071433906}) at /usr/src/sys/i386/i386/trap.c:466
#14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
#15 0xc023374e in softdep_update_inodeblock (ip=0xc1760c00, bp=0xc6564930, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
#16 0xc022e6cd in ffs_update (vp=0xcdd3a440, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
#17 0xc022e9b1 in ffs_truncate (vp=0xcdd3a440, length=0, flags=0, cred=0x0, p=0xcd93e8a0)
at /usr/src/sys/ufs/ffs/ffs_inode.c:201
#18 0xc0238dc8 in ufs_inactive (ap=0xcda0aeb0) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
#19 0xc023e2c1 in ufs_vnoperate (ap=0xcda0aeb0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
#20 0xc01bfb2c in vput (vp=0xcdd3a440) at vnode_if.h:815
#21 0xc01c2e15 in unlink (p=0xcd93e8a0, uap=0xcda0af80) at /usr/src/sys/kern/vfs_syscalls.c:1504
#22 0xc027901a in syscall2 (frame={tf_fs = -1078001617, tf_es = 47, tf_ds = -1078001617, tf_edi = 672627448, tf_esi = 0,
tf_ebp = -1077937936, tf_isp = -845107244, tf_ebx = 1, tf_edx = 672628888, tf_ecx = 135524352, tf_eax = 10,
tf_trapno = 12, tf_err = 2, tf_eip = 672239144, tf_cs = 31, tf_eflags = 647, tf_esp = -1077940028, tf_ss = 47})
at /usr/src/sys/i386/i386/trap.c:1175
#23 0xc026cd75 in Xint0x80_syscall ()
#24 0x805d23c in ?? ()
#25 0x804b714 in ?? ()
#26 0x804c40d in ?? ()
#27 0x804a871 in ?? ()
(kgdb) up 14
#14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
467 panic("worklist_remove: lock not held");
(kgdb) l
462 worklist_remove(item)
463 struct worklist *item;
464 {
465
466 if (lk.lkt_held == -1)
467 panic("worklist_remove: lock not held");
468 if ((item->wk_state & ONWORKLIST) == 0) {
469 FREE_LOCK(&lk);
470 panic("worklist_remove: not on list");
471 }
(kgdb) p *item
Cannot access memory at address 0x0.
(kgdb) p lk
$1 = {lkt_spl = 0, lkt_held = -1}
(kgdb) up
#15 0xc023374e in softdep_update_inodeblock (ip=0xc1760c00, bp=0xc6564930, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
3847 WORKLIST_REMOVE(wk);
(kgdb) l
3842 * operations dependent on the inode being written to disk
3843 * can be moved to the id_bufwait so that they will be
3844 * processed when the buffer I/O completes.
3845 */
3846 while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
3847 WORKLIST_REMOVE(wk);
3848 WORKLIST_INSERT(&inodedep->id_bufwait, wk);
3849 }
3850 /*
3851 * Newly allocated inodes cannot be written until the bitmap
(kgdb) p wk
$2 = (struct worklist *) 0x68c040
(kgdb) p *wk
Cannot access memory at address 0x68c040.
# vmstat -m -M vmcore.3
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 1116 164 424115 1280 0
32 488 280 4533610 640 0
64 19590 6586 35473023 320 18958
128 1244 4100 6899026 160 348464
256 17591 905 5822574 80 2631
512 548 124 15562 40 277
1K 101 7 4403849 20 13
2K 61 99 2956 10 1304
4K 30 3 624293 5 0
8K 3 1 10 5 0
16K 5 0 85 5 0
32K 8 0 8 5 0
64K 10 0 10 5 0
128K 3 0 4 5 0
Memory usage type by bucket size
Size Type(s)
16 uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl,
ether_multi, BPF, vnodes, mount, pcb, soname, accf, shm, rman, bus,
sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld
32 atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag,
indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl,
ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname,
accf, taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl,
uidinfo, subproc, pgrp, temp, devbuf, proc-args, sigio, kld
64 isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon,
NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes,
vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc,
session, temp, devbuf, lockf, proc-args, file
128 ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF,
vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred,
temp, devbuf, zombie, proc-args, dev_t, timecounter, kld
256 UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon,
NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes,
ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue,
file desc
512 UFS dirhash, UFS mount, NFS daemon, mount, BIO buffer, ptys, msg,
ioctlops, bus, uidinfo, ATA generic, temp, devbuf, prison, file desc
1K uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, ioctlops, bus,
temp, devbuf, kqueue
2K UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver, shm,
ioctlops, bus, proc, devbuf
4K memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus,
temp, devbuf
8K pagedep, syncache, shm, bus, temp
16K VM pgdata, indirdep, sem, msg, devbuf
32K BPF, sem, msg, temp, devbuf
64K ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf
128K vfscache, msg, SWAP
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
atkbddev 1 1K 1K 42378K 1 0 0 32
uc_devlist 14 2K 2K 42378K 14 0 0 16,1K
memdesc 1 4K 4K 42378K 1 0 0 4K
mbuf 1 4K 4K 42378K 1 0 0 4K
isadev 10 1K 1K 42378K 10 0 0 64
ZONE 16 2K 2K 42378K 16 0 0 128
VM pgdata 1 16K 16K 42378K 1 0 0 16K
ISOFS mount 1 64K 64K 42378K 1 0 0 64K
UFS dirhash 1088 216K 274K 42378K 4095 0 0 16,32,64,128,256,512,1K,2K,4K
UFS mount 21 46K 46K 42378K 21 0 0 512,2K,4K
UFS ihash 1 64K 64K 42378K 1 0 0 64K
FFS node 17283 4321K 4487K 42378K 1282170 0 0 256
dirrem 0 0K 3K 42378K 14638 0 0 32
mkdir 0 0K 1K 42378K 130 0 0 32
diradd 2 1K 3K 42378K 14962 0 0 32
freefile 0 0K 2K 42378K 8476 0 0 32
freeblks 1 1K 3K 42378K 6793 0 0 128
freefrag 0 0K 10K 42378K 3611 0 0 32
allocindir 0 0K 425K 42378K 32022 0 0 64
indirdep 0 0K 33K 42378K 410 0 0 32,16K
allocdirect 1 1K 4K 42378K 12040 0 0 64
bmsafemap 1 1K 1K 42378K 7652 0 0 32
newblk 1 1K 1K 42378K 44063 0 0 32,256
inodedep 4 65K 78K 42378K 13879 0 0 128,64K
pagedep 2 9K 10K 42378K 8084 0 0 64,8K
p1003.1b 1 1K 1K 42378K 1 0 0 16
NFS hash 1 64K 64K 42378K 1 0 0 64K
NQNFS Lease 1 1K 1K 42378K 1 0 0 1K
NFSV3 srvdesc 0 0K 1K 42378K 8 0 0 16,256
NFS daemon 11 4K 4K 42378K 11 0 0 64,256,512
NFS req 0 0K 1K 42378K 2212294 0 0 64
NFS srvsock 1 1K 1K 42378K 1 0 0 256
syncache 1 8K 8K 42378K 1 0 0 8K
tseg_qent 0 0K 1K 42378K 185 0 0 32
IpFw/IpAcct 53 8K 15K 42378K 2652 0 0 64,128,256,1K
in_multi 3 1K 1K 42378K 3 0 0 32
routetbl 140 20K 120K 42378K 2327 0 0 16,32,64,128,256
ether_multi 12 1K 1K 42378K 12 0 0 16,32,64
ifaddr 27 6K 6K 42378K 27 0 0 32,64,256,2K
BPF 15 137K 137K 42378K 15 0 0 16,32,128,4K,32K
Export Host 6 2K 2K 42378K 6 0 0 256
vnodes 26 6K 6K 42378K 337 0 0 16,32,64,128,256
mount 16 8K 8K 42378K 18 0 0 16,128,512
cluster_save buffer 0 0K 1K 42378K 1031 0 0 32
vfscache 18465 1287K 1696K 42378K 1339707 0 0 64,128,128K
BIO buffer 25 50K 248K 42378K 1868 0 0 512,2K
pcb 63 6K 6K 42378K 5919 0 0 16,32,64,2K
soname 34 3K 5K 42378K 426863 0 0 16,32,64,128
ACD driver 2 3K 3K 42378K 2 0 0 256,2K
accf 3 1K 1K 42378K 3 0 0 16,32
ptys 1 1K 1K 42378K 1 0 0 512
ttys 210 27K 27K 42378K 639 0 0 128,256
shm 14 33K 43K 42378K 1141 0 0 16,2K,8K
sem 3 88K 88K 42378K 3 0 0 16K,32K,64K
msg 4 137K 137K 42378K 4 0 0 512,16K,32K,128K
rman 58 4K 4K 42378K 401 0 0 16,64
iov 0 0K 1K 42378K 316 0 0 128
ioctlops 0 0K 4K 42378K 14 0 0 512,1K,2K,4K
taskqueue 1 1K 1K 42378K 1 0 0 32
SWAP 2 73K 145K 42378K 4 0 0 32,128K
ATAPI generic 1 1K 1K 42378K 2 0 0 32,128
kobj 1 4K 4K 42378K 1 0 0 4K
eventhandler 15 1K 1K 42378K 15 0 0 32,64
bus 375 34K 37K 42378K 696 0 0 16,32,64,128,256,512,1K,2K,4K,8K
sysctloid 10 1K 1K 42378K 10 0 0 16,64
sysctl 0 0K 1K 42378K 4790 0 0 16,32
uidinfo 10 1K 1K 42378K 708 0 0 32,512
cred 68 9K 12K 42378K 722640 0 0 128
subproc 210 14K 21K 42378K 8845542 0 0 32,64,256
proc 2 4K 4K 42378K 2 0 0 2K
session 46 3K 4K 42378K 4248 0 0 64
pgrp 46 2K 2K 42378K 4248 0 0 32
ATA generic 2 1K 1K 42378K 2 0 0 16,512
temp 299 138K 152K 42378K 9568290 0 0 16,32,64,128,256,512,1K,4K,8K,32K,64K
devbuf 506 412K 412K 42378K 1168 0 0 16,32,64,128,256,512,1K,2K,4K,16K,32K,64K
lockf 36 3K 4K 42378K 1546084 0 0 64
prison 3 2K 2K 42378K 3 0 0 512
atexit 1 1K 1K 42378K 1 0 0 16
zombie 0 0K 1K 42378K 4419948 0 0 128
proc-args 70 4K 6K 42378K 4491591 0 0 16,32,64,128,256
kqueue 4 4K 10K 42378K 5529 0 0 256,1K
sigio 1 1K 1K 42378K 121 0 0 32
file 646 41K 52K 42378K 18711429 0 0 64
file desc 120 31K 45K 42378K 4422440 0 0 256,512
dev_t 663 83K 83K 42378K 663 0 0 128
timecounter 10 2K 2K 42378K 10 0 0 128
kld 4 1K 1K 42378K 35 0 0 16,32,128
Memory Totals: In Use Free Requests
7564K 1449K 58199125
Here is the kernel:
machine i386
cpu I586_CPU
cpu I686_CPU
ident NEWMONSTER
maxusers 128
makeoptions DEBUG=-g
options CPU_ENABLE_SSE
options INET
options FFS
options FFS_ROOT
options NFS
options SOFTUPDATES
options PROCFS
options CD9660
options COMPAT_43
options UCONSOLE
options USERCONFIG
options VISUAL_USERCONFIG
options P1003_1B
options _KPOSIX_PRIORITY_SCHEDULING
options _KPOSIX_VERSION=199309L
options KTRACE
options PERFMON
options SYSVSHM
options SYSVMSG
options SYSVSEM
options SHMMAXPGS=8192 # max amount of shared memory pages (4k on i386)
options SHMALL=33554432 # max amount of shared memory (bytes)
options SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)"
options SHMMNI=256 # max shared mem id's per system
options SHMSEG=256 # max shared mem id's per process
options MSGMNB=8192 # max # of bytes in a queue
options MSGMNI=256 # number of message queue identifiers
options MSGSEG=256 # number of message segments per queue
options MSGSSZ=64 # size of a message segment
options MSGTQL=8192 # max messages in system
options SEMMAP=256
options SEMMNI=256
options SEMMNS=512
options SEMMNU=256
options INCLUDE_CONFIG_FILE
options IPFILTER
options IPFILTER_LOG
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFW2
options RANDOM_IP_ID
options ICMP_BANDLIM
options ACCEPT_FILTER_HTTP
options VESA
options PANIC_REBOOT_WAIT_TIME=20
options SMBFS
options LIBMCHAIN
options LIBICONV
options NETSMB
options NETSMBCRYPTO
options UFS_DIRHASH
options SHOW_BUSYBUFS
options HZ=1000
device isa
device pci
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device fd1 at fdc0 drive 1
device ata0 at isa? port IO_WD1 irq 14
device ata1 at isa? port IO_WD2 irq 15
device ata
device atadisk
device atapicd
options ATA_STATIC_ID
device ahc
device aic0 at isa?
device scbus
device da
device sa
device cd
device pass
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1
device vga0 at isa?
device sc0 at isa? flags 0x100
device npx0 at nexus? port IO_NPX irq 13
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
# device sio2 at isa? disable port IO_COM3 irq 5
# device sio3 at isa? disable port IO_COM4 irq 9
device miibus
device dc
device fxp
pseudo-device loop
pseudo-device ether
pseudo-device tun
pseudo-device pty
pseudo-device bpf 8
pseudo-device vn 2
pseudo-device gzip
pseudo-device splash
device smbus
device intpm
device alpm
device ichsmb
device viapm
device smb
device iicbus
device iicbb
device ic
device iic
device iicsmb
device apm0 at nexus?
This machine runs squid && pgsql - I cleared softupdates from squid and pgsql filesystems but the panic chase me.
--
VAMPIRO-RIPN
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200209090230.g892U4Cl067557>