From owner-freebsd-bugs  Sat Aug 31 21:20:34 2002
Delivered-To: freebsd-bugs@hub.freebsd.org
Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id A0A2237B400
	for <freebsd-bugs@hub.freebsd.org>; Sat, 31 Aug 2002 21:20:02 -0700 (PDT)
Received: from freefall.freebsd.org (freefall.FreeBSD.org [216.136.204.21])
	by mx1.FreeBSD.org (Postfix) with ESMTP id BEE1943E3B
	for <freebsd-bugs@hub.freebsd.org>; Sat, 31 Aug 2002 21:20:01 -0700 (PDT)
	(envelope-from gnats@FreeBSD.org)
Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1])
	by freefall.freebsd.org (8.12.4/8.12.4) with ESMTP id g814K1JU092529
	for <freebsd-bugs@freefall.freebsd.org>; Sat, 31 Aug 2002 21:20:01 -0700 (PDT)
	(envelope-from gnats@freefall.freebsd.org)
Received: (from gnats@localhost)
	by freefall.freebsd.org (8.12.4/8.12.4/Submit) id g814K11Q092528;
	Sat, 31 Aug 2002 21:20:01 -0700 (PDT)
Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id B80FF37B400
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 31 Aug 2002 21:17:30 -0700 (PDT)
Received: from vampiro.rootshell.ru (vampiro.rootshell.ru [195.162.58.222])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 9752743E3B
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 31 Aug 2002 21:17:28 -0700 (PDT)
	(envelope-from vampiro@rootshell.ru)
Received: by vampiro.rootshell.ru (Sendmail for UK-NC (RT11-SJ), from userid 1111)
	id 7C3385430; Sun,  1 Sep 2002 11:17:22 +0700 (OMSST)
Message-Id: <20020901041722.7C3385430@vampiro.rootshell.ru>
Date: Sun,  1 Sep 2002 11:17:22 +0700 (OMSST)
From: El Vampiro <vampiro@rootshell.ru>
Reply-To: El Vampiro <vampiro@rootshell.ru>
To: FreeBSD-gnats-submit@FreeBSD.org
X-Send-Pr-Version: 3.113
Subject: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
Sender: owner-freebsd-bugs@FreeBSD.ORG
Precedence: bulk
List-ID: <freebsd-bugs.FreeBSD.ORG>
List-Archive: <http://docs.freebsd.org/mail/> (Web Archive)
List-Help: <mailto:majordomo@FreeBSD.ORG?subject=help> (List Instructions)
List-Subscribe: <mailto:majordomo@FreeBSD.ORG?subject=subscribe%20freebsd-bugs>
List-Unsubscribe: <mailto:majordomo@FreeBSD.ORG?subject=unsubscribe%20freebsd-bugs>
X-Loop: FreeBSD.org


>Number:         42277
>Category:       kern
>Synopsis:       Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Aug 31 21:20:01 PDT 2002
>Closed-Date:
>Last-Modified:
>Originator:     El Vampiro
>Release:        FreeBSD 4.6-STABLE i386
>Organization:
>Environment:
System: FreeBSD rshb.com.ru 4.6-STABLE FreeBSD 4.6-STABLE #0: Sat Aug 31 13:17:44 OMSST 2002 vampiro@vampiro.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER  i386

Hardware: new Intel brand server

Kernel:
	machine		i386
	cpu		I586_CPU
	cpu		I686_CPU
	ident		NEWMONSTER
	maxusers	128
	makeoptions	DEBUG=-g
	options		CPU_ENABLE_SSE
	options 	INET
	options 	FFS
	options 	FFS_ROOT
	options		NFS
	options 	SOFTUPDATES
	options 	PROCFS
	options		CD9660
	options 	COMPAT_43
	options 	UCONSOLE
	options 	USERCONFIG
	options 	VISUAL_USERCONFIG
	options 	P1003_1B
	options 	_KPOSIX_PRIORITY_SCHEDULING
	options		_KPOSIX_VERSION=199309L
	options		KTRACE
	options		PERFMON
	options 	SYSVSHM
	options 	SYSVMSG
	options 	SYSVSEM
	options		SHMMAXPGS=8192  # max amount of shared memory pages (4k on i386)
	options		SHMMNI=512      # max shared mem id's per system
	options		SHMSEG=256      # max shared mem id's per process
	options		MSGMNB=8192     # max # of bytes in a queue
	options		MSGMNI=256      # number of message queue identifiers
	options		MSGSEG=256      # number of message segments per queue
	options		MSGSSZ=64       # size of a message segment
	options		MSGTQL=8192     # max messages in system
	options		SEMMAP=256
	options		SEMMNI=256
	options		SEMMNS=512
	options		SEMMNU=256
	options		INCLUDE_CONFIG_FILE
	options		IPFILTER
	options		IPFILTER_LOG
	options         IPFIREWALL
	options         IPFIREWALL_VERBOSE
	options         IPFIREWALL_DEFAULT_TO_ACCEPT
	options		IPFW2
	options		RANDOM_IP_ID
	options		ICMP_BANDLIM
	options		ACCEPT_FILTER_HTTP
	options         VESA
	options         PANIC_REBOOT_WAIT_TIME=20
	options         SMBFS
	options         LIBMCHAIN
	options         LIBICONV
	options         NETSMB
	options         NETSMBCRYPTO
	options         UFS_DIRHASH
	options         SHOW_BUSYBUFS
	options		HZ=1000
	device		isa
	device		pci
	device		fdc0	at isa? port IO_FD1 irq 6 drq 2
	device		fd0	at fdc0 drive 0
	device		fd1	at fdc0 drive 1
	device		ata0	at isa? port IO_WD1 irq 14
	device		ata1	at isa? port IO_WD2 irq 15
	device		ata
	device		atadisk
	device          atapicd
	options 	ATA_STATIC_ID
	device          ahc
	device          aic0    at isa?
	device          scbus
	device          da
	device          sa
	device          cd
	device          pass
	device		atkbdc0	at isa? port IO_KBD
	device		atkbd0	at atkbdc? irq 1
	device		vga0	at isa?
	device		sc0	at isa? flags 0x100
	device		npx0	at nexus? port IO_NPX irq 13
	device		sio0	at isa? port IO_COM1 flags 0x10 irq 4
	device		sio1	at isa? port IO_COM2 irq 3
	device 		miibus
	device 		dc
	device 		fxp
	pseudo-device	loop
	pseudo-device	ether
	pseudo-device	tun
	pseudo-device	pty
	pseudo-device	bpf	8
	pseudo-device	vn	2
	pseudo-device   gzip
	pseudo-device   splash
	device          smbus
	device          intpm
	device          alpm
	device          ichsmb
	device          viapm
	device          smb
	device          iicbus
	device          iicbb
	device          ic
	device          iic
	device          iicsmb
	device		apm0 at nexus?

Copyright (c) 1992-2002 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 4.6-STABLE #0: Sat Aug 31 13:17:44 OMSST 2002
    vampiro@vampiro.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER
Timecounter "i8254"  frequency 1193182 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (999.72-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x68a  Stepping = 10
  Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory  = 268369920 (262080K bytes)
avail memory = 257720320 (251680K bytes)
Preloaded elf kernel "kernel" at 0xc0360000.
VESA: v2.0, 4096k memory, flags:0x0, mode table:0xc02fe6c2 (1000022)
VESA: ATI MACH64
netsmb_dev: loaded
Pentium Pro MTRR support enabled
Using $PIR table, 268435454 entries at 0xc00fdf10
apm: protected mode connections are not supported
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GV graphics accelerator> at 2.0 irq 11
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0x5800-0x583f mem 0xfb000000-0xfb0fffff,0xfb101000-0xfb101fff irq 10 at device 3.0 on pci0
fxp0: Ethernet address 00:d0:b7:b8:ab:64
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
dc0: <Davicom DM9102A 10/100BaseTX> port 0x5400-0x54ff mem 0xfb102000-0xfb1020ff irq 7 at device 7.0 on pci0
dc0: Ethernet address: 00:80:ad:08:12:2b
miibus1: <MII bus> on dc0
ukphy0: <Generic IEEE 802.3u media interface> on miibus1
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks ROSB4 ATA33 controller> port 0x5840-0x584f,0x374-0x377,0x170-0x177 at device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pcib1: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6000-0x60ff mem 0xfd000000-0xfd000fff irq 9 at device 4.0 on pci1
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6400-0x64ff mem 0xfd001000-0xfd001fff irq 5 at device 4.1 on pci1
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcf7ff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ipfw2 initialized, divert disabled, rule-based forwarding enabled, default to accept, logging unlimited
IP Filter: v3.4.27 initialized.  Default = pass all, Logging = enabled
acd0: CDROM <SONY CD-ROM CDU5221> at ata0-master PIO4
Waiting 2 seconds for SCSI devices to settle
sa0 at ahc1 bus 0 target 0 lun 0
sa0: <ARCHIVE Python 04106-XXX 7550> Removable Sequential Access SCSI-2 device 
sa0: 10.000MB/s transfers (10.000MHz, offset 15)
da0 at ahc0 bus 0 target 0 lun 0
da0: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device 
da0: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
da0: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
da1 at ahc0 bus 0 target 1 lun 0
da1: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device 
da1: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
da1: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
Mounting root from ufs:/dev/da0s1a

>Description:
	Several kernel panics per day with panicstr: softdep_lock: locki ng against myself
	Upon reboot fsck discovers many filesystem errors. File damage occurs often.
	$ gdb -k kernel.debug vmcore.0 
	GNU gdb 4.18 (FreeBSD)
	Copyright 1998 Free Software Foundation, Inc.
	GDB is free software, covered by the GNU General Public License, and you are
	welcome to change it and/or distribute copies of it under certain conditions.
	Type "show copying" to see the conditions.
	There is absolutely no warranty for GDB.  Type "show warranty" for details.
	This GDB was configured as "i386-unknown-freebsd"...

	/big1/vampiro/crashes/4-NEW/vmcore.0: Permission denied.
	(kgdb) quit
	$ gdb -k kernel.debug vmcore.0 
	GNU gdb 4.18 (FreeBSD)
	Copyright 1998 Free Software Foundation, Inc.
	GDB is free software, covered by the GNU General Public License, and you are
	welcome to change it and/or distribute copies of it under certain conditions.
	Type "show copying" to see the conditions.
	There is absolutely no warranty for GDB.  Type "show warranty" for details.
	This GDB was configured as "i386-unknown-freebsd"...
	IdlePTD at phsyical address 0x0037f000
	initial pcb at physical address 0x002ea800
	panicstr: softdep_lock: locking against myself
	panic messages:
	---
	Fatal trap 12: page fault while in kernel mode
	fault virtual address   = 0xffff000a
	fault code              = supervisor read, page not present
	instruction pointer     = 0x8:0xc022988c
	stack pointer           = 0x10:0xcd0f9d10
	frame pointer           = 0x10:0xcd0f9d10
	code segment            = base 0x0, limit 0xfffff, type 0x1b
				= DPL 0, pres 1, def32 1, gran 1
	processor eflags        = interrupt enabled, resume, IOPL = 0
	current process         = 6 (syncer)
	interrupt mask          = bio 
	trap number             = 12
	panic: page fault

	syncing disks... panic: softdep_lock: locking against myself
	Uptime: 3h29m7s

	dumping to dev #da/0x20009, offset 128
	dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
	---
	#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
	487             if (dumping++) {
	(kgdb) where
	#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
	#1  0xc01888d3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
	#2  0xc0188cf8 in poweroff_wait (junk=0xc02ae0c0, howto=-1053066752)
	    at /usr/src/sys/kern/kern_shutdown.c:595
	#3  0xc02295be in acquire_lock (lk=0xc02d8c7c)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
	#4  0xc022d6d8 in softdep_update_inodeblock (ip=0xc13b7a00, bp=0xc64ec464, 
	    waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
	#5  0xc022870d in ffs_update (vp=0xcdbcefc0, waitfor=0)
	    at /usr/src/sys/ufs/ffs/ffs_inode.c:106
	#6  0xc0232105 in ffs_fsync (ap=0xcd0f9bb0)
	    at /usr/src/sys/ufs/ffs/ffs_vnops.c:273
	#7  0xc02309e3 in ffs_sync (mp=0xc12a1a00, waitfor=2, cred=0xc0a3c880, 
	    p=0xc02ff300) at vnode_if.h:558
	#8  0xc01b8f97 in sync (p=0xc02ff300, uap=0x0)
	    at /usr/src/sys/kern/vfs_syscalls.c:576
	#9  0xc0188646 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
	#10 0xc0188cf8 in poweroff_wait (junk=0xc02b868c, howto=-1070890577)
	    at /usr/src/sys/kern/kern_shutdown.c:595
	#11 0xc0274866 in trap_fatal (frame=0xcd0f9cd0, eva=4294901770)
	    at /usr/src/sys/i386/i386/trap.c:974
	#12 0xc0274539 in trap_pfault (frame=0xcd0f9cd0, usermode=0, eva=4294901770)
	    at /usr/src/sys/i386/i386/trap.c:867
	#13 0xc02740f7 in trap (frame={tf_fs = -854654960, tf_es = -1053884400, 
	      tf_ds = 16, tf_edi = 0, tf_esi = -1053851648, tf_ebp = -854614768, 
	      tf_isp = -854614788, tf_ebx = -65536, tf_edx = 0, tf_ecx = -65536, 
	      tf_eax = -1051831424, tf_trapno = 12, tf_err = 0, tf_eip = -1071474548, 
	      tf_cs = 8, tf_eflags = 66067, tf_esp = -854614736, tf_ss = -1071458418})
	    at /usr/src/sys/i386/i386/trap.c:466
	#14 0xc022988c in worklist_remove (item=0xffff0000)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
	#15 0xc022d78e in softdep_update_inodeblock (ip=0xc12f8000, bp=0xc651ab80, 
	    waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
	#16 0xc022870d in ffs_update (vp=0xcdd565c0, waitfor=0)
	    at /usr/src/sys/ufs/ffs/ffs_inode.c:106
	#17 0xc02289f1 in ffs_truncate (vp=0xcdd565c0, length=0, flags=0, cred=0x0, 
	    p=0xcc00a5e0) at /usr/src/sys/ufs/ffs/ffs_inode.c:201
	#18 0xc0232e08 in ufs_inactive (ap=0xcd0f9ed8)
	    at /usr/src/sys/ufs/ufs/ufs_inode.c:89
	#19 0xc0238301 in ufs_vnoperate (ap=0xcd0f9ed8)
	    at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
	#20 0xc01b70e8 in vput (vp=0xcdd565c0) at vnode_if.h:815
	#21 0xc022c594 in handle_workitem_remove (dirrem=0xc14034e0)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
	#22 0xc0229c0d in process_worklist_item (matchmnt=0x0, flags=0)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
	#23 0xc0229ab2 in softdep_process_worklist (matchmnt=0x0)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
	#24 0xc01b6a0f in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177
	(kgdb) up 14   
	#14 0xc022988c in worklist_remove (item=0xffff0000)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
	467                     panic("worklist_remove: lock not held");
	(kgdb) l
	462     worklist_remove(item)
	463             struct worklist *item;
	464     {
	465
	466             if (lk.lkt_held == -1)
	467                     panic("worklist_remove: lock not held");
	468             if ((item->wk_state & ONWORKLIST) == 0) {
	469                     FREE_LOCK(&lk);
	470                     panic("worklist_remove: not on list");
	471             }
	(kgdb) p item
	$1 = (struct worklist *) 0x0
	(kgdb) p lk
	$2 = {lkt_spl = 0, lkt_held = -1}
	(kgdb) up
	#15 0xc022d78e in softdep_update_inodeblock (ip=0xc12f8000, bp=0xc651ab80, 
	    waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
	3847                    WORKLIST_REMOVE(wk);
	(kgdb) l
	3842             * operations dependent on the inode being written to disk
	3843             * can be moved to the id_bufwait so that they will be
	3844             * processed when the buffer I/O completes.
	3845             */
	3846            while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
	3847                    WORKLIST_REMOVE(wk);
	3848                    WORKLIST_INSERT(&inodedep->id_bufwait, wk);
	3849            }
	3850            /*
	3851             * Newly allocated inodes cannot be written until the bitmap
	(kgdb) p wk
	$3 = (struct worklist *) 0x68c040
	(kgdb) p inodedep
	$4 = (struct inodedep *) 0xc14e5380
	(kgdb) p inodedep->id_inowait
	$5 = {lh_first = 0xffff0000}
	(kgdb) p inodedep->id_bufwait
	$6 = {lh_first = 0x0}

	I can provide more info upon request.
	
>How-To-Repeat:
	I found no way to repeat this. System can panic even upon minimal activity and can work upon hard load.	
>Fix:
	The only way I found is turn softupdates off

>Release-Note:
>Audit-Trail:
>Unformatted:

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message