Date: Sun, 18 Jan 2004 22:20:02 -0700 From: Scott Long <scottl@freebsd.org> To: "Robin P. Blanchard" <robin.blanchard@gactr.uga.edu> Cc: current@freebsd.org Subject: Re: Panic with this morning's (~9am EDT, 15 jan 2004) sources. Message-ID: <400B6902.7040203@freebsd.org> In-Reply-To: <EE3D3FBAFFCAED448C21C398FDAD91AC0108DB@EBE1.gc.nat> References: <EE3D3FBAFFCAED448C21C398FDAD91AC0108DB@EBE1.gc.nat>
next in thread | previous in thread | raw e-mail | index | archive | help
Do you have DIAGNOSTIC enabled? It's generally safe to run without it;
this particular bug is well known and will hopefully be fixed.
Scott
Robin P. Blanchard wrote:
> still panicing at boot with this morning's (~10am EDT, 18 jan 04) sources;
> but have found that a "cont" within db allows the machine to succesfully
> recover and come up 'normally'.
>
> Mounting root from ufs:/dev/twed0s1a
> start_init: trying /sbin/init
> lock order reversal
> 1st 0xc06c20a0 dont_sleep_in_callout (dont_sleep_in_callout) @
> /usr/src/sys/ker
> n/kern_timeout.c:224
> 2nd 0xc06c1480 allproc (allproc) @ /usr/src/sys/kern/kern_synch.c:588
> Stack backtrace:
> backtrace(c0673b76,c06c1480,c066fc3d,c066fc3d,c0670c87) at backtrace+0x17
> witness_lock(c06c1480,0,c0670c87,24c,c1560070) at witness_lock+0x672
> _sx_slock(c06c1480,c0670c87,24c,e0,0) at _sx_slock+0xae
> loadav(0,0,c06715a8,e0,c06737ca) at loadav+0x25
> softclock(0,0,c066e0c1,23e,c1568528) at softclock+0x1fb
> ithread_loop(c3fb6300,d7254d48,c066df27,315,0) at ithread_loop+0x192
> fork_exit(c04f9950,c3fb6300,d7254d48) at fork_exit+0xb5
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0x1, eip = 0, esp = 0xd7254d7c, ebp = 0 ---
> Debugger("witness_lock")
> Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0
> db> show locks
> exclusive sleep mutex dont_sleep_in_callout r = 0 (0xc06c20a0) locked @
> /usr/src
> /sys/kern/kern_timeout.c:224
> exclusive sleep mutex Giant r = 0 (0xc06c01a0) locked @
> /usr/src/sys/kern/kern_t
> imeout.c:217
> db> tr
> Debugger(c064a64d,c06c1480,c066fc3d,c066fc3d,c0670c87) at Debugger+0x55
> witness_lock(c06c1480,0,c0670c87,24c,c1560070) at witness_lock+0x687
> _sx_slock(c06c1480,c0670c87,24c,e0,0) at _sx_slock+0xae
> loadav(0,0,c06715a8,e0,c06737ca) at loadav+0x25
> softclock(0,0,c066e0c1,23e,c1568528) at softclock+0x1fb
> ithread_loop(c3fb6300,d7254d48,c066df27,315,0) at ithread_loop+0x192
> fork_exit(c04f9950,c3fb6300,d7254d48) at fork_exit+0xb5
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0x1, eip = 0, esp = 0xd7254d7c, ebp = 0 ---
> db>
>
> -----Original Message-----
> From: Robin P. Blanchard
> Sent: Sat 1/17/2004 5:19 PM
> To: Robin P. Blanchard; Dag-Erling Smørgrav; Scott Long
> Cc: current@freebsd.org
> Subject: RE: Panic with this morning's (~9am EDT, 15 jan 2004)
> sources.
>
>
> Still no luck....cvsup'ed earlier this morning...still fails to boot
> (box in question is dell poweredge 350 with add-on twe raid controller).
>
> # ident /boot/kernel/kernel |fgrep des
> $FreeBSD: src/sys/kern/kern_descrip.c,v 1.223 2004/01/17
> 00:58:36 des Exp $
> $FreeBSD: src/sys/kern/sys_pipe.c,v 1.163 2004/01/15 10:15:03
> des Exp $
> $FreeBSD: src/sys/kern/uipc_syscalls.c,v 1.169 2004/01/15
> 10:15:03 des Exp $
> $FreeBSD: src/sys/kern/uipc_usrreq.c,v 1.115 2004/01/17 00:59:04
> des Exp $
> $FreeBSD: src/sys/kern/vfs_syscalls.c,v 1.336 2004/01/15
> 10:15:04 des Exp $
> $FreeBSD: src/sys/libkern/ffsl.c,v 1.3 2004/01/14 07:47:10 des
> Exp $
> $FreeBSD: src/sys/libkern/flsl.c,v 1.3 2004/01/14 07:47:10 des
> Exp $
>
> ...
> Mounting root from ufs:/dev/twed0s1a
> start_init: trying /sbin/init
> lock order reversal
> 1st 0xc06c22e0 dont_sleep_in_callout (dont_sleep_in_callout) @
> /usr/src/sys/ker
> n/kern_timeout.c:224
> 2nd 0xc06c16c0 allproc (allproc) @
> /usr/src/sys/kern/kern_synch.c:588
> Stack backtrace:
> backtrace(c0673ded,c06c16c0,c066feb4,c066feb4,c0670efe) at
> backtrace+0x17
> witness_lock(c06c16c0,0,c0670efe,24c,c155e070) at witness_lock+0x672
> _sx_slock(c06c16c0,c0670efe,24c,e0,0) at _sx_slock+0xae
> loadav(0,0,c067181f,e0,c0673a41) at loadav+0x25
> softclock(0,0,c066e338,23e,c1566528) at softclock+0x1fb
> ithread_loop(c3fb5300,d7250d48,c066e19e,315,0) at ithread_loop+0x192
> fork_exit(c04f9c90,c3fb5300,d7250d48) at fork_exit+0xb5
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0x1, eip = 0, esp = 0xd7250d7c, ebp = 0 ---
> Debugger("witness_lock")
> Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0
> db> show locks
> exclusive sleep mutex dont_sleep_in_callout r = 0 (0xc06c22e0) locked
> @ /usr/src
> /sys/kern/kern_timeout.c:224
> exclusive sleep mutex Giant r = 0 (0xc06c03e0) locked @
> /usr/src/sys/kern/kern_t
> imeout.c:217
> db> tr
> Debugger(c064a8cd,c06c16c0,c066feb4,c066feb4,c0670efe) at
> Debugger+0x55
> witness_lock(c06c16c0,0,c0670efe,24c,c155e070) at witness_lock+0x687
> _sx_slock(c06c16c0,c0670efe,24c,e0,0) at _sx_slock+0xae
> loadav(0,0,c067181f,e0,c0673a41) at loadav+0x25
> softclock(0,0,c066e338,23e,c1566528) at softclock+0x1fb
> ithread_loop(c3fb5300,d7250d48,c066e19e,315,0) at ithread_loop+0x192
> fork_exit(c04f9c90,c3fb5300,d7250d48) at fork_exit+0xb5
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0x1, eip = 0, esp = 0xd7250d7c, ebp = 0 ---
> db> panic
> panic: from debugger
> cpuid = 0;
> Stack backtrace:
> backtrace(c065164d,0,c0664256,d7250a14,100) at backtrace+0x17
> panic(c0664256,d7250acc,c048ab32,c0614be5,0) at panic+0x141
> db_panic(c0614be5,0,ffffffff,d7250a40,d7250a3c) at db_panic+0x12
> db_command(c06b8220,c0691a00,c068b59c,c068b5a0,3) at db_command+0x272
> db_command_loop(c0614be5,d7250d7c,c078798e,8,0) at
> db_command_loop+0x95
> db_trap(3,0,0,46,0) at db_trap+0xc5
> kdb_trap(3,0,d7250b98,4,0) at kdb_trap+0x14c
> trap(d7250018,c0610010,10,c06efe58,c06c16c0) at trap+0x4ca
> calltrap() at calltrap+0x5
> --- trap 0x3, eip = 0xc0614be5, esp = 0xd7250bd8, ebp = 0xd7250be4
> ---
> Debugger(c064a8cd,c06c16c0,c066feb4,c066feb4,c0670efe) at
> Debugger+0x55
> witness_lock(c06c16c0,0,c0670efe,24c,c155e070) at witness_lock+0x687
> _sx_slock(c06c16c0,c0670efe,24c,e0,0) at _sx_slock+0xae
> loadav(0,0,c067181f,e0,c0673a41) at loadav+0x25
> softclock(0,0,c066e338,23e,c1566528) at softclock+0x1fb
> ithread_loop(c3fb5300,d7250d48,c066e19e,315,0) at ithread_loop+0x192
> fork_exit(c04f9c90,c3fb5300,d7250d48) at fork_exit+0xb5
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0x1, eip = 0, esp = 0xd7250d7c, ebp = 0 ---
> evxfevnt-0222 [09] AcpiDisable : ACPI mode disabled
> syncing disks, buffers remaining... 44 44 44 44 44 44 44 44 44 44 44
> 44 44 44 44
> 44 44 44 44 44
>
>
>
>
> -----Original Message-----
> From: owner-freebsd-current@freebsd.org on behalf of Robin P.
> Blanchard
> Sent: Fri 1/16/2004 4:03 PM
> To: Dag-Erling Smørgrav; Scott Long
> Cc: current@freebsd.org
> Subject: RE: Panic with this morning's (~9am EDT, 15 jan
> 2004) sources.
>
>
>
> Ok...
>
> I got the machine back up and running (kernel and kernel.old
> were both
> botched): grabbed the kernel from
>
> ftp://snapshots.jp.freebsd.org/pub/FreeBSD/snapshots/i386/ISO-IMAGES/live-5.2
> -CURRENT-20040115-JPSNAP.iso.
>
> Below is a comparison between the two. Please let me know
> what else I can do
> to help figure this one out.
>
> # ident /boot/rescue/kernel > ident.rescue
> # ident /boot/kernel/kernel > ident.panic
> # diff -u ident.rescue ident.panic |fgrep "+" |fgrep
> "FreeBSD"
>
> + $FreeBSD: src/sys/cam/scsi/scsi_pt.c,v 1.39 2003/06/10
> 18:14:05 obrien
> Exp $
> + $FreeBSD: src/sys/cam/scsi/scsi_targ_bh.c,v 1.20
> 2003/09/30 08:03:52
> simokawa Exp $
> + $FreeBSD: src/sys/cam/scsi/scsi_target.c,v 1.58
> 2003/11/09 09:17:20
> tanimura Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_cpu.c,v 1.29
> 2003/12/28 22:15:24 njl
> Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_ec.c,v 1.42
> 2004/01/03 02:01:39 njl
> Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_isab.c,v 1.4
> 2003/08/24 17:48:01
> obrien Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_lid.c,v 1.15
> 2003/10/25 05:03:24 njl
> Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_pci.c,v 1.6
> 2003/09/17 08:32:44
> iwasaki Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_pci_link.c,v 1.12
> 2003/12/23 02:29:46
> peter Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_pcib.c,v 1.36
> 2003/12/18 17:04:11 kan
> Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_pcib_pci.c,v 1.5
> 2003/08/24 17:48:01
> obrien Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_powerres.c,v 1.16
> 2003/08/28 16:06:30
> njl Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_resource.c,v 1.20
> 2003/09/26 05:24:55
> njl Exp $
> + $FreeBSD: src/sys/dev/acpica/acpi_thermal.c,v 1.35
> 2004/01/12 17:59:10
> njl Exp $
> + $FreeBSD: src/sys/dev/ata/ata-all.c,v 1.201 2004/01/14
> 21:26:34 sos Exp
> $
> + $FreeBSD: src/sys/dev/ata/ata-queue.c,v 1.16 2004/01/14
> 21:26:35 sos
> Exp $
> + $FreeBSD: src/sys/dev/ata/ata-lowlevel.c,v 1.27
> 2004/01/14 21:26:35 sos
> Exp $
> + $FreeBSD: src/sys/dev/ata/ata-isa.c,v 1.20 2004/01/14
> 21:26:35 sos Exp
> $
> + $FreeBSD: src/sys/dev/ata/ata-pci.c,v 1.74 2004/01/14
> 21:26:35 sos Exp
> $
> + $FreeBSD: src/sys/dev/ata/ata-chipset.c,v 1.56
> 2004/01/14 21:26:34 sos
> Exp $
> + $FreeBSD: src/sys/dev/ata/ata-dma.c,v 1.125 2004/01/14
> 21:26:35 sos Exp
> $
> + $FreeBSD: src/sys/dev/ata/ata-disk.c,v 1.167 2004/01/14
> 21:26:34 sos
> Exp $
> + $FreeBSD: src/sys/dev/ata/atapi-cd.c,v 1.160 2004/01/14
> 21:26:35 sos
> Exp $
> + $FreeBSD: src/sys/dev/ata/atapi-fd.c,v 1.92 2004/01/14
> 21:26:35 sos Exp
> $
> + $FreeBSD: src/sys/dev/ata/atapi-tape.c,v 1.87
> 2004/01/14 21:26:35 sos
> Exp $
> + $FreeBSD: src/sys/dev/ata/atapi-cam.c,v 1.31 2004/01/14
> 21:26:35 sos
> Exp $
> + $FreeBSD: src/sys/dev/ppbus/immio.c,v 1.17 2003/08/24
> 17:54:16 obrien
> Exp $
> + $FreeBSD: src/sys/dev/ppbus/vpo.c,v 1.29 2003/08/24
> 17:54:16 obrien Exp
> $
> + $FreeBSD: src/sys/dev/ppbus/vpoio.c,v 1.18 2003/08/24
> 17:54:16 obrien
> Exp $
> + $FreeBSD: src/sys/kern/init_main.c,v 1.239 2004/01/15
> 10:15:03 des Exp
> $
> + $FreeBSD: src/sys/kern/kern_descrip.c,v 1.221
> 2004/01/16 08:45:56 des
> Exp $
> + $FreeBSD: src/sys/kern/sched_ule.c,v 1.94 2003/12/20
> 22:35:20 jeff Exp
> $
> + $FreeBSD: src/sys/kern/sys_pipe.c,v 1.163 2004/01/15
> 10:15:03 des Exp $
> + $FreeBSD: src/sys/kern/uipc_cow.c,v 1.16 2003/11/16
> 06:11:25 alc Exp $
> + $FreeBSD: src/sys/kern/uipc_syscalls.c,v 1.169
> 2004/01/15 10:15:03 des
> Exp $
> + $FreeBSD: src/sys/kern/uipc_usrreq.c,v 1.114 2004/01/15
> 10:15:03 des
> Exp $
> + $FreeBSD: src/sys/kern/vfs_syscalls.c,v 1.336
> 2004/01/15 10:15:04 des
> Exp $
> + $FreeBSD: src/sys/kern/uipc_sem.c,v 1.9 2003/06/11
> 00:56:58 obrien Exp
> $
> + $FreeBSD: src/sys/i386/acpica/OsdEnvironment.c,v 1.9
> 2003/08/28
> 16:26:24 njl Exp $
> + $FreeBSD: src/sys/i386/acpica/acpi_machdep.c,v 1.15
> 2003/11/01 00:18:29
> njl Exp $
> + $FreeBSD: src/sys/i386/acpica/acpi_wakeup.c,v 1.30
> 2003/11/03 22:18:57
> jhb Exp $
> + $FreeBSD: src/sys/i386/acpica/madt.c,v 1.9 2003/12/09
> 03:04:19 njl Exp
> $
>
> >
> > Grabbed new code from cvsup1 this morning (~9am EDT, 16 jan
>
>>04) and am now panicing in a different location:
>
> >
> > FreeBSD/i386 bootstrap loader, Revision 1.1
> > (root@bsdfs5.gactr.uga.edu, Fri Jan 16 11:10:18 EST 2004)
> > Loading /boot/defaults/loader.conf /boot/kernel/kernel
> > text=0x277880 data=0x2a264+0x56838
> syms=[0x4+0x36e70+0x4+0x4
> > 3c03] /boot/modules/if_fxp.ko text=0x77b4 data=0x1014+0xc
> > syms=[0x4+0xcf0+0x4+0xdbb] loading required module 'miibus'
>
>>/boot/modules/miibus.ko text=0x15820 data=0xc84+0x68
>
> > syms=[0x4+0x1c60+0x4+0x2328 ] /boot/modules/twe.ko
> > text=0x7398 data=0x5ac+0x8c syms=[0x4+0xdb0+0x4+0xd1d]
> > Loading /boot/defaults/loader.conf
> > -
> > Hit [Enter] to boot immediately, or any other key for
> command
> > prompt. Booting [/boot/kernel/kernel]...
> > ACPI_DEBUG: set 'ACPI_EVENTS'
> > ACPI_DEBUG: set 'ACPI_LV_ALL_EXCEPTIONS'
> > ACPI debug layer 0x4 debug level 0x1f
> > Copyright (c) 1992-2004 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991,
> 1992,
> > 1993, 1994
> > The Regents of the University of California. All
> > rights reserved. FreeBSD 5.2-CURRENT #0: Fri Jan 16
> 11:58:59 EST 2004
> >
> root@bsdfs5.gactr.uga.edu:/usr/obj/usr/src/sys/ule.debug
> > Preloaded elf kernel "/boot/kernel/kernel" at 0xc07a7000.
> > Preloaded elf module "/boot/modules/if_fxp.ko" at
> 0xc07a726c.
> > Preloaded elf module "/boot/modules/miibus.ko" at
> 0xc07a7318.
> > Preloaded elf module "/boot/modules/twe.ko" at 0xc07a73c4.
> > Timecounter "i8254" frequency 1193182 Hz quality 0
> > CPU: Intel Pentium III (851.94-MHz 686-class CPU)
> > Origin = "GenuineIntel" Id = 0x68a Stepping = 10
> >
> >
> Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR
> > ,PGE,MCA,CMOV,P
> > A
> > T,PSE36,MMX,FXSR,SSE>
> > real memory = 536608768 (511 MB)
> > avail memory = 519680000 (495 MB)
> > acquiring duplicate lock of same type: "filedesc structure"
>
>> 1st filedesc structure @ /usr/src/sys/kern/kern_descrip.c:1377
>
> > 2nd filedesc structure @
> /usr/src/sys/kern/kern_descrip.c:1431
> > Stack backtrace:
> > backtrace(c0662f9a,c065e006,c065e5b2,597,1d8) at
> backtrace+0x17
> > witness_lock(c1568838,8,c065e5b2,597,c1568800) at
> witness_lock+0x5aa
> > _mtx_lock_flags(c1568838,0,c065e5b2,597,a) at
> _mtx_lock_flags+0x6a
> > fdcopy(c06a9120,0,c065e9a0,1a5,c0685924) at fdcopy+0x46
> > fork1(c06a8d60,60034,0,c0821d50,64) at fork1+0x61f
> > kthread_create(c04f9930,0,0,40000,0,c065edc1) at
> kthread_create+0x48
> > ktrace_init(0,81e000,81ec00,81e000,0) at ktrace_init+0xf5
> > mi_startup() at mi_startup+0xb5
> > begin() at begin+0x2c
> > Debugger("witness_lock")
> > Stopped at Debugger+0x55: xchgl %ebx,in_Debugger.0
> > db> show locks
> > exclusive sleep mutex filedesc structure r = 0 (0xc06a9158)
>
>>locked @ /usr/src/sy s/kern/kern_descrip.c:1377 exclusive
>
> > sleep mutex Giant r = 1 (0xc06ac580) locked @
> > /usr/src/sys/kern/kern_f ork.c:218
> > db> tr
> > Debugger(c063c331,c065e006,c065e5b2,597,1d8) at
> Debugger+0x55
> > witness_lock(c1568838,8,c065e5b2,597,c1568800) at
> witness_lock+0x5bf
> > _mtx_lock_flags(c1568838,0,c065e5b2,597,a) at
> _mtx_lock_flags+0x6a
> > fdcopy(c06a9120,0,c065e9a0,1a5,c0685924) at fdcopy+0x46
> > fork1(c06a8d60,60034,0,c0821d50,64) at fork1+0x61f
> > kthread_create(c04f9930,0,0,40000,0,c065edc1) at
> kthread_create+0x48
> > ktrace_init(0,81e000,81ec00,81e000,0) at ktrace_init+0xf5
> > mi_startup() at mi_startup+0xb5
> > begin() at begin+0x2c
> > db> panic
> > panic: from debugger
> > cpuid = 0;
> > Stack backtrace:
> > backtrace(c0642d67,0,c0655970,c0821a94,100) at
> backtrace+0x17
> > panic(c0655970,c0821b4c,c048a582,c0607825,0) at panic+0x141
>
>>db_panic(c0607825,0,ffffffff,c0821ac0,c0821abc) at db_panic+0x12
>
> > db_command(c06a43e0,c067dc20,c06777c4,c06777c8,3) at
> db_command+0x272
> > db_command_loop(c0607825,c0642d67,2,c0430859,0) at
> > db_command_loop+0x95
> > db_trap(3,0,0,80046,0) at db_trap+0xc5
> > kdb_trap(3,0,c0821c18,31,0) at kdb_trap+0x14c
> > trap(c0820018,c0600010,10,c068797c,c1568838) at trap+0x47a
> > calltrap() at calltrap+0x5
> > --- trap 0x3, eip = 0xc0607825, esp = 0xc0821c58, ebp =
> 0xc0821c64 ---
> > Debugger(c063c331,c065e006,c065e5b2,597,1d8) at
> Debugger+0x55
> > witness_lock(c1568838,8,c065e5b2,597,c1568800) at
> witness_lock+0x5bf
> > _mtx_lock_flags(c1568838,0,c065e5b2,597,a) at
> _mtx_lock_flags+0x6a
> > _mtx_lock_flags(c1568838,0,c065e5b2,597,a) at
> _mtx_lock_flags+0x6a
> > fork1(c06a8d60,60034,0,c0821d50,64) at fork1+0x61f
> > kthread_create(c04f9930,0,0,40000,0,c065edc1) at
> kthread_create+0x48
> > ktrace_init(0,81e000,81ec00,81e000,0) at ktrace_init+0xf5
> > mi_startup() at mi_startup+0xb5
> > begin() at begin+0x2c
> > Uptime: 1s
> > WARNING: Driver mistake: make_dev(geom.ctl) called before
> > SI_SUB_DRIVERS kernel trap 12 with interrupts disabled
> >
> >
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = 00
> > fault virtual address = 0x14
> > fault code = supervisor read, page not present
>
>>instruction pointer = 0x8:0xc04f53e4
>
> > stack pointer = 0x10:0xc0821928
> > frame pointer = 0x10:0xc082196c
> > code segment = base 0x0, limit 0xfffff, type
> 0x1b
> > = DPL 0, pres 1, def32 1, gran 1
> > processor eflags = resume, IOPL = 0
> > current process = 0 (swapper)
> > kernel: type 12 trap, code=0
> > Stopped at fork1+0x414: cmpl %ebx,0x14(%eax)
> > db>
>
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?400B6902.7040203>