Date: Thu, 8 Mar 2007 06:59:27 -0600 From: Tillman Hodgson <tillman@seekingfire.com> To: freebsd-current@FreeBSD.org Subject: Experiencing hangs on SMP box with no console messages given for clues. Details inside. Message-ID: <20070308125927.GA1265@seekingfire.com>
next in thread | raw e-mail | index | archive | help
Howdy folks, These has been happening every few days for a few weeks now. When it occurs, there's no messages logged to the console or to syslog -- it just silently hangs. I added the break-to-debugger option so that I can at least reboot it remotely via the serial console. I've been following the -current kernel fairly closely in hopes that it was just due to a transitory -current problem. I don't mind rebuilding a kernel with special options if it's useful -- I'll be rebuilding this morning with WITNESS and INVARIANTS for sure. I have the core saved, though I'm inexperienced with gdb. FreeBSD/i386 (athena.seekingfire.prv) (ttyd0) login: telnet> send brk KDB: enter: Line break on console [thread pid 11 tid 100005 ] Stopped at kdb_enter+0x2c: leave db> ? Bad character ? db> help print p examine x search set write w delete d break b dwatch watch dhwatch hwatch step s continue c until next match trace t alltrace where bt call show ps gdb halt reboot reset kill watchdog thread panic ahd_dump ahd_out ahd_in ahd_unpause ahd_pause ahd_sunit db> bt Tracing pid 11 tid 100005 td 0xc3afe6c0 kdb_enter(c0956f95,c0,c3afe6c0,c3af7cc8,c3afb880,...) at kdb_enter+0x2c siointr1(c3cb7b80,e25f0c84,c08cd60f,c3cb4000,c3afe6c0,...) at siointr1+0x3be siointr(c3cb4000,c3afe6c0,0,0,c3bfb400,...) at siointr+0x4c intr_execute_handlers(c3af7cc8,e25f0c94) at intr_execute_handlers+0xf3 Xapic_isr1() at Xapic_isr1+0x34 --- interrupt, eip = 0xc0baf599, esp = 0xe25f0cd4, ebp = 0xe25f0cd4 --- acpi_cpu_c1(e25f0cec,c06e382d,c0a5cb60,c3afe6c0,c06e3ccc,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,e25f0d24,c06b5db1,0,e25f0d38,...) at acpi_cpu_idle+0x15a sched_idletd(0,e25f0d38,0,c3afdb40,0,...) at sched_idletd+0x8a fork_exit(c06e3ccc,0,e25f0d38) at fork_exit+0x61 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe25f0d70, ebp = 0 --- db> show proc Process 11 (idle: cpu0) at 0xc3afdb40: state: NORMAL uid: 0 gids: 0 parent: pid 0 at 0xc0a58d80 ABI: null threads: 1 100005 Run CPU 0 [idle: cpu0] db> panic panic: from debugger cpuid = 0 Uptime: 2d22h24m3s Physical memory: 1015 MB Dumping 200 MB: 185 169 153 137 121 105 89 73 57 41 25 9 Dump complete Automatic reboot in 15 seconds - press a key on the console to abort [root@athena ~]# uname -a FreeBSD athena.seekingfire.prv 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Sun Mar 4 21:08:19 CST 2007 toor@athena.seekingfire.prv (/usr/src was synced the same day) [root@athena /usr/src/sys/i386/conf]# diff ATHENA GENERIC 24c24 < ident ATHENA --- > ident GENERIC 29c29 < ### makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols --- > makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols 67,73c67,70 < ###options INVARIANTS # Enable calls of extra sanity checking < ###options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS < ###options WITNESS # Enable checks to detect deadlocks and cycles < ###options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed < < ### Tillman added 26Feb07 as per http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html < options BREAK_TO_DEBUGGER --- > options INVARIANTS # Enable calls of extra sanity checking > options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS > options WITNESS # Enable checks to detect deadlocks and cycles > options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed [root@athena ~]# dmesg Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-CURRENT #0: Sun Mar 4 21:08:19 CST 2007 toor@athena.seekingfire.prv:/usr/obj/usr/src/sys/ATHENA ACPI APIC Table: <VIA694 AWRDACPI> Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel Pentium III (997.17-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x68a Stepping = 10 Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE> real memory = 1073676288 (1023 MB) avail memory = 1041326080 (993 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 <Version 1.1> irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: <VIA694 AWRDACPI> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 acpi_button0: <Power Button> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0x4000-0x407f,0x4080-0x40ff,0x5000-0x500f,0x6000-0x607f on acpi0 pci0: <ACPI PCI bus> on pcib0 agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> on hostb0 pcib1: <PCI-PCI bridge> at device 1.0 on pci0 pci1: <PCI bus> on pcib1 vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem 0xf4000000-0xf4ffffff,0xf6241000-0xf6241fff irq 19 at device 6.0 on pci0 isab0: <PCI-ISA bridge> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd400-0xd40f at device 7.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] uhci0: <VIA 83C572 USB controller> port 0xd800-0xd81f irq 12 at device 7.2 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: <VIA 83C572 USB controller> on uhci0 usb0: USB revision 1.0 uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 12 at device 7.3 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: <VIA 83C572 USB controller> on uhci1 usb1: USB revision 1.0 uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 uhub1: 2 ports with 2 removable, self powered pci0: <bridge> at device 7.4 (no driver attached) fxp0: <Intel 82559 Pro/100 Ethernet> port 0xe000-0xe03f mem 0xf6240000-0xf6240fff,0xf6000000-0xf60fffff irq 17 at device 13.0 on pci0 miibus0: <MII bus> on fxp0 inphy0: <i82555 10/100 media interface> PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:e0:81:21:ad:e0 fxp0: [ITHREAD] fxp1: <Intel 82559 Pro/100 Ethernet> port 0xe400-0xe43f mem 0xf6242000-0xf6242fff,0xf6100000-0xf61fffff irq 18 at device 14.0 on pci0 miibus1: <MII bus> on fxp1 inphy1: <i82555 10/100 media interface> PHY 1 on miibus1 inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: Ethernet address: 00:e0:81:21:ad:e1 fxp1: [ITHREAD] em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xe800-0xe83f mem 0xf6200000-0xf621ffff,0xf6220000-0xf623ffff irq 18 at device 16.0 on pci0 em0: Ethernet address: 00:0e:0c:c2:ce:4f em0: [FILTER] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] pmtimer0 on isa0 orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ad0: 38166MB <Seagate ST340016A 3.75> at ata0-master UDMA100 acd0: CDROM <CDU5211/YYS7> at ata1-master UDMA33 SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/ad0s1a WARNING: / was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted /var: mount pending error: blocks 56 files 3 USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 97.6 0.0 0 8 ?? RL 6:39AM 13:14.72 [idle: cpu0] -T -- "To be nobody but yourself in a world which is doing its best to make you everybody else, means to fight the hardest human battle ever and to never stop fighting." -- e.e. cummings
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070308125927.GA1265>