From owner-freebsd-current@FreeBSD.ORG Thu Mar 8 13:30:55 2007 Return-Path: X-Original-To: freebsd-current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6831516A400 for ; Thu, 8 Mar 2007 13:30:55 +0000 (UTC) (envelope-from tillman@seekingfire.com) Received: from mail.seekingfire.com (thoth.seekingfire.com [24.89.83.9]) by mx1.freebsd.org (Postfix) with ESMTP id 1D93613C461 for ; Thu, 8 Mar 2007 13:30:54 +0000 (UTC) (envelope-from tillman@seekingfire.com) Received: by mail.seekingfire.com (Postfix, from userid 500) id EA6F63980B; Thu, 8 Mar 2007 06:59:27 -0600 (CST) Date: Thu, 8 Mar 2007 06:59:27 -0600 From: Tillman Hodgson To: freebsd-current@FreeBSD.org Message-ID: <20070308125927.GA1265@seekingfire.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Habeas-SWE-1: winter into spring X-Habeas-SWE-2: brightly anticipated X-Habeas-SWE-3: like Habeas SWE (tm) X-Habeas-SWE-4: Copyright 2002 Habeas (tm) X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this X-Habeas-SWE-6: email in exchange for a license for this Habeas X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this X-Habeas-SWE-9: mark in spam to . X-GPG-Key-ID: 828AFC7B X-GPG-Fingerprint: 5584 14BA C9EB 1524 0E68 F543 0F0A 7FBC 828A FC7B X-GPG-Key: http://www.seekingfire.com/personal/gpg_key.asc X-Urban-Legend: There is lots of hidden information in headers X-Tillman-rules: yes he does User-Agent: Mutt/1.5.14 (2007-02-12) Cc: Subject: Experiencing hangs on SMP box with no console messages given for clues. Details inside. X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Mar 2007 13:30:55 -0000 Howdy folks, These has been happening every few days for a few weeks now. When it occurs, there's no messages logged to the console or to syslog -- it just silently hangs. I added the break-to-debugger option so that I can at least reboot it remotely via the serial console. I've been following the -current kernel fairly closely in hopes that it was just due to a transitory -current problem. I don't mind rebuilding a kernel with special options if it's useful -- I'll be rebuilding this morning with WITNESS and INVARIANTS for sure. I have the core saved, though I'm inexperienced with gdb. FreeBSD/i386 (athena.seekingfire.prv) (ttyd0) login: telnet> send brk KDB: enter: Line break on console [thread pid 11 tid 100005 ] Stopped at kdb_enter+0x2c: leave db> ? Bad character ? db> help print p examine x search set write w delete d break b dwatch watch dhwatch hwatch step s continue c until next match trace t alltrace where bt call show ps gdb halt reboot reset kill watchdog thread panic ahd_dump ahd_out ahd_in ahd_unpause ahd_pause ahd_sunit db> bt Tracing pid 11 tid 100005 td 0xc3afe6c0 kdb_enter(c0956f95,c0,c3afe6c0,c3af7cc8,c3afb880,...) at kdb_enter+0x2c siointr1(c3cb7b80,e25f0c84,c08cd60f,c3cb4000,c3afe6c0,...) at siointr1+0x3be siointr(c3cb4000,c3afe6c0,0,0,c3bfb400,...) at siointr+0x4c intr_execute_handlers(c3af7cc8,e25f0c94) at intr_execute_handlers+0xf3 Xapic_isr1() at Xapic_isr1+0x34 --- interrupt, eip = 0xc0baf599, esp = 0xe25f0cd4, ebp = 0xe25f0cd4 --- acpi_cpu_c1(e25f0cec,c06e382d,c0a5cb60,c3afe6c0,c06e3ccc,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,e25f0d24,c06b5db1,0,e25f0d38,...) at acpi_cpu_idle+0x15a sched_idletd(0,e25f0d38,0,c3afdb40,0,...) at sched_idletd+0x8a fork_exit(c06e3ccc,0,e25f0d38) at fork_exit+0x61 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe25f0d70, ebp = 0 --- db> show proc Process 11 (idle: cpu0) at 0xc3afdb40: state: NORMAL uid: 0 gids: 0 parent: pid 0 at 0xc0a58d80 ABI: null threads: 1 100005 Run CPU 0 [idle: cpu0] db> panic panic: from debugger cpuid = 0 Uptime: 2d22h24m3s Physical memory: 1015 MB Dumping 200 MB: 185 169 153 137 121 105 89 73 57 41 25 9 Dump complete Automatic reboot in 15 seconds - press a key on the console to abort [root@athena ~]# uname -a FreeBSD athena.seekingfire.prv 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Sun Mar 4 21:08:19 CST 2007 toor@athena.seekingfire.prv (/usr/src was synced the same day) [root@athena /usr/src/sys/i386/conf]# diff ATHENA GENERIC 24c24 < ident ATHENA --- > ident GENERIC 29c29 < ### makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols --- > makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols 67,73c67,70 < ###options INVARIANTS # Enable calls of extra sanity checking < ###options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS < ###options WITNESS # Enable checks to detect deadlocks and cycles < ###options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed < < ### Tillman added 26Feb07 as per http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html < options BREAK_TO_DEBUGGER --- > options INVARIANTS # Enable calls of extra sanity checking > options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS > options WITNESS # Enable checks to detect deadlocks and cycles > options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed [root@athena ~]# dmesg Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-CURRENT #0: Sun Mar 4 21:08:19 CST 2007 toor@athena.seekingfire.prv:/usr/obj/usr/src/sys/ATHENA ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel Pentium III (997.17-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x68a Stepping = 10 Features=0x387fbff real memory = 1073676288 (1023 MB) avail memory = 1041326080 (993 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: on acpi0 cpu1: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff,0x4000-0x407f,0x4080-0x40ff,0x5000-0x500f,0x6000-0x607f on acpi0 pci0: on pcib0 agp0: on hostb0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xd000-0xd0ff mem 0xf4000000-0xf4ffffff,0xf6241000-0xf6241fff irq 19 at device 6.0 on pci0 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd400-0xd40f at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] uhci0: port 0xd800-0xd81f irq 12 at device 7.2 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xdc00-0xdc1f irq 12 at device 7.3 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered pci0: at device 7.4 (no driver attached) fxp0: port 0xe000-0xe03f mem 0xf6240000-0xf6240fff,0xf6000000-0xf60fffff irq 17 at device 13.0 on pci0 miibus0: on fxp0 inphy0: PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:e0:81:21:ad:e0 fxp0: [ITHREAD] fxp1: port 0xe400-0xe43f mem 0xf6242000-0xf6242fff,0xf6100000-0xf61fffff irq 18 at device 14.0 on pci0 miibus1: on fxp1 inphy1: PHY 1 on miibus1 inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: Ethernet address: 00:e0:81:21:ad:e1 fxp1: [ITHREAD] em0: port 0xe800-0xe83f mem 0xf6200000-0xf621ffff,0xf6220000-0xf623ffff irq 18 at device 16.0 on pci0 em0: Ethernet address: 00:0e:0c:c2:ce:4f em0: [FILTER] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ad0: 38166MB at ata0-master UDMA100 acd0: CDROM at ata1-master UDMA33 SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/ad0s1a WARNING: / was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted /var: mount pending error: blocks 56 files 3 USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 97.6 0.0 0 8 ?? RL 6:39AM 13:14.72 [idle: cpu0] -T -- "To be nobody but yourself in a world which is doing its best to make you everybody else, means to fight the hardest human battle ever and to never stop fighting." -- e.e. cummings