From owner-freebsd-stable@FreeBSD.ORG Wed May 7 17:19:35 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 58533106566B for ; Wed, 7 May 2008 17:19:35 +0000 (UTC) (envelope-from yani@pi-greece.eu) Received: from rosebud.otenet.gr (rosebud.otenet.gr [195.170.0.94]) by mx1.freebsd.org (Postfix) with ESMTP id 214C38FC0A for ; Wed, 7 May 2008 17:19:33 +0000 (UTC) (envelope-from yani@pi-greece.eu) Received: from techmx01.pi-greece.eu (athedsl-4448179.home.otenet.gr [79.129.207.163]) by rosebud.otenet.gr (8.13.8/8.13.8/Debian-3) with SMTP id m47Gj94h019464 for ; Wed, 7 May 2008 19:45:09 +0300 Received: (qmail 2005 invoked by uid 0); 7 May 2008 19:45:08 +0300 Received: from 192.168.1.15 by techserver.pi-greece.eu (envelope-from , uid 0) with qmail-scanner-1.25 (clamdscan: 0.93/6817. spamassassin: 3.2.4. Clear:RC:0(192.168.1.15):SA:0(-4.4/5.0):. Processed in 5.144842 secs); 07 May 2008 16:45:08 -0000 X-Spam-Status: No, hits=-4.4 required=5.0 X-Qmail-Scanner-Mail-From: yani@pi-greece.eu via techserver.pi-greece.eu X-Qmail-Scanner: 1.25 (Clear:RC:0(192.168.1.15):SA:0(-4.4/5.0):. Processed in 5.144842 secs) Received: from unimatrix.pi-office (HELO ?192.168.1.15?) (yani@pi-greece.eu@192.168.1.15) by techserver.pi-greece.eu with SMTP; 7 May 2008 19:45:03 +0300 Message-ID: <4821DC99.1050008@pi-greece.eu> Date: Wed, 07 May 2008 19:45:13 +0300 From: Yani Karydis User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <20080506120014.60FED10656C9@hub.freebsd.org> In-Reply-To: <20080506120014.60FED10656C9@hub.freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Kernel panic - em0 culprit? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 May 2008 17:19:35 -0000 Hello, My server is experiencing occasional kernel panics when under moderate load. I'm attaching a crash dump and the dmesg output. I'm not sure how to read the kernel backtrace but it looks like the Intel NIC (em0) caused the problem. Occasionally, I used to get a "em0: watchdog timeout -- resetting" error message, but I never had a kernel panic. The problem started last week when I updated the server to the latest 7-STABLE. Can anyone help pinpoint the problem? Thanks, Yani kgdb output: [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". There is no member named pathname. Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/miibus.ko...Reading symbols from /boot/kernel/miibus.ko.symbols...done. done. Loaded symbols for /boot/kernel/miibus.ko Reading symbols from /boot/kernel/if_vr.ko...Reading symbols from /boot/kernel/if_vr.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_vr.ko Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done. done. Loaded symbols for /boot/kernel/accf_http.ko Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done. done. Loaded symbols for /boot/kernel/bridgestp.ko Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_bridge.ko Reading symbols from /boot/kernel/wlan_wep.ko...Reading symbols from /boot/kernel/wlan_wep.ko.symbols...done. done. Loaded symbols for /boot/kernel/wlan_wep.ko Reading symbols from /boot/kernel/wlan_tkip.ko...Reading symbols from /boot/kernel/wlan_tkip.ko.symbols...done. done. Loaded symbols for /boot/kernel/wlan_tkip.ko Reading symbols from /boot/kernel/wlan_ccmp.ko...Reading symbols from /boot/kernel/wlan_ccmp.ko.symbols...done. done. Loaded symbols for /boot/kernel/wlan_ccmp.ko Reading symbols from /boot/kernel/wlan_xauth.ko...Reading symbols from /boot/kernel/wlan_xauth.ko.symbols...done. done. Loaded symbols for /boot/kernel/wlan_xauth.ko Reading symbols from /boot/kernel/wlan_acl.ko...Reading symbols from /boot/kernel/wlan_acl.ko.symbols...done. done. Loaded symbols for /boot/kernel/wlan_acl.ko Reading symbols from /boot/kernel/aio.ko...Reading symbols from /boot/kernel/aio.ko.symbols...done. done. Loaded symbols for /boot/kernel/aio.ko Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/linprocfs.ko Reading symbols from /boot/kernel/nfslockd.ko...Reading symbols from /boot/kernel/nfslockd.ko.symbols...done. done. Loaded symbols for /boot/kernel/nfslockd.ko Reading symbols from /boot/kernel/krpc.ko...Reading symbols from /boot/kernel/krpc.ko.symbols...done. done. Loaded symbols for /boot/kernel/krpc.ko Unread portion of the kernel message buffer: em0: watchdog timeout -- resetting <5>em0: link state changed to DOWN kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x14 fault code = supervisor read, page not present instruction pointer = 0x20:0xc05bc167 stack pointer = 0x28:0xe403ec10 frame pointer = 0x28:0xe403ec2c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 23 (em0 taskq) trap number = 12 panic: page fault Uptime: 1d2h53m51s Physical memory: 1015 MB Dumping 224 MB: 209 193 177 161 145 129 113 97 81 65 49 33 17 1 #0 doadump () at pcpu.h:195 195 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) list 190 static __inline struct thread * 191 __curthread(void) 192 { 193 struct thread *td; 194 195 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); 196 return (td); 197 } 198 #define curthread (__curthread()) 199 (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0xc058ad84 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc058af84 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc07ccb5c in trap_fatal (frame=0xe403ebd0, eva=20) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc07cd4dd in trap (frame=0xe403ebd0) at /usr/src/sys/i386/i386/trap.c:280 #5 0xc07b721b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc05bc167 in propagate_priority (td=0xc3ab7880) at /usr/src/sys/kern/subr_turnstile.c:272 #7 0xc05bcab8 in turnstile_wait (ts=0xc3a9d370, owner=0xc3ab7880, queue=Variable "queue" is not available. ) at /usr/src/sys/kern/subr_turnstile.c:739 #8 0xc057e88d in _mtx_lock_sleep (m=0xc3baa2f4, tid=3283517440, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:416 #9 0xc04b02d7 in em_handle_rxtx (context=0xc3baa000, pending=1) at /usr/src/sys/dev/em/if_em.c:1666 #10 0xc05bb032 in taskqueue_run (queue=0xc3b94100) at /usr/src/sys/kern/subr_taskqueue.c:255 #11 0xc05bb20d in taskqueue_thread_loop (arg=0xc3baa368) at /usr/src/sys/kern/subr_taskqueue.c:374 #12 0xc0569d34 in fork_exit (callout=0xc05bb190 , arg=0xc3baa368, frame=0xe403ed38) at /usr/src/sys/kern/kern_fork.c:783 #13 0xc07b7290 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205 dmesg: Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #11: Tue May 6 15:34:11 EEST 2008 root@techserver.pi-greece.eu:/usr/obj/usr/src/sys/TECHSERVER7 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Sempron(tm) 2300+ (1585.75-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x681 Stepping = 1 Features=0x383fbff AMD Features=0xc0480800 real memory = 1073676288 (1023 MB) avail memory = 1041485824 (993 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard wlan: mac acl policy registered kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 3fef0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: on hostb0 agp0: aperture size is 32M pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xa000-0xa0ff mem 0xd0000000-0xd7ffffff,0xe0000000-0xe000ffff at device 0.0 on pci1 em0: port 0xb000-0xb03f mem 0xe1000000-0xe101ffff,0xe1020000-0xe103ffff irq 17 at device 9.0 on pci0 em0: [FILTER] em0: Ethernet address: 00:07:e9:49:18:b3 aac0: mem 0xd8000000-0xdbffffff irq 18 at device 10.0 on pci0 aac0: Enable Raw I/O aac0: New comm. interface enabled aac0: [ITHREAD] aac0: Adaptec 2610SA, aac driver 2.0.0-1 ahc0: port 0xb400-0xb4ff mem 0xe1042000-0xe1042fff irq 19 at device 11.0 on pci0 ahc0: [ITHREAD] aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs ral0: mem 0xe1040000-0xe1041fff irq 17 at device 13.0 on pci0 ral0: MAC/BBP RT2560 (rev 0x04), RF RT2525 ral0: Ethernet address: 00:11:50:14:b7:66 ral0: [ITHREAD] atapci0: port 0xb800-0xb807,0xbc00-0xbc03,0xc000-0xc007,0xc400-0xc403,0xc800-0xc80f,0xcc00-0xccff irq 20 at device 15.0 on pci0 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 15.1 on pci0 ata0: on atapci1 ata0: [ITHREAD] ata1: on atapci1 ata1: [ITHREAD] uhci0: port 0xd400-0xd41f irq 21 at device 16.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xd800-0xd81f irq 21 at device 16.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xdc00-0xdc1f irq 21 at device 16.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xe000-0xe01f irq 21 at device 16.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xe1043000-0xe10430ff irq 21 at device 16.4 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered isab0: at device 17.0 on pci0 isa0: on isab0 vr0: port 0xe400-0xe4ff mem 0xe1044000-0xe10440ff irq 23 at device 18.0 on pci0 vr0: Quirks: 0x0 vr0: Revision: 0x78 miibus0: on vr0 ukphy0: PHY 1 on miibus0 ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto vr0: Ethernet address: 00:0f:ea:e0:eb:cc vr0: [ITHREAD] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse Explorer, device ID 4 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcc000-0xccfff,0xcd000-0xd0fff,0xd1000-0xd17ff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 ppbus0: [ITHREAD] lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] Timecounter "TSC" frequency 1585747239 Hz quality 800 Timecounters tick every 1.000 msec Waiting 5 seconds for SCSI devices to settle ad0: 305244MB at ata0-master UDMA100 acd0: DVDR at ata1-master UDMA33 aacd0: on aac0 aacd0: 953812MB (1953406976 sectors) sa0 at ahc0 bus 0 target 3 lun 0 sa0: Removable Sequential Access SCSI-3 device sa0: 20.000MB/s transfers (10.000MHz, offset 8, 16bit) Trying to mount root from ufs:/dev/ad0s1a WARNING: / was not properly dismounted bridge0: Ethernet address: 86:5b:d2:24:75:72 em0: link state changed to UP