From owner-freebsd-bugs@FreeBSD.ORG Sat Jan 13 09:40:14 2007 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2849216A415 for ; Sat, 13 Jan 2007 09:40:14 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id 046C513C44C for ; Sat, 13 Jan 2007 09:40:13 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l0D9eDob027048 for ; Sat, 13 Jan 2007 09:40:13 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l0D9eDER027047; Sat, 13 Jan 2007 09:40:13 GMT (envelope-from gnats) Resent-Date: Sat, 13 Jan 2007 09:40:13 GMT Resent-Message-Id: <200701130940.l0D9eDER027047@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Mikle Davidkin Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E3E4916A403 for ; Sat, 13 Jan 2007 09:39:41 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [69.147.83.33]) by mx1.freebsd.org (Postfix) with ESMTP id D3D3613C465 for ; Sat, 13 Jan 2007 09:39:41 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l0D9ddYJ041070 for ; Sat, 13 Jan 2007 09:39:39 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id l0D9ddeu041069; Sat, 13 Jan 2007 09:39:39 GMT (envelope-from nobody) Message-Id: <200701130939.l0D9ddeu041069@www.freebsd.org> Date: Sat, 13 Jan 2007 09:39:39 GMT From: Mikle Davidkin To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.0 Cc: Subject: kern/107864: kernel panic in 6.2-RC2 on heavy network load X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jan 2007 09:40:14 -0000 >Number: 107864 >Category: kern >Synopsis: kernel panic in 6.2-RC2 on heavy network load >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Jan 13 09:40:13 GMT 2007 >Closed-Date: >Last-Modified: >Originator: Mikle Davidkin >Release: 6.2-RC2 >Organization: Vector Telecom >Environment: FreeBSD router05.vt.net.ru 6.2-RC2 FreeBSD 6.2-RC2 #0: Mon Jan 8 13:17:00 MSK 2007 root@router05.vt.net.ru:/usr/src/sys/i386/compile/skykernel i386 >Description: Kernel panics almost every evening when the most heavy network load is on the net. Machine is just a software router for 3 subnets with routed, ipfw, pf (for binat) and no more... Similar simptoms I have on other software router - will report in other PR and add link to it here. Kernel is GENERIC with some devices disabled (SCSI/RAID) and few options added: options DEVICE_POLLING options HZ=1000 options IPFIREWALL options DUMMYNET options IPFIREWALL_FORWARD dmesg ============== Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RC2 #0: Mon Jan 8 13:17:00 MSK 2007 root@router05.vt.net.ru:/usr/src/sys/i386/compile/skykernel ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) XP 2000+ (1661.94-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x681 Stepping = 1 Features=0x383fbff AMD Features=0xc0400800 real memory = 268369920 (255 MB) avail memory = 252968960 (241 MB) ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: on acpi0 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: mem 0xe5000000-0xe53fffff at device 0.0 on pci0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pci1: at device 0.0 (no driver attached) stge0: port 0xc000-0xc0ff mem 0xe5703000-0xe57030ff irq 17 at device 9.0 on pci0 miibus0: on stge0 ip1000phy0: on miibus0 ip1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto stge0: Ethernet address: 00:15:f2:ef:de:a9 fxp0: port 0xc400-0xc43f mem 0xe5701000-0xe5701fff,0xe5400000-0xe54fffff irq 18 at device 10.0 on pci0 miibus1: on fxp0 inphy0: on miibus1 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:d0:43:7a:fd:77 fxp1: port 0xc800-0xc83f mem 0xe5700000-0xe5700fff,0xe5500000-0xe55fffff irq 19 at device 11.0 on pci0 miibus2: on fxp1 inphy1: on miibus2 inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: Ethernet address: 00:02:b3:bc:f2:60 fxp2: port 0xcc00-0xcc3f mem 0xe5702000-0xe5702fff,0xe5600000-0xe56fffff irq 16 at device 12.0 on pci0 miibus3: on fxp2 inphy2: on miibus3 inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp2: Ethernet address: 00:d0:43:7a:fa:99 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 15.0 on pci0 ata0: on atapci0 ata1: on atapci0 uhci0: port 0xd400-0xd41f at device 16.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xd800-0xd81f at device 16.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xdc00-0xdc1f at device 16.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0xe000-0xe01f at device 16.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xe5704000-0xe57040ff at device 16.4 on pci0 ehci0: [GIANT-LOCKED] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub4: 8 ports with 8 removable, self powered isab0: at device 17.0 on pci0 isa0: on isab0 vr0: port 0xe400-0xe4ff mem 0xe5705000-0xe57050ff irq 23 at device 18.0 on pci0 miibus4: on vr0 ukphy0: on miibus4 ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto vr0: Ethernet address: 00:11:5b:93:c2:3d acpi_tz0: on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd27ff,0xd3000-0xd3fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 1661943183 Hz quality 800 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to deny, logging disabled ad0: 38166MB at ata0-master UDMA100 ================= backtrace ============================ 1) Jan 8 20:26 # kgdb /boot/kernel/kernel.debug /var/crash/vmcore.6 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x78 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0578abd stack pointer = 0x28:0xd40b8ab4 frame pointer = 0x28:0xd40b8ab8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 11 (swi1: net) trap number = 12 panic: page fault Uptime: 6h42m36s Physical memory: 247 MB Dumping 34 MB: 19 3 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc05584ee in boot (howto=260) at ../../../kern/kern_shutdown.c:409 #2 0xc0558784 in panic (fmt=0xc0738cc7 "%s") at ../../../kern/kern_shutdown.c:565 #3 0xc070fa60 in trap_fatal (frame=0xd40b8a74, eva=120) at ../../../i386/i386/trap.c:837 #4 0xc070f242 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -737440936, tf_esi = -1038653568, tf_ebp = -737441096, tf_isp = -737441120, tf_ebx = -1038686208, tf_edx = -1038686208, tf_ecx = 4, tf_eax = -1038653536, tf_trapno = 12, tf_err = 0, tf_eip = -1068004675, tf_cs = 32, tf_eflags = 589831, tf_esp = -1038653568, tf_ss = -737441060}) at ../../../i386/i386/trap.c:270 #5 0xc06fe3fa in calltrap () at ../../../i386/i386/exception.s:139 #6 0xc0578abd in turnstile_setowner (ts=0xc216e800, owner=0x4) at ../../../kern/subr_turnstile.c:432 #7 0xc0578db4 in turnstile_wait (lock=0xc28de168, owner=0x4) at ../../../kern/subr_turnstile.c:591 #8 0xc054ec70 in _mtx_lock_sleep (m=0xc28de168, tid=3256313728, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:579 #9 0xc05d5434 in rt_check (lrt=0xc21767a0, lrt0=0xd40b8b58, dst=0xc288e210) at ../../../net/route.c:1280 #10 0xc05d8432 in arpresolve (ifp=0xc2245400, rt0=0xc25fd420, m=0xc24baa00, dst=0xc288e210, desten=0xd40b8b78 "") at ../../../netinet/if_ether.c:404 #11 0xc05c7633 in ether_output (ifp=0xc2245400, m=0xc24baa00, dst=0xc288e210, rt0=0xc216e800) at ../../../net/if_ethersubr.c:174 #12 0xc05ed765 in ip_output (m=0xc24baa00, opt=0xc2245400, ro=0xd40b8bd4, flags=1, imo=0x0, inp=0x0) at ../../../netinet/ip_output.c:777 #13 0xc05ecb8c in ip_forward (m=0xc24baa00, srcrt=0) at ../../../netinet/ip_input.c:1917 #14 0xc05eb6bb in ip_input (m=0xc24baa00) at ../../../netinet/ip_input.c:688 #15 0xc05cff6b in netisr_processqueue (ni=0xc07fc7b8) at ../../../net/netisr.c:236 #16 0xc05d0166 in swi_net (dummy=0x0) at ../../../net/netisr.c:349 #17 0xc05428fd in ithread_execute_handlers (p=0xc2175a78, ie=0xc21a9300) at ../../../kern/kern_intr.c:682 #18 0xc0542a08 in ithread_loop (arg=0xc2143720) at ../../../kern/kern_intr.c:765 #19 0xc0541884 in fork_exit (callout=0xc05429b4 , arg=0xc2143720, frame=0xd40b8d38) at ../../../kern/kern_fork.c:821 #20 0xc06fe45c in fork_trampoline () at ../../../i386/i386/exception.s:208 (kgdb) list *0xc0578abd 0xc0578abd is in turnstile_setowner (../../../kern/subr_turnstile.c:433). 428 429 mtx_assert(&td_contested_lock, MA_OWNED); 430 MPASS(owner->td_proc->p_magic == P_MAGIC); 431 MPASS(ts->ts_owner == NULL); 432 ts->ts_owner = owner; 433 LIST_INSERT_HEAD(&owner->td_contested, ts, ts_link); 434 } 435 436 /* 437 * Malloc a turnstile for a new thread, initialize it and return it. 2) Jan 9 15:26 # kgdb /boot/kernel/kernel.debug /var/crash/vmcore.7 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc05584ee in boot (howto=260) at ../../../kern/kern_shutdown.c:409 #2 0xc0558784 in panic (fmt=0xc0738cc7 "%s") at ../../../kern/kern_shutdown.c:565 #3 0xc070fa60 in trap_fatal (frame=0xd40b8a74, eva=120) at ../../../i386/i386/trap.c:837 #4 0xc070f242 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -737440936, tf_esi = -1038653568, tf_ebp = -737441096, tf_isp = -737441120, tf_ebx = -1038686208, tf_edx = -1038686208, tf_ecx = 4, tf_eax = -1038653536, tf_trapno = 12, tf_err = 0, tf_eip = -1068004675, tf_cs = 32, tf_eflags = 589831, tf_esp = -1038653568, tf_ss = -737441060}) at ../../../i386/i386/trap.c:270 #5 0xc06fe3fa in calltrap () at ../../../i386/i386/exception.s:139 #6 0xc0578abd in turnstile_setowner (ts=0xc216e800, owner=0x4) at ../../../kern/subr_turnstile.c:432 #7 0xc0578db4 in turnstile_wait (lock=0xc2806378, owner=0x4) at ../../../kern/subr_turnstile.c:591 #8 0xc054ec70 in _mtx_lock_sleep (m=0xc2806378, tid=3256313728, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:579 #9 0xc05d5434 in rt_check (lrt=0xc21767a0, lrt0=0xd40b8b58, dst=0xc266ad50) at ../../../net/route.c:1280 #10 0xc05d8432 in arpresolve (ifp=0xc2245400, rt0=0xc2806000, m=0xc25b4100, dst=0xc266ad50, desten=0xd40b8b78 "") at ../../../netinet/if_ether.c:404 #11 0xc05c7633 in ether_output (ifp=0xc2245400, m=0xc25b4100, dst=0xc266ad50, rt0=0xc216e800) at ../../../net/if_ethersubr.c:174 #12 0xc05ed765 in ip_output (m=0xc25b4100, opt=0xc2245400, ro=0xd40b8bd4, flags=1, imo=0x0, inp=0x0) at ../../../netinet/ip_output.c:777 #13 0xc05ecb8c in ip_forward (m=0xc25b4100, srcrt=0) at ../../../netinet/ip_input.c:1917 #14 0xc05eb6bb in ip_input (m=0xc25b4100) at ../../../netinet/ip_input.c:688 #15 0xc05cff6b in netisr_processqueue (ni=0xc07fc7b8) at ../../../net/netisr.c:236 #16 0xc05d0166 in swi_net (dummy=0x0) at ../../../net/netisr.c:349 #17 0xc05428fd in ithread_execute_handlers (p=0xc2175a78, ie=0xc21a9300) at ../../../kern/kern_intr.c:682 #18 0xc0542a08 in ithread_loop (arg=0xc2143720) at ../../../kern/kern_intr.c:765 #19 0xc0541884 in fork_exit (callout=0xc05429b4 , arg=0xc2143720, frame=0xd40b8d38) at ../../../kern/kern_fork.c:821 #20 0xc06fe45c in fork_trampoline () at ../../../i386/i386/exception.s:208 (kgdb) quit >How-To-Repeat: panic occurs in 1-3 days uptume >Fix: >Release-Note: >Audit-Trail: >Unformatted: