From owner-freebsd-bugs@FreeBSD.ORG Sat Jan 13 10:00:28 2007 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D8FA016A40F for ; Sat, 13 Jan 2007 10:00:28 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id 90FF013C457 for ; Sat, 13 Jan 2007 10:00:28 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l0DA0Rn4027869 for ; Sat, 13 Jan 2007 10:00:27 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l0DA0R7o027868; Sat, 13 Jan 2007 10:00:27 GMT (envelope-from gnats) Resent-Date: Sat, 13 Jan 2007 10:00:27 GMT Resent-Message-Id: <200701131000.l0DA0R7o027868@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Mikle Davidkin Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3F63E16A403 for ; Sat, 13 Jan 2007 09:51:18 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [69.147.83.33]) by mx1.freebsd.org (Postfix) with ESMTP id 2F75E13C455 for ; Sat, 13 Jan 2007 09:51:18 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l0D9pIMR043890 for ; Sat, 13 Jan 2007 09:51:18 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id l0D9pIKh043889; Sat, 13 Jan 2007 09:51:18 GMT (envelope-from nobody) Message-Id: <200701130951.l0D9pIKh043889@www.freebsd.org> Date: Sat, 13 Jan 2007 09:51:18 GMT From: Mikle Davidkin To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.0 Cc: Subject: kern/107865: kernel panic in 6.2-RC2 on heavy network load X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jan 2007 10:00:29 -0000 >Number: 107865 >Category: kern >Synopsis: kernel panic in 6.2-RC2 on heavy network load >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Jan 13 10:00:27 GMT 2007 >Closed-Date: >Last-Modified: >Originator: Mikle Davidkin >Release: 6.2-RC2 >Organization: Vector Telecom >Environment: FreeBSD router03.vt.net.ru 6.2-RC2 FreeBSD 6.2-RC2 #0: Mon Jan 8 13:29:30 MSK 2007 root@router03.vt.net.ru:/usr/src/sys/i386/compile/skykernel i386 >Description: Kernel panics almost every evening when the most heavy network load is on the net. Machine is just a software router for 3 subnets with routed, ipfw, pf (for binat) and no more... May be it's related to my other PR - 107864, because routers are almost the same and doing similar work just on different subnets of my network... Kernel is GENERIC with some devices disabled (SCSI/RAID) and few options added: options DEVICE_POLLING options HZ=1000 options IPFIREWALL options DUMMYNET options IPFIREWALL_FORWARD dmesg ============== Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RC2 #0: Mon Jan 8 13:29:30 MSK 2007 root@router03.vt.net.ru:/usr/src/sys/i386/compile/skykernel ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Sempron(tm) Processor 2600+ (1607.34-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20fc2 Stepping = 2 Features=0x78bfbff Features2=0x1 AMD Features=0xe2500800 AMD Features2=0x1 real memory = 268369920 (255 MB) avail memory = 252952576 (241 MB) ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0 pci0: on pcib0 agp0: mem 0xee000000-0xefffffff at device 0.0 on pci0 isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xf6002000-0xf6002fff irq 20 at device 2.0 on pci0 ohci0: [GIANT-LOCKED] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 4 ports with 4 removable, self powered ohci1: mem 0xf6003000-0xf6003fff irq 21 at device 2.1 on pci0 ohci1: [GIANT-LOCKED] usb1: OHCI version 1.0, legacy support usb1: SMM does not respond, resetting usb1: on ohci1 usb1: USB revision 1.0 uhub1: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 4 ports with 4 removable, self powered ehci0: mem 0xf6004000-0xf60040ff irq 22 at device 2.2 on pci0 ehci0: [GIANT-LOCKED] usb2: EHCI version 1.0 usb2: companion controllers, 4 ports each: usb0 usb1 usb2: on ehci0 usb2: USB revision 2.0 uhub2: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub2: 8 ports with 8 removable, self powered nve0: port 0xb800-0xb807 mem 0xf6000000-0xf6000fff irq 20 at device 5.0 on pci0 nve0: Ethernet address 00:04:61:a5:08:01 miibus0: on nve0 rlphy0: on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto nve0: Ethernet address: 00:04:61:a5:08:01 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 8.0 on pci0 ata0: on atapci0 ata1: on atapci0 pcib1: at device 11.0 on pci0 pci1: on pcib1 pci1: at device 0.0 (no driver attached) pcib2: at device 14.0 on pci0 pci2: on pcib2 em0: port 0x9000-0x903f mem 0xf5100000-0xf511ffff,0xf5120000-0xf513ffff irq 19 at device 7.0 on pci2 em0: Ethernet address: 00:07:e9:0f:65:32 xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0x9400-0x947f mem 0xf5143000-0xf514307f irq 16 at device 8.0 on pci2 miibus1: on xl0 ukphy0: on miibus1 ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl0: Ethernet address: 00:0a:5e:4f:f3:d6 xl1: <3Com 3c905C-TX Fast Etherlink XL> port 0x9800-0x987f mem 0xf5140000-0xf514007f irq 17 at device 9.0 on pci2 miibus2: on xl1 ukphy1: on miibus2 ukphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl1: Ethernet address: 00:0a:5e:4f:f3:d9 xl2: <3Com 3c905C-TX Fast Etherlink XL> port 0x9c00-0x9c7f mem 0xf5141000-0xf514107f irq 18 at device 10.0 on pci2 miibus3: on xl2 ukphy2: on miibus3 ukphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl2: Ethernet address: 00:0a:5e:4f:f3:e2 fxp0: port 0xa000-0xa03f mem 0xf5142000-0xf5142fff,0xf5000000-0xf50fffff irq 19 at device 11.0 on pci2 miibus4: on fxp0 inphy0: on miibus4 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:d0:b7:07:c1:cc acpi_tz0: on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A ppc0: port 0x378-0x37f,0x778-0x77b irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd17ff,0xd2000-0xd27ff,0xd3000-0xd37ff,0xd4000-0xd4fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 1607340759 Hz quality 800 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to deny, logging disabled ad0: 38204MB at ata0-master UDMA100 ================= backtrace ============================ kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x78 fault code = supervisor read, page not present instruction pointer = 0x20:0xc05719ad stack pointer = 0x28:0xd6e18a7c frame pointer = 0x28:0xd6e18a80 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 306 (routed) trap number = 12 panic: page fault Uptime: 2d13h32m45s Physical memory: 247 MB Dumping 65 MB: 50 34 18 2 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc05513de in boot (howto=260) at ../../../kern/kern_shutdown.c:409 #2 0xc0551674 in panic (fmt=0xc07318a5 "%s") at ../../../kern/kern_shutdown.c:565 #3 0xc0708790 in trap_fatal (frame=0xd6e18a3c, eva=120) at ../../../i386/i386/trap.c:837 #4 0xc0707f72 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 16, tf_esi = -1035997184, tf_ebp = -689862016, tf_isp = -689862040, tf_ebx = -1032266368, tf_edx = -1032266368, tf_ecx = 4, tf_eax = -1035997152, tf_trapno = 12, tf_err = 0, tf_eip = -1068033619, tf_cs = 32, tf_eflags = 65539, tf_esp = -1035997184, tf_ss = -689861980}) at ../../../i386/i386/trap.c:270 #5 0xc06f712a in calltrap () at ../../../i386/i386/exception.s:139 #6 0xc05719ad in turnstile_setowner (ts=0xc278dd80, owner=0x4) at ../../../kern/subr_turnstile.c:432 #7 0xc0571ca4 in turnstile_wait (lock=0xc2809168, owner=0x4) at ../../../kern/subr_turnstile.c:591 #8 0xc0547b60 in _mtx_lock_sleep (m=0xc2809168, tid=3258970112, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:579 #9 0xc05cdcdc in rt_setgate (rt=0xc27e0bdc, dst=0xc27b40e0, gate=0xc363b66c) at ../../../net/route.c:1041 #10 0xc05ceecd in route_output (m=0xc36fed00, so=0xc247b000) at ../../../net/rtsock.c:487 #11 0xc05cc634 in raw_usend (so=0x4, flags=0, m=0xc278dd80, nam=0x0, control=0xc23ff020, td=0xc23ff000) at ../../../net/raw_usrreq.c:263 #12 0xc05ce7cb in rts_send (so=0xc247b000, flags=0, m=0xc36fed00, nam=0x0, control=0x0, td=0xc23ff000) at ../../../net/rtsock.c:269 #13 0xc058a4d3 in sosend (so=0xc247b000, addr=0x0, uio=0xd6e18cbc, top=0xc36fed00, control=0x0, flags=0, td=0xc23ff000) at ../../../kern/uipc_socket.c:836 #14 0xc0578d5e in soo_write (fp=0xc23ff020, uio=0xd6e18cbc, active_cred=0xc2173d80, flags=0, td=0xc23ff000) at ../../../kern/sys_socket.c:118 #15 0xc0573467 in dofilewrite (td=0xc23ff000, fd=3, fp=0xc2424288, auio=0xd6e18cbc, offset=Unhandled dwarf expression opcode 0x93 ) at file.h:252 #16 0xc057330b in kern_writev (td=0xc23ff000, fd=3, auio=0xd6e18cbc) at ../../../kern/sys_generic.c:402 #17 0xc0573231 in write (td=0xc23ff000, uap=0x4) at ../../../kern/sys_generic.c:326 #18 0xc0708aa7 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = -1024, tf_esi = 3, tf_ebp = -1077941176, tf_isp = -689861276, tf_ebx = 1, tf_edx = -1077941220, tf_ecx = 2, tf_eax = 4, tf_trapno = 22, tf_err = 2, tf_eip = 672494419, tf_cs = 51, tf_eflags = 642, tf_esp = -1077941380, tf_ss = 59}) at ../../../i386/i386/trap.c:983 #19 0xc06f717f in Xint0x80_syscall () at ../../../i386/i386/exception.s:200 #20 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) list *0xc05719ad 0xc05719ad is in turnstile_setowner (../../../kern/subr_turnstile.c:433). 428 * Malloc a turnstile for a new thread, initialize it and return it. 429 */ 430 struct turnstile * 431 turnstile_alloc(void) 432 { 433 struct turnstile *ts; 434 435 ts = malloc(sizeof(struct turnstile), M_TURNSTILE, M_WAITOK | M_ZERO); 436 TAILQ_INIT(&ts->ts_blocked); 437 TAILQ_INIT(&ts->ts_pending); >How-To-Repeat: panic occurs in 1-3 days uptume >Fix: >Release-Note: >Audit-Trail: >Unformatted: