From owner-freebsd-bugs@FreeBSD.ORG Tue Oct 24 18:40:26 2006 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4521E16A4B3 for ; Tue, 24 Oct 2006 18:40:26 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id DAB9243D49 for ; Tue, 24 Oct 2006 18:40:23 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k9OIeImQ014309 for ; Tue, 24 Oct 2006 18:40:18 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k9OIeIFT014305; Tue, 24 Oct 2006 18:40:18 GMT (envelope-from gnats) Resent-Date: Tue, 24 Oct 2006 18:40:18 GMT Resent-Message-Id: <200610241840.k9OIeIFT014305@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Kai Gallasch Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2B93B16A412 for ; Tue, 24 Oct 2006 18:36:01 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4CB5D43DC0 for ; Tue, 24 Oct 2006 18:35:46 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k9OIZj8W048674 for ; Tue, 24 Oct 2006 18:35:45 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id k9OIZjBR048673; Tue, 24 Oct 2006 18:35:45 GMT (envelope-from nobody) Message-Id: <200610241835.k9OIZjBR048673@www.freebsd.org> Date: Tue, 24 Oct 2006 18:35:45 GMT From: Kai Gallasch To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.0 Cc: Subject: kern/104765: kernel panic 6.2 prerelease-20061017 amd64 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Oct 2006 18:40:26 -0000 >Number: 104765 >Category: kern >Synopsis: kernel panic 6.2 prerelease-20061017 amd64 >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Oct 24 18:40:17 GMT 2006 >Closed-Date: >Last-Modified: >Originator: Kai Gallasch >Release: 6.2 prerelease (checkout 20061017) >Organization: FREE! >Environment: FreeBSD geldkraft.free.de 6.2-PRERELEASE FreeBSD 6.2-PRERELEASE #0: Sun Oct 22 13:36:38 CEST 2006 houdini@geldkraft.free.de:/usr/obj/usr/src/sys/SMP amd64 >Description: Kernel panics after 1-3 days uptime with trap number 12 - page fault. kernel config: -------------- GENERIC (SMP) with "makeoptions DEBUG=-g" $FreeBSD: src/sys/amd64/conf/GENERIC,v 1.439.2.14 2006/10/09 18:41:36 simon Exp $ Hardware: --------- HP/Compaq DL385 Dual Opteron (Dual Core) with ServeRaid 6 (Raid 5) and 1G RAM. dmesg: ------ Copyright (c) 1992-2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-PRERELEASE #0: Sun Oct 22 13:36:38 CEST 2006 houdini@geldkraft.free.de:/usr/obj/usr/src/sys/SMP ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Opteron(tm) Processor 280 (2405.47-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x20f12 Stepping = 2 Features=0x178bfbff Features2=0x1 AMD Features=0xe2500800 AMD Features2=0x2 Cores per package: 2 real memory = 1073709056 (1023 MB) avail memory = 1023938560 (976 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-27 on motherboard ioapic2 irqs 28-31 on motherboard ioapic3 irqs 32-35 on motherboard ioapic4 irqs 36-39 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: on acpi0 pci0: on pcib0 pcib1: at device 3.0 on pci0 pci1: on pcib1 ohci0: mem 0xf7df0000-0xf7df0fff irq 19 at device 0.0 on pci1 ohci0: [GIANT-LOCKED] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 3 ports with 3 removable, self powered ohci1: mem 0xf7de0000-0xf7de0fff irq 19 at device 0.1 on pci1 ohci1: [GIANT-LOCKED] usb1: OHCI version 1.0, legacy support usb1: SMM does not respond, resetting usb1: on ohci1 usb1: USB revision 1.0 uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 3 ports with 3 removable, self powered pci1: at device 2.0 (no driver attached) pci1: at device 2.2 (no driver attached) pci1: at device 3.0 (no driver attached) isab0: at device 4.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2000-0x200f at device 4.1 on pci0 ata0: on atapci0 ata1: on atapci0 pci0: at device 4.3 (no driver attached) pcib2: at device 7.0 on pci0 pci2: on pcib2 ciss0: port 0x5000-0x50ff mem 0xf7ef0000-0xf7ef1fff,0xf7e80000-0xf7ebffff irq 24 at device 4.0 on pci2 ciss0: [GIANT-LOCKED] pci0: at device 7.1 (no driver attached) pcib3: at device 8.0 on pci0 pci3: on pcib3 bge0: mem 0xf7ff0000-0xf7ffffff irq 28 at device 6.0 on pci3 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge0: Ethernet address: 00:17:a4:8f:27:68 bge1: mem 0xf7fe0000-0xf7feffff irq 29 at device 6.1 on pci3 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge1: Ethernet address: 00:17:a4:8f:27:67 pci0: at device 8.1 (no driver attached) pcib4: on acpi0 pci4: on pcib4 pcib5: at device 9.0 on pci4 pci5: on pcib5 pci4: at device 9.1 (no driver attached) pcib6: at device 10.0 on pci4 pci6: on pcib6 pci4: at device 10.1 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: flags 0x1 irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse, device ID 3 sio0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console fdc0: port 0x3f2-0x3f5 irq 6 drq 2 on acpi0 fdc0: [FAST] orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xee000-0xeffff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: CDROM at ata0-master PIO4 SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! da0 at ciss0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da0: 135.168MB/s transfers da0: 17200MB (35226720 512 byte sectors: 255H 32S/T 4317C) da1 at ciss0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-0 device da1: 135.168MB/s transfers da1: 17200MB (35226720 512 byte sectors: 255H 32S/T 4317C) da2 at ciss0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-0 device da2: 135.168MB/s transfers da2: 69499MB (142334880 512 byte sectors: 255H 32S/T 17443C) da3 at ciss0 bus 0 target 3 lun 0 da3: Fixed Direct Access SCSI-0 device da3: 135.168MB/s transfers da3: 69499MB (142334880 512 byte sectors: 255H 32S/T 17443C) da4 at ciss0 bus 0 target 4 lun 0 da4: Fixed Direct Access SCSI-0 device da4: 135.168MB/s transfers da4: 139799MB (286309920 512 byte sectors: 255H 32S/T 35087C) backtrace: ---------- [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: d, page not present instruction pointer = 0x8:0xffffffff803eea47 stack pointer = 0x10:0xffffffffa814a8b0 frame pointer = 0x10:0x4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 27596 (tcpserver) trap number = 12 panic: page fault cpuid = 3 Uptime: 2h12m0s Dumping 1023 MB (2 chunks) chunk 0: 1MB (156 pages) ... ok chunk 1: 1023MB (261880 pages) 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 #0 doadump () at pcpu.h:172 172 pcpu.h: No such file or directory. in pcpu.h (kgdb) quit geldkraft:/etc # mount /usr/src/ geldkraft:/etc # cd /usr/src/sys/amd64/conf/ geldkraft:/usr/src/sys/amd64/conf # kgdb SMP /var/crash/vmcore.0 kgdb: bad namelist - no kernbase geldkraft:/usr/src/sys/amd64/conf # kgdb /boot/kernel/kernel /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: d, page not present instruction pointer = 0x8:0xffffffff803eea47 stack pointer = 0x10:0xffffffffa814a8b0 frame pointer = 0x10:0x4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 27596 (tcpserver) trap number = 12 panic: page fault cpuid = 3 Uptime: 2h12m0s Dumping 1023 MB (2 chunks) chunk 0: 1MB (156 pages) ... ok chunk 1: 1023MB (261880 pages) 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 #0 doadump () at pcpu.h:172 172 pcpu.h: No such file or directory. in pcpu.h (kgdb) list *0xffffffff803eea47 0xffffffff803eea47 is in _mtx_lock_sleep (/usr/src/sys/kern/kern_mutex.c:548). 543 * If the current owner of the lock is executing on another 544 * CPU, spin instead of blocking. 545 */ 546 owner = (struct thread *)(v & MTX_FLAGMASK); 547 #ifdef ADAPTIVE_GIANT 548 if (TD_IS_RUNNING(owner)) { 549 #else 550 if (m != &Giant && TD_IS_RUNNING(owner)) { 551 #endif 552 turnstile_release(&m->mtx_object); (kgdb) backtrace #0 doadump () at pcpu.h:172 #1 0x0000000000000004 in ?? () #2 0xffffffff803f8fd7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff803f9671 in panic (fmt=0xffffff0002116980 "X?J:") at /usr/src/sys/kern/kern_shutdown.c:565 #4 0xffffffff80618b3f in trap_fatal (frame=0xffffff0002116980, eva=18446742975175902040) at /usr/src/sys/amd64/amd64/trap.c:660 #5 0xffffffff80619066 in trap (frame= {tf_rdi = 11, tf_rsi = -1099476932224, tf_rdx = 6, tf_rcx = 0, tf_r8 = 4, tf_r9 = -1098475933086, tf_rax = 1, tf_rbx = -1099415090280, tf_rbp = 4, tf_r10 = 4, tf_r11 = 4, tf_r12 = -1099476932224, tf_r13 = -1098728017152, tf_r14 = 0, tf_r15 = 1, tf_trapno = 12, tf_addr = 396, tf_flags = -2141616351, tf_err = 0, tf_rip = -2143360441, tf_cs = 8, tf_rflags = 65538, tf_rsp = -1475041088, tf_ss = 16}) at /usr/src/sys/amd64/amd64/trap.c:238 #6 0xffffffff8060442b in calltrap () at /usr/src/sys/amd64/amd64/exception.S:168 #7 0xffffffff803eea47 in _mtx_lock_sleep (m=0xffffff0005c10b98, tid=18446742974232619392, opts=6, file=0x0, line=4) at /usr/src/sys/kern/kern_mutex.c:546 #8 0xffffffff804bb51d in ip_ctloutput (so=0xb, sopt=0xffffffffa814ab30) at /usr/src/sys/netinet/ip_output.c:1193 #9 0xffffffff804ccad5 in tcp_ctloutput (so=0xffffff0024a0d268, sopt=0xffffffffa814ab30) at /usr/src/sys/netinet/tcp_usrreq.c:1038 #10 0xffffffff804416b8 in sosetopt (so=0xffffff0024a0d268, sopt=0xffffffffa814ab30) at /usr/src/sys/kern/uipc_socket.c:1563 #11 0xffffffff80447b93 in kern_setsockopt (td=0xffffff0002116980, s=616888072, level=4, name=0, val=0x4, valseg=1035694690, valsize=11) at /usr/src/sys/kern/uipc_syscalls.c:1351 #12 0xffffffff80447bfe in setsockopt (td=0xb, uap=0xffffff0002116980) at /usr/src/sys/kern/uipc_syscalls.c:1307 #13 0xffffffff80619991 in syscall (frame= {tf_rdi = 0, tf_rsi = 0, tf_rdx = 1, tf_rcx = 0, tf_r8 = 0, tf_r9 = 140737488350072, tf_rax = 105, tf_rbx = 0, tf_rbp = 3, tf_r10 = -3689348814741910323, tf_r11 = 514, tf_r12 = 140737488350480, tf_r13 = 34368406752, tf_r14 = 0, tf_r15 = 0, tf_trapno = 12, tf_addr = 5283944, tf_flags = 12, tf_err = 2, tf_rip = 34366834188, tf_cs = 43, tf_rflags = 518, tf_rsp = 140737488350184, tf_ss = 35}) at /usr/src/sys/amd64/amd64/trap.c:792 #14 0xffffffff806045c8 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:270 #15 0x00000008006c460c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) >How-To-Repeat: problem occurs in between 1-3 days uptime of server >Fix: Raising some sysctl values seems to lengthen the intervals between crashes. Although I might be mistaken that tweaking them has some effect on the problem. # default war 12328 #kern.maxfiles=80000 # default 128 #kern.ipc.somaxconn=384 # default war 11095 #kern.maxfilesperproc=50000 >Release-Note: >Audit-Trail: >Unformatted: