From owner-freebsd-bugs@FreeBSD.ORG Sun Dec 30 08:40:00 2012 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C73D4D85 for ; Sun, 30 Dec 2012 08:40:00 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 9B7278FC0A for ; Sun, 30 Dec 2012 08:40:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qBU8e0pQ083471 for ; Sun, 30 Dec 2012 08:40:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qBU8e0nW083470; Sun, 30 Dec 2012 08:40:00 GMT (envelope-from gnats) Resent-Date: Sun, 30 Dec 2012 08:40:00 GMT Resent-Message-Id: <201212300840.qBU8e0nW083470@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Adam Twardowski Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1BB8BD83 for ; Sun, 30 Dec 2012 08:39:43 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id E96138FC12 for ; Sun, 30 Dec 2012 08:39:42 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.5/8.14.5) with ESMTP id qBU8dghm070035 for ; Sun, 30 Dec 2012 08:39:42 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.5/8.14.5/Submit) id qBU8dgj5070034; Sun, 30 Dec 2012 08:39:42 GMT (envelope-from nobody) Message-Id: <201212300839.qBU8dgj5070034@red.freebsd.org> Date: Sun, 30 Dec 2012 08:39:42 GMT From: Adam Twardowski To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: kern/174822: Page fault in tcp_discardcb under high traffic X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2012 08:40:00 -0000 >Number: 174822 >Category: kern >Synopsis: Page fault in tcp_discardcb under high traffic >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Dec 30 08:40:00 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Adam Twardowski >Release: 9.1-RELEASE >Organization: >Environment: FreeBSD ID11278.choopa.com 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >Description: Random kernel page fault's under high network traffic traffic. I'm running haproxy 1.4.22 to load balance traffic to a cluster of web servers. I'm seeing the same error on multiple machines and I already ran memtest to verify that there wasn't any issues with the memory. [root@ID11278 /var/crash]# kgdb /boot/kernel/kernel.symbols vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x28 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80a7a951 stack pointer = 0x28:0xffffff88c1836780 frame pointer = 0x28:0xffffff88c1836800 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 22494 (haproxy) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff809208a6 at kdb_backtrace+0x66 #1 0xffffffff808ea8be at panic+0x1ce #2 0xffffffff80bd8240 at trap_fatal+0x290 #3 0xffffffff80bd857d at trap_pfault+0x1ed #4 0xffffffff80bd8b9e at trap+0x3ce #5 0xffffffff80bc315f at calltrap+0x8 #6 0xffffffff80a80ffa at tcp_usr_detach+0x11a #7 0xffffffff80952f84 at sofree+0x134 #8 0xffffffff80953a81 at soclose+0x3c1 #9 0xffffffff808a9653 at _fdrop+0x23 #10 0xffffffff808aa952 at closef+0x52 #11 0xffffffff808ab192 at kern_close+0x172 #12 0xffffffff80bd7ae6 at amd64_syscall+0x546 #13 0xffffffff80bc3447 at Xfast_syscall+0xf7 Uptime: 3d2h59m9s Dumping 1980 out of 32726 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_mirror.ko Reading symbols from /boot/modules/dscp_rewrite.ko...done. Loaded symbols for /boot/modules/dscp_rewrite.ko Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done. done. Loaded symbols for /boot/kernel/ipfw.ko #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 #1 0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 #2 0xffffffff808ea897 in panic (fmt=0x1
) at /usr/src/sys/kern/kern_shutdown.c:636 #3 0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0xffffffff80bd857d in trap_pfault (frame=0xffffff88c18366d0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773 #5 0xffffffff80bd8b9e in trap (frame=0xffffff88c18366d0) at /usr/src/sys/amd64/amd64/trap.c:456 #6 0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #7 0xffffffff80a7a951 in tcp_discardcb (tp=0x0) at /usr/src/sys/netinet/tcp_subr.c:839 #8 0xffffffff80a80ffa in tcp_usr_detach (so=Variable "so" is not available. ) at /usr/src/sys/netinet/tcp_usrreq.c:205 #9 0xffffffff80952f84 in sofree (so=0xfffffe05e6dbe7f8) at /usr/src/sys/kern/uipc_socket.c:647 #10 0xffffffff80953a81 in soclose (so=0xfffffe05e6dbe7f8) at /usr/src/sys/kern/uipc_socket.c:741 #11 0xffffffff808a9653 in _fdrop (fp=0xfffffe01d5f18280, td=Variable "td" is not available. ) at file.h:316 #12 0xffffffff808aa952 in closef (fp=0xfffffe01d5f18280, td=0xfffffe000fcd28e0) at /usr/src/sys/kern/kern_descrip.c:2250 #13 0xffffffff808ab192 in kern_close (td=0xfffffe000fcd28e0, fd=Variable "fd" is not available. ) at /usr/src/sys/kern/kern_descrip.c:1236 #14 0xffffffff80bd7ae6 in amd64_syscall (td=0xfffffe000fcd28e0, traced=0) at subr_syscall.c:135 #15 0xffffffff80bc3447 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387 #16 0x0000000800ff1cfc in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) >How-To-Repeat: Not easily repeatable, but it seems to happen to me every couple of days at least. >Fix: >Release-Note: >Audit-Trail: >Unformatted: