From owner-freebsd-amd64@FreeBSD.ORG Tue Aug 25 17:40:01 2009 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B99A81065690 for ; Tue, 25 Aug 2009 17:40:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 942448FC18 for ; Tue, 25 Aug 2009 17:40:01 +0000 (UTC) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n7PHe1Ms044181 for ; Tue, 25 Aug 2009 17:40:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n7PHe1iH044180; Tue, 25 Aug 2009 17:40:01 GMT (envelope-from gnats) Resent-Date: Tue, 25 Aug 2009 17:40:01 GMT Resent-Message-Id: <200908251740.n7PHe1iH044180@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Frank Sheiness Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70053106568B for ; Tue, 25 Aug 2009 17:31:11 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 5E6528FC26 for ; Tue, 25 Aug 2009 17:31:11 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n7PHVB17094489 for ; Tue, 25 Aug 2009 17:31:11 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id n7PHVB4o094488; Tue, 25 Aug 2009 17:31:11 GMT (envelope-from nobody) Message-Id: <200908251731.n7PHVB4o094488@www.freebsd.org> Date: Tue, 25 Aug 2009 17:31:11 GMT From: Frank Sheiness To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Tue, 25 Aug 2009 17:44:23 +0000 Cc: Subject: amd64/138177: FreeBSD crashing repeatedly in pcpu.h:195 X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2009 17:40:01 -0000 >Number: 138177 >Category: amd64 >Synopsis: FreeBSD crashing repeatedly in pcpu.h:195 >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Aug 25 17:40:01 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Frank Sheiness >Release: 7.2-STABLE >Organization: >Environment: FreeBSD ROUTER 7.2-STABLE FreeBSD 7.2-STABLE #0: Mon Aug 24 20:47:37 UTC 2009 root@BUILD-7.0:/usr/obj/usr/src2/src/sys/BUILD amd64 >Description: We have FreeBSD servers with this code deployed at several sites. It is only crashing at one site, but on both servers (original and spare) at that site. After booting, the server is fine until the interrupts start climbing on the CPU. Eventually, the server panics. It can take a few minutes or hours before it crashes. We suspect some kind of traffic is triggering it. We blocked everything that has an ether type other than IPv4 or ARP on the switch and it hasn't crashed since (around 12 hours). Here is the backtrace: Unread portion of the kernel message buffer: ge not present instruction pointer = 0x8:0xffffffff801b8f68 stack pointer = 0x10:0xffffff800002d560 frame pointer = 0x10:0x14cb code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 15 (swi1: net) trap number = 12 panic: page fault cpuid = 0 Uptime: 20m47s Physical memory: 1004 MB Dumping 1181 MB: 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:195 195 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); (kgdb) where #0 doadump () at pcpu.h:195 #1 0x0000000000000004 in ?? () #2 0xffffffff804067fe in boot (howto=260) at /usr/src2/src/sys/kern/kern_shutdown.c:418 #3 0xffffffff80406c12 in panic (fmt=0x104
) at /usr/src2/src/sys/kern/kern_shutdown.c:574 #4 0xffffffff806a9f73 in trap_fatal (frame=0xffffff0001191390, eva=Variable "eva" is not available. ) at /usr/src2/src/sys/amd64/amd64/trap.c:756 #5 0xffffffff806aa345 in trap_pfault (frame=0xffffff800002d4b0, usermode=0) at /usr/src2/src/sys/amd64/amd64/trap.c:672 #6 0xffffffff806aac84 in trap (frame=0xffffff800002d4b0) at /usr/src2/src/sys/amd64/amd64/trap.c:443 #7 0xffffffff8068f73e in calltrap () at /usr/src2/src/sys/amd64/amd64/exception.S:218 #8 0xffffffff801b8f68 in nat_new (fin=0xffffff800002d700, np=Variable "np" is not available. ) at /usr/src2/src/sys/contrib/ipfilter/netinet/ip_nat.c:2577 #9 0xffffffff801bcac0 in fr_checknatout (fin=0xffffff800002d700, passp=0xffffff800002d7ec) at /usr/src2/src/sys/contrib/ipfilter/netinet/ip_nat.c:3828 #10 0xffffffff801b02aa in fr_check (ip=0x2, hlen=Variable "hlen" is not available. ) at /usr/src2/src/sys/contrib/ipfilter/netinet/fil.c:2624 #11 0xffffffff804d58a1 in pfil_run_hooks (ph=Variable "ph" is not available. ) at /usr/src2/src/sys/net/pfil.c:78 #12 0xffffffff8051d64d in ip_output (m=0xffffff00013b6100, opt=Variable "opt" is not available. ) at /usr/src2/src/sys/netinet/ip_output.c:443 #13 0xffffffff8051a63d in ip_forward (m=0xffffff00013b6100, srcrt=Variable "srcrt" is not available. ) at /usr/src2/src/sys/netinet/ip_input.c:1366 #14 0xffffffff8051be06 in ip_input (m=0xffffff00013b6100) at /usr/src2/src/sys/netinet/ip_input.c:609 #15 0xffffffff804c8561 in ether_demux (ifp=0xffffff000c771000, m=0xffffff00013b6100) at /usr/src2/src/sys/net/if_ethersubr.c:834 #16 0xffffffff804c87de in ether_input (ifp=0xffffff000c771000, m=0xffffff00013b6100) at /usr/src2/src/sys/net/if_ethersubr.c:692 #17 0xffffffff804c8486 in ether_demux (ifp=0xffffff00011a9800, m=0xffffff00013b6100) at /usr/src2/src/sys/net/if_ethersubr.c:743 #18 0xffffffff804c87de in ether_input (ifp=0xffffff00011a9800, m=0xffffff00013b6100) at /usr/src2/src/sys/net/if_ethersubr.c:692 #19 0xffffffff80257b02 in em_rxeof (adapter=0xffffff8000211000, count=2) at /usr/src2/src/sys/dev/e1000/if_em.c:4539 #20 0xffffffff80259c19 in em_poll (ifp=0xffffff00011a9800, cmd=Variable "cmd" is not available. ) at /usr/src2/src/sys/dev/e1000/if_em.c:1594 #21 0xffffffff803fb1ad in netisr_poll () at /usr/src2/src/sys/kern/kern_poll.c:432 #22 0xffffffff804d3c1a in swi_net (dummy=Variable "dummy" is not available. ) at /usr/src2/src/sys/net/netisr.c:254 #23 0xffffffff803e22a0 in ithread_loop (arg=0xffffff000117dae0) at /usr/src2/src/sys/kern/kern_intr.c:1126 #24 0xffffffff803de94f in fork_exit ( callout=0xffffffff803e2130 , arg=0xffffff000117dae0, frame=0xffffff800002dc80) at /usr/src2/src/sys/kern/kern_fork.c:811 #25 0xffffffff8068fb1e in fork_trampoline () at /usr/src2/src/sys/amd64/amd64/exception.S:554 #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000001 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000000 in ?? () #44 0x0000000000000000 in ?? () #45 0x0000000000000000 in ?? () #46 0x0000000000000000 in ?? () #47 0x0000000000000000 in ?? () #48 0x0000000000000000 in ?? () #49 0x0000000000000000 in ?? () #50 0x0000000000af4000 in ?? () #51 0xffffffff80981cc0 in tdg_maxid () #52 0xffffffff8098e4c0 in tdq_cpu () #53 0xffffffff8098e4c0 in tdq_cpu () #54 0xffffff0001191390 in ?? () #55 0xffffff00011916e0 in ?? () #56 0xffffff800002cd08 in ?? () #57 0x0000000000000000 in ?? () #58 0xffffffff8042e2d8 in sched_switch (td=0xffffffff803e2130, newtd=0x80054b1d0, flags=Variable "flags" is not available. ) at /usr/src2/src/sys/kern/sched_ule.c:1938 #59 0x0000000000000000 in ?? () #60 0x0000000000000000 in ?? () #61 0x0000000000000000 in ?? () #62 0x0000000000000000 in ?? () #63 0x0000000000000000 in ?? () #64 0x0000000000000000 in ?? () #65 0x0000000000000000 in ?? () #66 0x0000000000000000 in ?? () #67 0x0000000000000000 in ?? () #68 0x0000000000000000 in ?? () #69 0x0000000000000000 in ?? () #70 0x0000000000000000 in ?? () #71 0x0000000000000000 in ?? () #72 0x0000000000000000 in ?? () #73 0x0000000000000000 in ?? () #74 0x0000000000000000 in ?? () #75 0x0000000000000000 in ?? () #76 0x0000000000000000 in ?? () #77 0x0000000000000000 in ?? () #78 0x0000000000000000 in ?? () #79 0x0000000000000000 in ?? () #80 0x0000000000000000 in ?? () #81 0x0000000000000000 in ?? () #82 0x0000000000000000 in ?? () #83 0x0000000000000000 in ?? () #84 0x0000000000000000 in ?? () #85 0x0000000000000000 in ?? () #86 0x0000000000000000 in ?? () #87 0x0000000000000000 in ?? () #88 0x0000000000000000 in ?? () #89 0x0000000000000000 in ?? () #90 0x0000000000000000 in ?? () #91 0x0000000000000000 in ?? () #92 0x0000000000000000 in ?? () #93 0x0000000000000000 in ?? () #94 0x0000000000000000 in ?? () #95 0x0000000000000000 in ?? () #96 0x0000000000000000 in ?? () #97 0x0000000000000000 in ?? () #98 0x0000000000000000 in ?? () #99 0x0000000000000000 in ?? () #100 0x0000000000000000 in ?? () #101 0x0000000000000000 in ?? () #102 0x0000000000000000 in ?? () #103 0x0000000000000000 in ?? () #104 0x0000000000000000 in ?? () #105 0x0000000000000000 in ?? () #106 0x0000000000000000 in ?? () #107 0x0000000000000000 in ?? () #108 0x0000000000000000 in ?? () #109 0x0000000000000000 in ?? () #110 0x0000000000000000 in ?? () #111 0x0000000000000000 in ?? () #112 0x0000000000000000 in ?? () #113 0x0000000000000000 in ?? () #114 0x0000000000000000 in ?? () #115 0x0000000000000000 in ?? () #116 0x0000000000000000 in ?? () #117 0x0000000000000000 in ?? () #118 0x0000000000000000 in ?? () #119 0x0000000000000000 in ?? () #120 0x0000000000000000 in ?? () #121 0x0000000000000000 in ?? () #122 0x0000000000000000 in ?? () #123 0x0000000000000000 in ?? () #124 0x0000000000000000 in ?? () #125 0x0000000000000000 in ?? () #126 0x0000000000000000 in ?? () Cannot access memory at address 0xffffff800002e000 >How-To-Repeat: Unknown >Fix: Unknown >Release-Note: >Audit-Trail: >Unformatted: