From owner-freebsd-bugs@FreeBSD.ORG Sat Apr 14 06:20:07 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CAFD9106566B for ; Sat, 14 Apr 2012 06:20:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A2F348FC0A for ; Sat, 14 Apr 2012 06:20:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3E6K7Tr051319 for ; Sat, 14 Apr 2012 06:20:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3E6K7a5051318; Sat, 14 Apr 2012 06:20:07 GMT (envelope-from gnats) Resent-Date: Sat, 14 Apr 2012 06:20:07 GMT Resent-Message-Id: <201204140620.q3E6K7a5051318@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Dennis Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C47FA106566C for ; Sat, 14 Apr 2012 06:18:13 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id AEC868FC0C for ; Sat, 14 Apr 2012 06:18:13 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3E6IDMd074729 for ; Sat, 14 Apr 2012 06:18:13 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q3E6IDYS074728; Sat, 14 Apr 2012 06:18:13 GMT (envelope-from nobody) Message-Id: <201204140618.q3E6IDYS074728@red.freebsd.org> Date: Sat, 14 Apr 2012 06:18:13 GMT From: Dennis To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/166937: [panic] Random and frequent kernel crash, reason unknown X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Apr 2012 06:20:07 -0000 >Number: 166937 >Category: kern >Synopsis: [panic] Random and frequent kernel crash, reason unknown >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Apr 14 06:20:07 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Dennis >Release: 8.2-STABLE >Organization: >Environment: FreeBSD utm.leskolovo.ru 8.2-STABLE FreeBSD 8.2-STABLE #4: Sun Mar 25 10:26:56 MSK 2012 root@utm.leskolovo.ru:/usr/obj/usr/src/sys/ROUTER_HOME_NETS amd64 >Description: Our server start frequently and randomly crashing and we don't know why. :( That server do billing (netflow, apache, mysql) and gateway (nat, policy-based routing with 2 WANs, ipfw, dummynet) for relatively small network (summary traffic up to 100 Mbit/sec) Part of crash.txt.* info: ------------------------------------------------------------------------- Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xffffffff8032b15f stack pointer = 0x28:0xffffff80ed216a50 frame pointer = 0x28:0xffffff80ed216aa0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (dummynet) trap number = 9 panic: general protection fault cpuid = 0 Uptime: 8d1h40m32s Dumping 667 out of 3318 MB:..3%..12%..22%..32%..41%..51%..63%..72%..82%..92% [...skipped....] #0 doadump () at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:224 #1 0xffffffff802629f0 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:441 #2 0xffffffff80262e73 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:614 #3 0xffffffff803dd08d in trap_fatal (frame=0xffffffff805d2840, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:825 #4 0xffffffff803dd54a in trap (frame=0xffffff80ed2169a0) at /usr/src/sys/amd64/amd64/trap.c:621 #5 0xffffffff803c4f14 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #6 0xffffffff8032b15f in dn_ht_scan_bucket (ht=0xffffff00069a3300, bucket=Variable "bucket" is not available. ) at /usr/src/sys/netinet/ipfw/dn_heap.c:537 #7 0xffffffff8032e4e3 in drain_scheduler_sch_cb (_s=Variable "_s" is not available. ) at /usr/src/sys/netinet/ipfw/ip_dummynet.c:1985 #8 0xffffffff8032b16a in dn_ht_scan_bucket (ht=0xffffff0001d1f400, bucket=Variable "bucket" is not available. ) at /usr/src/sys/netinet/ipfw/dn_heap.c:537 #9 0xffffffff8032e230 in dn_drain_scheduler () at /usr/src/sys/netinet/ipfw/ip_dummynet.c:2001 #10 0xffffffff80331a25 in dummynet_task (context=Variable "context" is not available. ) at /usr/src/sys/netinet/ipfw/ip_dn_io.c:608 #11 0xffffffff8029f325 in taskqueue_run_locked (queue=0xffffff0001d1aa80) at /usr/src/sys/kern/subr_taskqueue.c:250 #12 0xffffffff8029f4be in taskqueue_thread_loop (arg=Variable "arg" is not available. ) at /usr/src/sys/kern/subr_taskqueue.c:387 #13 0xffffffff80239cdf in fork_exit ( callout=0xffffffff8029f470 , arg=0xffffffff805fc080, frame=0xffffff80ed216c50) at /usr/src/sys/kern/kern_fork.c:876 #14 0xffffffff803c545e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #15 0x0000000000000000 in ?? () #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0xffffffff805f7038 in sleepq_chains () #40 0xffffff0001c87430 in ?? () #41 0x0000000000000000 in ?? () #42 0xffffff0001c87000 in ?? () #43 0xffffff80ed216b00 in ?? () #44 0xffffff80ed216aa8 in ?? () #45 0xffffff0001ff88c0 in ?? () #46 0xffffffff802879f2 in sched_switch (td=0xffffffff8029f470, newtd=0xffffffff805fc080, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1861 Previous frame inner to this frame (corrupt stack?) (kgdb) ------------------------------------------------------------------------- Shortly after change was made in net.isr.direct 1 -> 0: ------------------------------------------------------------------------- Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xffffffff8081b13e stack pointer = 0x28:0xffffff80000484c0 frame pointer = 0x28:0xffffff8000048520 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi1: netisr 0) trap number = 9 panic: general protection fault cpuid = 0 Uptime: 1d1h50m1s .. #0 doadump () at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:224 #1 0xffffffff802629f0 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:441 #2 0xffffffff80262e73 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:614 #3 0xffffffff803dd08d in trap_fatal (frame=0xffffff0001855000, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:825 #4 0xffffffff803dd54a in trap (frame=0xffffff8000048410) at /usr/src/sys/amd64/amd64/trap.c:621 #5 0xffffffff803c4f14 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #6 0xffffffff8081b13e in _FindLinkIn (la=0xffffff800096e000, dst_addr= {s_addr = 406912350}, alias_addr={s_addr = 4224137940}, dst_port=8645, alias_port=57626, link_type=6, replace_partial_links=1) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:1215 #7 0xffffffff8081b314 in FindLinkIn (la=0xffffff800096e000, dst_addr=Variable "dst_addr" is not available. ) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:1304 #8 0xffffffff8081b56b in FindUdpTcpIn (la=0xffffff800096e000, dst_addr=Variable "dst_addr" is not available. ) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:1522 #9 0xffffffff80818886 in LibAliasInLocked (la=0xffffff800096e000, ptr=0xffffff005e3af810 "E ", maxpacketsize=2032) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:924 #10 0xffffffff8081934d in LibAliasIn (la=0xffffff800096e000, ptr=0xffffff005e3af810 "E ", maxpacketsize=2032) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:1302 #11 0xffffffff808303b1 in ng_nat_rcvdata (hook=0xffffff0001dcde00, item=0xffffff0049feed00) at /usr/src/sys/modules/netgraph/nat/../../../netgraph/ng_nat.c:712 #12 0xffffffff80825bb0 in ng_apply_item (node=0xffffff004c107300, item=0xffffff0049feed00, rw=1) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2327 #13 0xffffffff80824c7e in ng_snd_item (item=Variable "item" is not available. ) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2244 #14 0xffffffff80339d07 in ipfw_check_hook (arg=Variable "arg" is not available. ) at /usr/src/sys/netinet/ipfw/ip_fw_pfil.c:225 #15 0xffffffff8031192c in pfil_run_hooks (ph=Variable "ph" is not available. ) at /usr/src/sys/net/pfil.c:82 #16 0xffffffff8033f07e in ip_input (m=0xffffff003f63f300) at /usr/src/sys/netinet/ip_input.c:532 #17 0xffffffff803112d9 in swi_net (arg=Variable "arg" is not available. ) at /usr/src/sys/net/netisr.c:653 #18 0xffffffff8023cab4 in intr_event_execute_handlers (p=Variable "p" is not available. ) at /usr/src/sys/kern/kern_intr.c:1216 #19 0xffffffff8023e145 in ithread_loop (arg=0xffffff00018537a0) at /usr/src/sys/kern/kern_intr.c:1229 #20 0xffffffff80239cdf in fork_exit ( callout=0xffffffff8023e0b0 , arg=0xffffff00018537a0, frame=0xffffff8000048c50) at /usr/src/sys/kern/kern_fork.c:876 #21 0xffffffff803c545e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000001 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000000 in ?? () #44 0x0000000000000000 in ?? () #45 0x0000000000000000 in ?? () #46 0xffffffff805dce00 in affinity () #47 0x0000000000000000 in ?? () #48 0x0000000000000000 in ?? () #49 0xffffff00018598c0 in ?? () #50 0xffffff8000047c30 in ?? () #51 0xffffff8000047bd8 in ?? () #52 0xffffff0001c88460 in ?? () #53 0xffffffff802879f2 in sched_switch (td=0xffffffff8023e0b0, newtd=0xffffff00018537a0, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1861 Previous frame inner to this frame (corrupt stack?) ------------------------------------------------------------------------- Once more: ------------------------------------------------------------------------- panic: page fault [...skipped (there was no unread portion of the kernel message buffer)...] #0 doadump () at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:224 #1 0xffffffff802629f0 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:441 #2 0xffffffff80262e73 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:614 #3 0xffffffff803dd08d in trap_fatal (frame=0xffffffff805d2840, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:825 #4 0xffffffff803dd3c1 in trap_pfault (frame=0xffffff80ed2363d0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:741 #5 0xffffffff803dd79f in trap (frame=0xffffff80ed2363d0) at /usr/src/sys/amd64/amd64/trap.c:478 #6 0xffffffff803c4f14 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #7 0xffffffff80819cf1 in DeleteLink (lnk=0xffffff009cffec80) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:859 #8 0xffffffff80819f01 in HouseKeeping (la=0xffffff8000980000) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias_db.c:849 #9 0xffffffff808175ab in LibAliasOutLocked (la=0xffffff8000980000, ptr=dwarf2_read_address: Corrupted DWARF expression. ) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:1451 #10 0xffffffff808185e4 in LibAliasOut (la=0xffffff8000980000, ptr=0xffffff009936a010 "E", maxpacketsize=2032) at /usr/src/sys/modules/libalias/libalias/../../../netinet/libalias/alias.c:1418 #11 0xffffffff80830348 in ng_nat_rcvdata (hook=0xffffff0001dcc580, item=0xffffff004e38eb00) at /usr/src/sys/modules/netgraph/nat/../../../netgraph/ng_nat.c:722 #12 0xffffffff80825bb0 in ng_apply_item (node=0xffffff0032ad6000, item=0xffffff004e38eb00, rw=1) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2327 #13 0xffffffff80824c7e in ng_snd_item (item=Variable "item" is not available. ) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2244 #14 0xffffffff80339d07 in ipfw_check_hook (arg=Variable "arg" is not available. ) at /usr/src/sys/netinet/ipfw/ip_fw_pfil.c:225 #15 0xffffffff8031192c in pfil_run_hooks (ph=Variable "ph" is not available. ) at /usr/src/sys/net/pfil.c:82 #16 0xffffffff80341410 in ip_output (m=0xffffff00324c8100, opt=Variable "opt" is not available. ) at /usr/src/sys/netinet/ip_output.c:511 #17 0xffffffff8033178f in dummynet_send (m=0xffffff00324c8100) at /usr/src/sys/netinet/ipfw/ip_dn_io.c:652 #18 0xffffffff80331996 in dummynet_task (context=Variable "context" is not available. ) at /usr/src/sys/netinet/ipfw/ip_dn_io.c:615 #19 0xffffffff8029f325 in taskqueue_run_locked (queue=0xffffff0001d18a80) at /usr/src/sys/kern/subr_taskqueue.c:250 #20 0xffffffff8029f4be in taskqueue_thread_loop (arg=Variable "arg" is not available. ) at /usr/src/sys/kern/subr_taskqueue.c:387 #21 0xffffffff80239cdf in fork_exit ( callout=0xffffffff8029f470 , arg=0xffffffff805fc080, frame=0xffffff80ed236c50) at /usr/src/sys/kern/kern_fork.c:876 #22 0xffffffff803c545e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000000 in ?? () #44 0x0000000000000000 in ?? () #45 0x0000000000000000 in ?? () #46 0x0000000000000000 in ?? () #47 0xffffffff805f6b38 in sleepq_chains () #48 0xffffff0001c87430 in ?? () #49 0x0000000000000000 in ?? () #50 0xffffff0001c87000 in ?? () #51 0xffffff80ed236b00 in ?? () #52 0xffffff80ed236aa8 in ?? () #53 0xffffff0001858000 in ?? () #54 0xffffffff802879f2 in sched_switch (td=0xffffffff8029f470, newtd=0xffffffff805fc080, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1861 Previous frame inner to this frame (corrupt stack?) (kgdb) ------------------------------------------------------------------------- And more: ------------------------------------------------------------------------- Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xffffffff8032e431 stack pointer = 0x28:0xffffff80ed236a90 frame pointer = 0x28:0xffffff80ed236ac0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (dummynet) trap number = 9 panic: general protection fault cpuid = 0 Uptime: 2d4h4m21s [...skipped...] #0 doadump () at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:224 #1 0xffffffff802629f0 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:441 #2 0xffffffff80262e73 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:614 #3 0xffffffff803dd08d in trap_fatal (frame=0xffffffff805d2840, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:825 #4 0xffffffff803dd54a in trap (frame=0xffffff80ed2369e0) at /usr/src/sys/amd64/amd64/trap.c:621 #5 0xffffffff803c4f14 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #6 0xffffffff8032e431 in drain_scheduler_cb (_si=0x1100006b8bc26b, arg=0x0) at /usr/src/sys/netinet/ipfw/ip_dummynet.c:1958 #7 0xffffffff8032e4a5 in drain_scheduler_sch_cb (_s=Variable "_s" is not available. ) at /usr/src/sys/netinet/ipfw/ip_dummynet.c:1990 #8 0xffffffff8032b16a in dn_ht_scan_bucket (ht=0xffffff0001d1d400, bucket=Variable "bucket" is not available. ) at /usr/src/sys/netinet/ipfw/dn_heap.c:537 #9 0xffffffff8032e230 in dn_drain_scheduler () at /usr/src/sys/netinet/ipfw/ip_dummynet.c:2001 #10 0xffffffff80331a25 in dummynet_task (context=Variable "context" is not available. ) at /usr/src/sys/netinet/ipfw/ip_dn_io.c:608 #11 0xffffffff8029f325 in taskqueue_run_locked (queue=0xffffff0001d18a80) at /usr/src/sys/kern/subr_taskqueue.c:250 #12 0xffffffff8029f4be in taskqueue_thread_loop (arg=Variable "arg" is not available. ) at /usr/src/sys/kern/subr_taskqueue.c:387 #13 0xffffffff80239cdf in fork_exit ( callout=0xffffffff8029f470 , arg=0xffffffff805fc080, frame=0xffffff80ed236c50) at /usr/src/sys/kern/kern_fork.c:876 #14 0xffffffff803c545e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #15 0x0000000000000000 in ?? () #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0xffffffff805f6b38 in sleepq_chains () #40 0xffffff0001c87430 in ?? () #41 0x0000000000000000 in ?? () #42 0xffffff0001c87000 in ?? () #43 0xffffff80ed236b00 in ?? () #44 0xffffff80ed236aa8 in ?? () #45 0xffffff00018598c0 in ?? () #46 0xffffffff802879f2 in sched_switch (td=0xffffffff8029f470, newtd=0xffffffff805fc080, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1861 Previous frame inner to this frame (corrupt stack?) (kgdb) ------------------------------------------------------------------------- kernel config: options CONFIG_AUTOGENERATED ident ROUTER_HOME_NETS machine amd64 cpu HAMMER makeoptions DEBUG=-g options HWPMC_HOOKS options ENABLE_ALART options KDB_UNATTENDED options HZ=1000 options DUMMYNET options IPDIVERT options IPFIREWALL_FORWARD options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_VERBOSE_LIMIT=400 options IPFIREWALL_VERBOSE options IPFIREWALL options CONSPEED=115200 options ATA_STATIC_ID options ADAPTIVE_LOCKMGRS options ACCEPT_FILTER_HTTP options ACCEPT_FILTER_DATA options INCLUDE_CONFIG_FILE options AUDIT options ZERO_COPY_SOCKETS options SC_HISTORY_SIZE=8192 options SC_KERNEL_CONS_ATTR=(FG_YELLOW|BG_BLACK) options SC_NORM_ATTR=(FG_GREEN|BG_BLACK) options SC_DISABLE_REBOOT options ROUTETABLES=4 options FLOWTABLE options KBD_INSTALL_CDEV options _KPOSIX_PRIORITY_SCHEDULING options SYSVSEM options SYSVMSG options SYSVSHM options STACK options KTRACE options GEOM_PART_GPT options GEOM_LABEL options PSEUDOFS options PROCFS options CD9660 options MSDOSFS options MD_ROOT options UFS_GJOURNAL options UFS_DIRHASH options SOFTUPDATES options FFS options INET options IPI_PREEMPTION options PREEMPTION options SCHED_ULE options SMP options GEOM_PART_MBR options GEOM_PART_EBR_COMPAT options GEOM_PART_EBR options GEOM_PART_BSD device isa device mem device io device uart_ns8250 device pci device acpi device ata device atadisk device atapicd device atkbdc device atkbd device psm device vga device sc device re device miibus device loop device random device ether device pty device md device bpf device intpm device iicbus device smbus device iicsmb device hwpmc /boot/loader.conf: autoboot_delay="1" if_em_load="YES" geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8)) vfs.root.mountfrom="ufs:/dev/mirror/root" cc_chd_load="YES" coretemp_load="YES" net.graph.maxdata=16384 net.isr.numthreads=2 net.isr.maxthreads=2 net.isr.bindthreads=1 hw.em.rxd=4096 hw.em.txd=4096 /etc/sysctl.conf: dev.em.0.rx_abs_int_delay=4000 dev.em.0.rx_int_delay=200 dev.em.0.rx_processing_limit=4096 dev.em.0.tx_abs_int_delay=4000 dev.em.0.tx_int_delay=200 kern.ipc.maxsockbuf=83886080 kern.ipc.nmbclusters=262144 net.graph.maxdgram=8388608 net.graph.recvspace=8388608 net.inet.icmp.drop_redirect=1 net.inet.icmp.icmplim=2048 net.inet.icmp.log_redirect=1 net.inet.ip.dummynet.expire=0 net.inet.ip.dummynet.hash_size=512 net.inet.ip.dummynet.io_fast=1 net.inet.ip.dummynet.pipe_byte_limit=2097152 net.inet.ip.dummynet.pipe_slot_limit=1000 net.inet.ip.fw.dyn_max=32768 net.inet.ip.fw.one_pass=0 net.inet.ip.fw.verbose=1 net.inet.ip.fw.verbose_limit=100 net.inet.ip.redirect=0 net.inet.tcp.blackhole=1 net.inet.udp.blackhole=1 net.isr.direct=0 net.isr.direct_force=0 net.link.ether.ipfw=0 ipfw list: 00100 allow ip from any to any via lo0 00200 allow ip from 10.1.0.0/24 to 10.1.0.0/24 00400 setfib 1 ip from any to any via vlan500 // WAN-2 00500 setfib 1 ip from table(2) to any // WAN-2 00600 setfib 1 ip from x.x.199.248/29 to any // WAN-2 00700 setfib 1 ip from any to x.x.199.248/29 // WAN-2 00800 fwd x.x.199.249 ip from x.x.199.250 to any xmit vlan720 // My traffic to WAN-2 00900 setfib 1 ip from x.x.27.0/24 to any // WAN-2's real ips 01000 count log logamount 29999 tcp from any to any dst-port 25 setup out recv vlan100 // Dumb antispam rule 01200 allow tcp from any to me dst-port 22 setup keep-state // SSH 01400 allow tcp from table(11) to me dst-port 20,21,22,3306 setup keep-state // FTP and HTTP for trusted clients 01600 allow tcp from any to me dst-port 443 setup keep-state // HTTPS for any clients (ACLs in Apache, for payments) 02000 fwd 127.0.0.1,3128 tcp from any to 212.193.229.77,92.241.171.120 dst-port 80 recv vlan100 // TProxy for real gismeteo.ru 03000 skipto 6000 ip from table(1) to any // Skip deny rules for allowed ips 03100 unreach filter-prohib ip from any to not me in recv vlan100 // Verbose deny for internal lan 04000 netgraph 4000 ip from any to me recv vlan720 // Inside NAT 04100 netgraph 4100 ip from any to x.x.199.250 recv vlan500 // Inside NAT 06000 pipe tablearg ip from any to table(4) xmit vlan100 06000 pipe tablearg ip from table(5) to any xmit vlan720 06000 pipe tablearg ip from table(5) to any xmit vlan500 06500 ngtee 1 ip from table(1) to any // Netflow Accounting from users 06600 ngtee 2 ip from any to table(1) in recv vlan720 // Netflow Accounting to users from WAN-1 06600 ngtee 2 ip from any to table(1) in recv vlan500 // Netflow Accounting to users from WAN-2 07000 netgraph 7000 ip from 10.1.0.0/16 to any xmit vlan720 // Outside NAT 07100 netgraph 7100 ip from 10.1.0.0/16 to any xmit vlan500 // Outside NAT 08000 allow ip from me to any xmit vlan720 // Allow all from me 08100 allow ip from x.x.199.248/29,x.x.27.0/24 to any xmit vlan500 08150 allow tcp from me to any setup keep-state // Tcp from me 08160 allow udp from any to me keep-state // Open my udp 08170 allow udp from me to any keep-state // Open my udp 08200 allow ip from any to any established 08300 allow ip from table(1) to any 08400 allow ip from any to table(1) 08500 allow icmp from any to any icmptypes 0,3,4,8,11 65530 deny log logamount 100 ip from any to any 65535 allow ip from any to any >How-To-Repeat: Don't know >Fix: >Release-Note: >Audit-Trail: >Unformatted: