From owner-freebsd-bugs@FreeBSD.ORG Tue Jan 4 05:30:13 2011 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C597F1065675 for ; Tue, 4 Jan 2011 05:30:13 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8454F8FC1A for ; Tue, 4 Jan 2011 05:30:13 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p045UDCD043374 for ; Tue, 4 Jan 2011 05:30:13 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p045UDNG043373; Tue, 4 Jan 2011 05:30:13 GMT (envelope-from gnats) Resent-Date: Tue, 4 Jan 2011 05:30:13 GMT Resent-Message-Id: <201101040530.p045UDNG043373@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, eugen Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DBC3106566B for ; Tue, 4 Jan 2011 05:21:12 +0000 (UTC) (envelope-from eugen@k-45-pc-1.sd.rdtc.ru) Received: from k-45-pc-1.sd.rdtc.ru (k-45-pc-1.sd.rdtc.ru [62.231.160.108]) by mx1.freebsd.org (Postfix) with ESMTP id B9CDC8FC14 for ; Tue, 4 Jan 2011 05:21:09 +0000 (UTC) Received: from k-45-pc-1.sd.rdtc.ru (localhost [127.0.0.1]) by k-45-pc-1.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id p045L7iF061222 for ; Tue, 4 Jan 2011 11:21:07 +0600 (NOVT) (envelope-from eugen@k-45-pc-1.sd.rdtc.ru) Received: (from eugen@localhost) by k-45-pc-1.sd.rdtc.ru (8.14.4/8.14.4/Submit) id p045L2pJ061221; Tue, 4 Jan 2011 11:21:02 +0600 (NOVT) (envelope-from eugen) Message-Id: <201101040521.p045L2pJ061221@k-45-pc-1.sd.rdtc.ru> Date: Tue, 4 Jan 2011 11:21:02 +0600 (NOVT) From: eugen To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: Subject: kern/153671: [em] [panic] 8.2-PRERELEASE repeatable kernel in if_em(4) X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: eugen List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2011 05:30:13 -0000 >Number: 153671 >Category: kern >Synopsis: [em] [panic] 8.2-PRERELEASE repeatable kernel in if_em(4) >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Jan 04 05:30:13 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Eugene Grosbein >Release: FreeBSD 8.2-PRERELEASE amd64 >Organization: RDTC JSC >Environment: System: FreeBSD k-45-pc-1.sd.rdtc.ru 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #4: Thu Dec 30 15:02:04 NOVT 2010 root@k-45-pc-1.sd.rdtc.ru:/usr/local/obj/home/src/sys/PPPOE amd64 >Description: 8.2-PRERELEASE with stock em(4) driver version 7.1.8 panices under moderate network load in the driver. The system is PPPoE network access server using mpd-5.5 and NETGRAPH. User sessions are created/deleted quite quickly. The system has only two up and running interfaces corresponding physical NICs: em0 is uplink having IP address, em1 has not IP address but has many vlans on it carrying PPPoE frames. vlans do not have IP addresses too. Both interfaces are configured in /etc/rc.conf as such: ifconfig_em0="inet $myip netmask 255.255.255.248 -wol" ifconfig_em1="-wol" I've got full crashdump for total 4GB of router memory. Here comes gdb backtrace. Then dmesg output, kernel config file, loader.conf and sysctl.conf. Script started on Tue Jan 4 10:56:28 2011 kgdb kernel.debug /var/crash/vmcore.1 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 04 fault virtual address = 0x1400000001 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80349eda stack pointer = 0x28:0xffffff80e77929b0 frame pointer = 0x28:0xffffff80e77929c0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (irq264: em1:tx 0) trap number = 12 panic: page fault cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff8019eaac = db_trace_self_wrapper+0x2a kdb_backtrace() at 0xffffffff8031e4cc = kdb_backtrace+0x37 panic() at 0xffffffff802ee7b8 = panic+0x182 trap_fatal() at 0xffffffff8048aca0 = trap_fatal+0x292 trap_pfault() at 0xffffffff8048b055 = trap_pfault+0x286 trap() at 0xffffffff8048b4ff = trap+0x3cb calltrap() at 0xffffffff80473e54 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff80349eda, rsp = 0xffffff80e77929b0, rbp = 0xffffff80e77929c0 --- m_tag_delete() at 0xffffffff80349eda = m_tag_delete+0x1a m_tag_delete_chain() at 0xffffffff80349f10 = m_tag_delete_chain+0x1c uma_zfree_arg() at 0xffffffff80450247 = uma_zfree_arg+0x3a m_freem() at 0xffffffff80347a58 = m_freem+0x51 em_txeof() at 0xffffffff801dd6a4 = em_txeof+0x15f em_msix_tx() at 0xffffffff801dfaff = em_msix_tx+0x4a intr_event_execute_handlers() at 0xffffffff802c9f33 = intr_event_execute_handlers+0xfc ithread_loop() at 0xffffffff802cb4d6 = ithread_loop+0x95 fork_exit() at 0xffffffff802c790d = fork_exit+0x118 fork_trampoline() at 0xffffffff8047432e = fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80e7792cf0, rbp = 0 --- Uptime: 4d6h2m45s Dumping 3893 MB (6 chunks) chunk 0: 1MB (151 pages) ... ok chunk 1: 3251MB (832247 pages) 3236 3220 3204 3188 3172 3156 3140 3124 3108 3092 3076 3060 3044 3028 3012 2996 2980 2964 2948 2932 2916 2900 2884 2868 2852 2836 2820 2804 2788 2772 2756 2740 2724 2708 2692 2676 2660 2644 2628 2612 2596 2580 2564 2548 2532 2516 2500 2484 2468 2452 2436 2420 2404 2388 2372 2356 2340 2324 2308 2292 2276 2260 2244 2228 2212 2196 2180 2164 2148 2132 2116 2100 2084 2068 2052 2036 2020 2004 1988 1972 1956 1940 1924 1908 1892 1876 1860 1844 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652 1636 1620 1604 1588 1572 1556 1540 1524 1508 1492 1476 1460 1444 1428 1412 1396 1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204 1188 1172 1156 1140 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676 660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84 68 52 36 20 4 ... ok chunk 2: 1MB (2 pages) ... ok chunk 3: 1MB (1 pages) ... ok chunk 4: 2MB (381 pages) ... ok chunk 5: 640MB (163840 pages) 625 609 593 577 561 545 529 513 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1 Reading symbols from /boot/modules/if_lagg.ko...done. Loaded symbols for /boot/modules/if_lagg.ko #0 doadump () at pcpu.h:224 224 __asm("movq %%gs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:224 #1 0xffffffff802ee333 in boot (howto=260) at /home/src/sys/kern/kern_shutdown.c:419 #2 0xffffffff802ee7a2 in panic (fmt=Variable "fmt" is not available. ) at /home/src/sys/kern/kern_shutdown.c:592 #3 0xffffffff8048aca0 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. ) at /home/src/sys/amd64/amd64/trap.c:783 #4 0xffffffff8048b055 in trap_pfault (frame=0xffffff80e7792900, usermode=0) at /home/src/sys/amd64/amd64/trap.c:699 #5 0xffffffff8048b4ff in trap (frame=0xffffff80e7792900) at /home/src/sys/amd64/amd64/trap.c:449 #6 0xffffffff80473e54 in calltrap () at /home/src/sys/amd64/amd64/exception.S:224 #7 0xffffffff80349eda in m_tag_delete (m=0xffffff007a9cd100, t=0x1400000001) at mbuf.h:991 #8 0xffffffff80349f10 in m_tag_delete_chain (m=0xffffff007a9cd100, t=0x1400000001) at /home/src/sys/kern/uipc_mbuf2.c:356 #9 0xffffffff80450247 in uma_zfree_arg (zone=0xffffff00cb3eb300, item=0xffffff007a9cd100, udata=0x0) at /home/src/sys/vm/uma_core.c:2548 #10 0xffffffff80347a58 in m_freem (mb=0xffffff0005d7be00) at uma.h:333 #11 0xffffffff801dd6a4 in em_txeof (txr=0xffffff00019ee400) at /home/src/sys/dev/e1000/if_em.c:3714 #12 0xffffffff801dfaff in em_msix_tx (arg=Variable "arg" is not available. ) at /home/src/sys/dev/e1000/if_em.c:1540 #13 0xffffffff802c9f33 in intr_event_execute_handlers (p=Variable "p" is not available. ) at /home/src/sys/kern/kern_intr.c:1220 #14 0xffffffff802cb4d6 in ithread_loop (arg=0xffffff0001b897c0) at /home/src/sys/kern/kern_intr.c:1233 #15 0xffffffff802c790d in fork_exit (callout=0xffffffff802cb441 , arg=0xffffff0001b897c0, frame=0xffffff80e7792c40) at /home/src/sys/kern/kern_fork.c:845 #16 0xffffffff8047432e in fork_trampoline () at /home/src/sys/amd64/amd64/exception.S:566 #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000001 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0xffffffff806e9240 in affinity () #42 0x0000000000000000 in ?? () #43 0x0000000000000000 in ?? () #44 0xffffff0001b86460 in ?? () ---Type to continue, or q to quit--- #45 0xffffff80e7792560 in ?? () #46 0xffffff80e7792508 in ?? () #47 0xffffff0001e00000 in ?? () #48 0xffffffff80312544 in sched_switch (td=0xffffffff802cb441, newtd=0xffffff0001b897c0, flags=Variable "flags" is not available. ) at /home/src/sys/kern/sched_ule.c:1852 Previous frame inner to this frame (corrupt stack?) (kgdb) frame 11 #11 0xffffffff801dd6a4 in em_txeof (txr=0xffffff00019ee400) at /home/src/sys/dev/e1000/if_em.c:3714 3714 m_freem(tx_buffer->m_head); (kgdb) p *tx_buffer->m_head $1 = {m_hdr = {mh_next = 0xffffff0005d7be00, mh_nextpkt = 0x0, mh_data = 0xffffff007a9cd1e8 "", mh_len = 20, mh_flags = 65538, mh_type = 1, pad = "\000\000\000\000\000"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xffffff0001915800, header = 0x0, len = 73, flowid = 0, csum_flags = 3840, csum_data = 65535, tso_segsz = 0, PH_vt = {vt_vtag = 393, vt_nrecs = 393}, tags = {slh_first = 0xffffff007ad33900}}, MH_dat = {MH_ext = { ext_buf = 0x40b536d4050045
, ext_free = 0x79f224d4e73e067f, ext_arg1 = 0x4c07c50ec4e524d4, ext_arg2 = 0xc2d0137048e5aadd, ext_size = 3045003344, ref_cnt = 0x137900001378, ext_type = 0}, MH_databuf = "E\000\005Ô6µ@\000\177\006>çÔ$òyÔ$åÄ\016Å\aLݪåHp\023ÐÂP\020\177µü\000\000\000x\023\000\000y\023", '\0' , "î\035\000\000\fS\026\000\2030\000\000Çß\024", '\0' , "ÿ\023\021\036ì\017½\000þ\022TE\002\032±m'\232\223\224·þ\026ú\aêéªÍ|\004ó\214\025\v, "î\035\000\000\fS\026\000\2030\000\000Çß\024", '\0' , "ÿ\023\021\036ì\017½\000þ\022TE\002\032±m'\232\223\224·þ\026ú\aêéªÍ|\004ó\214\025\vm_head->M_dat.MH.MH_pkthdr.rcvif $2 = {if_softc = 0xffffff80003bc000, if_l2com = 0xffffff00019e7010, if_vnet = 0x0, if_link = {tqe_next = 0xffffff0004040800, tqe_prev = 0xffffff0001847818}, if_xname = "em0", '\0' , if_dname = 0xffffff00018af0d8 "em", if_dunit = 0, if_refcount = 1, if_addrhead = {tqh_first = 0xffffff00019eda00, tqh_last = 0xffffff00050e0cb8}, if_pcount = 0, if_carp = 0x0, if_bpf = 0xffffff0001a11c80, if_index = 3, if_timer = 0, if_vlantrunk = 0x0, if_flags = 34819, if_capabilities = 80283, if_capenable = 411, if_linkmib = 0x0, if_linkmiblen = 0, if_data = { ifi_type = 6 '\006', ifi_physical = 0 '\0', ifi_addrlen = 6 '\006', ifi_hdrlen = 18 '\022', ifi_link_state = 2 '\002', ifi_spare_char1 = 0 '\0', ifi_spare_char2 = 0 '\0', ifi_datalen = 152 '\230', ifi_mtu = 1500, ifi_metric = 0, ifi_baudrate = 1000000000, ifi_ipackets = 7558629784, ifi_ierrors = 0, ifi_opackets = 7037328617, ifi_oerrors = 0, ifi_collisions = 0, ifi_ibytes = 7274365128582, ifi_obytes = 5168997597383, ifi_imcasts = 134715, ifi_omcasts = 82322, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 38, ifi_epoch = 1, ifi_lastchange = {tv_sec = 1293736260, tv_usec = 525554}}, if_multiaddrs = {tqh_first = 0xffffff007ad8b480, tqh_last = 0xffffff00050e5400}, if_amcount = 0, if_output = 0xffffffff8039d6cf , if_input = 0xffffffff8039d379 , if_start = 0xffffffff801deb33 , if_ioctl = 0xffffffff801e3700 , if_watchdog = 0, if_init = 0xffffffff801e3674 , if_resolvemulti = 0xffffffff8039cbd8 , if_qflush = 0xffffffff80398ace , if_transmit = 0xffffffff80398812 , if_reassign = 0, if_home_vnet = 0x0, if_addr = 0xffffff00019eda00, if_llsoftc = 0x0, if_drv_flags = 64, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 4095, ifq_drops = 0, ifq_mtx = {lock_object = { lo_name = 0xffffff0001915828 "em0", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, ifq_drv_head = 0x0, ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 4095, altq_type = 0, altq_flags = 1, altq_disc = 0x0, altq_ifp = 0xffffff0001915800, altq_enqueue = 0, altq_dequeue = 0, altq_request = 0, altq_clfier = 0x0, altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0}, if_broadcastaddr = 0xffffffff8051d600 "ÿÿÿÿÿÿgif_mtx", if_bridge = 0x0, if_label = 0x0, if_prefixhead = {tqh_first = 0x0, tqh_last = 0xffffff0001915a80}, if_afdata = {0x0, 0x0, 0xffffff0004a471e0, 0x0 }, if_afdata_initialized = 2, if_afdata_lock = {lock_object = { lo_name = 0xffffffff8051d02c "if_afdata", lo_flags = 69402624, lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff80397235 , ta_context = 0xffffff0001915800}, if_addr_mtx = {lock_object = { lo_name = 0xffffffff8051d020 "if_addr_mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {tqh_first = 0xffffff00019e6a20, tqh_last = 0xffffff00019e6a28}, if_pf_kif = 0x0, if_lagg = 0x0, if_alloctype = 6 '\006', if_cspare = "\000\000", if_description = 0x0, if_pspare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, if_ispare = {0, 0, 0, 0}} (kgdb) quit Script done on Tue Jan 4 11:00:41 2011 Now dmesg output: em0: link state changed to DOWN em0: link state changed to UP ifa_add_loopback_route: insertion failed em0: link state changed to DOWN em0: link state changed to UP ifa_add_loopback_route: insertion failed em0: link state changed to DOWN em0: link state changed to UP arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) ng_generic_msg: number of nodes changed arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) arp: unknown hardware address format (0x1100) Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 04 fault virtual address = 0x1400000001 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80349eda stack pointer = 0x28:0xffffff80e77929b0 frame pointer = 0x28:0xffffff80e77929c0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (irq264: em1:tx 0) trap number = 12 panic: page fault cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff8019eaac = db_trace_self_wrapper+0x2a kdb_backtrace() at 0xffffffff8031e4cc = kdb_backtrace+0x37 panic() at 0xffffffff802ee7b8 = panic+0x182 trap_fatal() at 0xffffffff8048aca0 = trap_fatal+0x292 trap_pfault() at 0xffffffff8048b055 = trap_pfault+0x286 trap() at 0xffffffff8048b4ff = trap+0x3cb calltrap() at 0xffffffff80473e54 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff80349eda, rsp = 0xffffff80e77929b0, rbp = 0xffffff80e77929c0 --- m_tag_delete() at 0xffffffff80349eda = m_tag_delete+0x1a m_tag_delete_chain() at 0xffffffff80349f10 = m_tag_delete_chain+0x1c uma_zfree_arg() at 0xffffffff80450247 = uma_zfree_arg+0x3a m_freem() at 0xffffffff80347a58 = m_freem+0x51 em_txeof() at 0xffffffff801dd6a4 = em_txeof+0x15f em_msix_tx() at 0xffffffff801dfaff = em_msix_tx+0x4a intr_event_execute_handlers() at 0xffffffff802c9f33 = intr_event_execute_handlers+0xfc ithread_loop() at 0xffffffff802cb4d6 = ithread_loop+0x95 fork_exit() at 0xffffffff802c790d = fork_exit+0x118 fork_trampoline() at 0xffffffff8047432e = fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80e7792cf0, rbp = 0 --- Uptime: 4d6h2m45s Dumping 3893 MB (6 chunks) chunk 0: 1MB (151 pages) ... ok chunk 1: 3251MB (832247 pages) 3236 3220 3204 3188 3172 3156 3140 3124 3108 3092 3076 3060 3044 3028 3012 2996 2980 2964 2948 2932 2916 2900 2884 2868 2852 2836 2820 2804 2788 2772 2756 2740 2724 2708 2692 2676 2660 2644 2628 2612 2596 2580 2564 2548 2532 2516 2500 2484 2468 2452 2436 2420 2404 2388 2372 2356 2340 2324 2308 2292 2276 2260 2244 2228 2212 2196 2180 2164 2148 2132 2116 2100 2084 2068 2052 2036 2020 2004 1988 1972 1956 1940 1924 1908 1892 1876 1860 1844 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652 1636 1620 1604 1588 1572 1556 1540 1524 1508 1492 1476 1460 1444 1428 1412 1396 1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204 1188 1172 1156 1140 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676 660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84 68 52 36 20 4 ... ok chunk 2: 1MB (2 pages) ... ok chunk 3: 1MB (1 pages) ... ok chunk 4: 2MB (381 pages) ... ok chunk 5: 640MB (163840 pages) 625 609 593 577 561 545 529 513 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-PRERELEASE #4: Thu Dec 30 15:02:04 NOVT 2010 root@k-45-pc-1.sd.rdtc.ru:/usr/local/obj/home/src/sys/PPPOE amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz (3059.03-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x20652 Family = 6 Model = 25 Stepping = 2 Features=0xbfebfbff Features2=0x98e3bd AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3902296064 (3721 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 4 ACPI Warning: 32/64X FACS address mismatch in FADT - 0xCB603F40/0x00000000CB603E40, using 32 (20101013/tbfadt-586) ioapic0 irqs 0-23 on motherboard netisr_init: forcing maxthreads from 4 to 2 ichwd module loaded kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 ACPI Warning: Incorrect checksum in table [SSDT] - 0x3F, should be 0xE0 (20101013/tbutils-354) cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 igb0: port 0xe020-0xe03f mem 0xfbc20000-0xfbc3ffff,0xfb800000-0xfbbfffff,0xfbc44000-0xfbc47fff irq 16 at device 0.0 on pci1 igb0: Using MSIX interrupts with 3 vectors igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: Ethernet address: 00:1b:21:63:55:0c igb1: port 0xe000-0xe01f mem 0xfbc00000-0xfbc1ffff,0xfb000000-0xfb3fffff,0xfbc40000-0xfbc43fff irq 17 at device 0.1 on pci1 igb1: Using MSIX interrupts with 3 vectors igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: Ethernet address: 00:1b:21:63:55:0d vgapci0: port 0xf1e0-0xf1e7 mem 0xfa800000-0xfabfffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0 pci0: at device 22.0 (no driver attached) atapci0: port 0xf1d0-0xf1d7,0xf1c0-0xf1c3,0xf1b0-0xf1b7,0xf1a0-0xf1a3,0xf190-0xf19f irq 18 at device 22.2 on pci0 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] pci0: at device 22.3 (no driver attached) em0: port 0xf120-0xf13f mem 0xfbe00000-0xfbe1ffff,0xfbe28000-0xfbe28fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: 70:71:bc:09:84:a7 uhci0: port 0xf100-0xf11f irq 21 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 uhci1: port 0xf0e0-0xf0ff irq 18 at device 26.1 on pci0 uhci1: [ITHREAD] usbus1: on uhci1 uhci2: port 0xf0c0-0xf0df irq 19 at device 26.2 on pci0 uhci2: [ITHREAD] usbus2: on uhci2 ehci0: mem 0xfbe27000-0xfbe273ff irq 16 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 pci0: at device 27.0 (no driver attached) pcib2: irq 17 at device 28.0 on pci0 pci2: on pcib2 pcib3: irq 17 at device 28.4 on pci0 pci3: on pcib3 em1: port 0xd000-0xd01f mem 0xfbdc0000-0xfbddffff,0xfbd00000-0xfbd7ffff,0xfbde0000-0xfbde3fff irq 16 at device 0.0 on pci3 em1: Using MSIX interrupts with 3 vectors em1: [ITHREAD] em1: [ITHREAD] em1: [ITHREAD] em1: Ethernet address: 00:1b:21:6d:29:5b uhci3: port 0xf0a0-0xf0bf irq 19 at device 29.0 on pci0 uhci3: [ITHREAD] usbus4: on uhci3 uhci4: port 0xf080-0xf09f irq 16 at device 29.1 on pci0 uhci4: [ITHREAD] usbus5: on uhci4 uhci5: port 0xf060-0xf07f irq 18 at device 29.2 on pci0 uhci5: [ITHREAD] usbus6: on uhci5 uhci6: port 0xf040-0xf05f irq 23 at device 29.3 on pci0 uhci6: [ITHREAD] usbus7: on uhci6 ehci1: mem 0xfbe26000-0xfbe263ff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus8: EHCI version 1.0 usbus8: on ehci1 pcib4: at device 30.0 on pci0 pci4: on pcib4 isab0: at device 31.0 on pci0 isa0: on isab0 atapci1: port 0xf170-0xf177,0xf160-0xf163,0xf150-0xf157,0xf140-0xf143,0xf020-0xf03f mem 0xfbe25000-0xfbe257ff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] atapci1: AHCI called from vendor specific driver atapci1: AHCI v1.30 controller with 6 3Gbps ports, PM not supported ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] ata7: on atapci1 ata7: [ITHREAD] ichsmb0: port 0xf000-0xf01f mem 0xfbe24000-0xfbe240ff irq 18 at device 31.3 on pci0 ichsmb0: [ITHREAD] smbus0: on ichsmb0 smb0: on smbus0 acpi_button0: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] atrtc0: port 0x70-0x71 irq 8 on acpi0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] coretemp0: on cpu0 coretemp0: Tj(target) value 105 does not seem right. est0: on cpu0 p4tcc0: on cpu0 coretemp1: on cpu1 coretemp1: Tj(target) value 105 does not seem right. est1: on cpu1 p4tcc1: on cpu1 Timecounters tick every 1.000 msec ipfw2 initialized, divert loadable, nat loadable, rule-based forwarding enabled, default to deny, logging disabled load_dn_sched dn_sched FIFO loaded load_dn_sched dn_sched PRIO loaded load_dn_sched dn_sched QFQ loaded load_dn_sched dn_sched RR loaded load_dn_sched dn_sched WF2Q+ loaded usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 12Mbps Full Speed USB v1.0 usbus8: 480Mbps High Speed USB v2.0 ad8: 305244MB at ata4-master UDMA100 SATA 3Gb/s SMP: AP CPU #1 Launched! ugen1.1: at usbus1ugen0.1: at usbus0 uhub0: on usbus1 uhub1: on usbus0 ugen2.1: at usbus2ugen3.1: at usbus3 uhub2: on usbus2 uhub3: on usbus3 ugen4.1: at usbus4ugen5.1: at usbus5 uhub4: on usbus4 uhub5: on usbus5 ugen6.1: at usbus6ugen7.1: at usbus7 uhub6: on usbus6 uhub7: on usbus7 ugen8.1: at usbus8 uhub8: on usbus8 Root mount waiting for: usbus8 usbus7 usbus6 usbus5 usbus4 usbus3 usbus2 usbus1 usbus0 uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered uhub7: 2 ports with 2 removable, self powered Root mount waiting for: usbus8 usbus3 Root mount waiting for: usbus8 usbus3 uhub3: 6 ports with 6 removable, self powered uhub8: 8 ports with 8 removable, self powered Trying to mount root from ufs:/dev/ufs/root WARNING: / was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /var was not properly dismounted WARNING: /usr/local was not properly dismounted WARNING: /home was not properly dismounted em1: link state changed to UP vlan748: link state changed to UP vlan680: link state changed to UP vlan136: link state changed to UP vlan68: link state changed to UP vlan204: link state changed to UP vlan408: link state changed to UP vlan612: link state changed to UP vlan749: link state changed to UP vlan681: link state changed to UP vlan69: link state changed to UP vlan205: link state changed to UP vlan273: link state changed to UP vlan409: link state changed to UP vlan613: link state changed to UP vlan750: link state changed to UP vlan682: link state changed to UP vlan70: link state changed to UP vlan2: link state changed to UP vlan206: link state changed to UP vlan274: link state changed to UP vlan342: link state changed to UP vlan410: link state changed to UP vlan546: link state changed to UP vlan614: link state changed to UP vlan751: link state changed to UP vlan683: link state changed to UP vlan139: link state changed to UP vlan71: link state changed to UP vlan3: link state changed to UP vlan207: link state changed to UP vlan275: link state changed to UP vlan880: link state changed to UP vlan684: link state changed to UP vlan140: link state changed to UP vlan64: link state changed to UP vlan4: link state changed to UP vlan200: link state changed to UP vlan276: link state changed to UP vlan336: link state changed to UP vlan548: link state changed to UP vlan608: link state changed to UP vlan881: link state changed to UP vlan685: link state changed to UP vlan65: link state changed to UP vlan5: link state changed to UP vlan201: link state changed to UP vlan277: link state changed to UP vlan337: link state changed to UP vlan413: link state changed to UP vlan549: link state changed to UP vlan609: link state changed to UP vlan882: link state changed to UP vlan746: link state changed to UP vlan686: link state changed to UP vlan66: link state changed to UP vlan6: link state changed to UP vlan202: link state changed to UP vlan338: link state changed to UP vlan414: link state changed to UP vlan550: link state changed to UP vlan610: link state changed to UP vlan883: link state changed to UP vlan823: link state changed to UP vlan687: link state changed to UP vlan67: link state changed to UP vlan7: link state changed to UP vlan203: link state changed to UP vlan279: link state changed to UP vlan339: link state changed to UP vlan415: link state changed to UP vlan551: link state changed to UP vlan611: link state changed to UP vlan672: link state changed to UP vlan128: link state changed to UP vlan76: link state changed to UP vlan8: link state changed to UP vlan196: link state changed to UP vlan280: link state changed to UP vlan348: link state changed to UP vlan400: link state changed to UP vlan552: link state changed to UP vlan620: link state changed to UP vlan673: link state changed to UP vlan129: link state changed to UP vlan77: link state changed to UP vlan9: link state changed to UP vlan197: link state changed to UP vlan281: link state changed to UP vlan349: link state changed to UP vlan401: link state changed to UP vlan553: link state changed to UP vlan621: link state changed to UP vlan674: link state changed to UP vlan130: link state changed to UP vlan10: link state changed to UP vlan198: link state changed to UP vlan350: link state changed to UP vlan402: link state changed to UP vlan554: link state changed to UP vlan622: link state changed to UP vlan827: link state changed to UP vlan743: link state changed to UP vlan675: link state changed to UP vlan131: link state changed to UP vlan79: link state changed to UP vlan11: link state changed to UP vlan199: link state changed to UP vlan555: link state changed to UP vlan623: link state changed to UP vlan828: link state changed to UP vlan736: link state changed to UP vlan676: link state changed to UP vlan132: link state changed to UP vlan72: link state changed to UP vlan12: link state changed to UP vlan192: link state changed to UP vlan344: link state changed to UP vlan556: link state changed to UP vlan616: link state changed to UP vlan677: link state changed to UP vlan133: link state changed to UP vlan13: link state changed to UP vlan193: link state changed to UP vlan345: link state changed to UP vlan405: link state changed to UP vlan557: link state changed to UP vlan617: link state changed to UP vlan950: link state changed to UP vlan738: link state changed to UP vlan678: link state changed to UP vlan134: link state changed to UP vlan74: link state changed to UP vlan14: link state changed to UP vlan194: link state changed to UP vlan286: link state changed to UP vlan346: link state changed to UP vlan406: link state changed to UP vlan466: link state changed to UP vlan558: link state changed to UP vlan618: link state changed to UP vlan739: link state changed to UP vlan679: link state changed to UP vlan135: link state changed to UP vlan75: link state changed to UP vlan195: link state changed to UP vlan287: link state changed to UP vlan347: link state changed to UP vlan407: link state changed to UP vlan467: link state changed to UP vlan559: link state changed to UP vlan619: link state changed to UP vlan937: link state changed to UP vlan801: link state changed to UP vlan765: link state changed to UP vlan697: link state changed to UP vlan153: link state changed to UP vlan85: link state changed to UP vlan17: link state changed to UP vlan221: link state changed to UP vlan325: link state changed to UP vlan393: link state changed to UP vlan461: link state changed to UP vlan561: link state changed to UP vlan936: link state changed to UP vlan800: link state changed to UP vlan696: link state changed to UP vlan152: link state changed to UP vlan16: link state changed to UP vlan220: link state changed to UP vlan256: link state changed to UP vlan324: link state changed to UP vlan392: link state changed to UP vlan460: link state changed to UP vlan560: link state changed to UP vlan628: link state changed to UP vlan939: link state changed to UP vlan803: link state changed to UP vlan767: link state changed to UP vlan699: link state changed to UP vlan155: link state changed to UP vlan87: link state changed to UP vlan19: link state changed to UP vlan223: link state changed to UP vlan259: link state changed to UP vlan563: link state changed to UP vlan938: link state changed to UP vlan802: link state changed to UP vlan698: link state changed to UP vlan154: link state changed to UP vlan86: link state changed to UP vlan18: link state changed to UP vlan258: link state changed to UP vlan326: link state changed to UP vlan562: link state changed to UP vlan630: link state changed to UP vlan941: link state changed to UP vlan805: link state changed to UP vlan81: link state changed to UP vlan21: link state changed to UP vlan157: link state changed to UP vlan217: link state changed to UP vlan321: link state changed to UP vlan397: link state changed to UP vlan565: link state changed to UP vlan4000: link state changed to UP vlan940: link state changed to UP vlan804: link state changed to UP vlan700: link state changed to UP vlan156: link state changed to UP vlan80: link state changed to UP vlan216: link state changed to UP vlan320: link state changed to UP vlan396: link state changed to UP vlan624: link state changed to UP vlan807: link state changed to UP vlan83: link state changed to UP vlan23: link state changed to UP vlan159: link state changed to UP vlan219: link state changed to UP vlan263: link state changed to UP vlan323: link state changed to UP vlan399: link state changed to UP vlan627: link state changed to UP vlan942: link state changed to UP vlan806: link state changed to UP vlan702: link state changed to UP vlan82: link state changed to UP vlan22: link state changed to UP vlan158: link state changed to UP vlan218: link state changed to UP vlan262: link state changed to UP vlan322: link state changed to UP vlan398: link state changed to UP vlan566: link state changed to UP vlan626: link state changed to UP vlan689: link state changed to UP vlan145: link state changed to UP vlan93: link state changed to UP vlan25: link state changed to UP vlan213: link state changed to UP vlan333: link state changed to UP vlan385: link state changed to UP vlan453: link state changed to UP vlan569: link state changed to UP vlan637: link state changed to UP vlan928: link state changed to UP vlan808: link state changed to UP vlan688: link state changed to UP vlan24: link state changed to UP vlan212: link state changed to UP vlan332: link state changed to UP vlan384: link state changed to UP vlan452: link state changed to UP vlan931: link state changed to UP vlan691: link state changed to UP vlan95: link state changed to UP vlan27: link state changed to UP vlan215: link state changed to UP vlan335: link state changed to UP vlan571: link state changed to UP vlan639: link state changed to UP vlan998: link state changed to UP vlan690: link state changed to UP vlan94: link state changed to UP vlan26: link state changed to UP vlan214: link state changed to UP vlan334: link state changed to UP vlan386: link state changed to UP vlan570: link state changed to UP vlan638: link state changed to UP vlan993: link state changed to UP vlan933: link state changed to UP vlan753: link state changed to UP vlan693: link state changed to UP vlan149: link state changed to UP vlan89: link state changed to UP vlan29: link state changed to UP vlan209: link state changed to UP vlan633: link state changed to UP vlan992: link state changed to UP vlan872: link state changed to UP vlan692: link state changed to UP vlan148: link state changed to UP vlan88: link state changed to UP vlan28: link state changed to UP vlan208: link state changed to UP vlan328: link state changed to UP vlan388: link state changed to UP vlan448: link state changed to UP vlan572: link state changed to UP vlan632: link state changed to UP vlan935: link state changed to UP vlan815: link state changed to UP vlan695: link state changed to UP vlan151: link state changed to UP vlan91: link state changed to UP vlan211: link state changed to UP vlan331: link state changed to UP vlan391: link state changed to UP vlan575: link state changed to UP vlan635: link state changed to UP vlan934: link state changed to UP vlan694: link state changed to UP vlan150: link state changed to UP vlan90: link state changed to UP vlan30: link state changed to UP vlan270: link state changed to UP vlan390: link state changed to UP vlan574: link state changed to UP vlan634: link state changed to UP vlan718: link state changed to UP vlan102: link state changed to UP vlan34: link state changed to UP vlan170: link state changed to UP vlan238: link state changed to UP vlan306: link state changed to UP vlan374: link state changed to UP vlan442: link state changed to UP vlan510: link state changed to UP vlan514: link state changed to UP vlan582: link state changed to UP vlan650: link state changed to UP vlan923: link state changed to UP vlan35: link state changed to UP vlan171: link state changed to UP vlan307: link state changed to UP vlan375: link state changed to UP vlan443: link state changed to UP vlan511: link state changed to UP vlan515: link state changed to UP vlan583: link state changed to UP vlan651: link state changed to UP vlan920: link state changed to UP vlan100: link state changed to UP vlan168: link state changed to UP vlan372: link state changed to UP vlan440: link state changed to UP vlan508: link state changed to UP vlan512: link state changed to UP vlan580: link state changed to UP vlan648: link state changed to UP vlan921: link state changed to UP vlan717: link state changed to UP vlan101: link state changed to UP vlan33: link state changed to UP vlan169: link state changed to UP vlan237: link state changed to UP vlan305: link state changed to UP vlan373: link state changed to UP vlan513: link state changed to UP vlan581: link state changed to UP vlan649: link state changed to UP vlan714: link state changed to UP vlan98: link state changed to UP vlan38: link state changed to UP vlan174: link state changed to UP vlan234: link state changed to UP vlan370: link state changed to UP vlan446: link state changed to UP vlan506: link state changed to UP vlan518: link state changed to UP vlan578: link state changed to UP vlan654: link state changed to UP vlan927: link state changed to UP vlan715: link state changed to UP vlan39: link state changed to UP vlan175: link state changed to UP vlan235: link state changed to UP vlan311: link state changed to UP vlan371: link state changed to UP vlan447: link state changed to UP vlan507: link state changed to UP vlan519: link state changed to UP vlan579: link state changed to UP vlan655: link state changed to UP vlan848: link state changed to UP vlan712: link state changed to UP vlan96: link state changed to UP vlan172: link state changed to UP vlan232: link state changed to UP vlan308: link state changed to UP vlan504: link state changed to UP vlan516: link state changed to UP vlan576: link state changed to UP vlan652: link state changed to UP vlan925: link state changed to UP vlan849: link state changed to UP vlan713: link state changed to UP vlan97: link state changed to UP vlan37: link state changed to UP vlan173: link state changed to UP vlan233: link state changed to UP vlan309: link state changed to UP vlan369: link state changed to UP vlan445: link state changed to UP vlan517: link state changed to UP vlan577: link state changed to UP vlan653: link state changed to UP vlan914: link state changed to UP vlan710: link state changed to UP vlan110: link state changed to UP vlan42: link state changed to UP vlan162: link state changed to UP vlan230: link state changed to UP vlan314: link state changed to UP vlan382: link state changed to UP vlan502: link state changed to UP vlan522: link state changed to UP vlan590: link state changed to UP vlan642: link state changed to UP vlan711: link state changed to UP vlan111: link state changed to UP vlan43: link state changed to UP vlan163: link state changed to UP vlan231: link state changed to UP vlan315: link state changed to UP vlan435: link state changed to UP vlan503: link state changed to UP vlan523: link state changed to UP vlan591: link state changed to UP vlan643: link state changed to UP vlan912: link state changed to UP vlan708: link state changed to UP vlan108: link state changed to UP vlan40: link state changed to UP vlan160: link state changed to UP vlan228: link state changed to UP vlan312: link state changed to UP vlan380: link state changed to UP vlan432: link state changed to UP vlan500: link state changed to UP vlan640: link state changed to UP vlan913: link state changed to UP vlan709: link state changed to UP vlan109: link state changed to UP vlan41: link state changed to UP vlan161: link state changed to UP vlan229: link state changed to UP vlan381: link state changed to UP vlan433: link state changed to UP vlan501: link state changed to UP vlan521: link state changed to UP vlan641: link state changed to UP vlan918: link state changed to UP vlan106: link state changed to UP vlan166: link state changed to UP vlan226: link state changed to UP vlan318: link state changed to UP vlan378: link state changed to UP vlan526: link state changed to UP vlan586: link state changed to UP vlan646: link state changed to UP vlan919: link state changed to UP vlan107: link state changed to UP vlan47: link state changed to UP vlan167: link state changed to UP vlan227: link state changed to UP vlan319: link state changed to UP vlan379: link state changed to UP vlan499: link state changed to UP vlan527: link state changed to UP vlan587: link state changed to UP vlan647: link state changed to UP vlan104: link state changed to UP vlan164: link state changed to UP vlan224: link state changed to UP vlan316: link state changed to UP vlan376: link state changed to UP vlan436: link state changed to UP vlan644: link state changed to UP vlan705: link state changed to UP vlan45: link state changed to UP vlan165: link state changed to UP vlan225: link state changed to UP vlan377: link state changed to UP vlan497: link state changed to UP vlan525: link state changed to UP vlan585: link state changed to UP vlan645: link state changed to UP vlan907: link state changed to UP vlan839: link state changed to UP vlan735: link state changed to UP vlan667: link state changed to UP vlan51: link state changed to UP vlan187: link state changed to UP vlan291: link state changed to UP vlan359: link state changed to UP vlan427: link state changed to UP vlan495: link state changed to UP vlan531: link state changed to UP vlan599: link state changed to UP vlan906: link state changed to UP vlan770: link state changed to UP vlan734: link state changed to UP vlan666: link state changed to UP vlan118: link state changed to UP vlan50: link state changed to UP vlan186: link state changed to UP vlan290: link state changed to UP vlan358: link state changed to UP vlan426: link state changed to UP vlan494: link state changed to UP vlan530: link state changed to UP vlan598: link state changed to UP vlan905: link state changed to UP vlan733: link state changed to UP vlan665: link state changed to UP vlan117: link state changed to UP vlan49: link state changed to UP vlan185: link state changed to UP vlan253: link state changed to UP vlan289: link state changed to UP vlan357: link state changed to UP vlan425: link state changed to UP vlan493: link state changed to UP vlan529: link state changed to UP vlan597: link state changed to UP vlan732: link state changed to UP vlan664: link state changed to UP vlan116: link state changed to UP vlan48: link state changed to UP vlan184: link state changed to UP vlan252: link state changed to UP vlan288: link state changed to UP vlan356: link state changed to UP vlan424: link state changed to UP vlan492: link state changed to UP vlan911: link state changed to UP vlan835: link state changed to UP vlan731: link state changed to UP vlan671: link state changed to UP vlan115: link state changed to UP vlan55: link state changed to UP vlan191: link state changed to UP vlan431: link state changed to UP vlan491: link state changed to UP vlan535: link state changed to UP vlan595: link state changed to UP vlan910: link state changed to UP vlan774: link state changed to UP vlan730: link state changed to UP vlan670: link state changed to UP vlan54: link state changed to UP vlan190: link state changed to UP vlan250: link state changed to UP vlan294: link state changed to UP vlan354: link state changed to UP vlan430: link state changed to UP vlan490: link state changed to UP vlan594: link state changed to UP vlan773: link state changed to UP vlan729: link state changed to UP vlan669: link state changed to UP vlan113: link state changed to UP vlan53: link state changed to UP vlan189: link state changed to UP vlan249: link state changed to UP vlan293: link state changed to UP vlan429: link state changed to UP vlan489: link state changed to UP vlan533: link state changed to UP vlan593: link state changed to UP vlan908: link state changed to UP vlan772: link state changed to UP vlan728: link state changed to UP vlan668: link state changed to UP vlan112: link state changed to UP vlan52: link state changed to UP vlan188: link state changed to UP vlan248: link state changed to UP vlan292: link state changed to UP vlan428: link state changed to UP vlan532: link state changed to UP vlan592: link state changed to UP vlan899: link state changed to UP vlan847: link state changed to UP vlan727: link state changed to UP vlan127: link state changed to UP vlan59: link state changed to UP vlan179: link state changed to UP vlan247: link state changed to UP vlan299: link state changed to UP vlan367: link state changed to UP vlan419: link state changed to UP vlan539: link state changed to UP vlan607: link state changed to UP vlan659: link state changed to UP vlan966: link state changed to UP vlan846: link state changed to UP vlan778: link state changed to UP vlan726: link state changed to UP vlan126: link state changed to UP vlan58: link state changed to UP vlan178: link state changed to UP vlan246: link state changed to UP vlan298: link state changed to UP vlan366: link state changed to UP vlan418: link state changed to UP vlan538: link state changed to UP vlan606: link state changed to UP vlan658: link state changed to UP vlan897: link state changed to UP vlan845: link state changed to UP vlan725: link state changed to UP vlan125: link state changed to UP vlan57: link state changed to UP vlan177: link state changed to UP vlan245: link state changed to UP vlan297: link state changed to UP vlan365: link state changed to UP vlan537: link state changed to UP vlan605: link state changed to UP vlan657: link state changed to UP vlan896: link state changed to UP vlan844: link state changed to UP vlan724: link state changed to UP vlan56: link state changed to UP vlan176: link state changed to UP vlan244: link state changed to UP vlan416: link state changed to UP vlan536: link state changed to UP vlan604: link state changed to UP vlan656: link state changed to UP vlan903: link state changed to UP vlan843: link state changed to UP vlan783: link state changed to UP vlan723: link state changed to UP vlan663: link state changed to UP vlan123: link state changed to UP vlan183: link state changed to UP vlan243: link state changed to UP vlan303: link state changed to UP vlan423: link state changed to UP vlan543: link state changed to UP vlan603: link state changed to UP vlan782: link state changed to UP vlan722: link state changed to UP vlan122: link state changed to UP vlan62: link state changed to UP vlan182: link state changed to UP vlan242: link state changed to UP vlan302: link state changed to UP vlan362: link state changed to UP vlan422: link state changed to UP vlan602: link state changed to UP vlan662: link state changed to UP vlan901: link state changed to UP vlan841: link state changed to UP vlan781: link state changed to UP vlan721: link state changed to UP vlan121: link state changed to UP vlan61: link state changed to UP vlan181: link state changed to UP vlan301: link state changed to UP vlan421: link state changed to UP vlan541: link state changed to UP vlan601: link state changed to UP vlan661: link state changed to UP vlan960: link state changed to UP vlan900: link state changed to UP vlan840: link state changed to UP vlan780: link state changed to UP vlan720: link state changed to UP vlan120: link state changed to UP vlan60: link state changed to UP vlan180: link state changed to UP vlan300: link state changed to UP vlan360: link state changed to UP vlan600: link state changed to UP vlan660: link state changed to UP em0: link state changed to UP The kernel is mostly monolithic; if_lagg.ko is loaded but unused - this particular box has no lagg devices configured. Here comes kernel config file: # PPPOE kernel config cpu HAMMER ident PPPOE # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. # Use the following to compile in values accessible to the kernel # through getenv() (or kenv(1) in userland). The format of the file # is 'variable=value', see kenv(1) # # env "GENERIC.env" makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking #options INET6 # IPv6 communications protocols #options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device #options NFSCLIENT # Network Filesystem Client #options NFSSERVER # Network Filesystem Server #options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCLIENT #options MSDOSFS # MSDOS Filesystem #options CD9660 # ISO 9660 Filesystem #options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options GEOM_JOURNAL options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options COMPAT_FREEBSD32 # Compatible with i386 binaries #options COMPAT_FREEBSD4 # Compatible with FreeBSD4 #options COMPAT_FREEBSD5 # Compatible with FreeBSD5 #options COMPAT_FREEBSD6 # Compatible with FreeBSD6 #options COMPAT_FREEBSD7 # Compatible with FreeBSD7 #options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework #options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives #device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives #device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives #device atapifd # ATAPI floppy drives #device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI peripherals device scbus # SCSI bus (required for SCSI) #device ch # SCSI media changers device da # Direct Access (disks) #device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) #device ses # SCSI Environmental Services (and SAF-TE) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver #device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc #device agp # support several AGP chipsets # Serial (COM) ports device uart # Generic UART driver # PCI Ethernet NICs. device em # Intel PRO/1000 Gigabit Ethernet Family device igb # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support #device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling #device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device snp # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support #options USB_DEBUG # enable debug msgs options USB_VERBOSE device uhci # UHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse #options IPSEC #device crypto options NETGRAPH options NETGRAPH_ETHER options NETGRAPH_IFACE options NETGRAPH_MPPC_ENCRYPTION options NETGRAPH_PPP options NETGRAPH_PPPOE options NETGRAPH_SOCKET options NETGRAPH_TCPMSS options NETGRAPH_TEE options NETGRAPH_VJC options IPFIREWALL options IPFIREWALL_FORWARD #options IPDIVERT options DUMMYNET options VFS_AIO device smbus device smb device ichsmb device iicbus device iicbb device ic device iic device iicsmb device coretemp device ichwd device nvram #options DEVICE_POLLING options KDB options KDB_TRACE options KDB_UNATTENDED options DDB options DDB_NUMSYM #options INVARIANTS #options INVARIANT_SUPPORT #options DEBUG_MEMGUARD Now /boot/loader.conf: autoboot_delay="2" if_lagg_load="YES" hw.em.rxd=4096 hw.em.txd=4096 hw.igb.rxd=4096 hw.igb.txd=4096 vm.kmem_size=3G machdep.hyperthreading_allowed=0 # netgraph queue sizes tuning, see vmstat -z|egrep 'ITEM|NetGraph' net.graph.maxdata=65536 net.graph.maxalloc=65536 net.isr.maxthreads=4 # default value #net.isr.maxqlimit=10240 # for rtsock net.route.netisr_maxqlen=4096 # for other protocols (IP & PPPoE?) net.isr.defaultqlimit=4096 # default outgoing interface queue length # used by lagg etc. net.link.ifqmaxlen=10240 Now /etc/sysctl.conf: # force chipset reprogramming dev.em.0.rx_int_delay=50 dev.em.0.tx_int_delay=50 dev.em.0.rx_abs_int_delay=900 dev.em.0.tx_abs_int_delay=900 dev.em.0.rx_processing_limit=4000 dev.em.0.rx_int_delay=100 dev.em.0.tx_int_delay=100 dev.em.0.rx_abs_int_delay=1000 dev.em.0.tx_abs_int_delay=1000 dev.em.0.rx_processing_limit=4096 dev.em.1.rx_int_delay=50 dev.em.1.tx_int_delay=50 dev.em.1.rx_abs_int_delay=900 dev.em.1.tx_abs_int_delay=900 dev.em.1.rx_processing_limit=4000 dev.em.1.rx_int_delay=100 dev.em.1.tx_int_delay=100 dev.em.1.rx_abs_int_delay=1000 dev.em.1.tx_abs_int_delay=1000 dev.em.1.rx_processing_limit=4096 net.inet.ip.intr_queue_maxlen=10240 net.inet.ip.fastforwarding=1 net.inet.ip.dummynet.pipe_slot_limit=1000 net.inet.ip.dummynet.io_fast=1 kern.ipc.nmbclusters=100000 kern.ipc.nmbjumbop=100000 kern.ipc.maxsockbuf=83886080 net.graph.maxdgram=8388608 net.graph.recvspace=8388608 net.isr.direct=0 net.isr.direct_force=0 net.inet.ip.redirect=0 net.inet.ip.ttl=128 net.inet.ip.random_id=1 net.inet.tcp.blackhole=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.path_mtu_discovery=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.ecn.enable=1 net.inet.ip.ttl=130 net.inet.ip.fw.verbose=1 debug.minidump=0 >How-To-Repeat: It takes several days for panic to occur and seem to depend from load: more load, less time to panic. This particular panic occured at 4th day of uptime. >Fix: Unknown. >Release-Note: >Audit-Trail: >Unformatted: