From owner-freebsd-net@freebsd.org Wed Jan 6 18:00:22 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B9D3A64DCE for ; Wed, 6 Jan 2016 18:00:22 +0000 (UTC) (envelope-from dchagin@chd.heemeyer.club) Received: from heemeyer.club (heemeyer.club [108.61.204.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "heemeyer.club", Issuer "heemeyer.club" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B0E21E19 for ; Wed, 6 Jan 2016 18:00:21 +0000 (UTC) (envelope-from dchagin@chd.heemeyer.club) Received: from chd.heemeyer.club ([78.107.232.239]) by heemeyer.club (8.15.2/8.15.1) with ESMTPS id u06I09wo018565 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 6 Jan 2016 18:00:12 GMT (envelope-from dchagin@chd.heemeyer.club) X-Authentication-Warning: heemeyer.club: Host [78.107.232.239] claimed to be chd.heemeyer.club Received: from chd.heemeyer.club (localhost [127.0.0.1]) by chd.heemeyer.club (8.15.2/8.15.1) with ESMTPS id u06I09rC001398 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 6 Jan 2016 21:00:09 +0300 (MSK) (envelope-from dchagin@chd.heemeyer.club) Received: (from dchagin@localhost) by chd.heemeyer.club (8.15.2/8.15.1/Submit) id u06I09kB001397 for freebsd-net@freebsd.org; Wed, 6 Jan 2016 21:00:09 +0300 (MSK) (envelope-from dchagin) Date: Wed, 6 Jan 2016 21:00:08 +0300 From: Chagin Dmitry To: freebsd-net@freebsd.org Subject: long network segment call stack can fail Message-ID: <20160106180008.GA1363@chd.heemeyer.club> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jan 2016 18:00:22 -0000 Hi, all. Here's a panic that occurs during the massive network loads, as far as I can see network segment has eaten the whole stack, so only 0xeb0 bytes left for interrupt hanlder. #0 doadump (textdump=0x8151cd90) at /home/git/head/sys/kern/kern_shutdown.c:296 #1 0xffffffff803d4715 in db_fncall_generic (addr=0xffffffff80763f00, rv=0xffffffff8151cd60 , nargs=0x0, args=0xffffffff8151cd70 ) at /home/git/head/sys/ddb/db_command.c:568 #2 0xffffffff803d3e94 in db_fncall (dummy1=0xffffffff8151cea0, dummy2=0x0, dummy3=0x0, dummy4=0xffffffff8151cea0 "\n") at /home/git/head/sys/ddb/db_command.c:616 #3 0xffffffff803d371e in db_command (last_cmdp=0xffffffff812e3838 , cmd_table=0x0, dopager=0x1) at /home/git/head/sys/ddb/db_command.c:440 #4 0xffffffff803d32be in db_command_loop () at /home/git/head/sys/ddb/db_command.c:493 #5 0xffffffff803d7b53 in db_trap (type=0x3, code=0x0) at /home/git/head/sys/ddb/db_main.c:251 #6 0xffffffff807c568f in kdb_trap (type=0x3, code=0x0, tf=0xffffffff8151d430 ) at /home/git/head/sys/kern/subr_kdb.c:654 #7 0xffffffff80c7a9f1 in trap (frame=0xffffffff8151d430 ) at /home/git/head/sys/amd64/amd64/trap.c:549 #8 0xffffffff80c7bb5a in trap_check (frame=0xffffffff8151d430 ) at /home/git/head/sys/amd64/amd64/trap.c:628 #9 0xffffffff80c53f47 in calltrap () at /home/git/head/sys/amd64/amd64/exception.S:234 #10 0xffffffff807c4ee5 in breakpoint () at ./machine/cpufunc.h:63 #11 0xffffffff807c4ae7 in kdb_enter (why=0xffffffff80db94e3 "panic", msg=0xffffffff80db94e3 "panic") at /home/git/head/sys/kern/subr_kdb.c:443 #12 0xffffffff8076461b in vpanic (fmt=0xffffffff80e1067f "double fault", ap=0xffffffff8151d630 ) at /home/git/head/sys/kern/kern_shutdown.c:750 #13 0xffffffff807646c0 in panic (fmt=0xffffffff80e1067f "double fault") at /home/git/head/sys/kern/kern_shutdown.c:688 #14 0xffffffff80c7bc6a in dblfault_handler (frame=0xffffffff8151d6b0 ) at /home/git/head/sys/amd64/amd64/trap.c:861 #15 0xffffffff80c5403c in Xdblfault () at /home/git/head/sys/amd64/amd64/exception.S:290 #16 0xffffffff807a26c5 in cpu_search (cg=0xffffffff8146b148 , low=0xfffffe02d858b298, high=0x0, match=0x1) at /home/git/head/sys/kern/sched_ule.c:689 #17 cpu_search_lowest (cg=0xffffffff8146b148 , low=0xfffffe02d858b298) at /home/git/head/sys/kern/sched_ule.c:781 #18 0xffffffff807a286b in cpu_search (cg=0xffffffff8146b068 , low=0xfffffe02d858b418, high=0x0, match=0x1) at /home/git/head/sys/kern/sched_ule.c:708 #19 cpu_search_lowest (cg=0xffffffff8146b068 , low=0xfffffe02d858b418) at /home/git/head/sys/kern/sched_ule.c:781 #20 0xffffffff807a286b in cpu_search (cg=0xffffffff8146b030 , low=0xfffffe02d858b490, high=0x0, match=0x1) at /home/git/head/sys/kern/sched_ule.c:708 #21 cpu_search_lowest (cg=0xffffffff8146b030 , low=0xfffffe02d858b490) at /home/git/head/sys/kern/sched_ule.c:781 #22 0xffffffff807a9496 in sched_lowest (cg=0xffffffff8146b030 , mask=..., pri=0x78, maxload=0x7fffffff, prefer=0x0) at /home/git/head/sys/kern/sched_ule.c:813 #23 0xffffffff807a511b in sched_pickcpu (td=0xfffff8000bb3a4f0, flags=0x0) at /home/git/head/sys/kern/sched_ule.c:1290 #24 0xffffffff807a61c8 in sched_add (td=0xfffff8000bb3a4f0, flags=0x0) at /home/git/head/sys/kern/sched_ule.c:2426 #25 0xffffffff807a5e78 in sched_wakeup (td=0xfffff8000bb3a4f0) at /home/git/head/sys/kern/sched_ule.c:2058 #26 0xffffffff80777eb4 in setrunnable (td=0xfffff8000bb3a4f0) at /home/git/head/sys/kern/kern_synch.c:522 #27 0xffffffff807dcf99 in sleepq_resume_thread (sq=0xfffff800036fb680, td=0xfffff8000bb3a4f0, pri=0x0) at /home/git/head/sys/kern/subr_sleepqueue.c:776 #28 0xffffffff807db303 in sleepq_timeout (arg=0xfffff8000bb3a4f0) at /home/git/head/sys/kern/subr_sleepqueue.c:915 #29 0xffffffff8078bfac in softclock_call_cc (c=0xfffff8000bb3a8a8, cc=0xffffffff81589180 , direct=0x1) at /home/git/head/sys/kern/kern_timeout.c:723 #30 0xffffffff8078b78a in callout_process (now=0x393bab69d8fb) at /home/git/head/sys/kern/kern_timeout.c:493 #31 0xffffffff80cc8932 in handleevents (now=0x393bab69d8fb, fake=0x0) at /home/git/head/sys/kern/kern_clocksource.c:212 #32 0xffffffff80cc941c in timercb (et=0xffffffff81562d98 , arg=0x0) at /home/git/head/sys/kern/kern_clocksource.c:345 #33 0xffffffff80d1e9e8 in lapic_handle_timer (frame=0xfffffe02d858bdf0) at /home/git/head/sys/x86/x86/local_apic.c:1056 #34 0xffffffff80c54b3c in Xtimerint () at /home/git/head/sys/amd64/amd64/apic_vector.S:133 #35 0xffffffff80c0cc36 in uma_dbg_getslab (zone=0xfffff80003e15000, item=0xfffff801f0e4b800) at /home/git/head/sys/vm/uma_dbg.c:219 #36 0xffffffff80c0ca34 in uma_dbg_alloc (zone=0xfffff80003e15000, slab=0x0, item=0xfffff801f0e4b800) at /home/git/head/sys/vm/uma_dbg.c:242 #37 0xffffffff80c04e58 in uma_zalloc_arg (zone=0xfffff80003e15000, udata=0xfffffe02d858c070, flags=0x1) at /home/git/head/sys/vm/uma_core.c:2220 #38 0xffffffff8095ea38 in m_gethdr (how=0x1, type=0x1) at /home/git/head/sys/sys/mbuf.h:672 #39 0xffffffff8095e8b6 in ieee80211_mbuf_adjust (vap=0xfffff8000f2b1000, hdrsize=0x1a, key=0xfffffe0001817138, m=0xfffff8000f744400) at /home/git/head/sys/net80211/ieee80211_output.c:1113 #40 0xffffffff8095add2 in ieee80211_encap (vap=0xfffff8000f2b1000, ni=0xfffffe0001817000, m=0xfffff8000f744400) at /home/git/head/sys/net80211/ieee80211_output.c:1354 #41 0xffffffff8095a234 in ieee80211_vap_pkt_send_dest (vap=0xfffff8000f2b1000, m=0xfffff8000f744400, ni=0xfffffe0001817000) at /home/git/head/sys/net80211/ieee80211_output.c:258 #42 0xffffffff8095c91e in ieee80211_start_pkt (vap=0xfffff8000f2b1000, m=0xfffff8000f744400) at /home/git/head/sys/net80211/ieee80211_output.c:426 #43 0xffffffff8095c27b in ieee80211_vap_transmit (ifp=0xfffff8000f2b7000, m=0xfffff8000f744400) at /home/git/head/sys/net80211/ieee80211_output.c:486 #44 0xffffffff808db57b in bridge_enqueue (sc=0xfffff8000f2c5c00, dst_ifp=0xfffff8000f2b7000, m=0xfffff8000f744400) at /home/git/head/sys/net/if_bridge.c:1919 #45 0xffffffff808e02be in bridge_output (ifp=0xfffff8000f2b7000, m=0xfffff8000f744400, sa=0x0, rt=0x0) at /home/git/head/sys/net/if_bridge.c:2085 #46 0xffffffff808e4b52 in ether_output (ifp=0xfffff8000f2b7000, m=0xfffff8000f744400, dst=0xfffff8000befe6b0, ro=0xfffffe02d858d248) at /home/git/head/sys/net/if_ethersubr.c:376 #47 0xffffffff8095cab7 in ieee80211_output (ifp=0xfffff8000f2b7000, m=0xfffff8000f744400, dst=0xfffff8000befe6b0, ro=0xfffffe02d858d248) at /home/git/head/sys/net80211/ieee80211_output.c:576 #48 0xffffffff809fd851 in ip_output (m=0xfffff8000f744400, opt=0x0, ro=0xfffffe02d858d248, flags=0x0, imo=0x0, inp=0xfffff801f786c7d0) at /home/git/head/sys/netinet/ip_output.c:638 #49 0xffffffff80ade593 in tcp_output (tp=0xfffff8016b2bc418) at /home/git/head/sys/netinet/tcp_output.c:1414 #50 0xffffffff80ad59ee in tcp_do_segment (m=0xfffff8000f71d100, th=0xfffff800071f804a, so=0xfffff8016b5c92e8, tp=0xfffff8016b2bc418, drop_hdrlen=0x34, tlen=0x544, iptos=0x0, ti_locked=0x1) at /home/git/head/sys/netinet/tcp_input.c:1904 #51 0xffffffff80ad3c9f in tcp_input (mp=0xfffffe02d858de28, offp=0xfffffe02d858de00, proto=0x6) at /home/git/head/sys/netinet/tcp_input.c:1417 #52 0xffffffff809f8b91 in ip_input (m=0x0) at /home/git/head/sys/netinet/ip_input.c:786 #53 0xffffffff808f6b4e in netisr_dispatch_src (proto=0x1, source=0x0, m=0xfffff8000f71d100) at /home/git/head/sys/net/netisr.c:972 #54 0xffffffff808f7157 in netisr_dispatch (proto=0x1, m=0xfffff8000f71d100) at /home/git/head/sys/net/netisr.c:1063 #55 0xffffffff808e55b0 in ether_demux (ifp=0xfffff8000f2b7000, m=0xfffff8000f71d100) at /home/git/head/sys/net/if_ethersubr.c:805 #56 0xffffffff808e7231 in ether_input_internal (ifp=0xfffff8000f2b7000, m=0xfffff8000f71d100) at /home/git/head/sys/net/if_ethersubr.c:611 #57 0xffffffff808e6a3f in ether_nh_input (m=0xfffff8000f71d100) at /home/git/head/sys/net/if_ethersubr.c:641 #58 0xffffffff808f6b4e in netisr_dispatch_src (proto=0x5, source=0x0, m=0xfffff8000f71d100) at /home/git/head/sys/net/netisr.c:972 #59 0xffffffff808f7157 in netisr_dispatch (proto=0x5, m=0xfffff8000f71d100) at /home/git/head/sys/net/netisr.c:1063 #60 0xffffffff808e5b6c in ether_input (ifp=0xfffff8000f2b7000, m=0xfffff8000f71d100) at /home/git/head/sys/net/if_ethersubr.c:715 #61 0xffffffff80938d6a in ieee80211_deliver_data (vap=0xfffff8000f2b1000, ni=0xfffffe0001817000, m=0xfffff8000f71d100) at /home/git/head/sys/net80211/ieee80211_input.c:298 #62 0xffffffff80974e77 in sta_input (ni=0xfffffe0001817000, m=0xfffff8000f71d100, rxs=0xfffffe02d858e440, rssi=0x82, nf=0xb0) at /home/git/head/sys/net80211/ieee80211_sta.c:868 #63 0xffffffff809383b8 in ieee80211_input_mimo (ni=0xfffffe0001817000, m=0xfffff8000f71d100, rx=0xfffffe02d858e560) at /home/git/head/sys/net80211/ieee80211_input.c:103 #64 0xffffffff804c158f in iwm_mvm_rx_rx_mpdu (sc=0xfffffe000108f000, pkt=0xfffff800071f8000, data=0xfffffe00010ce928) at /home/git/head/sys/dev/iwm/if_iwm.c:2391 #65 0xffffffff804bfe88 in iwm_notif_intr (sc=0xfffffe000108f000) at /home/git/head/sys/dev/iwm/if_iwm.c:4092 #66 0xffffffff804bf55e in iwm_intr (arg=0xfffffe000108f000) at /home/git/head/sys/dev/iwm/if_iwm.c:4416 #67 0xffffffff8070f7c8 in intr_event_execute_handlers (p=0xfffff80003df7aa0, ie=0xfffff80003def900) at /home/git/head/sys/kern/kern_intr.c:1262 #68 0xffffffff807105b7 in ithread_execute_handlers (p=0xfffff80003df7aa0, ie=0xfffff80003def900) at /home/git/head/sys/kern/kern_intr.c:1275 #69 0xffffffff807103e6 in ithread_loop (arg=0xfffff80007002500) at /home/git/head/sys/kern/kern_intr.c:1356 #70 0xffffffff8070ad89 in fork_exit (callout=0xffffffff80710270 , arg=0xfffff80007002500, frame=0xfffffe02d858eac0) at /home/git/head/sys/kern/kern_fork.c:1010 #71 0xffffffff80c5447e in fork_trampoline () at /home/git/head/sys/amd64/amd64/exception.S:609 #71 0xffffffff80c5447e in fork_trampoline () at /home/git/head/sys/amd64/amd64/exception.S:609 up 71 609 call fork_exit print $sp $1 = (void *) 0xfffffe02d858eac0 up 35 #35 0xffffffff80c0cc36 in uma_dbg_getslab (zone=0xfffff80003e15000, item=0xfffff801f0e4b800) at /home/git/head/sys/vm/uma_dbg.c:219 219 if (keg->uk_flags & UMA_ZONE_HASH) print $sp $37 = (void *) 0xfffffe02d858beb0 so, 0x0eb0 bytes left on stack before fault at 0xfffffe02d858b000 up 16 #16 0xffffffff807a26c5 in cpu_search (cg=0xffffffff8146b148 , low=0xfffffe02d858b298, high=0x0, match=0x1) at /home/git/head/sys/kern/sched_ule.c:689 689 cpu = CPU_FFS(&cpumask) - 1; print $sp $56 = (void *) 0xfffffe02d858b000