From owner-freebsd-net@FreeBSD.ORG Mon Sep 8 19:34:05 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 049D17E1; Mon, 8 Sep 2014 19:34:05 +0000 (UTC) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id DB06B1F12; Mon, 8 Sep 2014 19:34:04 +0000 (UTC) Received: from marvin.lab.vangyzen.net (c-24-125-214-90.hsd1.va.comcast.net [24.125.214.90]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 90BF656444; Mon, 8 Sep 2014 14:34:03 -0500 (CDT) Message-ID: <540E04AA.80806@vangyzen.net> Date: Mon, 08 Sep 2014 15:34:02 -0400 From: Eric van Gyzen User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: sbruno@freebsd.org, FreeBSD Net Subject: Re: ixgbe(4) spin lock held too long References: <1410203348.1343.1.camel@bruno> <1410203965.1343.3.camel@bruno> In-Reply-To: <1410203965.1343.3.camel@bruno> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 19:34:05 -0000 On 09/08/2014 15:19, Sean Bruno wrote: > On Mon, 2014-09-08 at 12:09 -0700, Sean Bruno wrote: >> This sort of looks like the hardware failed to respond to us in time? >> Too busy? >> >> sean >> > This seems to be affecting my 10/stable machines from 15Aug2014. > > Not a lot of churn in the code so I don't think this is new. The > afflicted machines, quite a few by my count, appear to have not been > super busy (pushing about 200 Mb/s). > > sean > > > >> panic: spin lock held too long >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you >> are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> spin lock 0xffffffff812a0400 (callout) held by 0xfffff800151fe000 (tid >> 100003) too long TID 100003 is usually a kernel idle thread, which would seem to indicate a dangling lock. Can you enable WITNESS (without WITNESS_SKIPSPIN) on this box? >> panic: spin lock held too long >> cpuid = 4 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe1f292752b0 >> panic() at panic+0x155/frame 0xfffffe1f29275330 >> _mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0x243/frame >> 0xfffffe1f29275390 >> callout_lock() at callout_lock+0xd4/frame 0xfffffe1f292753d0 >> callout_reset_sbt_on() at callout_reset_sbt_on+0x10b/frame >> 0xfffffe1f29275450 >> tcp_timer_activate() at tcp_timer_activate+0xe7/frame 0xfffffe1f29275470 >> tcp_do_segment() at tcp_do_segment+0x96/frame 0xfffffe1f292755b0 >> tcp_input() at tcp_input+0xeed/frame 0xfffffe1f292756f0 >> ip_input() at ip_input+0x97/frame 0xfffffe1f29275740 >> netisr_dispatch_src() at netisr_dispatch_src+0x62/frame >> 0xfffffe1f292757b0 >> ether_demux() at ether_demux+0x126/frame 0xfffffe1f292757e0 >> ether_nh_input() at ether_nh_input+0x349/frame 0xfffffe1f29275840 >> netisr_dispatch_src() at netisr_dispatch_src+0x62/frame >> 0xfffffe1f292758b0 >> tcp_lro_flush() at tcp_lro_flush+0x198/frame 0xfffffe1f292758d0 >> ixgbe_rxeof() at ixgbe_rxeof+0x6b3/frame 0xfffffe1f29275990 >> ixgbe_msix_que() at ixgbe_msix_que+0xba/frame 0xfffffe1f292759e0 >> intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame >> 0xfffffe1f29275a20 >> ithread_loop() at ithread_loop+0x96/frame 0xfffffe1f29275a70 >> fork_exit() at fork_exit+0x9a/frame 0xfffffe1f29275ab0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1f29275ab0 >> --- trap 0, rip = 0, rsp = 0xfffffe1f29275b70, rbp = 0 --- >> Uptime: 8d20h4m58s >> >