From owner-freebsd-net@FreeBSD.ORG  Wed Sep 10 15:59:01 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id B6F9898;
 Wed, 10 Sep 2014 15:59:01 +0000 (UTC)
Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1])
 (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 8F8BA170B;
 Wed, 10 Sep 2014 15:59:01 +0000 (UTC)
Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net
 [173.70.85.31])
 by bigwig.baldwin.cx (Postfix) with ESMTPSA id 7B6E7B987;
 Wed, 10 Sep 2014 11:59:00 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: freebsd-net@freebsd.org
Subject: Re: ixgbe(4) spin lock held too long
Date: Wed, 10 Sep 2014 11:24:41 -0400
Message-ID: <1577813.IPE4JfnhZd@ralph.baldwin.cx>
User-Agent: KMail/4.10.5 (FreeBSD/10.0-STABLE; KDE/4.10.5; amd64; ; )
In-Reply-To: <540E04AA.80806@vangyzen.net>
References: <1410203348.1343.1.camel@bruno> <1410203965.1343.3.camel@bruno>
 <540E04AA.80806@vangyzen.net>
MIME-Version: 1.0
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
 (bigwig.baldwin.cx); Wed, 10 Sep 2014 11:59:00 -0400 (EDT)
Cc: Eric van Gyzen <eric@vangyzen.net>
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 10 Sep 2014 15:59:01 -0000

On Monday, September 08, 2014 03:34:02 PM Eric van Gyzen wrote:
> On 09/08/2014 15:19, Sean Bruno wrote:
> > On Mon, 2014-09-08 at 12:09 -0700, Sean Bruno wrote:
> >> This sort of looks like the hardware failed to respond to us in time?
> >> Too busy?
> >> 
> >> sean
> > 
> > This seems to be affecting my 10/stable machines from 15Aug2014.
> > 
> > Not a lot of churn in the code so I don't think this is new.  The
> > afflicted machines, quite a few by my count, appear to have not been
> > super busy (pushing about 200 Mb/s).
> > 
> > sean
> > 
> >> panic: spin lock held too long
> >> 
> >> GNU gdb 6.1.1 [FreeBSD]
> >> Copyright 2004 Free Software Foundation, Inc.
> >> GDB is free software, covered by the GNU General Public License, and you
> >> are
> >> welcome to change it and/or distribute copies of it under certain
> >> conditions.
> >> Type "show copying" to see the conditions.
> >> There is absolutely no warranty for GDB.  Type "show warranty" for
> >> details.
> >> This GDB was configured as "amd64-marcel-freebsd"...
> >> 
> >> Unread portion of the kernel message buffer:
> >> spin lock 0xffffffff812a0400 (callout) held by 0xfffff800151fe000 (tid
> >> 100003) too long
> 
> TID 100003 is usually a kernel idle thread, which would seem to indicate
> a dangling lock.  Can you enable WITNESS (without WITNESS_SKIPSPIN) on
> this box?

Also, do 'tid 100003' and 'bt' in kgdb to see what the thread holding the lock 
was doing.

-- 
John Baldwin