From owner-freebsd-net@FreeBSD.ORG Tue Sep 13 20:58:24 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E6951065670 for ; Tue, 13 Sep 2011 20:58:24 +0000 (UTC) (envelope-from lacombar@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id E33D48FC17 for ; Tue, 13 Sep 2011 20:58:23 +0000 (UTC) Received: by ywp17 with SMTP id 17so1012383ywp.13 for ; Tue, 13 Sep 2011 13:58:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=gDqwT14GIApksU9EysfYZOOYEt51LfsV/iLitqaz4Ew=; b=POADrmvezMAmg2LgJusm69BBzMrSWdYSSTvryRUqOJAAdBgQeoD5iZlO5z74vWkb8m 2F+m67JhpouUbMQjeKr5mry18E7p8eiH4WpxXjftkCanYBReI6oERd6obgO1hoS+Qwcw PU6NMbK6t1b1uhAN6iSrqWmUWQ/YaZYexCylw= MIME-Version: 1.0 Received: by 10.68.44.202 with SMTP id g10mr2207030pbm.63.1315947502741; Tue, 13 Sep 2011 13:58:22 -0700 (PDT) Received: by 10.142.12.18 with HTTP; Tue, 13 Sep 2011 13:58:22 -0700 (PDT) In-Reply-To: References: Date: Tue, 13 Sep 2011 16:58:22 -0400 Message-ID: From: Arnaud Lacombe To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: FreeBSD 7-STABLE mbuf corruption X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Sep 2011 20:58:24 -0000 Hi, On Tue, Sep 13, 2011 at 2:36 PM, Arnaud Lacombe wrote: > Hi, > > On Wed, Sep 7, 2011 at 7:57 PM, Jack Vogel wrote: >> I have seen this, but I don't have any hot ideas right off the top of my >> head yet :( >> > I've been running for 19h now the following patches: > =A0- backport of kmacy@'s buf_ring(9) API, from FreeBSD 8 (from [0], see > attachment for full diff) > =A0- conversion of igb(4), from CURRENT, to use buf_ring(9) on FreeBSD > 7.1 (see attachment) > =A0- all the original patches I already sent > > It did not crash, yet. The only downside is that after 3h30 and ~4h, > igb(4) queues' handler started spinning infinitely, breaking network > connectivity. > > I would be tempted to say that the infinite loop issue is an igb(4) > (separate from the original crashes), Here's a backtrace of the 3 "igb1 que" spinning on CPU0, CPU1 and CPU3, when the condition occurs: Tracing command igb1 que pid 44 tid 100043 td 0xc6a79460 cpustop_handler(9e0008,28,28,c0838783,1220,...) at cpustop_handler+0x32 Xcpustop() at Xcpustop+0x2d --- interrupt, eip =3D 0xc062f518, esp =3D 0xf4589bbc, ebp =3D 0xf4589bdc -= -- _mtx_unlock_flags(c69e7e9c,0,c0838783,1220,c6a3f8d4,...) at _mtx_unlock_flags+0x98 igb_rxeof(f4589ca4,c062f502,c6a73b9c,4,c0850e5e,...) at igb_rxeof+0x543 igb_handle_que(c6a3f8b4,1,c0856713,54,c6a73b84,...) at igb_handle_que+0x76 taskqueue_run(c6a73b80,c6a73b9c,0,c08492c5,0,...) at taskqueue_run+0xa1 taskqueue_thread_loop(c6a3f8e4,f4589d38,c084e8d3,31c,c6a7f570,...) at taskqueue_thread_loop+0x6a fork_exit(c06718f0,c6a3f8e4,f4589d38) at fork_exit+0xa6 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xf4589d70, ebp =3D 0 --- Tracing command igb1 que pid 42 tid 100041 td 0xc6a798c0 cpustop_handler(8,28,f4580028,0,0,...) at cpustop_handler+0x32 Xcpustop() at Xcpustop+0x2d --- interrupt, eip =3D 0xc07f6d5b, esp =3D 0xf4583c40, ebp =3D 0xf4583c44 -= -- spinlock_exit(c08e58b4,0,c0856087,108,f4583c7c,...) at spinlock_exit+0x2b sleepq_release(c6a73c00,0,ffffffff,0,c6a73c00,...) at sleepq_release+0x37 wakeup_one(c6a73c00,0,c0856713,4b,c6a73c1c,...) at wakeup_one+0x41 taskqueue_enqueue(c6a73c00,c6a3f898,c0856713,54,c6a73c04,...) at taskqueue_enqueue+0x169 taskqueue_run(c6a73c00,c6a73c1c,0,c08492c5,0,...) at taskqueue_run+0xa1 taskqueue_thread_loop(c6a3f8a8,f4583d38,c084e8d3,31c,c6a7fae0,...) at taskqueue_thread_loop+0x6a fork_exit(c06718f0,c6a3f8a8,f4583d38) at fork_exit+0xa6 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xf4583d70, ebp =3D 0 --- Tracing command igb1 que pid 40 tid 100039 td 0xc6a79d20 cpustop_handler(8,f4570028,28,0,c6a3f85c,...) at cpustop_handler+0x32 Xcpustop() at Xcpustop+0x2d --- interrupt, eip =3D 0xc07f6d5b, esp =3D 0xf457dc44, ebp =3D 0xf457dc48 -= -- spinlock_exit(c08e5f44,0,c0856087,108,f457dc88,...) at spinlock_exit+0x2b sleepq_release(c6a3f85c,c6a73c80,f457dc88,c066f7a7,c08e5f44,...) at sleepq_release+0x37 sleepq_broadcast(c6a3f85c,0,ffffffff,0,c6a3f85c,...) at sleepq_broadcast+0x= d7 wakeup(c6a3f85c,0,c0856713,4b,c6a73c84,...) at wakeup+0x32 taskqueue_run(c6a73c80,c6a73c9c,0,c08492c5,0,...) at taskqueue_run+0xdb taskqueue_thread_loop(c6a3f86c,f457dd38,c084e8d3,31c,c6a2a000,...) at taskqueue_thread_loop+0x6a fork_exit(c06718f0,c6a3f86c,f457dd38) at fork_exit+0xa6 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip =3D 0, esp =3D 0xf457dd70, ebp =3D 0 --- - Arnaud