From owner-freebsd-net@FreeBSD.ORG Sat Jan 24 18:56:20 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F6AF1C5 for ; Sat, 24 Jan 2015 18:56:20 +0000 (UTC) Received: from mail-ie0-x22a.google.com (mail-ie0-x22a.google.com [IPv6:2607:f8b0:4001:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32FB11C7 for ; Sat, 24 Jan 2015 18:56:20 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id y20so2737444ier.1 for ; Sat, 24 Jan 2015 10:56:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=Kw67UXRJq+ZdWtF/1MnJfvcBOheNGlTgQnLrcirCiv8=; b=oV8wzAtKxFMXYMoBLMEpTWzwWPO4hDUgaK43cd5WC5y1NtgqW17e+KD+Sevo1ycr1N UIfXyJLRjd3eaVHIWxSlaaHqEyGOqv8qfaVnwGgTjZC1gySZhBNgAA6eXh7G81K7G1Vv icyg/4u7cGMG6hFp09reaywnqdFKraRF4Du+5Y7++a8kv9Dc29bcCft5pzSwbr8A6594 /ew2891DCg/86/ZGA8/Go9f3JfUCFUXWc4hj0gML4ZM4no0S7K6zxhKPDLzZtDBhhHRK oj/D9p0VGHvj0RlVBQHRS5yUMhLsSoTywKr4VdrMovoSPrRgHdxroVEjX9lKD3kZzaot MtpQ== MIME-Version: 1.0 X-Received: by 10.42.201.78 with SMTP id ez14mr14306975icb.22.1422125779565; Sat, 24 Jan 2015 10:56:19 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.36.78.14 with HTTP; Sat, 24 Jan 2015 10:56:19 -0800 (PST) In-Reply-To: References: <5B08402C-67A7-49E7-ADF8-390C94DCF1D7@gmail.com> Date: Sat, 24 Jan 2015 10:56:19 -0800 X-Google-Sender-Auth: XtJUqD3dhwxByp8a_48OMS5swPI Message-ID: Subject: Re: network locks up with udp traffic From: Adrian Chadd To: Dmitry Sivachenko Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jan 2015 18:56:20 -0000 Hi! To be clear: * is your kernel modified in any way; and * did witness give you a full stacktrace as part of the lock order reversal? All of that would be good. Thanks, On 24 January 2015 at 09:53, Dmitry Sivachenko wrote: > >> On 24 =D1=8F=D0=BD=D0=B2. 2015 =D0=B3., at 20:29, Adrian Chadd wrote: >> >> Hi, >> >> Can you compile your kernel with WITNESS and repeat this test? It >> looks like it's stuck in some lock ordering thing between some UDP >> paths and an inpcb lookup. >> >> It should log in dmesg the first time a LOR occurs, regardless of >> whether it actively causes a hang. > > > > Here is what I got in dmesg just before my net locked up: > > lock order reversal: > 1st 0xffffffff80e79008 pcbinfohash (pcbinfohash) @ /opt/WRK/src/sys/neti= net6/udp6_usrreq.c:1202 > 2nd 0xffffffff80e78fb0 udp (udp) @ /opt/WRK/src/sys/netinet6/in6_pcb.c:6= 14 > lock order reversal: > 1st 0xffffffff80e79008 pcbinfohash (pcbinfohash) @ /opt/WRK/src/sys/neti= net6/udp6_usrreq.c:1202 > 2nd 0xffffffff80e78d58 tcp (tcp) @ /opt/WRK/src/sys/netinet6/in6_pcb.c:6= 14 > lock order reversal: > 1st 0xffffffff80e79008 pcbinfohash (pcbinfohash) @ /opt/WRK/src/sys/neti= net6/udp6_usrreq.c:1202 > 2nd 0xffffffff80e781c0 rip (rip) @ /opt/WRK/src/sys/netinet6/in6_pcb.c:6= 14 > > When I was saving this dmesg output to disk, I got another LOR: > > lock order reversal: > 1st 0xfffffe0baf12fd78 bufwait (bufwait) @ /opt/WRK/src/sys/kern/vfs_bio= .c:3065 > 2nd 0xfffff80011416c00 dirhash (dirhash) @ /opt/WRK/src/sys/ufs/ufs/ufs_= dirhash.c:284 > > > Please tell me if I can provide more information to help tracking this do= wn. > > Thanks!