From owner-freebsd-net@FreeBSD.ORG Fri Mar 21 02:47:45 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6C6F847D for ; Fri, 21 Mar 2014 02:47:45 +0000 (UTC) Received: from mail-qc0-x234.google.com (mail-qc0-x234.google.com [IPv6:2607:f8b0:400d:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2B4669EF for ; Fri, 21 Mar 2014 02:47:45 +0000 (UTC) Received: by mail-qc0-f180.google.com with SMTP id w7so2159824qcr.11 for ; Thu, 20 Mar 2014 19:47:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Scma7ukSPD53COwzAmXmTA23/pRgd1PDg9805tca3rg=; b=XJUwWDTZ61JWf1Bna8vEKShVlwJUmh+R0oSte6GlCQ48D6Eoofr89tuD/7IZUnvET7 3BkgofuvixLvMoIPLkOGnvUO/Jn0Z0h0VPeTZI1NQu4smj13e+/tmgeaiDMIafvKQB8M w5vBBfUI9ZAuvJafUfInVpUfbRHRregDf8pXO4k2ds7nXAAc6nBWeez0xR+C5nZhS3gY 2q4JReKwjjJa6gBFp6ekpz3OO/D18xnKQNlx1ZzIOt55N11pkdeOT1eUWcsMI7FBuWng qzSA4DepD0LfLwiFUiIY7GFWHeH5KYhoQZZYzt7eptUFSp8bVNsJ9cgM1elI+rPYHsN9 sCig== MIME-Version: 1.0 X-Received: by 10.224.57.81 with SMTP id b17mr55221730qah.44.1395370064385; Thu, 20 Mar 2014 19:47:44 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Thu, 20 Mar 2014 19:47:44 -0700 (PDT) In-Reply-To: References: <1543350122.637684.1395368002237.JavaMail.root@uoguelph.ca> Date: Thu, 20 Mar 2014 23:47:44 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Mar 2014 02:47:45 -0000 BTW - I think this will end up being a TSO issue, not the patch that Jack applied. When I boot Jack's patch (MJUM9BYTES removal) this is what netstat -m shows: 21489/2886/24375 mbufs in use (current/cache/total) 4080/626/4706/6127254 mbuf clusters in use (current/cache/total/max) 4080/587 mbuf+clusters out of packet secondary zone in use (current/cache) 16384/50/16434/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 79068K/2173K/81241K bytes allocated to network (current/cache/total) 18831/545/4542 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 15626/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile Here is an un-patched boot: 21550/7400/28950 mbufs in use (current/cache/total) 4080/3760/7840/6127254 mbuf clusters in use (current/cache/total/max) 4080/2769 mbuf+clusters out of packet secondary zone in use (current/cache) 0/42/42/3063627 4k (page size) jumbo clusters in use (current/cache/total/max) 16439/129/16568/907741 9k jumbo clusters in use (current/cache/total/max) 0/0/0/510604 16k jumbo clusters in use (current/cache/total/max) 161498K/10699K/172197K bytes allocated to network (current/cache/total) 18345/155/4099 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 3/3723/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile See how removing the MJUM9BYTES is just pushing the problem from the 9k jumbo cluster into the 4k jumbo cluster? Compare this to my FreeBSD 9.2 STABLE machine from ~ Dec 2013 : Exact same hardware, revisions, zpool size, etc. Just it's running an older FreeBSD. # uname -a FreeBSD SAN1.XXXXX 9.2-STABLE FreeBSD 9.2-STABLE #0: Wed Dec 25 15:12:14 AST 2013 aatech@FreeBSD-Update Server:/usr/obj/usr/src/sys/GENERIC amd64 root@SAN1:/san1 # uptime 7:44AM up 58 days, 38 mins, 4 users, load averages: 0.42, 0.80, 0.91 root@SAN1:/san1 # netstat -m 37930/15755/53685 mbufs in use (current/cache/total) 4080/10996/15076/524288 mbuf clusters in use (current/cache/total/max) 4080/5775 mbuf+clusters out of packet secondary zone in use (current/cache) 0/692/692/262144 4k (page size) jumbo clusters in use (current/cache/total/max) 32773/4257/37030/96000 9k jumbo clusters in use (current/cache/total/max) 0/0/0/508538 16k jumbo clusters in use (current/cache/total/max) 312599K/67011K/379611K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines Lastly, please note this link: http://lists.freebsd.org/pipermail/freebsd-net/2012-October/033660.html It's so old that I assume the TSO leak that he speaks of has been patched, but perhaps not. More things to look into tomorrow. On Thu, Mar 20, 2014 at 11:32 PM, Christopher Forgeron wrote: > Yes, there is something broken in TSO for sure, as disabling it allows me > to run without error. It is possible that the drop in performance is > allowing me to stay under a critical threshold for the problem, but I'd > feel happier testing to make sure. > > I understand what you're asking for in the patch, I'll make the edits > tomorrow and recompile a test kernel and see. > > Right now I'm running tests on the ixgbe that Jack sent. Even if his patch > fixes the issue, I wonder if something else isn't broken in TSO, as the > ixgbe code has had these lines for a long time, and it's only on this 10.0 > build that I have issues. > > I'll be following up tomorrow with info on either outcome. > > Thanks for your help.. your rusty networking is still better than mine. :-) > > > On Thu, Mar 20, 2014 at 11:13 PM, Rick Macklem wrote: > >> Christopher Forgeron wrote: >> > >> > Output from the patch you gave me (I have screens of it.. let me know >> > what you're hoping to see. >> > >> > >> > Mar 20 16:37:22 SAN0 kernel: after mbcnt=33 pklen=65538 actl=65538 >> > Mar 20 16:37:22 SAN0 kernel: before pklen=65538 actl=65538 >> Hmm. I think this means that the loop that generates TSO segments in >> tcp_output() is broken, since I'm pretty sure that the maximum size >> should be is IP_MAXPACKET (65535). >> >> Either that or some non-TCP socket is trying to send a packet that >> exceeds IP_MAXPACKET for some reason. >> >> Would it be possible to add a printf() for m->m_pkthdr.csum_flags >> to the before case, in the "if" that generates the before printf? >> I didn't think to put this in, but CSUM_TSO will be set if it >> is a TSO segment, I think? My networking is very rusty. >> (If how to add this isn't obvious, just email and I'll update >> the patch.) >> >> Thanks for doing this, rick >> >> >