From owner-freebsd-net@FreeBSD.ORG Mon Mar 24 16:23:23 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 19E8F5DA; Mon, 24 Mar 2014 16:23:23 +0000 (UTC) Received: from mail-qg0-x230.google.com (mail-qg0-x230.google.com [IPv6:2607:f8b0:400d:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B79759EA; Mon, 24 Mar 2014 16:23:22 +0000 (UTC) Received: by mail-qg0-f48.google.com with SMTP id j107so17266961qga.7 for ; Mon, 24 Mar 2014 09:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=HZC408ZRAR5ZKn5crL4/up8voxBDnFbZ86OpBmgiICY=; b=azkz133epfpFBehKlZK0Em5m6JWooSK/1uMfSA0x3Um1mbW1oiwNV+Jy8h0+o7PfBY dUCWANRGFXsOdjQA5ll5D0syve9dHENuV5EWb+nZW8Ef97uuDljFxBFHnzjmLGUGP/4L NKnfiBSHNRWlXs7D5nNMP9bXgwlCs1EMzonnvlimofTyuz3HhyXzTvkCYKhaVfMF6eeo /JDEwMoCzIvgPVS23SxIXiAaiWI48ovFhAwInQw6iOi0cJjaBmGwZmBjilmnRB+AZz4S ZiJFvRwG3iYL2wfaXfANcK9rCAwaJuPIipHA03pi8Gd12O7WojMIoXt4b2C/+jAinbxn F/ig== MIME-Version: 1.0 X-Received: by 10.140.29.68 with SMTP id a62mr58308775qga.57.1395678201389; Mon, 24 Mar 2014 09:23:21 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Mon, 24 Mar 2014 09:23:21 -0700 (PDT) In-Reply-To: <0BC10908-2081-45AC-A1C8-14220D81EC0A@hostpoint.ch> References: <1164414873.1690348.1395622026185.JavaMail.root@uoguelph.ca> <0BC10908-2081-45AC-A1C8-14220D81EC0A@hostpoint.ch> Date: Mon, 24 Mar 2014 13:23:21 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Markus Gebert Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Rick Macklem , Garrett Wollman , Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 16:23:23 -0000 I think making hw_tsomax a sysctl would be a good patch to commit - It could enable easy debugging/performance testing for the masses. I'm curious to hear how your environment is working with a tso turned off on your nics. My testbed just hit the 2 hour mark. With TSO off, I don't get a single packet over IP_MAXPACKET. That puts my confidence at around 95% in the statement 'turning off tso negates this issue for me'. I'm now rebooting into a +tso env to see if I can capture the bad packets. I am also sure that the netstat -m mbuf denied is a completely separate issue. I'm going around the lab and powering up different boxes with 10.0-RELEASE, and they all have mbuf/mbuf clusters denied on boot, and that number increases with network traffic. It's probably not helping the IP_MAXPACKET issue. I'll create a separate thread for that one shortly. On Mon, Mar 24, 2014 at 1:14 PM, Markus Gebert wrote: > > On 24.03.2014, at 16:21, Christopher Forgeron > wrote: > > > This is regarding the TSO patch that Rick suggested earlier. (With many > > thanks for his time and suggestion) > > > > As I mentioned earlier, it did not fix the issue on a 10.0 system. It did > > make it less of a problem on 9.2, but either way, I think it's not > needed, > > and shouldn't be considered as a patch for testing/etc. > > > > Patching TSO to anything other than a max value (and by default the code > > gives it IP_MAXPACKET) is confusing the matter, as the packet length > > ultimately needs to be adjusted for many things on the fly like TCP > > Options, etc. Using static header sizes won't be a good idea. > > > > Additionally, it seems that setting nic TSO will/may be ignored by code > > like this in sys/netinet/tcp_output.c: > > > > 10.0 Code: > > > > 780 if (len > tp->t_tsomax - hdrlen) > > { !! > > 781 len = tp->t_tsomax - > > hdrlen; !! > > 782 sendalot = > > 1; > > 783 } > > > > > > I've put debugging here, set the nic's max TSO as per Rick's patch ( set > to > > say 32k), and have seen that tp->t_tsomax == IP_MAXPACKET. It's being set > > someplace else, and thus our attempts to set TSO on the nic may be in > vain. > > > > It may have mattered more in 9.2, as I see the code doesn't use > > tp->t_tsomax in some locations, and may actually default to what the nic > is > > set to. > > > > The NIC may still win, I didn't walk through the code to confirm, it was > > enough to suggest to me that setting TSO wouldn't fix this issue. > > > I just applied Rick's ixgbe TSO patch and additionally wanted to be able > to easily change the value of hw_tsomax, so I made a sysctl out of it. > > While doing that, I asked myself the same question. Where and how will > this value actually be used and how comes that tcp_output() uses that other > value in struct tcpcb. > > The only place tcpcb->t_tsomax gets set, that I have found so far, is in > tcp_input.c's tcp_mss() function. Some subfunctions get called: > > tcp_mss() -> tcp_mss_update() -> tcp_maxmtu() > > Then tcp_maxmtu() indeed uses the interface's hw_tsomax value: > > 1746 cap->tsomax = ifp->if_hw_tsomax; > > It get's passed back to tcp_mss() where it is set on the connection level > which will be used in tcp_output() later on. > > tcp_mss() gets called from multiple places, I'll look into that later. I > will let you know if I find out more. > > > Markus > > > > However, this is still a TSO related issue, it's just not one related to > > the setting of TSO's max size. > > > > A 10.0-STABLE system with tso disabled on ix0 doesn't have a single > packet > > over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit longer to > > increase confidence in this assertion, but I don't want to waste time on > > this when I could be logging problem packets on a system with TSO > enabled. > > > > Comments are very welcome.. > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > >