From owner-freebsd-net@FreeBSD.ORG Mon Mar 24 16:14:17 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8701125D; Mon, 24 Mar 2014 16:14:17 +0000 (UTC) Received: from mail.adm.hostpoint.ch (mail.adm.hostpoint.ch [IPv6:2a00:d70:0:a::e0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 41C86909; Mon, 24 Mar 2014 16:14:17 +0000 (UTC) Received: from [2001:1620:2013:1:7810:eaed:8406:ff6] (port=49631) by mail.adm.hostpoint.ch with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1WS7Vt-0005dw-9M; Mon, 24 Mar 2014 17:14:13 +0100 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: 9.2 ixgbe tx queue hang From: Markus Gebert In-Reply-To: Date: Mon, 24 Mar 2014 17:14:08 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <0BC10908-2081-45AC-A1C8-14220D81EC0A@hostpoint.ch> References: <1164414873.1690348.1395622026185.JavaMail.root@uoguelph.ca> To: Christopher Forgeron X-Mailer: Apple Mail (2.1874) Cc: FreeBSD Net , Rick Macklem , Garrett Wollman , Jack Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 16:14:17 -0000 On 24.03.2014, at 16:21, Christopher Forgeron = wrote: > This is regarding the TSO patch that Rick suggested earlier. (With = many > thanks for his time and suggestion) >=20 > As I mentioned earlier, it did not fix the issue on a 10.0 system. It = did > make it less of a problem on 9.2, but either way, I think it's not = needed, > and shouldn't be considered as a patch for testing/etc. >=20 > Patching TSO to anything other than a max value (and by default the = code > gives it IP_MAXPACKET) is confusing the matter, as the packet length > ultimately needs to be adjusted for many things on the fly like TCP > Options, etc. Using static header sizes won't be a good idea. >=20 > Additionally, it seems that setting nic TSO will/may be ignored by = code > like this in sys/netinet/tcp_output.c: >=20 > 10.0 Code: >=20 > 780 if (len > tp->t_tsomax - hdrlen) > { !! > 781 len =3D tp->t_tsomax - > hdrlen; !! > 782 sendalot =3D > 1; > 783 } >=20 >=20 > I've put debugging here, set the nic's max TSO as per Rick's patch ( = set to > say 32k), and have seen that tp->t_tsomax =3D=3D IP_MAXPACKET. It's = being set > someplace else, and thus our attempts to set TSO on the nic may be in = vain. >=20 > It may have mattered more in 9.2, as I see the code doesn't use > tp->t_tsomax in some locations, and may actually default to what the = nic is > set to. >=20 > The NIC may still win, I didn't walk through the code to confirm, it = was > enough to suggest to me that setting TSO wouldn't fix this issue. I just applied Rick=92s ixgbe TSO patch and additionally wanted to be = able to easily change the value of hw_tsomax, so I made a sysctl out of = it. While doing that, I asked myself the same question. Where and how will = this value actually be used and how comes that tcp_output() uses that = other value in struct tcpcb. The only place tcpcb->t_tsomax gets set, that I have found so far, is in = tcp_input.c=92s tcp_mss() function. Some subfunctions get called: tcp_mss() -> tcp_mss_update() -> tcp_maxmtu() Then tcp_maxmtu() indeed uses the interface=92s hw_tsomax value: 1746 cap->tsomax =3D ifp->if_hw_tsomax; It get=92s passed back to tcp_mss() where it is set on the connection = level which will be used in tcp_output() later on. tcp_mss() gets called from multiple places, I=92ll look into that later. = I will let you know if I find out more. Markus > However, this is still a TSO related issue, it's just not one related = to > the setting of TSO's max size. >=20 > A 10.0-STABLE system with tso disabled on ix0 doesn't have a single = packet > over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit longer to > increase confidence in this assertion, but I don't want to waste time = on > this when I could be logging problem packets on a system with TSO = enabled. >=20 > Comments are very welcome.. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20