From owner-freebsd-net@FreeBSD.ORG Mon Mar 24 15:21:29 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3A22F874; Mon, 24 Mar 2014 15:21:29 +0000 (UTC) Received: from mail-qc0-x22b.google.com (mail-qc0-x22b.google.com [IPv6:2607:f8b0:400d:c01::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C53C9167; Mon, 24 Mar 2014 15:21:28 +0000 (UTC) Received: by mail-qc0-f171.google.com with SMTP id c9so6142708qcz.16 for ; Mon, 24 Mar 2014 08:21:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=05A6UjaX2avV4XRZJqQulpScg8ZrhsljB/RGsgcr6Po=; b=Gh0yWzmcN+6jt/QxaT+Rv1i8znbQHXkKWiw2FA8Go8nns+TT4tKaf1pXm/+aqDjXDh A14ynSpHcO3R2NXpjdz69oH7acIc2LCewMsLQ/HI/CsBEHWBwtM9e6NplnWoa4OosLqV re5b43n0nq/VwZtQs5UQZ28f1FNXzGiKMYVd7+pkDC7kylEA2XNqGhLz1UYrwZpR/TSJ 2nltOYWtxcec59AcJVSRRXbuzAnntS6WOFUxmkQx9WJPau9/MpvJWvGXj1jyZ/dHRyQy q+4/xU2pwLVlyFostQkMsaZfwq7Zj/KRLi1jNrqxeKmuvw80E22I9747finkzbjA+GVu MflA== MIME-Version: 1.0 X-Received: by 10.224.69.66 with SMTP id y2mr9605259qai.25.1395674486649; Mon, 24 Mar 2014 08:21:26 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Mon, 24 Mar 2014 08:21:26 -0700 (PDT) In-Reply-To: References: <1164414873.1690348.1395622026185.JavaMail.root@uoguelph.ca> Date: Mon, 24 Mar 2014 12:21:26 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Garrett Wollman , Jack Vogel , Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2014 15:21:29 -0000 This is regarding the TSO patch that Rick suggested earlier. (With many thanks for his time and suggestion) As I mentioned earlier, it did not fix the issue on a 10.0 system. It did make it less of a problem on 9.2, but either way, I think it's not needed, and shouldn't be considered as a patch for testing/etc. Patching TSO to anything other than a max value (and by default the code gives it IP_MAXPACKET) is confusing the matter, as the packet length ultimately needs to be adjusted for many things on the fly like TCP Options, etc. Using static header sizes won't be a good idea. Additionally, it seems that setting nic TSO will/may be ignored by code like this in sys/netinet/tcp_output.c: 10.0 Code: 780 if (len > tp->t_tsomax - hdrlen) { !! 781 len = tp->t_tsomax - hdrlen; !! 782 sendalot = 1; 783 } I've put debugging here, set the nic's max TSO as per Rick's patch ( set to say 32k), and have seen that tp->t_tsomax == IP_MAXPACKET. It's being set someplace else, and thus our attempts to set TSO on the nic may be in vain. It may have mattered more in 9.2, as I see the code doesn't use tp->t_tsomax in some locations, and may actually default to what the nic is set to. The NIC may still win, I didn't walk through the code to confirm, it was enough to suggest to me that setting TSO wouldn't fix this issue. However, this is still a TSO related issue, it's just not one related to the setting of TSO's max size. A 10.0-STABLE system with tso disabled on ix0 doesn't have a single packet over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit longer to increase confidence in this assertion, but I don't want to waste time on this when I could be logging problem packets on a system with TSO enabled. Comments are very welcome..