From owner-freebsd-bugs@FreeBSD.ORG Thu Apr 26 14:00:22 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 93367106564A for ; Thu, 26 Apr 2012 14:00:22 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5CB248FC1B for ; Thu, 26 Apr 2012 14:00:22 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3QE0MYt029174 for ; Thu, 26 Apr 2012 14:00:22 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3QE0MfC029173; Thu, 26 Apr 2012 14:00:22 GMT (envelope-from gnats) Resent-Date: Thu, 26 Apr 2012 14:00:22 GMT Resent-Message-Id: <201204261400.q3QE0MfC029173@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Greg Becker Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AEAB1065670 for ; Thu, 26 Apr 2012 13:56:21 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 0CBA38FC18 for ; Thu, 26 Apr 2012 13:56:21 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3QDuKsf024135 for ; Thu, 26 Apr 2012 13:56:20 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q3QDuKLq024134; Thu, 26 Apr 2012 13:56:20 GMT (envelope-from nobody) Message-Id: <201204261356.q3QDuKLq024134@red.freebsd.org> Date: Thu, 26 Apr 2012 13:56:20 GMT From: Greg Becker To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/167325: sosend sometimes return EINVAL with TSO and VLAN on 82599 NIC X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 14:00:22 -0000 >Number: 167325 >Category: kern >Synopsis: sosend sometimes return EINVAL with TSO and VLAN on 82599 NIC >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Apr 26 14:00:22 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Greg Becker >Release: FreeBSD-8.2 >Organization: CacheIQ >Environment: FreeBSD perf-node-1.i.cacheiq.com 8.2-RELEASE FreeBSD 8.2-RELEASE #84: Wed Apr 25 20:34:52 CDT 2012 greg@groover.i.cacheiq.com:/build/trunk/out/PHOENIX/usr/obj/build/trunk/freebsd/RELENG_8/src/sys/PHOENIX amd64 >Description: At CacheIQ we discovered a problem where sosend() would occassionally return EINVAL. This is pretty bad, as it leads our software to believe something was wrong with the calling arguments or perhaps the socket is in a problematic state. Digging into it we found that every now and then requests larger than dmat->maxsize are rejected by bus_dmamap_load_mbuf_sg(). It's important to note that in our environment we are using the Intel 82599 10Gb/s adapter with TSO enabled and VLAN tagging (QINQ). The problem only occurs for TSO packets generated by tcp_output(). When the failure occurs, we see that tcp_ouput() submits an mbuf chain to ip_output() that is 65520 bytes long. ip_output() adds on the link header bringing it to 65534 bytes. Finally, vlan_tag() adds 4 more bytes bringing it to 65538 bytes, and thus failing the lower level checks. As a quick fix we factor in max_linkhdr into the TSO length adjustment in tcp_output(). >How-To-Repeat: Enable TSO and VLAN tagging for 82599 NIC, and drive a lot of data through the system. We use a custom tool, but any networking tool that can keep the link saturated should illustrate the problem. Note, our software is based on FreeBSD-8.2, but includes a number of fixes and select updates. >Fix: Factor max_linkhdr into the TSO length adjustment in tcp_output(). Patch attached with submission follows: Index: freebsd/RELENG_8/src/sys/netinet/tcp_output.c =================================================================== --- freebsd/RELENG_8/src/sys/netinet/tcp_output.c (revision 5803) +++ freebsd/RELENG_8/src/sys/netinet/tcp_output.c (working copy) @@ -747,9 +747,9 @@ if (len + optlen + ipoptlen > tp->t_maxopd) { flags &= ~TH_FIN; if (tso) { - if (len > TCP_MAXWIN - hdrlen - optlen) { - len = TCP_MAXWIN - hdrlen - optlen; - len = len - (len % (tp->t_maxopd - optlen)); + if (len > TCP_MAXWIN - hdrlen - optlen - max_linkhdr) { + len = TCP_MAXWIN - hdrlen - optlen - max_linkhdr; + len = len - (len % (tp->t_maxopd - optlen - max_linkhdr)); sendalot = 1; } else if (tp->t_flags & TF_NEEDFIN) sendalot = 1; >Release-Note: >Audit-Trail: >Unformatted: