From owner-freebsd-net@freebsd.org Fri Apr 20 21:30:42 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95406FA3E2D for ; Fri, 20 Apr 2018 21:30:42 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by mx1.freebsd.org (Postfix) with ESMTP id 36EB081D66 for ; Fri, 20 Apr 2018 21:30:41 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id E80A617FC2; Fri, 20 Apr 2018 14:20:45 -0700 (PDT) Date: Fri, 20 Apr 2018 14:20:45 -0700 From: hiren panchasara To: Garrett Wollman Cc: freebsd-net@freebsd.org Subject: Re: Diagnosing terrible ixl performance Message-ID: <20180420212045.GB3653@strugglingcoder.info> References: <23257.26265.720293.659892@hergotha.csail.mit.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="xXmbgvnjoT4axfJE" Content-Disposition: inline In-Reply-To: <23257.26265.720293.659892@hergotha.csail.mit.edu> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 21:30:42 -0000 --xXmbgvnjoT4axfJE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 04/20/18 at 12:03P, Garrett Wollman wrote: > I'm commissioning a new NFS server with an Intel dual-40G XL710 > interface, running 11.1. I have a few other servers with this > adapter, although not running 40G, and they work fine so long as you > disable TSO. This one ... not so much. On the receive side, it gets > about 600 Mbit/s with lots of retransmits. On the *sending* side, > though, it's not even able to sustain 10 Mbit/s -- but there's no > evidence of retransmissions, it's just sending really really slowly. > (Other machines with XL710 adapters are able to sustain full 10G.) > There is no evidence of any errors on either the adapter or the switch > it's connected to. >=20 > So far, I've tried: >=20 > - Using the latest Intel driver (no change) > - Using the latest Intel firmware (breaks the adapter) > - Disabling performance tweaks in loader.conf and sysctl.conf > - Changing congestion-control algorithms >=20 > Anyone have suggestions while I still have time to test this? (My > plan B is to fall back to an X520 card that I have in my spares kit, > because I *know* those work great with no faffing about.) Any > relevant MIBs to inspect? >=20 > The test I'm doing here is simple iperf over TCP, with MTU 9120. It > takes about 10 seconds for the sending side to complete, but buffers > are severely constipated for 20 seconds after that (delaying all > traffic, including ssh connections). >=20 > I'm at the point of trying different switch ports just to eliminate > that as a possibility. You are already trying to check if the switch in-between isn't causing the problem. A few other (probably obvious) things to try: - sysctl -a | grep hw.ixl or dev.ixl to see if you find anything useful (actual name might not be ixl, but you get the point) - Try with lower mtu to see if that's causing anything interesting - If you can reproduce easily, a single stream pcap might be useful from both send and recv side to understand the slowness. Cheers, Hiren --xXmbgvnjoT4axfJE Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJa2lmqXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lzoYH/jyuNt1ZOMMBBm03TdHaIGZj xqj/A3mWnKFsk20UtzRhwtyg31ShWYXLpKoXDS9nTiKNhqX5HRo3A8QmcD6jLa6o rTiYHjKn1orHhEB1lcUwU/LAn5SA9tRKzyVuIVpeyEYt1OG8yaK4V/3zzOVMaLYK DXzrRK9ecjyN1RlRlLpX6ymUxLMbymDUE+KidQ+I87jQTf388JP+gpFBr44qM7gm J4IYlqC+XULWn0XlayAsDmcD8zydrrHmpo2fblPSeleRXVEzh2woW4DGI+cEVtM2 LdLbUHelCEzAmQMfGGfl4VFGmHMVcIE7KBsGZ2j40RzArJL3kZzkBkqzE81WlgY= =Sefu -----END PGP SIGNATURE----- --xXmbgvnjoT4axfJE--