From owner-freebsd-net@freebsd.org Fri Apr 20 04:03:39 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2186F9FB43 for ; Fri, 20 Apr 2018 04:03:39 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4730C73F9D for ; Fri, 20 Apr 2018 04:03:39 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.15.2/8.15.2) with ESMTP id w3K43bci032886 for ; Fri, 20 Apr 2018 00:03:37 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.15.2/8.14.4/Submit) id w3K43brZ032885; Fri, 20 Apr 2018 00:03:37 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <23257.26265.720293.659892@hergotha.csail.mit.edu> Date: Fri, 20 Apr 2018 00:03:37 -0400 From: Garrett Wollman To: freebsd-net@freebsd.org Subject: Diagnosing terrible ixl performance X-Mailer: VM 8.2.0b under 25.3.1 (amd64-portbld-freebsd10.3) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (hergotha.csail.mit.edu [127.0.0.1]); Fri, 20 Apr 2018 00:03:38 -0400 (EDT) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hergotha.csail.mit.edu X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 04:03:39 -0000 I'm commissioning a new NFS server with an Intel dual-40G XL710 interface, running 11.1. I have a few other servers with this adapter, although not running 40G, and they work fine so long as you disable TSO. This one ... not so much. On the receive side, it gets about 600 Mbit/s with lots of retransmits. On the *sending* side, though, it's not even able to sustain 10 Mbit/s -- but there's no evidence of retransmissions, it's just sending really really slowly. (Other machines with XL710 adapters are able to sustain full 10G.) There is no evidence of any errors on either the adapter or the switch it's connected to. So far, I've tried: - Using the latest Intel driver (no change) - Using the latest Intel firmware (breaks the adapter) - Disabling performance tweaks in loader.conf and sysctl.conf - Changing congestion-control algorithms Anyone have suggestions while I still have time to test this? (My plan B is to fall back to an X520 card that I have in my spares kit, because I *know* those work great with no faffing about.) Any relevant MIBs to inspect? The test I'm doing here is simple iperf over TCP, with MTU 9120. It takes about 10 seconds for the sending side to complete, but buffers are severely constipated for 20 seconds after that (delaying all traffic, including ssh connections). I'm at the point of trying different switch ports just to eliminate that as a possibility. -GAWollman