From owner-freebsd-net@freebsd.org Fri Apr 20 06:16:00 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84010FA9242 for ; Fri, 20 Apr 2018 06:16:00 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 251A47334B for ; Fri, 20 Apr 2018 06:15:59 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (220-253-154-11.dyn.iinet.net.au [220.253.154.11]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w3K6FdE7041366 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 19 Apr 2018 23:15:42 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: Diagnosing terrible ixl performance To: Garrett Wollman , freebsd-net@freebsd.org References: <23257.26265.720293.659892@hergotha.csail.mit.edu> From: Julian Elischer Message-ID: Date: Fri, 20 Apr 2018 14:15:34 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <23257.26265.720293.659892@hergotha.csail.mit.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 06:16:00 -0000 On 20/4/18 12:03 pm, Garrett Wollman wrote: > I'm commissioning a new NFS server with an Intel dual-40G XL710 > interface, running 11.1. I have a few other servers with this > adapter, although not running 40G, and they work fine so long as you > disable TSO. This one ... not so much. On the receive side, it gets > about 600 Mbit/s with lots of retransmits. On the *sending* side, > though, it's not even able to sustain 10 Mbit/s -- but there's no > evidence of retransmissions, it's just sending really really slowly. > (Other machines with XL710 adapters are able to sustain full 10G.) > There is no evidence of any errors on either the adapter or the switch > it's connected to. > > So far, I've tried: > > - Using the latest Intel driver (no change) > - Using the latest Intel firmware (breaks the adapter) > - Disabling performance tweaks in loader.conf and sysctl.conf > - Changing congestion-control algorithms replacing  card?  obvious but you don't list it... > > Anyone have suggestions while I still have time to test this? (My > plan B is to fall back to an X520 card that I have in my spares kit, > because I *know* those work great with no faffing about.) Any > relevant MIBs to inspect? > > The test I'm doing here is simple iperf over TCP, with MTU 9120. It > takes about 10 seconds for the sending side to complete, but buffers > are severely constipated for 20 seconds after that (delaying all > traffic, including ssh connections). > > I'm at the point of trying different switch ports just to eliminate > that as a possibility. that is also a possibility, as well as swap cables with a good machine... > > -GAWollman > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >