From owner-freebsd-net@freebsd.org Fri Jul 22 13:52:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45A8ABA0792 for ; Fri, 22 Jul 2016 13:52:23 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from 004.lax.mailroute.net (004.lax.mailroute.net [199.89.1.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mailroute.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1539A162D for ; Fri, 22 Jul 2016 13:52:22 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from localhost (localhost.localdomain [127.0.0.1]) by 004.lax.mailroute.net (Postfix) with ESMTP id 3rwsSF50tbz15KC6 for ; Fri, 22 Jul 2016 13:48:45 +0000 (UTC) X-Virus-Scanned: by MailRoute Received: from 004.lax.mailroute.net ([199.89.1.7]) by localhost (004.lax.mailroute.net [127.0.0.1]) (mroute_mailscanner, port 10026) with LMTP id HLCn-0y-bFdM for ; Fri, 22 Jul 2016 13:48:44 +0000 (UTC) Received: from zimbra.earthside.net (unknown [50.251.189.243]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by 004.lax.mailroute.net (Postfix) with ESMTPS id 3rwsSC6nr7z15KBq for ; Fri, 22 Jul 2016 13:48:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id B1A2F50267B for ; Fri, 22 Jul 2016 09:48:42 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id QENgP6oNx-ss for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id CDADE50267C for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id sxnSN9G02ob7 for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Received: from zimbra.earthside.net (zimbra.earthside.net [10.11.12.148]) by zimbra.earthside.net (Postfix) with ESMTP id 598CC50267B for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Date: Fri, 22 Jul 2016 09:48:40 -0400 (EDT) From: Chris Dunbar To: freebsd-net Message-ID: <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> In-Reply-To: References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC MIME-Version: 1.0 X-Originating-IP: [10.11.12.148] X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC51 (Mac)/8.6.0_GA_1153) Thread-Topic: Slow performance with Intel X540-T2 10Gb NIC Thread-Index: AEXdseC/lXu8ZlsAGuMNm0T6Ia8d1w== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 13:52:23 -0000 Hi Sami,=20 I haven't actually fixed anything yet. I have only demonstrated that the po= or performance does not appear to happen between two FreeBSD boxes and poss= ibly between a Linux and FreeBSD, but I am going to confirm that now. I hav= e also seen good performance between the Windows box and Linux so that does= n't quite add up either. I may have to break out Wireshark and make some pa= cket captures to see if I can tell what's going on. If I find anything, I w= ill be sure to share it.=20 Regards,=20 Chris=20 From: "Sami Halabi" =20 To: "chris" =20 Cc: "freebsd-net" =20 Sent: Friday, July 22, 2016 8:34:30 AM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 hi,=20 would you share what was wrong in the windows side and how you solved it?= =20 Sami=20 =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 22 =D7=91=D7=99=D7=95=D7=9C=D7=99 2016= 12:33 AM,=E2=80=8F "Chris Dunbar" < chris@dunbar.net > =D7=9B=D7=AA=D7=91:= =20 Hello again,=20 I have good news and bad news:=20 The bad news first: I am an idiot and I have wasted some of your time for w= hich I apologize.=20 The good news: Testing now between two FreeBSD 10.3 systems, I am achieving= blistering speeds with iperf3. I apparently fell into the trap of assuming= the new thing (FreeBSD is new to me) was broken. Now I see that I was assu= ming Windows was working fine and focusing all my attention on FreeBSD. Loo= king back over everything I have done to troubleshoot this situation I must= conclude that the performance issue was on the Windows side and not the Fr= eeBSD side. I am less concerned about that because my ultimate goal is to i= nstall my three X540s into one FreeBSD server and two VMware ESXi hosts. I = am now fairly confident performance will be great.=20 Many thanks for your collective attention and the suggestions I received fr= om Eric and others.=20 Regards,=20 Chris=20 ----- Original Message -----=20 From: "chris" < chris@dunbar.net >=20 To: "freebsd-net" < freebsd-net@freebsd.org >=20 Sent: Thursday, July 21, 2016 4:53:40 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 Eric, et al:=20 I haven't tried netperf yet, but I do have some new information to share. I= have two systems that I am using for testing: the new server and an older = (not too old) desktop PC. I installed CentOS on the new server again becaus= e I know it can achieve >9 GB/s with the X540. I replaced Windows on the de= sktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 again. I wa= s able to achieve >9 GB/s so I know the problem isn't the X540 and I know t= he problem isn't anything with the default installation of FreeBSD 10.3. So= , what in the world might be nutty in my BIOS settings (or elsewhere) that = would cause the new server + FreeBSD 10.3 + X540 to equal slow performance?= =20 Regards,=20 Chris=20 From: "Eric Joyner" < erj@freebsd.org >=20 To: "chris" < chris@dunbar.net >, "freebsd-net" < freebsd-net@freebsd.org >= =20 Sent: Thursday, July 21, 2016 1:27:10 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 (Replying-all this time)=20 Did you try to set these settings that ESnet recommends? https://fasterdata= .es.net/host-tuning/freebsd/=20 We don't use iperf3 here at Intel (we use netperf instead), so I'm not sure= I can be much help diagnosing what's wrong.=20 On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote:=20 Hello,=20 I am new to FreeBSD and recently built a file server out of new components = running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am experie= ncing what I consider to be slow transfer speeds. I am using iperf3 to meas= ure the speed and test the results of modifications. So far nothing I have = done has made a noticeable difference. If I run iperf3 -s on the FreeBSD se= rver, I see transfer speeds of approximately 1.6 Gb/s. If I run iperf3 in c= lient mode, the speed improves to ~2.75 Gb/s. However, if I replace FreeBSD= with CentOS 7 on the same hardware, I see iperf3 speeds surpassing 8 GB/s.= The other end of my iperf3 test is a Windows 10 box that also has an Intel= X540-T2 installed.=20 I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes = a slightly older Intel driver (v3.1.13-k). I managed to build a custom kern= el that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me to ma= nually load the latest 3.1.14 driver downloaded from Intel's web site. Unfo= rtunately that did not produce any improvements. I am working my way throug= h man tuning() and some other articles on network performance. So far nothi= ng I tweak makes a noticeable difference. I'm increasingly skeptical that I= am going to find a setting or two that more than doubles the speed I am cu= rrently experiencing.=20 I am open to any and all suggestions at this point.=20 Thank you!=20 Chris=20 _______________________________________________=20 freebsd-net@freebsd.org mailing list=20 https://lists.freebsd.org/mailman/listinfo/freebsd-net=20 To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org "=20 _______________________________________________=20 freebsd-net@freebsd.org mailing list=20 https://lists.freebsd.org/mailman/listinfo/freebsd-net=20 To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org "=20 _______________________________________________=20 freebsd-net@freebsd.org mailing list=20 https://lists.freebsd.org/mailman/listinfo/freebsd-net=20 To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org "=20