From owner-freebsd-net@freebsd.org Tue Aug 18 07:17:32 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAE3D9A0522; Tue, 18 Aug 2015 07:17:32 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44F84616; Tue, 18 Aug 2015 07:17:31 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from mbpro-w.cs.huji.ac.il ([132.65.80.91]) by kabab.cs.huji.ac.il with esmtp id 1ZRb99-0000T0-QC; Tue, 18 Aug 2015 10:17:23 +0300 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance From: Daniel Braniss In-Reply-To: <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> Date: Tue, 18 Aug 2015 10:07:23 +0300 Cc: FreeBSD Net , Slawa Olhovchenkov , FreeBSD stable , Christopher Forgeron Message-Id: <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Aug 2015 07:17:32 -0000 > On Aug 18, 2015, at 12:49 AM, Rick Macklem = wrote: >=20 > Daniel Braniss wrote: >>=20 >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem = wrote: >>>=20 >>> Daniel Braniss wrote: >>>>=20 >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron = >>>>> wrote: >>>>>=20 >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and = FreeBSD >>>>> 10.1. Before 10.1 it was less. >>>>>=20 >>>>=20 >>>> this is NOT iperf/3 where i do get close to wire speed, >>>> it=E2=80=99s NFS writes, i.e., almost real work :-) >>>>=20 >>>>> I used to tweak the card settings, but now it's just stock. You = may want >>>>> to >>>>> check your settings, the Mellanox may just have better defaults = for your >>>>> switch. >>>>>=20 >>> Have you tried disabling TSO for the Intel? With TSO enabled, it = will be >>> copying >>> every transmitted mbuf chain to a new chain of mbuf clusters via. >>> m_defrag() when >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be = the >>> 82599 chip >>> these days?) >>>=20 >>=20 >> hi Rick >>=20 >> how can i check the chip? >>=20 > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling = TSO helped, > I'll bet you don't have a 82598.) >=20 >>> This has been fixed in the driver very recently, but those fixes = won't be >>> in 10.1. >>>=20 >>> rick >>> ps: If you could test with 10.2, it would be interesting to see how = the ix >>> does with >>> the current driver fixes in it? >>=20 >> I new TSO was involved! >> ok, firstly, it=E2=80=99s 10.2 stable. >> with TSO enabled, ix is bad, around 64MGB/s. >> disabling TSO it=E2=80=99s better, around 130 >>=20 > Hmm, could you check to see of these lines are in = sys/dev/ixgbe/if_ix.c at around > line#2500? > /* TSO parameters */ > 2572 ifp->if_hw_tsomax =3D 65518; > 2573 ifp->if_hw_tsomaxsegcount =3D = IXGBE_82599_SCATTER; > 2574 ifp->if_hw_tsomaxsegsize =3D 2048; >=20 > They are in stable/10. I didn't look at releng/10.2. (And if they're = in a #ifdef > for FreeBSD11, take the #ifdef away.) > If they are there and not ifdef'd, I can't explain why disabling TSO = would help. > Once TSO is fixed so that it handles the 64K transmit segments without = copying all > the mbufs, I suspect you might get better perf. with it enabled? >=20 this is 10.2 : they are on lines 2509-2511 and I don=E2=80=99t see any #ifdefs around = it. the plot thickens :-) danny > Good luck with it, rick >=20 >> still, mlxen0 is about 250! with and without TSO >>=20 >>=20 >>>=20 >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov = >>>> > wrote: >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: >>>>>=20 >>>>>> hi, >>>>>> I have a host (Dell R730) with both cards, connected to an = HP8200 >>>>>> switch at 10Gb. >>>>>> when writing to the same storage (netapp) this is what I get: >>>>>> ix0: ~130MGB/s >>>>>> mlxen0 ~330MGB/s >>>>>> this is via nfs/tcpv3 >>>>>>=20 >>>>>> I can get similar (bad) performance with the mellanox if I = increase >>>>>> the file size >>>>>> to 512MGB. >>>>>=20 >>>>> Look like mellanox have internal beffer for caching and do ACK >>>>> acclerating. >>>>>=20 >>>>>> so at face value, it seems the mlxen does a better use of = resources >>>>>> than the intel. >>>>>> Any ideas how to improve ix/intel's performance? >>>>>=20 >>>>> Are you sure about netapp performance? >>>>> _______________________________________________ >>>>> freebsd-net@freebsd.org mailing = list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>> >>>>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org >>>>> " >>>>>=20 >>>>=20 >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"