From owner-freebsd-net@freebsd.org Wed Aug 19 02:42:01 2015 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB1929BD78A; Wed, 19 Aug 2015 02:42:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 74DB3191C; Wed, 19 Aug 2015 02:42:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:eIbFqhDlnn1Crzv6DF2DUyQJP3N1i/DPJgcQr6AfoPdwSPn6p8bcNUDSrc9gkEXOFd2CrakU0KyK7uu+AiQp2tWojjMrSNR0TRgLiMEbzUQLIfWuLgnFFsPsdDEwB89YVVVorDmROElRH9viNRWJ+iXhpQAbFhi3DwdpPOO9QteU1JTskbzvsMOIKyxzxxODIppKZC2sqgvQssREyaBDEY0WjiXzn31TZu5NznlpL1/A1zz158O34YIxu38I46Fp34d6XK77Z6U1S6BDRHRjajhtpZ6jiB/YUAHa5mcASn5E1V1MAhPZ91f0RJr8uDD28O1n126fNMzySLkyHjCj9LtqThHvzykdOjMz622SkdB5hqZW8y+nvAF1lo7IfJmOZr05eqLGYchcS3BMU8xKW2pGGIz7aoIOC+8IO6FcrpLhpl0AqlywHwShDvjjjyRUj3Xy0P4H1f88G1TGwBA4BIBJ93DVt8nucqkIXO2/16WOyi/MKPZf2DP44Y6PdhE6vfCKU7U3f9DcxEM0G0bDg0nDlYuwEzqT1+kJ+0KB5uxhTvnn32IurQdgijO0gMcxiIiPj4lTy1SSsW1ZyYAubeW1VFJ2e5afHZ9ZrCKLf992Wc4mSnprqQ400LALs4W3Oi8Qx8J06QTYbqm9coOLqjfqX+WVLDIw0Ghgcbm8gxu32VWnxfDxUtG0ll1D+HkW2uLQv2wAgkSAovOMTeFwqwL4gW6C X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2A6AgBW7NNV/61jaINdDoNhaQaDH7pfAQmBbQqFMUoCgXYUAQEBAQEBAQGBCYIdggcBAQQBAQEgBCcgCxACAQgOCgICDRYDAgIhBgEJFRECBAEHBwQBHASHeAMSDbp5kDQNhVcBAQEBAQEBAwEBAQEBAQEXBIEiijGCT4FiAQYBAQcVATMHgmmBQwWHI41/hQSFBnWDN5Evg0+DZQImgz9aIjMHfgEIFyOBBAEBAQ X-IronPort-AV: E=Sophos;i="5.15,706,1432612800"; d="scan'208";a="231577093" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 18 Aug 2015 22:41:59 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id A1A3F15F55D; Tue, 18 Aug 2015 22:41:59 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id wmVkoLeVSGkc; Tue, 18 Aug 2015 22:41:58 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7FE1B15F571; Tue, 18 Aug 2015 22:41:58 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ZSabU6gBtopC; Tue, 18 Aug 2015 22:41:58 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 6054715F55D; Tue, 18 Aug 2015 22:41:58 -0400 (EDT) Date: Tue, 18 Aug 2015 22:41:58 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Daniel Braniss <danny@cs.huji.ac.il>, Hans Petter Selasky <hselasky@freebsd.org> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Christopher Forgeron <csforgeron@gmail.com>, FreeBSD stable <freebsd-stable@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru> Message-ID: <333280926.25456572.1439952118371.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <CAB2_NwBOLcL4EVjFN6=BvBC_YN=gmfZMweVbmb5ZPCsK4Hnx1g@mail.gmail.com> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: zLdb+6CG04LhJxMghkR/pohBzDKCLQ== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 19 Aug 2015 02:42:02 -0000 Daniel Braniss wrote: >=20 > > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem@uoguelph.ca> wrote= : > >=20 > > Daniel Braniss wrote: > >>=20 > >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem@uoguelph.ca> wrot= e: > >>>=20 > >>> Daniel Braniss wrote: > >>>>=20 > >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron > >>>>> <csforgeron@gmail.com> > >>>>> wrote: > >>>>>=20 > >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and Fre= eBSD > >>>>> 10.1. Before 10.1 it was less. > >>>>>=20 > >>>>=20 > >>>> this is NOT iperf/3 where i do get close to wire speed, > >>>> it=E2=80=99s NFS writes, i.e., almost real work :-) > >>>>=20 > >>>>> I used to tweak the card settings, but now it's just stock. You may > >>>>> want > >>>>> to > >>>>> check your settings, the Mellanox may just have better defaults for > >>>>> your > >>>>> switch. > >>>>>=20 > >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will= be > >>> copying > >>> every transmitted mbuf chain to a new chain of mbuf clusters via. > >>> m_defrag() when > >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be t= he > >>> 82599 chip > >>> these days?) > >>>=20 Oops, I think I screwed up. It looks like t_maxopd is limited to somewhat l= ess than the mtu. If that is the case, the code block wouldn't do what I thought it would do. However, if_hw_tsomaxsegcount does need to be one less than the limit for t= he driver, since the tcp/ip header isn't yet prepended when it is counted. I think the code in tcp_output() should subtract 1, but you can change it i= n the driver to test this. Thanks for doing this, rick > >>=20 > >> hi Rick > >>=20 > >> how can i check the chip? > >>=20 > > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling T= SO > > helped, > > I'll bet you don't have a 82598.) > >=20 > >>> This has been fixed in the driver very recently, but those fixes won'= t be > >>> in 10.1. > >>>=20 > >>> rick > >>> ps: If you could test with 10.2, it would be interesting to see how t= he > >>> ix > >>> does with > >>> the current driver fixes in it? > >>=20 > >> I new TSO was involved! > >> ok, firstly, it=E2=80=99s 10.2 stable. > >> with TSO enabled, ix is bad, around 64MGB/s. > >> disabling TSO it=E2=80=99s better, around 130 > >>=20 > > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c= at > > around > > line#2500? > > /* TSO parameters */ > > 2572 =09 =09 ifp->if_hw_tsomax =3D 65518; > > 2573 =09 =09 ifp->if_hw_tsomaxsegcount =3D IXGBE_82599_SCATTER= ; > > 2574 =09 =09 ifp->if_hw_tsomaxsegsize =3D 2048; > >=20 > > They are in stable/10. I didn't look at releng/10.2. (And if they're in= a > > #ifdef > > for FreeBSD11, take the #ifdef away.) > > If they are there and not ifdef'd, I can't explain why disabling TSO wo= uld > > help. > > Once TSO is fixed so that it handles the 64K transmit segments without > > copying all > > the mbufs, I suspect you might get better perf. with it enabled? > >=20 >=20 > this is 10.2 : > they are on lines 2509-2511 and I don=E2=80=99t see any #ifdefs around i= t. >=20 > the plot thickens :-) >=20 > danny >=20 > > Good luck with it, rick > >=20 > >> still, mlxen0 is about 250! with and without TSO > >>=20 > >>=20 > >>>=20 > >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw@zxy.spb.ru > >>>>> <mailto:slw@zxy.spb.ru>> wrote: > >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: > >>>>>=20 > >>>>>> hi, > >>>>>> I have a host (Dell R730) with both cards, connected to an HP8= 200 > >>>>>> switch at 10Gb. > >>>>>> when writing to the same storage (netapp) this is what I get: > >>>>>> ix0: ~130MGB/s > >>>>>> mlxen0 ~330MGB/s > >>>>>> this is via nfs/tcpv3 > >>>>>>=20 > >>>>>> I can get similar (bad) performance with the mellanox if I > >>>>>> increase > >>>>>> the file size > >>>>>> to 512MGB. > >>>>>=20 > >>>>> Look like mellanox have internal beffer for caching and do ACK > >>>>> acclerating. > >>>>>=20 > >>>>>> so at face value, it seems the mlxen does a better use of > >>>>>> resources > >>>>>> than the intel. > >>>>>> Any ideas how to improve ix/intel's performance? > >>>>>=20 > >>>>> Are you sure about netapp performance? > >>>>> _______________________________________________ > >>>>> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing li= st > >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net> > >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.o= rg > >>>>> <mailto:freebsd-net-unsubscribe@freebsd.org>" > >>>>>=20 > >>>>=20 > >>>> _______________________________________________ > >>>> freebsd-stable@freebsd.org mailing list > >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >>>> To unsubscribe, send any mail to > >>>> "freebsd-stable-unsubscribe@freebsd.org" > >>=20 > >> _______________________________________________ > >> freebsd-stable@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.o= rg" >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"