Date: Mon, 17 Aug 2015 16:29:20 +0300 From: Daniel Braniss <danny@cs.huji.ac.il> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Christopher Forgeron <csforgeron@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> In-Reply-To: <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <CAB2_NwBOLcL4EVjFN6=BvBC_YN=gmfZMweVbmb5ZPCsK4Hnx1g@mail.gmail.com> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem@uoguelph.ca> = wrote: >=20 > Daniel Braniss wrote: >>=20 >>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron = <csforgeron@gmail.com> >>> wrote: >>>=20 >>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and = FreeBSD >>> 10.1. Before 10.1 it was less. >>>=20 >>=20 >> this is NOT iperf/3 where i do get close to wire speed, >> it=E2=80=99s NFS writes, i.e., almost real work :-) >>=20 >>> I used to tweak the card settings, but now it's just stock. You may = want to >>> check your settings, the Mellanox may just have better defaults for = your >>> switch. >>>=20 > Have you tried disabling TSO for the Intel? With TSO enabled, it will = be copying > every transmitted mbuf chain to a new chain of mbuf clusters via. = m_defrag() when > TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be = the 82599 chip > these days?) >=20 hi Rick how can i check the chip? > This has been fixed in the driver very recently, but those fixes won't = be in 10.1. >=20 > rick > ps: If you could test with 10.2, it would be interesting to see how = the ix does with > the current driver fixes in it? I new TSO was involved!=20 ok, firstly, it=E2=80=99s 10.2 stable. with TSO enabled, ix is bad, around 64MGB/s. disabling TSO it=E2=80=99s better, around 130 still, mlxen0 is about 250! with and without TSO >=20 >>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw@zxy.spb.ru >>> <mailto:slw@zxy.spb.ru>> wrote: >>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: >>>=20 >>>> hi, >>>> I have a host (Dell R730) with both cards, connected to an = HP8200 >>>> switch at 10Gb. >>>> when writing to the same storage (netapp) this is what I get: >>>> ix0: ~130MGB/s >>>> mlxen0 ~330MGB/s >>>> this is via nfs/tcpv3 >>>>=20 >>>> I can get similar (bad) performance with the mellanox if I = increase >>>> the file size >>>> to 512MGB. >>>=20 >>> Look like mellanox have internal beffer for caching and do ACK = acclerating. >>>=20 >>>> so at face value, it seems the mlxen does a better use of = resources >>>> than the intel. >>>> Any ideas how to improve ix/intel's performance? >>>=20 >>> Are you sure about netapp performance? >>> _______________________________________________ >>> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing = list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>; >>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org >>> <mailto:freebsd-net-unsubscribe@freebsd.org>" >>>=20 >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7F892C70-9C04-4468-9514-EDBFE75CF2C6>