Date: Mon, 17 Aug 2015 08:21:12 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Daniel Braniss <danny@cs.huji.ac.il> Cc: Christopher Forgeron <csforgeron@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>, FreeBSD stable <freebsd-stable@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <CAB2_NwBOLcL4EVjFN6=BvBC_YN=gmfZMweVbmb5ZPCsK4Hnx1g@mail.gmail.com> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
Daniel Braniss wrote:
>=20
> > On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron@gmail.com=
>
> > wrote:
> >=20
> > FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
> > 10.1. Before 10.1 it was less.
> >=20
>=20
> this is NOT iperf/3 where i do get close to wire speed,
> it=E2=80=99s NFS writes, i.e., almost real work :-)
>=20
> > I used to tweak the card settings, but now it's just stock. You may wan=
t to
> > check your settings, the Mellanox may just have better defaults for you=
r
> > switch.
> >=20
Have you tried disabling TSO for the Intel? With TSO enabled, it will be co=
pying
every transmitted mbuf chain to a new chain of mbuf clusters via. m_defrag(=
) when
TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the 825=
99 chip
these days?)
This has been fixed in the driver very recently, but those fixes won't be i=
n 10.1.
rick
ps: If you could test with 10.2, it would be interesting to see how the ix =
does with
the current driver fixes in it?
> > On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw@zxy.spb.ru
> > <mailto:slw@zxy.spb.ru>> wrote:
> > On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> >=20
> > > hi,
> > > I have a host (Dell R730) with both cards, connected to an HP82=
00
> > > switch at 10Gb.
> > > when writing to the same storage (netapp) this is what I get:
> > > ix0: ~130MGB/s
> > > mlxen0 ~330MGB/s
> > > this is via nfs/tcpv3
> > >
> > > I can get similar (bad) performance with the mellanox if I incr=
ease
> > > the file size
> > > to 512MGB.
> >=20
> > Look like mellanox have internal beffer for caching and do ACK acclerat=
ing.
> >=20
> > > so at face value, it seems the mlxen does a better use of resou=
rces
> > > than the intel.
> > > Any ideas how to improve ix/intel's performance?
> >=20
> > Are you sure about netapp performance?
> > _______________________________________________
> > freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > <https://lists.freebsd.org/mailman/listinfo/freebsd-net>;
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org
> > <mailto:freebsd-net-unsubscribe@freebsd.org>"
> >=20
>=20
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?473274181.23263108.1439814072514.JavaMail.zimbra>