Date: Mon, 17 Aug 2015 17:49:10 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Daniel Braniss <danny@cs.huji.ac.il> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru>, FreeBSD stable <freebsd-stable@freebsd.org>, Christopher Forgeron <csforgeron@gmail.com> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <CAB2_NwBOLcL4EVjFN6=BvBC_YN=gmfZMweVbmb5ZPCsK4Hnx1g@mail.gmail.com> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
Daniel Braniss wrote: > > > On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote: > > > > Daniel Braniss wrote: > >> > >>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron@gmail.com> > >>> wrote: > >>> > >>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD > >>> 10.1. Before 10.1 it was less. > >>> > >> > >> this is NOT iperf/3 where i do get close to wire speed, > >> it’s NFS writes, i.e., almost real work :-) > >> > >>> I used to tweak the card settings, but now it's just stock. You may want > >>> to > >>> check your settings, the Mellanox may just have better defaults for your > >>> switch. > >>> > > Have you tried disabling TSO for the Intel? With TSO enabled, it will be > > copying > > every transmitted mbuf chain to a new chain of mbuf clusters via. > > m_defrag() when > > TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the > > 82599 chip > > these days?) > > > > hi Rick > > how can i check the chip? > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO helped, I'll bet you don't have a 82598.) > > This has been fixed in the driver very recently, but those fixes won't be > > in 10.1. > > > > rick > > ps: If you could test with 10.2, it would be interesting to see how the ix > > does with > > the current driver fixes in it? > > I new TSO was involved! > ok, firstly, it’s 10.2 stable. > with TSO enabled, ix is bad, around 64MGB/s. > disabling TSO it’s better, around 130 > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at around line#2500? /* TSO parameters */ 2572 ifp->if_hw_tsomax = 65518; 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; 2574 ifp->if_hw_tsomaxsegsize = 2048; They are in stable/10. I didn't look at releng/10.2. (And if they're in a #ifdef for FreeBSD11, take the #ifdef away.) If they are there and not ifdef'd, I can't explain why disabling TSO would help. Once TSO is fixed so that it handles the 64K transmit segments without copying all the mbufs, I suspect you might get better perf. with it enabled? Good luck with it, rick > still, mlxen0 is about 250! with and without TSO > > > > > >>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw@zxy.spb.ru > >>> <mailto:slw@zxy.spb.ru>> wrote: > >>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: > >>> > >>>> hi, > >>>> I have a host (Dell R730) with both cards, connected to an HP8200 > >>>> switch at 10Gb. > >>>> when writing to the same storage (netapp) this is what I get: > >>>> ix0: ~130MGB/s > >>>> mlxen0 ~330MGB/s > >>>> this is via nfs/tcpv3 > >>>> > >>>> I can get similar (bad) performance with the mellanox if I increase > >>>> the file size > >>>> to 512MGB. > >>> > >>> Look like mellanox have internal beffer for caching and do ACK > >>> acclerating. > >>> > >>>> so at face value, it seems the mlxen does a better use of resources > >>>> than the intel. > >>>> Any ideas how to improve ix/intel's performance? > >>> > >>> Are you sure about netapp performance? > >>> _______________________________________________ > >>> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>; > >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > >>> <mailto:freebsd-net-unsubscribe@freebsd.org>" > >>> > >> > >> _______________________________________________ > >> freebsd-stable@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?805850043.24018217.1439848150695.JavaMail.zimbra>