Date: Tue, 18 Aug 2015 08:53:01 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Daniel Braniss <danny@cs.huji.ac.il> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Christopher Forgeron <csforgeron@gmail.com>, FreeBSD stable <freebsd-stable@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <1721122651.24481798.1439902381663.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <CAB2_NwBOLcL4EVjFN6=BvBC_YN=gmfZMweVbmb5ZPCsK4Hnx1g@mail.gmail.com> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
Daniel Braniss wrote:
>
> > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> >
> > Daniel Braniss wrote:
> >>
> >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> >>>
> >>> Daniel Braniss wrote:
> >>>>
> >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
> >>>>> <csforgeron@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
> >>>>> 10.1. Before 10.1 it was less.
> >>>>>
> >>>>
> >>>> this is NOT iperf/3 where i do get close to wire speed,
> >>>> it’s NFS writes, i.e., almost real work :-)
> >>>>
> >>>>> I used to tweak the card settings, but now it's just stock. You may
> >>>>> want
> >>>>> to
> >>>>> check your settings, the Mellanox may just have better defaults for
> >>>>> your
> >>>>> switch.
> >>>>>
> >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be
> >>> copying
> >>> every transmitted mbuf chain to a new chain of mbuf clusters via.
> >>> m_defrag() when
> >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the
> >>> 82599 chip
> >>> these days?)
> >>>
> >>
> >> hi Rick
> >>
> >> how can i check the chip?
> >>
> > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO
> > helped,
> > I'll bet you don't have a 82598.)
> >
> >>> This has been fixed in the driver very recently, but those fixes won't be
> >>> in 10.1.
> >>>
> >>> rick
> >>> ps: If you could test with 10.2, it would be interesting to see how the
> >>> ix
> >>> does with
> >>> the current driver fixes in it?
> >>
> >> I new TSO was involved!
> >> ok, firstly, it’s 10.2 stable.
> >> with TSO enabled, ix is bad, around 64MGB/s.
> >> disabling TSO it’s better, around 130
> >>
> > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at
> > around
> > line#2500?
> > /* TSO parameters */
> > 2572 ifp->if_hw_tsomax = 65518;
> > 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER;
> > 2574 ifp->if_hw_tsomaxsegsize = 2048;
> >
> > They are in stable/10. I didn't look at releng/10.2. (And if they're in a
> > #ifdef
> > for FreeBSD11, take the #ifdef away.)
> > If they are there and not ifdef'd, I can't explain why disabling TSO would
> > help.
> > Once TSO is fixed so that it handles the 64K transmit segments without
> > copying all
> > the mbufs, I suspect you might get better perf. with it enabled?
> >
>
> this is 10.2 :
> they are on lines 2509-2511 and I don’t see any #ifdefs around it.
>
> the plot thickens :-)
>
If this is just a test machine, maybe you could test with these lines (at about #880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will disable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
/*
* In case there are too many small fragments
* don't use TSO:
*/
if (len <= max_len) {
len = max_len;
sendalot = 1;
tso = 0;
}
This was added along with the other stuff that did the if_hw_tsomaxsegcount, etc and I
never noticed it until now (not my patch).
rick
> danny
>
> > Good luck with it, rick
> >
> >> still, mlxen0 is about 250! with and without TSO
> >>
> >>
> >>>
> >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw@zxy.spb.ru
> >>>>> <mailto:slw@zxy.spb.ru>> wrote:
> >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> >>>>>
> >>>>>> hi,
> >>>>>> I have a host (Dell R730) with both cards, connected to an HP8200
> >>>>>> switch at 10Gb.
> >>>>>> when writing to the same storage (netapp) this is what I get:
> >>>>>> ix0: ~130MGB/s
> >>>>>> mlxen0 ~330MGB/s
> >>>>>> this is via nfs/tcpv3
> >>>>>>
> >>>>>> I can get similar (bad) performance with the mellanox if I
> >>>>>> increase
> >>>>>> the file size
> >>>>>> to 512MGB.
> >>>>>
> >>>>> Look like mellanox have internal beffer for caching and do ACK
> >>>>> acclerating.
> >>>>>
> >>>>>> so at face value, it seems the mlxen does a better use of
> >>>>>> resources
> >>>>>> than the intel.
> >>>>>> Any ideas how to improve ix/intel's performance?
> >>>>>
> >>>>> Are you sure about netapp performance?
> >>>>> _______________________________________________
> >>>>> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list
> >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>;
> >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org
> >>>>> <mailto:freebsd-net-unsubscribe@freebsd.org>"
> >>>>>
> >>>>
> >>>> _______________________________________________
> >>>> freebsd-stable@freebsd.org mailing list
> >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-stable-unsubscribe@freebsd.org"
> >>
> >> _______________________________________________
> >> freebsd-stable@freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1721122651.24481798.1439902381663.JavaMail.zimbra>