Date: Tue, 18 Aug 2015 18:04:25 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Hans Petter Selasky <hps@selasky.org> Cc: Daniel Braniss <danny@cs.huji.ac.il>, FreeBSD Net <freebsd-net@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru>, FreeBSD stable <freebsd-stable@freebsd.org>, Christopher Forgeron <csforgeron@gmail.com> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <805386587.25297673.1439935465127.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55D331A5.9050601@selasky.org> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> <1721122651.24481798.1439902381663.JavaMail.zimbra@uoguelph.ca> <55D331A5.9050601@selasky.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hans Petter Selasky wrote:
> On 08/18/15 14:53, Rick Macklem wrote:
> > If this is just a test machine, maybe you could test with these lines (at
> > about #880)
> > in sys/netinet/tcp_output.c commented out? (It looks to me like this will
> > disable TSO
> > for almost all the NFS writes.)
> > - around line #880 in sys/netinet/tcp_output.c:
> > /*
> > * In case there are too many small fragments
> > * don't use TSO:
> > */
> > if (len <= max_len) {
> > len = max_len;
> > sendalot = 1;
> > tso = 0;
> > }
> >
> > This was added along with the other stuff that did the
> > if_hw_tsomaxsegcount, etc and I
> > never noticed it until now (not my patch).
>
> FYI:
>
> These lines are needed by other hardware, like the mlxen driver. If you
> remove them mlxen will start doing m_defrag(). I believe if you set the
> correct parameters in the "struct ifnet" for the TSO size/count limits
> this problem will go away. If you print the "len" and "max_len" and also
> the cases where TSO limits are reached, you'll see what parameter is
> triggering it and needs to be increased.
>
Well, if the driver isn't setting if_hw_tsomaxsegcount correctly, then it
is the driver that needs to be fixed.
Having the above code block disable TSO for all of the NFS writes, including
the ones that set if_hw_tsomaxsegcount correctly doesn't make sense to me.
If the driver authors don't set these, the drivers do lots of m_defrag()
calls. I have posted more than once to freebsd-net@ asking the driver authors
to set these and some now have. (I can't do it, because I don't have the
hardware to test it with.)
I do think that most/all of them don't subtract 1 for the tcp/ip header and
I don't think they should be expected to, since the driver isn't supposed to
worry about the protocol at that level.
--> I think tcp_output() should subtract one from the if_hw_tsomaxsegcount
provided by the driver to handle this, since it chooses to count mbufs
(the while() loop at around line #825 in sys/netinet/tcp_output.c.)
before it prepends the tcp/ip header mbuf.
rick
> --HPS
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?805386587.25297673.1439935465127.JavaMail.zimbra>