Date: Wed, 19 Aug 2015 16:30:10 +0900 From: Yonghyeon PYUN <pyunyh@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Hans Petter Selasky <hps@selasky.org>, Daniel Braniss <danny@cs.huji.ac.il>, FreeBSD Net <freebsd-net@freebsd.org>, Christopher Forgeron <csforgeron@gmail.com>, FreeBSD stable <freebsd-stable@freebsd.org>, Slawa Olhovchenkov <slw@zxy.spb.ru> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <20150819073010.GA964@michelle.fasterthan.com> In-Reply-To: <805386587.25297673.1439935465127.JavaMail.zimbra@uoguelph.ca> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> <9D8B0503-E8FA-43CA-88F0-01F184F84D9B@cs.huji.ac.il> <1721122651.24481798.1439902381663.JavaMail.zimbra@uoguelph.ca> <55D331A5.9050601@selasky.org> <805386587.25297673.1439935465127.JavaMail.zimbra@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Aug 18, 2015 at 06:04:25PM -0400, Rick Macklem wrote:
> Hans Petter Selasky wrote:
> > On 08/18/15 14:53, Rick Macklem wrote:
> > > If this is just a test machine, maybe you could test with these lines (at
> > > about #880)
> > > in sys/netinet/tcp_output.c commented out? (It looks to me like this will
> > > disable TSO
> > > for almost all the NFS writes.)
> > > - around line #880 in sys/netinet/tcp_output.c:
> > > /*
> > > * In case there are too many small fragments
> > > * don't use TSO:
> > > */
> > > if (len <= max_len) {
> > > len = max_len;
> > > sendalot = 1;
> > > tso = 0;
> > > }
> > >
> > > This was added along with the other stuff that did the
> > > if_hw_tsomaxsegcount, etc and I
> > > never noticed it until now (not my patch).
> >
> > FYI:
> >
> > These lines are needed by other hardware, like the mlxen driver. If you
> > remove them mlxen will start doing m_defrag(). I believe if you set the
> > correct parameters in the "struct ifnet" for the TSO size/count limits
> > this problem will go away. If you print the "len" and "max_len" and also
> > the cases where TSO limits are reached, you'll see what parameter is
> > triggering it and needs to be increased.
> >
> Well, if the driver isn't setting if_hw_tsomaxsegcount correctly, then it
> is the driver that needs to be fixed.
> Having the above code block disable TSO for all of the NFS writes, including
> the ones that set if_hw_tsomaxsegcount correctly doesn't make sense to me.
> If the driver authors don't set these, the drivers do lots of m_defrag()
> calls. I have posted more than once to freebsd-net@ asking the driver authors
> to set these and some now have. (I can't do it, because I don't have the
> hardware to test it with.)
>
Thanks for reminder. I have generated a diff against HEAD.
https://people.freebsd.org/~yongari/tso.param.diff
The diff restores optimal TSO parameters which were lost in r271946
for drivers that relied on sane default values. I'll commit it
after some testing.
> I do think that most/all of them don't subtract 1 for the tcp/ip header and
> I don't think they should be expected to, since the driver isn't supposed to
> worry about the protocol at that level.
I agree.
> --> I think tcp_output() should subtract one from the if_hw_tsomaxsegcount
> provided by the driver to handle this, since it chooses to count mbufs
> (the while() loop at around line #825 in sys/netinet/tcp_output.c.)
> before it prepends the tcp/ip header mbuf.
>
> rick
>
> > --HPS
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150819073010.GA964>