Date: Thu, 30 Jan 2014 19:53:03 -0800 From: John-Mark Gurney <jmg@funkthat.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Adrian Chadd <adrian@freebsd.org> Subject: Re: 64K NFS I/O generates a 34mbuf list for TCP which breaks TSO Message-ID: <20140131035303.GT93141@funkthat.com> In-Reply-To: <1856284835.584005.1391139152133.JavaMail.root@uoguelph.ca> References: <CAJ-VmonGR-KQBGyKCUn=k8PDH3skB5N8br3JyDTD27%2Bz=UwJJw@mail.gmail.com> <1856284835.584005.1391139152133.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Rick Macklem wrote this message on Thu, Jan 30, 2014 at 22:32 -0500:
> Adrian Chadd wrote:
> > On 30 January 2014 07:06, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> > > Hi, just adding one more idea on what to do about this
> > > to the list:
> > > - Add a if_hw_tsomaxseg and modify the loop in tcp_output()
> > > so that it uses both if_hw_tsomax and if_hw_tsomaxseg to
> > > decide how much to hand to the device driver in each mbuf list.
> > > (I haven't looked to see how easy it would be to change this
> > > loop.)
> >
> > I don't think that's a hack. I think adding that and setting
> > tsomaxseg
> > to say 30 for now would be a good comprimise.
> >
> Well, my TCP is very rusty and I have no way to test it (I don't
> have anything that does TSO), but I've attached a stab at a patch
> to do this.
>
> Maybe it can be used as a starting point for this, if others think
> it makes sense.
>
> The "#ifdef notyet" in the patch would become something like:
> # if __FreeBSD_version >= NNNN
> when a change to add if_hw_tsomaxseg is done, was what I was
> thinking.
Definately need to make sure you fix the drivers that support large
enough sg arrays like ixgb which supports 100...
Just a sampling of ones that use a _SCATTER define:
./e1000/if_igb.h:#define IGB_MAX_SCATTER 64
./e1000/if_lem.h:#define EM_MAX_SCATTER 64
./e1000/if_em.h:#define EM_MAX_SCATTER 32
./nfe/if_nfereg.h:#define NFE_MAX_SCATTER 32
./ixgbe/ixgbe.h:#define IXGBE_82598_SCATTER 100
./ixgbe/ixgbe.h:#define IXGBE_82599_SCATTER 32
./ixgb/if_ixgb.h:#define IXGB_MAX_SCATTER 100
I wonder how many of these are hardware limits, or just I don't
want to allocate too much space on the stack, as 16 bytes per
bus_dma_segment_t (on amd64) adds up...
The other question is should the drivers w/ a limit on the segments
reduce the size of the TSO packet so that we don't need to
m_defrag/m_collapse which are expensive operations...
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140131035303.GT93141>