Date: Mon, 27 Jan 2014 16:28:26 -0800 From: John-Mark Gurney <jmg@funkthat.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-net@freebsd.org, Adam McDougall <mcdouga9@egr.msu.edu> Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <20140128002826.GU13704@funkthat.com> In-Reply-To: <222089865.17245782.1390866430479.JavaMail.root@uoguelph.ca> References: <20140127032338.GP13704@funkthat.com> <222089865.17245782.1390866430479.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Rick Macklem wrote this message on Mon, Jan 27, 2014 at 18:47 -0500:
> John-Mark Gurney wrote:
> > Rick Macklem wrote this message on Sun, Jan 26, 2014 at 21:16 -0500:
> > > Btw, thanks go to Garrett Wollman for suggesting the change to
> > > MJUMPAGESIZE
> > > clusters.
> > >
> > > rick
> > > ps: If the attachment doesn't make it through and you want the
> > > patch, just
> > > email me and I'll send you a copy.
> >
> > The patch looks good, but we probably shouldn't change _readlink..
> > The chances of a link being >2k are pretty slim, and the chances of
> > the link being >32k are even smaller...
> >
> Yea, I already thought of that, actually. However, see below w.r.t.
> NFSv4.
>
> However, at this point I
> mostly want to find out if it the long mbuf chain that causes problems
> for TSO enabled network interfaces.
I agree, though a long mbuf chain is more of a driver issue than an
NFS issue...
> > In fact, we might want to switch _readlink to MGET (could be
> > conditional
> > upon cnt) so that if it fits in an mbuf we don't allocate a cluster
> > for
> > it...
> >
> For NFSv4, what was an RPC for NFSv3 becomes one of several Ops. in
> a compound RPC. As such, there is no way to know how much additional
> RPC message there will be. So, although the readlink reply won't use
> much of the 4K allocation, replies for subsequent Ops. in the compound
> certainly could. (Is it more efficient to allocate 4K now and use
> part of it for subsequent message reply stuff or allocate additional
> mbuf clusters later for subsequent stuff, as required? On a small
> memory constrained machine, I suspect the latter is correct, but for
> the kind of hardware that has TSO scatter/gather enabled network
> interfaces, I'm not so sure. At this point, I wouldn't even say
> that using 4K clusters is going to be a win and my hunch is that
> any win wouldn't apply to small memory constrained machines.)
Though the code that was patched wasn't using any partial buffers,
it was always allocating a new buffer... If the code in
_read/_readlinks starts using a previous mbuf chain, then obviously
things are different and I'd agree, always allocating a 2k/4k
cluster makes sense...
> My test server has 256Mbytes of ram and it certainly doesn't show
> any improvement (big surprise;-), but it also doesn't show any
> degradation for the limited testing I've done.
I'm not too surprised, unless you're on a heavy server pushing
>200MB/sec, the allocation cost is probably cheap enough that it
doesn't show up... going to 4k means immediately half as many mbufs
are needed/allocated, and as they are page sized, don't have the
problems of physical memory fragmentation, nor do they have to do an
IPI/tlb shoot down in the case of multipage allocations... (I'm
dealing w/ this for geli.)
> Again, my main interest at this point is whether reducing the
> number of mbufs in the chain fixes the TSO issues. I think
> the question of whether or not 4K clusters are performance
> improvement in general, is an interesting one that comes later.
Another thing I noticed is that we are getting an mbuf and then
allocating a cluster... Is there a reason we aren't using something
like m_getm or m_getcl? We have a special uma zone that has
mbuf and mbuf cluster already paired meaning we save some lock
operations for each segment allocated...
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140128002826.GU13704>