Date: Tue, 30 Mar 2010 13:44:05 -0700 From: Garrett Cooper <yanefbsd@gmail.com> To: Bruce Evans <brde@optusnet.com.au> Cc: freebsd-bugs@freebsd.org, freebsd-gnats-submit@freebsd.org, Rich <rercola@acm.jhu.edu> Subject: Re: misc/145189: nfsd performs abysmally under load Message-ID: <7d6fde3d1003301344n52199cbdw583c35d2f63ce46e@mail.gmail.com> In-Reply-To: <20100331064843.Q2145@besplex.bde.org> References: <201003300501.o2U51afE033587@www.freebsd.org> <20100331021023.Y919@besplex.bde.org> <5da0588e1003300929k50be2963h64d520f9f942e3a2@mail.gmail.com> <20100331064843.Q2145@besplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Mar 30, 2010 at 1:11 PM, Bruce Evans <brde@optusnet.com.au> wrote:
> On Tue, 30 Mar 2010, Rich wrote:
>
>> On Tue, Mar 30, 2010 at 11:50 AM, Bruce Evans <brde@optusnet.com.au>
>> wrote:
>
>>>> For instance, copying a 4GB file over NFSv3 from a ZFS filesystem with
>>>> the
>>>> following flags
>>>>
>>>> [rw,nosuid,hard,intr,nofsc,tcp,vers=3D3,rsize=3D8192,wsize=3D8192,slop=
py,addr=3DX.X.X.X](Linux
>>>> client, the above is the server), I achieve 2 MB/s, fluctuating betwee=
n
>>>> 1
>>>> and 3. (pv reports 2.23 MB/s avg)
>
> I also tried various nfs r/w sizes and tcp/udp. =A0The best sizes are
> probably the fs block size or twice that (normally 16K for ffs). =A0Old
> versions of FreeBSD had even more bugs in this area and gave surprising
> performance differences depending on the nfs r/w sizes or application
> i/o sizes. =A0In some cases smaller sizes worked best, apparently because
> they avoided the stalls.
>
>>>> ...
>>>
>>> Enabling polling is a good way to destroy latency. =A0A ping latency of
>>> ...
>
>> Actually, we noticed that throughput appeared to get marginally better
>> while
>> causing occasional bursts of crushing latency, but yes, we have it on in
>> the
>> kernel without using it in any actual NICs at present. :)
>>
>> But yes, I'm getting 40-90+ MB/s, occasionally slowing to 20-30 MB/s,
>> average after copying a 6.5 GB file of 52.7 MB/s, on localhost IPv4,
>> with no additional mount flags. {r,w}size=3D8192 on localhost goes up to
>> 80-100 MB/s, with occasional sinks to 60 (average after copying
>> another, separate 6.5 GB file: 77.3 MB/s).
>
> I thought you said you often got 1-3MB/S.
>
>> Also:
>> 64 bytes from 127.0.0.1: icmp_seq=3D0 ttl=3D64 time=3D0.015 ms
>> 64 bytes from 127.0.0.1: icmp_seq=3D1 ttl=3D64 time=3D0.049 ms
>> 64 bytes from 127.0.0.1: icmp_seq=3D2 ttl=3D64 time=3D0.012 ms
>
> Fairly normal slowness for -current.
>
>> 64 bytes from [actual IP]: icmp_seq=3D0 ttl=3D64 time=3D0.019 ms
>> 64 bytes from [actual IP]: icmp_seq=3D1 ttl=3D64 time=3D0.015 ms
>
> Are these with external hardware NICs? =A0Then 15 uS is excellent. =A0Bet=
ter
> than I've ever seen. =A0Very good hardware might be able to do this, but
> I suspect it is for the local machine. =A0BTW, I don't like the times
> been reported in ms and sub-uS times not being supported. =A0I sometimes
> run Linux or cygwin ping and don't like it not supporting sub-mS times,
> so that it always reports 0 for my average latency of 100 uS.
>
>>> After various tuning and bug fixing (now partly committed by others) I
>>> get
>>> improvements like the following on low-end systems with ffs (I don't us=
e
>>> zfs):
>>> - very low end with 100Mbps ethernet: little change; bulk transfers
>>> always
>>> =A0went at near wire speed (about 10 MB/S)
>>> - low end with 1Gbps/S: bulk transfers up from 20MB/S to 45MB/S (local
>>> ffs
>>> =A050MB/S). =A0buildworld over nfs of 5.2 world down from 1200 seconds =
to 800
>>> =A0seconds (this one is very latency-sensitive. =A0Takes about 750 seco=
nds on
>>> =A0local ffs).
>>
>> Is this on 9.0-CURRENT, or RELENG_8, or something else?
>
> Mostly with 7-CURRENT or 8-CURRENT a couple of years ago. =A0Sometimes wi=
th
> a ~5.2-SERVER. =A0nfs didn't vary much with the server, except there were
> surprising differences due to latency that I never tracked down.
>
> I forgot to mention another thing you can try easily:
>
> - negative name caching. =A0Improves latency. =A0I used this to reduce ma=
keworld
> =A0times significantly, and it is now standard in -current but not
> =A0enabled by default.
Have you also tried tuning via sysctl (vfs.nfs* ?)
Thanks,
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7d6fde3d1003301344n52199cbdw583c35d2f63ce46e>