Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 20 Oct 2012 15:52:03 +0300
From:      Nikolay Denev <ndenev@gmail.com>
To:        Ivan Voras <ivoras@freebsd.org>
Cc:        "freebsd-hackers@freebsd.org Hackers" <freebsd-hackers@freebsd.org>, Rick Macklem <rmacklem@uoguelph.ca>
Subject:   Re: NFS server bottlenecks
Message-ID:  <942B9B96-7F2B-4833-865F-33DDCCA3500A@gmail.com>
In-Reply-To: <CAF-QHFWY0drcrUpo7GGD1zQNSDWsEeB_LHAjEbUKrX2ovQHNxw@mail.gmail.com>
References:  <937460294.2185822.1350093954059.JavaMail.root@erie.cs.uoguelph.ca> <302BF685-4B9D-49C8-8000-8D0F6540C8F7@gmail.com> <k5gtdh$nc0$1@ger.gmane.org> <0857D79A-6276-433F-9603-D52125CF190F@gmail.com> <CAF-QHFUU0hhtRNK1_p9zks2w%2Be22bfWOtv%2BXaqgFqTiURcJBbQ@mail.gmail.com> <6DAAB1E6-4AC7-4B08-8CAD-0D8584D039DE@gmail.com> <23D7CB3A-BD66-427E-A7F5-6C9D3890EE1B@gmail.com> <CAF-QHFWY0drcrUpo7GGD1zQNSDWsEeB_LHAjEbUKrX2ovQHNxw@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

On Oct 20, 2012, at 3:11 PM, Ivan Voras <ivoras@freebsd.org> wrote:

> On 20 October 2012 13:42, Nikolay Denev <ndenev@gmail.com> wrote:
>=20
>> Here are the results from testing both patches : =
http://home.totalterror.net/freebsd/nfstest/results.html
>> Both tests ran for about 14 hours ( a bit too much, but I wanted to =
compare different zfs recordsize settings ),
>> and were done first after a fresh reboot.
>> The only noticeable difference seems to be much more context switches =
with Ivan's patch.
>=20
> Thank you very much for your extensive testing!
>=20
> I don't know how to interpret the rise in context switches; as this is
> kernel code, I'd expect no context switches. I hope someone else can
> explain.
>=20
> But, you have also shown that my patch doesn't do any better than
> Rick's even on a fairly large configuration, so I don't think there's
> value in adding the extra complexity, and Rick knows NFS much better
> than I do.
>=20
> But there are a few things other than that I'm interested in: like why
> does your load average spike almost to 20-ties, and how come that with
> 24 drives in RAID-10 you only push through 600 MBit/s through the 10
> GBit/s Ethernet. Have you tested your drive setup locally (AESNI
> shouldn't be a bottleneck, you should be able to encrypt well into
> Gbyte/s range) and the network?
>=20
> If you have the time, could you repeat the tests but with a recent
> Samba server and a CIFS mount on the client side? This is probably not
> important, but I'm just curious of how would it perform on your
> machine.

I've now started this test locally.
But from previous different iozone runs, I remember locally the speed =
was much better,
but I will wait for this test to finish, as the comparison will be =
better.

But I think there is still something fishy=85 I have cases where I have =
reached 1000MB/s over NFS
(from network stats, not local machine stats), but sometimes it is very =
slow even for=20
file completely in ARC. Rick mentioned that this could be due to RPC =
overhead and network round trip time, but
earlier in this thread I've done a test only on the server by mounting =
the NFS exported ZFS dataset locally and did some tests with "dd":

> To take the network out of the equation I redid the test by mounting =
the same filesystem over NFS on the server:
>=20
> [18:23]root@goliath:~#  mount -t nfs -o =
rw,hard,intr,tcp,nfsv3,rsize=3D1048576,wsize=3D1048576 =
localhost:/tank/spa_db/undo /mnt
> [18:24]root@goliath:~# dd if=3D/mnt/data.dbf of=3D/dev/null bs=3D1M=20
> 30720+1 records in
> 30720+1 records out
> 32212262912 bytes transferred in 79.793343 secs (403696120 bytes/sec)
> [18:25]root@goliath:~# dd if=3D/mnt/data.dbf of=3D/dev/null bs=3D1M
> 30720+1 records in
> 30720+1 records out
> 32212262912 bytes transferred in 12.033420 secs (2676900110 bytes/sec)
>=20
> During the first run I saw several nfsd threads in top, along with dd =
and again zero disk I/O.
> There was increase in memory usage because of the double buffering =
ARC->buffercahe.
> The second run was with all of the nfsd threads totally idle, and read =
directly from the buffercache.






Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?942B9B96-7F2B-4833-865F-33DDCCA3500A>