Date: Wed, 3 Aug 2016 08:49:57 -0500 (CDT) From: "Valeri Galtsev" <galtsev@kicp.uchicago.edu> To: Gerrit =?iso-8859-1?Q?K=FChn?= <gerrit.kuehn@aei.mpg.de> Cc: "Alan Somers" <asomers@freebsd.org>, "FreeBSD Net" <freebsd-net@freebsd.org> Subject: Re: NFS on 10G interfaces still painfully slow Message-ID: <38638.128.135.52.6.1470232197.squirrel@cosmo.uchicago.edu> In-Reply-To: <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> <CAOtMX2hTGkm68rY7=hWDNG6YeuE-xd4%2BrV7%2BLoaywaPBo5yjdQ@mail.gmail.com> <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, August 3, 2016 12:59 am, Gerrit Kühn wrote: > On Tue, 2 Aug 2016 08:45:07 -0600 Alan Somers <asomers@freebsd.org> wrote > about Re: NFS on 10G interfaces still painfully slow: > >> > Is there anyone around here who can confirm that nfs can go faster >> > over 10G links? >> > Any hints for further tuning/debugging are greatly appreciated. > > AS> I can get 1GB/s over NFS on a 10G link, so it's not always slow. > AS> There's probably something about your setup that's slowing it down. > AS> What is your NFS client? > > This time, FreeBSD 10.3 on both client and server (to make debugging > easier). > > AS> What benchmark are you using to measure that speed? > > Right now only very simple things like using dd from /dev/zero or copying > large files. When I had trouble (too slow data throughput, never mind on what medium) using dd - it was: not specifying bs=[big number] in dd command. It turned out without that dd sends stuff down the pipe in very small chunks and wouldn't send next chunk till acknowledgement on previous is received. Imagine hard drive wrighting 4 kb at a time send 1 byte at a time, or RAID expected to write 8 x 64 kB striped in parallel sent 1 byte at a time. In your case this [big number] ideally should be equal or slightly smaller than size of jumbo packet of you net connection. Just speculating (remembered really trivial thing _I_ hit myself ;-) Thanks. Valeri > In my experience it is useless to go for more sophisticated > benchmarks, if these simple things already don't work as expected. > > AS> Did you remember to start lockd and statd? > > Yes. > > AS> If you post your /etc/exports and the client's /etc/fstab, that might > AS> reveal something. > > exports on the server side: > > V4: /mt-rear -sec=sys 192.168.1.11 > /mt-right 192.168.1.11 -maproot=root > /mt-rear -maproot=root 192.168.1.11 > /mt-left 192.168.1.11 -maproot=root > > > fstab on the client does not tell you anything, I still use commandline > mounts during testing. This is what nfsstat -m will tell (V4 is not > mounted right now): > > tom:/mt-rear on /net/mt-rear > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=4,wcommitsize=50000000,timeout=120,retrans=2 > tom:/mt-right on /net/mt-right > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 > tom:/mt-left on /net/mt-left > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 > > > This is what, e.g., dd gives: > > root@crest:~ # dd if=/dev/zero of=/net/mt-rear/Z bs=1024k count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec) > > root@crest:~ # dd if=/dev/zero of=/net/mt-right/Z bs=1024k > count=1000 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec) > > > And yes (before that question pops up :-), I'm using zfs on the server > side, but I disabled syncing for testing purposes. > > > cu > Gerrit > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?38638.128.135.52.6.1470232197.squirrel>