From owner-freebsd-net@freebsd.org Wed Aug 3 14:21:26 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EF35BBAD987 for ; Wed, 3 Aug 2016 14:21:26 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.70.90]) by mx1.freebsd.org (Postfix) with ESMTP id AF08C1432; Wed, 3 Aug 2016 14:21:26 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id A467DCB8CB5; Wed, 3 Aug 2016 08:49:57 -0500 (CDT) Received: from 128.135.52.6 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Wed, 3 Aug 2016 08:49:57 -0500 (CDT) Message-ID: <38638.128.135.52.6.1470232197.squirrel@cosmo.uchicago.edu> In-Reply-To: <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> Date: Wed, 3 Aug 2016 08:49:57 -0500 (CDT) Subject: Re: NFS on 10G interfaces still painfully slow From: "Valeri Galtsev" To: Gerrit =?iso-8859-1?Q?K=FChn?= Cc: "Alan Somers" , "FreeBSD Net" Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 14:21:27 -0000 On Wed, August 3, 2016 12:59 am, Gerrit Kühn wrote: > On Tue, 2 Aug 2016 08:45:07 -0600 Alan Somers wrote > about Re: NFS on 10G interfaces still painfully slow: > >> > Is there anyone around here who can confirm that nfs can go faster >> > over 10G links? >> > Any hints for further tuning/debugging are greatly appreciated. > > AS> I can get 1GB/s over NFS on a 10G link, so it's not always slow. > AS> There's probably something about your setup that's slowing it down. > AS> What is your NFS client? > > This time, FreeBSD 10.3 on both client and server (to make debugging > easier). > > AS> What benchmark are you using to measure that speed? > > Right now only very simple things like using dd from /dev/zero or copying > large files. When I had trouble (too slow data throughput, never mind on what medium) using dd - it was: not specifying bs=[big number] in dd command. It turned out without that dd sends stuff down the pipe in very small chunks and wouldn't send next chunk till acknowledgement on previous is received. Imagine hard drive wrighting 4 kb at a time send 1 byte at a time, or RAID expected to write 8 x 64 kB striped in parallel sent 1 byte at a time. In your case this [big number] ideally should be equal or slightly smaller than size of jumbo packet of you net connection. Just speculating (remembered really trivial thing _I_ hit myself ;-) Thanks. Valeri > In my experience it is useless to go for more sophisticated > benchmarks, if these simple things already don't work as expected. > > AS> Did you remember to start lockd and statd? > > Yes. > > AS> If you post your /etc/exports and the client's /etc/fstab, that might > AS> reveal something. > > exports on the server side: > > V4: /mt-rear -sec=sys 192.168.1.11 > /mt-right 192.168.1.11 -maproot=root > /mt-rear -maproot=root 192.168.1.11 > /mt-left 192.168.1.11 -maproot=root > > > fstab on the client does not tell you anything, I still use commandline > mounts during testing. This is what nfsstat -m will tell (V4 is not > mounted right now): > > tom:/mt-rear on /net/mt-rear > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=4,wcommitsize=50000000,timeout=120,retrans=2 > tom:/mt-right on /net/mt-right > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 > tom:/mt-left on /net/mt-left > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 > > > This is what, e.g., dd gives: > > root@crest:~ # dd if=/dev/zero of=/net/mt-rear/Z bs=1024k count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec) > > root@crest:~ # dd if=/dev/zero of=/net/mt-right/Z bs=1024k > count=1000 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec) > > > And yes (before that question pops up :-), I'm using zfs on the server > side, but I disabled syncing for testing purposes. > > > cu > Gerrit > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++