From owner-freebsd-net@freebsd.org Wed Aug 3 05:59:58 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECEFCBADA8C for ; Wed, 3 Aug 2016 05:59:58 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7CF6318EB; Wed, 3 Aug 2016 05:59:58 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 254E52000BA; Wed, 3 Aug 2016 07:59:55 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 0D5E2406ADE; Wed, 3 Aug 2016 07:59:55 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id BB747406ADB; Wed, 3 Aug 2016 07:59:54 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016080307595428-63260 ; Wed, 3 Aug 2016 07:59:54 +0200 Date: Wed, 3 Aug 2016 07:59:54 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Alan Somers Cc: FreeBSD Net Subject: Re: NFS on 10G interfaces still painfully slow Message-Id: <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> In-Reply-To: References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 03/08/2016 07:59:54, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 03/08/2016 07:59:54, Serialize complete at 03/08/2016 07:59:54 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.8.3.55116 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, SUPERLONG_LINE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_2000_2999 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, IN_REP_TO 0, LEGITIMATE_NEGATE 0, LEGITIMATE_SIGNS 0, MSG_THREAD 0, MULTIPLE_REAL_RCPTS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __C230066_P5 0, __CC_NAME 0, __CC_NAME_DIFF_FROM_ACC 0, __CC_REAL_NAMES 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_CC_HDR 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __REFERENCES 0, __SANE_MSGID 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __TO_NAME 0, __TO_NAME_DIFF_FROM_ACC 0, __TO_REAL_NAMES 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 05:59:59 -0000 On Tue, 2 Aug 2016 08:45:07 -0600 Alan Somers wrote about Re: NFS on 10G interfaces still painfully slow: > > Is there anyone around here who can confirm that nfs can go faster > > over 10G links? > > Any hints for further tuning/debugging are greatly appreciated. AS> I can get 1GB/s over NFS on a 10G link, so it's not always slow. AS> There's probably something about your setup that's slowing it down. AS> What is your NFS client? This time, FreeBSD 10.3 on both client and server (to make debugging easier). AS> What benchmark are you using to measure that speed? Right now only very simple things like using dd from /dev/zero or copying large files. In my experience it is useless to go for more sophisticated benchmarks, if these simple things already don't work as expected. AS> Did you remember to start lockd and statd? Yes. AS> If you post your /etc/exports and the client's /etc/fstab, that might AS> reveal something. exports on the server side: V4: /mt-rear -sec=sys 192.168.1.11 /mt-right 192.168.1.11 -maproot=root /mt-rear -maproot=root 192.168.1.11 /mt-left 192.168.1.11 -maproot=root fstab on the client does not tell you anything, I still use commandline mounts during testing. This is what nfsstat -m will tell (V4 is not mounted right now): tom:/mt-rear on /net/mt-rear nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=4,wcommitsize=50000000,timeout=120,retrans=2 tom:/mt-right on /net/mt-right nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 tom:/mt-left on /net/mt-left nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 This is what, e.g., dd gives: root@crest:~ # dd if=/dev/zero of=/net/mt-rear/Z bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec) root@crest:~ # dd if=/dev/zero of=/net/mt-right/Z bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec) And yes (before that question pops up :-), I'm using zfs on the server side, but I disabled syncing for testing purposes. cu Gerrit