From owner-freebsd-fs@FreeBSD.ORG Tue Jul 9 23:38:08 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 25FFCCC8 for ; Tue, 9 Jul 2013 23:38:08 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id E290B15C2 for ; Tue, 9 Jul 2013 23:38:07 +0000 (UTC) X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=u+Bwc9JL7tMNtl/i9xObSTPSFclN5AOtXcIZY5dPsHA= c=1 sm=2 a=4x594vOIrDwA:10 a=FKkrIqjQGGEA:10 a=V5z4IuhVU5kA:10 a=IkcTkHD0fZMA:10 a=ybZZDoGAAAAA:8 a=GzJd4s-eAAAA:8 a=9Pf269GSlW1ixed9Z58A:9 a=QEXdDO2ut3YA:10 a=qIVjreYYsbEA:10 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIEAD6e3FGDaFve/2dsb2JhbABbgztNgwi+C4ErdIIjAQEEASNWBRYYAgINGQJZBhOICQapA5EaBIEmjhE0B4JWgR4DqR2DLSCBbA X-IronPort-AV: E=Sophos;i="4.87,1031,1363147200"; d="scan'208";a="39661795" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 09 Jul 2013 19:38:01 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 1E23DB408F; Tue, 9 Jul 2013 19:38:01 -0400 (EDT) Date: Tue, 9 Jul 2013 19:38:01 -0400 (EDT) From: Rick Macklem To: Berend de Boer Message-ID: <818900293.3878290.1373413081112.JavaMail.root@uoguelph.ca> In-Reply-To: <877gh0yvhm.wl%berend@pobox.com> Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jul 2013 23:38:08 -0000 Berend de Boer wrote: > >>>>> "Berend" == Berend de Boer writes: > > Berend> 1. Does not effect nfs3. > > One update: with zfs sync=standard, I can get nfs3 to perform as bad > as the nfs4+patch. > Hmm, this is interesting. ken@'s file handle affinity patch works for NFSv3, but not NFSv4. If I understood his posts correctly, the fh affinity patch was needed, so ZFS's heuristic for recognizing sequential reading would function correctly. (A file handle affinity patch for NFSv4 will take some time, since all RPCs in NFSv4 are compounds, with reads/writes imbedded in them, along with other ops.) If you could do some testing where you export a UFS volume, the results might help to isolate the issue to ZFS vs nfsd. > 1. without nolock, without async, sync=standard: 8m21s > > 2. with nolock, without async, sync=standard: 5m37s > > 3. without nolock, with async, sync=standard: 4m56s. > > 4. with nolock, with async, sync=standard: 4m23s. > > 5. without nolock, without async, sync=disabled: 1m57. > > > PS: the nfs4 test was done with sync=disabled. > W.r.t. CPU overheads and the size of vfs.nfsd.tcphighwater. The size of the NFSRVCACHE_HASHSIZE was increased to 500 by the patch. 500 seems about right for a vfs.nfsd.tcphighwater of a few thousand. If you are going to use a very large value (100000), I'd suggest you increase NFSRVCACHE_HASHSIZE to something like 10000. (You have to edit sys/fs/nfs/nfsrvcache.h and recompile to change this.) I'd suggset you try something like: vfs.nfsd.tcphighwater=5000 vfs.nfsd.tcpcachetimeo=300 (5 minutes instead of 12hrs) Also, I can't remember if you've bumped up the # of nfsd threads, but I'd go for 256. nfs_server_flags="-u -t -n 256" - in your /etc/rc.conf I never use ZFS, so I can't help w.r.t. ZFS tuning, rick > > -- > All the best, > > Berend de Boer > > > ------------------------------------------------------ > Awesome Drupal hosting: https://www.xplainhosting.com/ >