From owner-freebsd-fs@FreeBSD.ORG Mon Jul 8 00:31:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7BBA3F53 for ; Mon, 8 Jul 2013 00:31:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 42A7C1D5D for ; Mon, 8 Jul 2013 00:31:39 +0000 (UTC) X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=ME3lrcP4jFDzpPiCSQywCMKJiHtpRWeRXBDIYmR1BZg= c=1 sm=2 a=4x594vOIrDwA:10 a=FKkrIqjQGGEA:10 a=V5z4IuhVU5kA:10 a=IkcTkHD0fZMA:10 a=6I5d2MoRAAAA:8 a=GzJd4s-eAAAA:8 a=v161nJs2HSYVeKKg6NEA:9 a=QEXdDO2ut3YA:10 a=SV7veod9ZcQA:10 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqQEAOIH2lGDaFve/2dsb2JhbABaFoMlSoMIvViBIXSCIwEBAQEDAQEBIAQnIAsbGAICDRkCKQEJJgYIBwQBHASHbgynSJA7gSaNE340B4JTgRwDlQ6DbpAfgy0gMoEDNw X-IronPort-AV: E=Sophos;i="4.87,1016,1363147200"; d="scan'208";a="38684045" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 07 Jul 2013 20:31:16 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id B19C3B4055; Sun, 7 Jul 2013 20:31:16 -0400 (EDT) Date: Sun, 7 Jul 2013 20:31:16 -0400 (EDT) From: Rick Macklem To: Berend de Boer Message-ID: <50381589.2918947.1373243476714.JavaMail.root@uoguelph.ca> In-Reply-To: <580122426.2916694.1373242759482.JavaMail.root@uoguelph.ca> Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jul 2013 00:31:40 -0000 I wrote: > Berend de Boer wrote: > > Hi All, > > > > I've just completed a round of NFS testing on FreeBSD 9.1 on AWS. > > The > > underlying file system is ZFS. I have a real nfs killer test: > > doing > > an "svn update" of a directory of 3541 files. > > > > Performance on the NFS server itself is good: checking this out > > takes > > 11 seconds (doing the same on a not really comparable Linux NFS > > server > > is 43 seconds). > > > > Doing this on a client writing to an NFS mounted home directory is > > however terrible. Really terrible. This takes 25 minutes! With > > "sync=disable" it's 16 minutes. > > > > (doing this against the underpowered Linux NFS server is about 4.5 > > minutes). > > > > The problem might be that the NFS server (nfsd) runs at 70-80% CPU. > > So > > my writing speed is cpu bound (go figure). > > > See below w.r.t. patch that reduces cpu overheads in the DRC. > > > Repeating this with nfs4 + udp: doesn't work at all, get > > input/output > > error after a few files. > > > The RFCs for NFSv4 require use of transport protocols that include > congestion control --> no UDP support. What client are you using? > (If it is a FreeBSD one, I need to patch it to make the mount fail. > Since NFSv4 over UDP was done in early testing, the client might > still have that bogus code in it.) > > > nfs3: 1m55s (nfsd cpu is in the 6%-8% range) > > > > Varying tcp/udp, lock/nolock, sync/async, or enabling zfs > > sync=standard doesn't make any particular difference. All timing > > were > > within the 1m49s - 2m3s range. > > > > So what's up with NFS4 on FreeBSD? > > > Please try this patch: > http://people.freebsd.org/rmacklem/~drc4.patch Oops, it actually is: http://people.freebsd.org/~rmacklem/drc4.patch > - After you apply the patch and boot the rebuilt kernel, the cpu > overheads should be reduced after you increase the value of > vfs.nfsd.tcphighwater. The larger you make it, the more space > (mbuf clusters and other kernel malloc'd data structures) the DRC > uses, but hopefully with reduced CPU overheads. > The plan is to commit a patch semantically similar to this to head > and > then MFC it someday. ivoras@ has a similar patch, but written in > cleaner > C. However, I've never gotten around to combining the patches into a > version for head. Someday I'll get to it, but not in time for 9.2. > > Good luck with it, rick > ps: There is also ken@'s file handle affinity patch, which is in head > (and I think stable/9), but it only works for NFSv3 at this > point. > Hopefully we'll come up with a patch for NFSv4 for it someday. > > > -- > > All the best, > > > > Berend de Boer > > > > > > ------------------------------------------------------ > > Awesome Drupal hosting: https://www.xplainhosting.com/ > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >