From owner-freebsd-fs@FreeBSD.ORG Mon Jul 8 00:19:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 54F55CFA for ; Mon, 8 Jul 2013 00:19:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 1BD8C1CF0 for ; Mon, 8 Jul 2013 00:19:20 +0000 (UTC) X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=ME3lrcP4jFDzpPiCSQywCMKJiHtpRWeRXBDIYmR1BZg= c=1 sm=2 a=ctSXsGKhotwA:10 a=FKkrIqjQGGEA:10 a=V5z4IuhVU5kA:10 a=IkcTkHD0fZMA:10 a=6I5d2MoRAAAA:8 a=GzJd4s-eAAAA:8 a=jhuFN07rW3Fs5YaUhqQA:9 a=QEXdDO2ut3YA:10 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIEAJQE2lGDaFve/2dsb2JhbABaFoMlSoMIvViBIXSCIwEBBAEjBFIFFhgCAg0ZAlkGiBwGDKdHkDuBJo4RNAeCU4EcA5h8kB+DLSCBbA X-IronPort-AV: E=Sophos;i="4.87,1016,1363147200"; d="scan'208";a="38683382" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 07 Jul 2013 20:19:19 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 7921EB402C; Sun, 7 Jul 2013 20:19:19 -0400 (EDT) Date: Sun, 7 Jul 2013 20:19:19 -0400 (EDT) From: Rick Macklem To: Berend de Boer Message-ID: <580122426.2916694.1373242759482.JavaMail.root@uoguelph.ca> In-Reply-To: <87y59i0yni.wl%berend@pobox.com> Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jul 2013 00:19:21 -0000 Berend de Boer wrote: > Hi All, > > I've just completed a round of NFS testing on FreeBSD 9.1 on AWS. The > underlying file system is ZFS. I have a real nfs killer test: doing > an "svn update" of a directory of 3541 files. > > Performance on the NFS server itself is good: checking this out takes > 11 seconds (doing the same on a not really comparable Linux NFS > server > is 43 seconds). > > Doing this on a client writing to an NFS mounted home directory is > however terrible. Really terrible. This takes 25 minutes! With > "sync=disable" it's 16 minutes. > > (doing this against the underpowered Linux NFS server is about 4.5 > minutes). > > The problem might be that the NFS server (nfsd) runs at 70-80% CPU. > So > my writing speed is cpu bound (go figure). > See below w.r.t. patch that reduces cpu overheads in the DRC. > Repeating this with nfs4 + udp: doesn't work at all, get input/output > error after a few files. > The RFCs for NFSv4 require use of transport protocols that include congestion control --> no UDP support. What client are you using? (If it is a FreeBSD one, I need to patch it to make the mount fail. Since NFSv4 over UDP was done in early testing, the client might still have that bogus code in it.) > nfs3: 1m55s (nfsd cpu is in the 6%-8% range) > > Varying tcp/udp, lock/nolock, sync/async, or enabling zfs > sync=standard doesn't make any particular difference. All timing were > within the 1m49s - 2m3s range. > > So what's up with NFS4 on FreeBSD? > Please try this patch: http://people.freebsd.org/rmacklem/~drc4.patch - After you apply the patch and boot the rebuilt kernel, the cpu overheads should be reduced after you increase the value of vfs.nfsd.tcphighwater. The larger you make it, the more space (mbuf clusters and other kernel malloc'd data structures) the DRC uses, but hopefully with reduced CPU overheads. The plan is to commit a patch semantically similar to this to head and then MFC it someday. ivoras@ has a similar patch, but written in cleaner C. However, I've never gotten around to combining the patches into a version for head. Someday I'll get to it, but not in time for 9.2. Good luck with it, rick ps: There is also ken@'s file handle affinity patch, which is in head (and I think stable/9), but it only works for NFSv3 at this point. Hopefully we'll come up with a patch for NFSv4 for it someday. > -- > All the best, > > Berend de Boer > > > ------------------------------------------------------ > Awesome Drupal hosting: https://www.xplainhosting.com/ >