From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 07:22:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 39664B0E for ; Wed, 31 Jul 2013 07:22:40 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E7CEB2E44 for ; Wed, 31 Jul 2013 07:22:39 +0000 (UTC) Received: from Julian-MBP3.local (etroy.elischer.org [121.45.226.51]) (authenticated bits=0) by vps1.elischer.org (8.14.7/8.14.6) with ESMTP id r6V7MX1n032062 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 31 Jul 2013 00:22:35 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <51F8BB33.2060901@freebsd.org> Date: Wed, 31 Jul 2013 15:22:27 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Berend de Boer Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 References: <8761wfvwml.wl%berend@pobox.com> <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> <874nbb7cys.wl%berend@pobox.com> In-Reply-To: <874nbb7cys.wl%berend@pobox.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 07:22:40 -0000 On 7/31/13 9:08 AM, Berend de Boer wrote: >>>>>> "Rick" == Rick Macklem writes: > Rick> I think you mentioned that you were using a Linux client, > Rick> but not what version. I'd suggest a recent kernel from > Rick> kernel.org. (Fedora tracks updates/fixes for NFSv4 pretty > Rick> closely, so the newest Fedora release should be pretty > Rick> current.) > > This was Ubuntu 10.04 LTS. > > Have just tried a FreeBSD 9.1 client. Similar numbers. NFSv3 is about > 30% slower on FreeBSD than Linux: 3m30s versus 2m10s. NFSv4 has the > same terribly slow performance, i.e. 21m56s for the same test. > > Interestingly, the nfsd cpu usage doesn't rise as high as with > Linux. But goes up to 20% (instead of over 50%). > > I had a look at collectd measurements as well, one cpu on the FreeBSD > server is spending a lot of time in IRQ (whatever that means). > > BTS, this was a FreeBSD NFS4 out-of-the-box server, not with the patch > (as the patch didn't do that much for me, it did some, but performance > was still 8 times slower than nfs3). > > > Rick> All I can suggest is capturing packets and then emailing be > Rick> the captured packet trace. You can use tcpdump to do the > Rick> capture, since wireshark will understand it: # tcpdump -s 0 > Rick> -w .pcap host and then emailing me > Rick> .pcap. > > Rick> I can take a look at the packet capture and maybe see what > Rick> is going on. > > Will email them shortly. Recent evidence with AWS is suggesting that the NOADAPTIVE_XXX options in the XENHVM kernel are now seriously hindering AWS performance all over the place. make sure you have tried with these options removed. > -- > All the best, > > Berend de Boer > > > ------------------------------------------------------ > Awesome Drupal hosting: https://www.xplainhosting.com/