From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 01:08:58 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 043DF7C2 for ; Wed, 31 Jul 2013 01:08:58 +0000 (UTC) (envelope-from berend@pobox.com) Received: from smtp.pobox.com (b-pb-sasl-quonix.pobox.com [208.72.237.35]) by mx1.freebsd.org (Postfix) with ESMTP id B29F3229B for ; Wed, 31 Jul 2013 01:08:57 +0000 (UTC) Received: from smtp.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 74F91292F2; Wed, 31 Jul 2013 01:08:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date :message-id:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=sasl; bh=tOKbqFCvyu8I0gHJ/C7YQKsqlR8=; b=yb5vUgbO5Kdxd8TSb/W8w++18cLE QGRDmNhz07RyiROul30r+qXe3EAvg8G8/jcBi58XZUPrRnRt+43lc64z7EjVg6Gf L3znje0MNItfMqSnGvaXgbmCJ19IrLZts7sDMZLKeNo+s98ZdcEW8Zbs1zcRSizw 2RMPunXj1Zsy09U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:message-id :from:to:cc:subject:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=sasl; b=f+yWfY YIY7C/oVXH0elruTAJJWKH5yw0PIrTjHHREo4TbLyLNZLHnyAfY9f8eVyUYDcb/x N3uMnM6/ruaaAc2nQGbVYb4DljJry9OD26kfEnW9EfVixv6wATv2Obg+uTTdZhGX oA5wsFMVgkcqYfDgn++fLlSAyYSYFdupApFbY= Received: from b-pb-sasl-quonix.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 67240292F1; Wed, 31 Jul 2013 01:08:49 +0000 (UTC) Received: from bmach.nederware.nl (unknown [27.252.146.198]) by b-sasl-quonix.pobox.com (Postfix) with ESMTPA id 57972292EF; Wed, 31 Jul 2013 01:08:48 +0000 (UTC) Received: from quadrio.nederware.nl (quadrio.nederware.nl [192.168.33.13]) by bmach.nederware.nl (Postfix) with ESMTP id 73F795C80; Wed, 31 Jul 2013 13:07:20 +1200 (NZST) Received: from quadrio.nederware.nl (quadrio.nederware.nl [127.0.0.1]) by quadrio.nederware.nl (Postfix) with ESMTP id 09202404431E; Wed, 31 Jul 2013 13:08:44 +1200 (NZST) Date: Wed, 31 Jul 2013 13:08:43 +1200 Message-ID: <874nbb7cys.wl%berend@pobox.com> From: Berend de Boer To: Rick Macklem Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 In-Reply-To: <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> References: <8761wfvwml.wl%berend@pobox.com> <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/24.3 (i686-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: Xplain Technology Ltd MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: multipart/signed; boundary="pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1"; micalg=pgp-sha256; protocol="application/pgp-signature" Content-Transfer-Encoding: 7bit X-Pobox-Relay-ID: C14AD838-F97D-11E2-99C0-E84251E3A03C-48001098!b-pb-sasl-quonix.pobox.com Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 01:08:58 -0000 --pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1 Content-Type: text/plain; charset=US-ASCII >>>>> "Rick" == Rick Macklem writes: Rick> I think you mentioned that you were using a Linux client, Rick> but not what version. I'd suggest a recent kernel from Rick> kernel.org. (Fedora tracks updates/fixes for NFSv4 pretty Rick> closely, so the newest Fedora release should be pretty Rick> current.) This was Ubuntu 10.04 LTS. Have just tried a FreeBSD 9.1 client. Similar numbers. NFSv3 is about 30% slower on FreeBSD than Linux: 3m30s versus 2m10s. NFSv4 has the same terribly slow performance, i.e. 21m56s for the same test. Interestingly, the nfsd cpu usage doesn't rise as high as with Linux. But goes up to 20% (instead of over 50%). I had a look at collectd measurements as well, one cpu on the FreeBSD server is spending a lot of time in IRQ (whatever that means). BTS, this was a FreeBSD NFS4 out-of-the-box server, not with the patch (as the patch didn't do that much for me, it did some, but performance was still 8 times slower than nfs3). Rick> All I can suggest is capturing packets and then emailing be Rick> the captured packet trace. You can use tcpdump to do the Rick> capture, since wireshark will understand it: # tcpdump -s 0 Rick> -w .pcap host and then emailing me Rick> .pcap. Rick> I can take a look at the packet capture and maybe see what Rick> is going on. Will email them shortly. -- All the best, Berend de Boer ------------------------------------------------------ Awesome Drupal hosting: https://www.xplainhosting.com/ --pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1 Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit Content-Description: OpenPGP Digital Signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAABCAAGBQJR+GObAAoJEKOfeD48G3g52tUQANGXsgDpVeOJaTCdCrL87Lfv JHU7KDVfHF2EsODVw0zV0nO/nd+iyggzjAO8sc4EUDXU9HljLOsZccmu9mUr1X/Q ytwuLN57ZL+RbQab9Mu7U/+NeGu0qhT1Y+RPAL4zeNYBRSXECAkqwD9z2pPmA/ca D7MAXRqssU6toPIb44xoTMyfxIPEWEjFRk5dgSWdJHXrsUmFgrfEQhAc4gU/1tS0 nH5Y/zEGi3Tp99pmZczncSrDIk6SkzA7ZxKyIfDSpZfdcd9cqgm8olZPXXy+OYHu o1PnFMHcCb5e77+ALlfExXvN+AQHfoA+VXXTK/XOeyPY9LH/lo6v6OlgnzEjcEd4 tZ9SSYTFF6UyDZc6fatzLLJwhcqRCZPwWdXKtPCdEGorUBONVZ626b304eUc4iG4 0CBIPU5Ziang+8+2PDnAxyDzJmUmHljWu8Cr54+hZjxs0m0T07hNHLaoDpqtLHB3 VMy1BSg2ABZ5+oOtBx5IS1uBDVqXMUGbxJfhvHxI/iyAdOhz18wgiv5e+HlVkvLn aOgCUgFdILBh8OquA1uUzfTWkadndcHHSpF7mN9OzM+Vc1i6KJIDyGpKmCIHLbVm yreJGrTDM/gS2WjqXpyuzf3mqFzVP5bwLsvstRIU//2zEGDqy8WLhLgvxvRFm9zv YaIA98hQsJKA6H2WqtzD =2Alm -----END PGP SIGNATURE----- --pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1--