From owner-freebsd-fs@FreeBSD.ORG Tue Jul 9 07:48:14 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 405B3440 for ; Tue, 9 Jul 2013 07:48:14 +0000 (UTC) (envelope-from berend@pobox.com) Received: from smtp.pobox.com (b-pb-sasl-quonix.pobox.com [208.72.237.35]) by mx1.freebsd.org (Postfix) with ESMTP id 0D2A61E3F for ; Tue, 9 Jul 2013 07:48:13 +0000 (UTC) Received: from smtp.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 76C952CF77; Tue, 9 Jul 2013 07:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date :message-id:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=sasl; bh=JuRPQ4vw3EGHFNvZY2fue7owmIc=; b=j8PU2LwsxwaIANQVCdu9BSMwO32a dPVuBaAD7z9UuLdizQ3dKqtCkjHxFSkUymAolkNGRxDRFGyyBz62yhnADK3pbBOo EbyH8hE1dBbeL0bp7SPY2ecnBOV53bFL4uFh85Ydsxz7cforOfUev/9pB+vCsdxB JsogOnCr3uUz+go= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:message-id :from:to:cc:subject:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=sasl; b=THRgw1 bHDJlSRQrbo/FA8C5hOrv2k36YREVCnKv/bf1E6DXfMEneSmnFPHqEQ3BaIGyE0V h5A27uMMnpALPyHgnstvCHYVA+irHXzgrdd7ZpOcxeBvtUm9R/aU+LPs0yWxSW2o M8ul4CNX/csKhvyG5ypRFF9hFTQBdlDAoI6Fk= Received: from b-pb-sasl-quonix.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 6DD7C2CF76; Tue, 9 Jul 2013 07:48:09 +0000 (UTC) Received: from bmach.nederware.nl (unknown [27.252.169.66]) by b-sasl-quonix.pobox.com (Postfix) with ESMTPA id 8AAD62CF6F; Tue, 9 Jul 2013 07:48:08 +0000 (UTC) Received: from quadrio.nederware.nl (quadrio.nederware.nl [192.168.33.13]) by bmach.nederware.nl (Postfix) with ESMTP id 324D85C55; Tue, 9 Jul 2013 19:48:01 +1200 (NZST) Received: from quadrio.nederware.nl (quadrio.nederware.nl [127.0.0.1]) by quadrio.nederware.nl (Postfix) with ESMTP id 7D86349FB97A; Tue, 9 Jul 2013 19:48:05 +1200 (NZST) Date: Tue, 09 Jul 2013 19:48:05 +1200 Message-ID: <87a9lwyy16.wl%berend@pobox.com> From: Berend de Boer To: Rick Macklem Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2 In-Reply-To: <580122426.2916694.1373242759482.JavaMail.root@uoguelph.ca> References: <87y59i0yni.wl%berend@pobox.com> <580122426.2916694.1373242759482.JavaMail.root@uoguelph.ca> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/24.3 (i686-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: Xplain Technology Ltd MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: multipart/signed; boundary="pgp-sign-Multipart_Tue_Jul__9_19:48:05_2013-1"; micalg=pgp-sha256; protocol="application/pgp-signature" Content-Transfer-Encoding: 7bit X-Pobox-Relay-ID: E59A31DE-E86B-11E2-B70A-E84251E3A03C-48001098!b-pb-sasl-quonix.pobox.com Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jul 2013 07:48:14 -0000 --pgp-sign-Multipart_Tue_Jul__9_19:48:05_2013-1 Content-Type: text/plain; charset=US-ASCII >>>>> "Rick" == Rick Macklem writes: Rick> After you Rick> apply the patch and boot the rebuilt kernel, the cpu Rick> overheads should be reduced after you increase the value of Rick> vfs.nfsd.tcphighwater. OK, completely disregard my previous email. I actually was testing against a server in a different data centre, didn't think it would matter too much, but clearly it does (ping times 2-3 times higher). So moved server + disks into the same data centre as the nfs client. 1. Does not effect nfs3. 2. When I do not set vfs.nfsd.tcphighwater, I get a "Remote I/O error" on the client. On server I see: nfsd server cache flooded, try to increase nfsrc_floodlevel (this just FYI). 3. With vfs.nfsd.tcphighwater set to 150,000. I get very high cpu, 50%. Performance is now about 8m15s. Which is better, but still twice above a lower spec Linux NFS4 server, and four times slower than nfs3 on the same box. 4. With Garrett's settings, I looked at when the cpu starts to increase. It starts slow, but raises quickly to 50% in about 1 minute. Time was similar 7m54s. 5. I lowered vfs.nfsd.tcphighwater to 10,000 but then it actually became worse, cpu quickly went to 70%, i.e. not much difference with FreeBSD without patch. Didn't keep this test running to see if it became slower over time. Making it 300,000 seems that the cpu increases are slower (but it keeps rising). So from what I observe from the patch is that it makes the rise in cpu increase slower, but doesn't stop it. I.e. after a few minutes, even with setting 300,000 the cpu is getting to 50%, but dropped a bit after a while to hover around 40%. Then it crept back to over 50%. 6. So the conclusion is: this patch helps somewhat, but nfs4 behaviour is still majorly impaired compared to nfs3. -- All the best, Berend de Boer ------------------------------------------------------ Awesome Drupal hosting: https://www.xplainhosting.com/ --pgp-sign-Multipart_Tue_Jul__9_19:48:05_2013-1 Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit Content-Description: OpenPGP Digital Signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAABCAAGBQJR28A1AAoJEKOfeD48G3g5yPIQANLfMW8UEy6PjEtkTuraF8JB 26WLNBr3+P7r/nOpJF+Wa148rQ+Snb338W4eLATcmxDvXseoN/Q3EZkDQaxl9kDu CDc3wiT0UNJ4bUcuF5gemeon8pPCdcy8NYkfBoz0bUzLP/nHO81i9twk4Evlsgzz sgEng3NxP4GYjykyqX8tWWiW83i8a/BNL3p5Oi1srJp/hbbzPF/dhv4FrfFGWHIK lIh6AUc12UZh7MTyHhrwdLWNMerYyOL5BH2WGAgs/2+Z9ZU/AvjjVpsrUJQE0nZh LA2hkm5CG3XzDHEW+8B8Qgz1G3HQXjRD4AGQ4ygOAaPS9UIrWKOxvbH/iTQ1vBXQ 6/6VhPBNWlFzsU0VLRHhXcGXtnmXg7N0E21WLipzKUCa9U/zD3wgfKxSoX2eIz13 JOROZ1jXZrPwmAcynYMh+WouZSyWCS4sN6rbVAGoJyExkmRRH18mGTfAJDZcVqgA jL2o9Onfe8mDADWRQU5mYoHDUGYP61pqyCuqTPantw1SSjKySCRuIfJL7hh3c0P6 UZu3JtXun8o4ojmg1o76qd0KN4i7Wrqn+lGn7J0QaYE6zDMA0sHgDMVf7zFvvKhj ZtOOzOapP5jmv6RBmtU985E5THVowfBGSWSzHGSNWCf2kh9OtAr0qKyS/YFWzS+/ Tc9ZBp6ezyVBwUJgszKi =WX8Z -----END PGP SIGNATURE----- --pgp-sign-Multipart_Tue_Jul__9_19:48:05_2013-1--