From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 16:37:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2D6771A9 for ; Thu, 18 Oct 2012 16:37:45 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id A08848FC0A for ; Thu, 18 Oct 2012 16:37:44 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1TOt6M-00065T-DR for freebsd-fs@freebsd.org; Thu, 18 Oct 2012 18:37:42 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1TOt6L-0008UY-Lt for freebsd-fs@freebsd.org; Thu, 18 Oct 2012 18:37:41 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) References: Date: Thu, 18 Oct 2012 18:37:40 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.02 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.0 X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_50 autolearn=disabled version=3.2.5 X-Scan-Signature: 7006e789400ccdeb5065f5d065827fb5 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 16:37:45 -0000 On Thu, 18 Oct 2012 18:16:16 +0200, Thomas Johnson wrote: > We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. Almost > immediately, we received reports from users of poor performance. The > upgraded hosts are PXE-booted, with an NFS-mounted root. Additionally, > they > mount a number of other NFS shares, which is where our users work from. > After a week of tweaking rsize/wsize/readahead parameters (per guidance), > it finally occurred to me that 9.0 defaults to the new NFS client and > server. I remounted the user shares using the oldnfs file type, and users > reported that performance returned to its expected level. > > This is obviously a workaround, rather than a solution. We would prefer > to > get our hosts using the newnfs client, since presumably oldnfs will be > deprecated at some point in the future. Is there some change that we > should > have made to our NFS configuration with the upgrade to 9.0, or is it > possible that our workload is exposing some deficiency with newnfs? We > tend > to deal with a huge number of tiny files (several KB in size). The NFS > server has been running 9.0 for some time (prior to the client upgrade) > without any issue. NFS is served from a zpool, backed by a Dell MD3000, > populated with 15k SAS disks. Clients and server are connected with Gig-E > links. The general hardware configuration has not changed in nearly 3 > years. > > As an example of the performance difference, here is some of the testing > I > did while troubleshooting. Given a directory containing 5671 zip files, > with an average size of 15KB. I append all files to an existing zip file. > Using the newnfs mount, I found that this operation generally takes ~30 > seconds (wall time). Switching the mount to oldnfs resulted in the same > operation taking ~10 seconds. > > tom@test-1:/test-> ls 53*zip | wc -l > 5671 > tom@test-1:/test-> ll -h BIG* > -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip > tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip > 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w > ...reset and repeat... > 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w > ...reset and repeat... > 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w > > tom@test-1:/-> cd / > tom@test-1:/-> sudo umount /test > tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test > tom@test-1:/-> mount | grep test > server:/array/test on /test (oldnfs) > tom@test-1:/-> cd /test > ...reset and repeat... > 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w > ...reset and repeat... > 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w > tom@test-1:/test-> ll -h BIG* > -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip > > Thanks! > You might find this thread from today interesting. http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015441.html Ronald.