From owner-freebsd-fs@FreeBSD.ORG Tue Oct 23 20:33:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CC2CE7DA for ; Tue, 23 Oct 2012 20:33:04 +0000 (UTC) (envelope-from tom@claimlynx.com) Received: from na3sys009aog121.obsmtp.com (na3sys009aog121.obsmtp.com [74.125.149.145]) by mx1.freebsd.org (Postfix) with SMTP id 286FD8FC12 for ; Tue, 23 Oct 2012 20:33:03 +0000 (UTC) Received: from mail-qc0-f200.google.com ([209.85.216.200]) (using TLSv1) by na3sys009aob121.postini.com ([74.125.148.12]) with SMTP ID DSNKUIb++FmRVsHunCHUxOsmIyK1XgiStd66@postini.com; Tue, 23 Oct 2012 13:33:04 PDT Received: by mail-qc0-f200.google.com with SMTP id l42so7920971qco.7 for ; Tue, 23 Oct 2012 13:32:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=Fzz8q6RTRYeJfpAT53t/OqfSf7R2NQYRcllgZhkHGBM=; b=YrCqO/6nM4X3zeeGQF7TutsTSbVbUSuiOlBnjMUtJg/9BjWfCKzga4sBXWVrWf6kwz Gf+36PgLDLHNq66826yREedlR6TCSWqAMjeBMq39Fm2LhmpYC73W64jxcZNzBsMaOF9x 50q6lTqzCLr9QwSiltyNjqVUrlOzOqzom65CqzmVa270g00P9s/CCXK0kxBvKWJr9XP3 4HVWEBLlSGFj/Iqh243KCevaxCdizypgPQ4Q0bGsRkWoFEMqHcHpTOSz4Ap7St5UQPXw kEM7CTjSRDJW9pOhd5eVPqUDN2c17y3Yuis6ARWjCp/W7QUEFbWl+VPU7G5f6SCdc4PT M2yg== Received: by 10.52.98.105 with SMTP id eh9mr18567725vdb.11.1351024376122; Tue, 23 Oct 2012 13:32:56 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.98.105 with SMTP id eh9mr18567708vdb.11.1351024375758; Tue, 23 Oct 2012 13:32:55 -0700 (PDT) Received: by 10.58.28.138 with HTTP; Tue, 23 Oct 2012 13:32:55 -0700 (PDT) In-Reply-To: <238159534.2504674.1350598282110.JavaMail.root@erie.cs.uoguelph.ca> References: <238159534.2504674.1350598282110.JavaMail.root@erie.cs.uoguelph.ca> Date: Tue, 23 Oct 2012 15:32:55 -0500 Message-ID: Subject: Re: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) From: Thomas Johnson To: Rick Macklem X-Gm-Message-State: ALoCoQnsNhAk7eVdmMsJDf1cR/o+L8zd3/yi78NFA2gzGiSVeavITU0HHHl0X5hSf5nAoFtDeIsCLca14oC2phESCK7jxRebO9E4W3z4pUUCoZdSw2eVsb2aRiW8bzLvKomWxvbRmwPDUcSUABCosCFj3o6Ylzehgw== Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 20:33:05 -0000 I built a test image based on 9.1-rc2, per your suggestion Rick. The results are below. I was not able to exactly reproduce the workload in my original message, so I have also included results for the new (very similar) workload on my 9.0 client image as well. To summarize, 9.1-rc2 using newnfs seems to perform better than 9.0-p4, but oldnfs appears to still be significantly faster in both cases. I will get packet traces to Rick, but I want to get new results to the list. -Tom root@test:/test-> uname -a FreeBSD test.claimlynx.com 9.1-RC2 FreeBSD 9.1-RC2 #1: Fri Oct 19 08:27:12 CDT 2012 root@builder.claimlynx.com:/usr/obj/usr/src/sys/GENERIC amd64 root@test:/-> mount | grep test server:/array/test on /test (nfs) root@test:/test-> zip BIGGER_PILE.zip BIG_PILE_53* adding: BIG_PILE_5306.zip (stored 0%) adding: BIG_PILE_5378.zip (stored 0%) adding: BIG_PILE_5386.zip (stored 0%) root@test:/test-> ll -h BIGGER_PILE.zip -rw-rw-r-- 1 root claimlynx 5.5M Oct 23 14:05 BIGGER_PILE.zip root@test:/test-> time zip BIGGER_PILE.zip 53*.zip > /dev/null 0.664u 1.693s 0:30.21 7.7% 296+3084k 0+2926io 0pf+0w 0.726u 0.989s 0:08.04 21.1% 230+2667k 0+2956io 0pf+0w 0.829u 1.268s 0:11.89 17.4% 304+3037k 0+2961io 0pf+0w 0.807u 0.902s 0:08.02 21.1% 233+2676k 0+2947io 0pf+0w 0.753u 1.354s 0:12.73 16.4% 279+2879k 0+2947io 0pf+0w root@test:/test-> ll -h BIGGER_PILE.zip -rw-rw-r-- 1 root claimlynx 89M Oct 23 14:03 BIGGER_PILE.zip root@test:/test-> mount | grep test server:/array/test on /test (oldnfs) root@test:/test-> time zip BIGGER_PILE.zip 53*.zip > /dev/null 0.645u 1.435s 0:08.05 25.7% 295+3044k 0+5299io 0pf+0w 0.783u 0.993s 0:06.48 27.3% 225+2499k 0+5320io 0pf+0w 0.787u 1.000s 0:06.28 28.3% 246+2884k 0+5317io 0pf+0w 0.707u 1.392s 0:07.94 26.3% 266+2743k 0+5313io 0pf+0w 0.709u 1.056s 0:06.08 28.7% 246+2814k 0+5318io 0pf+0w root@test:/home/tom-> uname -a FreeBSD test.claimlynx.com 9.0-RELEASE-p4 FreeBSD 9.0-RELEASE-p4 #0: Tue Sep 18 11:51:11 CDT 2012 root@builder.claimlynx.com:/usr/obj/usr/src/sys/GENERIC amd64 root@test:/test-> mount | grep test server:/array/test on /test (nfs) root@test:/test-> time zip BIGGER_PILE.zip 53*.zip > /dev/null 0.721u 1.819s 0:31.13 8.1% 284+2886k 0+2932io 0pf+0w 0.725u 1.386s 0:12.84 16.3% 247+2631k 0+2957io 0pf+0w 0.675u 1.392s 0:13.94 14.7% 300+3005k 0+2928io 0pf+0w 0.705u 1.206s 0:10.72 17.7% 278+2874k 0+2973io 0pf+0w 0.727u 1.200s 0:18.28 10.5% 274+2872k 0+2947io 0pf+0w root@test:/-> umount /test root@test:/-> mount -t oldnfs server:/array/test /test root@test:/-> mount | grep test server:/array/test on /test (oldnfs) root@test:/test-> time zip BIGGER_PILE.zip 53*.zip > /dev/null 0.694u 1.820s 0:10.82 23.1% 271+2964k 0+5320io 0pf+0w 0.726u 1.293s 0:06.37 31.5% 303+2998k 0+5322io 0pf+0w 0.717u 1.248s 0:06.08 32.0% 246+2607k 0+5354io 0pf+0w 0.733u 1.230s 0:06.17 31.7% 256+2536k 0+5311io 0pf+0w 0.549u 1.581s 0:08.02 26.4% 302+3116k 0+5321io 0pf+0w On Thu, Oct 18, 2012 at 5:11 PM, Rick Macklem wrote: > Ronald Klop wrote: > > On Thu, 18 Oct 2012 18:16:16 +0200, Thomas Johnson > > wrote: > > > > > We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. > > > Almost > > > immediately, we received reports from users of poor performance. The > > > upgraded hosts are PXE-booted, with an NFS-mounted root. > > > Additionally, > > > they > > > mount a number of other NFS shares, which is where our users work > > > from. > > > After a week of tweaking rsize/wsize/readahead parameters (per > > > guidance), > > > it finally occurred to me that 9.0 defaults to the new NFS client > > > and > > > server. I remounted the user shares using the oldnfs file type, and > > > users > > > reported that performance returned to its expected level. > > > > > > This is obviously a workaround, rather than a solution. We would > > > prefer > > > to > > > get our hosts using the newnfs client, since presumably oldnfs will > > > be > > > deprecated at some point in the future. Is there some change that we > > > should > > > have made to our NFS configuration with the upgrade to 9.0, or is it > > > possible that our workload is exposing some deficiency with newnfs? > > > We > > > tend > > > to deal with a huge number of tiny files (several KB in size). The > > > NFS > > > server has been running 9.0 for some time (prior to the client > > > upgrade) > > > without any issue. NFS is served from a zpool, backed by a Dell > > > MD3000, > > > populated with 15k SAS disks. Clients and server are connected with > > > Gig-E > > > links. The general hardware configuration has not changed in nearly > > > 3 > > > years. > > > > > > As an example of the performance difference, here is some of the > > > testing > > > I > > > did while troubleshooting. Given a directory containing 5671 zip > > > files, > > > with an average size of 15KB. I append all files to an existing zip > > > file. > > > Using the newnfs mount, I found that this operation generally takes > > > ~30 > > > seconds (wall time). Switching the mount to oldnfs resulted in the > > > same > > > operation taking ~10 seconds. > > > > > > tom@test-1:/test-> ls 53*zip | wc -l > > > 5671 > > > tom@test-1:/test-> ll -h BIG* > > > -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip > > > tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip > > > 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w > > > ...reset and repeat... > > > 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w > > > ...reset and repeat... > > > 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w > > > > > > tom@test-1:/-> cd / > > > tom@test-1:/-> sudo umount /test > > > tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test > > > tom@test-1:/-> mount | grep test > > > server:/array/test on /test (oldnfs) > > > tom@test-1:/-> cd /test > > > ...reset and repeat... > > > 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w > > > ...reset and repeat... > > > 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w > > > tom@test-1:/test-> ll -h BIG* > > > -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip > > > > > > Thanks! > > > > > > > > > You might find this thread from today interesting. > > http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015441.html > > > Yes, although I can't explain why Alexey's problem went away > when he went from 9.0->9.1 for his NFS server, it would be > interesting if Thomas could try the same thing? > > About the only thing different between the old and new NFS > clients is the default rsize/wsize. However, if Thomas tried > rsize=32768,wsize=32768 for the default (new) NFS client, then > that would be ruled out. To be honest, the new client uses code > cloned from the old one for all the caching etc (which is where > the clients are "smart"). They use different RPC parsing code, > since the new one does NFSv4 as well, but that code is pretty > straightforward, so I can't think why it would result in a > factor of 3 in performance. > > If Thomas were to capture a packet trace of the above test > for two clients and emailed them to me, I could take a look > and see if I can see what is going on. (For Alexey's case, > it was a whole bunch of Read RPCs without replies, but that > was a Linux client, of course. It also had a significant # of > TCP layer retransmits and out of order TCP segments in it.) > > It would be nice to figure this out, since I was thinking > that the old client might go away for 10.0 (can't if these > issues still exist). > > rick > > > Ronald. > > _______________________________________________ > > -- Thomas Johnson ClaimLynx, Inc.