From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 16:16:24 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 35DB6968 for ; Thu, 18 Oct 2012 16:16:24 +0000 (UTC) (envelope-from tom@claimlynx.com) Received: from na3sys009aog102.obsmtp.com (na3sys009aog102.obsmtp.com [74.125.149.69]) by mx1.freebsd.org (Postfix) with SMTP id 9B2228FC0A for ; Thu, 18 Oct 2012 16:16:23 +0000 (UTC) Received: from mail-vb0-f54.google.com ([209.85.212.54]) (using TLSv1) by na3sys009aob102.postini.com ([74.125.148.12]) with SMTP ID DSNKUIArUVucziojDpXjw3ePmvwiiSAoqExz@postini.com; Thu, 18 Oct 2012 09:16:23 PDT Received: by mail-vb0-f54.google.com with SMTP id v11so11412662vbm.13 for ; Thu, 18 Oct 2012 09:16:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=bp+3PfmWT3y9BOu3zvJHknxHDiavkrYvQAVnsSdpxkw=; b=g8gEDPr1rikMiJ0gSKducF9P7DqywE0t1e+INkKYSC+W2NZdpFEamyXSgt7Yzpi7uj +RmBmJOsjtUlKX3tmKt21B3fE90CalGEpzS+9IcMMYbTbYcvlrTXtK0Ek7Tp/irncoNa Br27wHrAweSBGzTHHWbCBV8HgOMkDgB+ZDzgogC20V22GLpqfxoDcpYL5zbXTbZHuJ7O xhAUd3o4C7sdZeynEThGJtJ5b9iPY29WWuuVkyJE+N5iYxNUiMcGmvHAP0umcJW+zNnj Y19M7IDoWGTWZ83gBF6Ybb8Y/ptkICaxYg5lOWI9FSOZJ0faEk6Af8NVxm5kw46exjua AS1A== MIME-Version: 1.0 Received: by 10.220.38.73 with SMTP id a9mr6312214vce.72.1350576976347; Thu, 18 Oct 2012 09:16:16 -0700 (PDT) Received: by 10.58.28.138 with HTTP; Thu, 18 Oct 2012 09:16:16 -0700 (PDT) Date: Thu, 18 Oct 2012 11:16:16 -0500 Message-ID: Subject: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) From: Thomas Johnson To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQntBVkBJ+bf5j6RfKCfcEJ0awMM9kFCY5Vqc8n+gPVQK0/CtEf0zcZQCsmmwRbp1/IRTsaa Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 16:16:24 -0000 We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. Almost immediately, we received reports from users of poor performance. The upgraded hosts are PXE-booted, with an NFS-mounted root. Additionally, they mount a number of other NFS shares, which is where our users work from. After a week of tweaking rsize/wsize/readahead parameters (per guidance), it finally occurred to me that 9.0 defaults to the new NFS client and server. I remounted the user shares using the oldnfs file type, and users reported that performance returned to its expected level. This is obviously a workaround, rather than a solution. We would prefer to get our hosts using the newnfs client, since presumably oldnfs will be deprecated at some point in the future. Is there some change that we should have made to our NFS configuration with the upgrade to 9.0, or is it possible that our workload is exposing some deficiency with newnfs? We tend to deal with a huge number of tiny files (several KB in size). The NFS server has been running 9.0 for some time (prior to the client upgrade) without any issue. NFS is served from a zpool, backed by a Dell MD3000, populated with 15k SAS disks. Clients and server are connected with Gig-E links. The general hardware configuration has not changed in nearly 3 years. As an example of the performance difference, here is some of the testing I did while troubleshooting. Given a directory containing 5671 zip files, with an average size of 15KB. I append all files to an existing zip file. Using the newnfs mount, I found that this operation generally takes ~30 seconds (wall time). Switching the mount to oldnfs resulted in the same operation taking ~10 seconds. tom@test-1:/test-> ls 53*zip | wc -l 5671 tom@test-1:/test-> ll -h BIG* -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w ...reset and repeat... 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w ...reset and repeat... 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w tom@test-1:/-> cd / tom@test-1:/-> sudo umount /test tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test tom@test-1:/-> mount | grep test server:/array/test on /test (oldnfs) tom@test-1:/-> cd /test ...reset and repeat... 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w ...reset and repeat... 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w tom@test-1:/test-> ll -h BIG* -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip Thanks! -- Thomas Johnson ClaimLynx, Inc.