From owner-freebsd-fs@FreeBSD.ORG Wed Jun 12 09:26:20 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 287DFA16; Wed, 12 Jun 2013 09:26:20 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 7BB051C6A; Wed, 12 Jun 2013 09:26:18 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id B21A5109600B; Wed, 12 Jun 2013 11:26:17 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 24.4369] X-CRM114-CacheID: sfid-20130612_11260_D61B9634 X-CRM114-Status: Good ( pR: 24.4369 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Wed Jun 12 11:26:17 2013 X-DSPAM-Confidence: 0.9965 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 51b83eb9254715214434796 X-DSPAM-Factors: 27, From*Attila Nagy , 0.00010, >+On, 0.00059, FreeBSD, 0.00074, FreeBSD, 0.00074, )+>, 0.00139, wrote+>>, 0.00147, wrote+>, 0.00218, On+Tue, 0.00242, >+of, 0.00279, >+of, 0.00279, >+>>, 0.00321, 2013+at, 0.00348, >+>, 0.00371, >+>, 0.00371, References*fsn.hu>, 0.00371, the+server, 0.00397, >>+>>, 0.00428, something+like, 0.00463, parameters, 0.00463, >+have, 0.00463, queue, 0.00529, wrote, 0.00535, wrote, 0.00535, ZFS, 0.00555, ZFS, 0.00555, >+If, 0.00555, X-Spambayes-Classification: ham; 0.00 Received: from japan.t-online.private (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id C30B31095FF4; Wed, 12 Jun 2013 11:26:06 +0200 (CEST) Message-ID: <51B83EAE.7060603@fsn.hu> Date: Wed, 12 Jun 2013 11:26:06 +0200 From: Attila Nagy MIME-Version: 1.0 To: "Kenneth D. Merry" Subject: Re: An order of magnitude higher IOPS needed with ZFS than UFS References: <51B79023.5020109@fsn.hu> <253074981.119060.1370985609747.JavaMail.root@erie.cs.uoguelph.ca> <20130611232124.GA42577@nargothrond.kdm.org> In-Reply-To: <20130611232124.GA42577@nargothrond.kdm.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2013 09:26:20 -0000 Hi, On 06/12/13 01:21, Kenneth D. Merry wrote: > On Tue, Jun 11, 2013 at 17:20:09 -0400, Rick Macklem wrote: >> >> ken@ recently committed a change to the new NFS server to add file >> handle affinity support to it. He reported that he had found that, >> without file handle affinity, that ZFS's sequential reading heuristic >> broke badly (or something like that, you can probably find the email >> thread or maybe he will chime in). > That is correct. The problem, if the I/O is sequential, is that simultaneous > requests for adjacent blocks in a file will get farmed out to different The IO is pretty much random, and the files aren't so big either (mean size around 400k). > threads in the NFS server. These can easily go down into ZFS out of order, > and make the ZFS prefetch code think that the file is not being read > sequentially. It blows away the zfetch stream, and you wind up with a lot > of I/O bandwidth getting used (with a lot of prefetching done and then > re-done), but not much performance. I've tried disabling prefetch, without any noticeable effects. > > Linux clients are more likely than FreeBSD and MacOS clients to queue a lot > of reads to the server. The clients are also FreeBSD (8.3 and 7.2 mostly). Running NFSv3 of course. > >> Anyhow, you could try switching the FreeBSD 9 system to use the old >> NFS server (assuming your clients are doing NFSv3 mounts) and see if >> that has a significant effect. (For FreeBSD9, the old server has file >> handle affinity, but the new server does not.) > If using the old NFS server helps, then the FHA code for the new server > will help as well. Perhaps more, because the default FHA tuning parameters > have changed somewhat and parallel writes are now possible. > > If you want to try out the FHA changes in stable/9, I just MFCed them, > change 251641. > Sure, I will try both 251641 and the old nfsd. Thanks,