From owner-freebsd-fs@FreeBSD.ORG Thu Jun 13 08:28:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4EAE5BF1 for ; Thu, 13 Jun 2013 08:28:33 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: from mail.bytecamp.net (mail.bytecamp.net [212.204.60.9]) by mx1.freebsd.org (Postfix) with ESMTP id DC9671F60 for ; Thu, 13 Jun 2013 08:28:32 +0000 (UTC) Received: (qmail 41990 invoked by uid 89); 13 Jun 2013 10:28:24 +0200 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with CAMELLIA256-SHA encrypted SMTP; 13 Jun 2013 10:28:24 +0200 Message-ID: <51B982A8.10605@bytecamp.net> Date: Thu, 13 Jun 2013 10:28:24 +0200 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130330 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: An order of magnitude higher IOPS needed with ZFS than UFS References: <51B79023.5020109@fsn.hu> <20130612114937.GA13688@icarus.home.lan> In-Reply-To: <20130612114937.GA13688@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2013 08:28:33 -0000 Hi, Am 12.06.2013 13:49, schrieb Jeremy Chadwick: > On Wed, Jun 12, 2013 at 06:40:32AM -0500, Mark Felder wrote: >> On Tue, 11 Jun 2013 16:01:23 -0500, Attila Nagy wrote: >> >> ZFS write performance can begin to drop pretty badly when you get >> around 80% full. I've not seen any benchmarks showing an improvement >> with a very fast and large ZIL or tons of memory, but I'd expect >> that would help significantly. Just note that you're right at the >> edge where performance gets impacted. > > Mark, do you have any references for this? I'd love to learn/read more > about this engineering/design aspect (I won't say flaw, I'll just say > aspect) to ZFS, as it's the first I've heard of it. this is even true when getting near a quota limit on a zfs, although there are e.g. 10/16 TB free in the pool. Just create a filesystem and set quota=1G, then do sequential invocations of dd to fill the fs with 100M files. You will see a sharp slowdown when the last twenty files are beeing created. Here are the results from the following short test: for i in `jot - 0 99` do dd if=/dev/zero of=/pool/quota-test/10M.$i bs=1M count=10 done 0..80: < 0.4 s 80 0.27 s 81 0.77 s 82 0.50 s 83 0.51 s 84 0.22 s 85 0.87 s 86 0.52 s 87 1.13 s 88 0.91 s 90 0.39 s 91 1.04 s 92 0.80 s 93 1.94 s 94 1.27 s 95 1.36 s 96 1.76 s 97 2.13 s 98 3.28 s 99 4.07 s of course, there are some small values beyond 80% utilisation, but I think the trend is clearly visible. In my opinion, hitting a quota limit should not give these results unless enough free physical disk space is available in the pool. This is a bug or a design flaw and creating serious problems when exporting quota'ed zfs over nfs. with kind regards, Robert Schulze -- /7\ bytecamp GmbH Geschwister-Scholl-Str. 10, 14776 Brandenburg a.d. Havel HRB15752, Amtsgericht Potsdam, Geschaeftsfuehrer: Bjoern Barnekow, Frank Rosenbaum, Sirko Zidlewitz tel +49 3381 79637-0 werktags 10-12,13-17 Uhr, fax +49 3381 79637-20 mail rs@bytecamp.net, web http://bytecamp.net/