From owner-freebsd-fs@FreeBSD.ORG Fri Jun 21 21:10:03 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 26B94C30 for ; Fri, 21 Jun 2013 21:10:03 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-la0-x234.google.com (mail-la0-x234.google.com [IPv6:2a00:1450:4010:c03::234]) by mx1.freebsd.org (Postfix) with ESMTP id 9D2F9166F for ; Fri, 21 Jun 2013 21:10:02 +0000 (UTC) Received: by mail-la0-f52.google.com with SMTP id fo12so7956711lab.39 for ; Fri, 21 Jun 2013 14:10:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=JVgfVh3NrPxOGfKyIUAP084/ocYp9DlSLpQxcU76UWQ=; b=CcZ+yLA66DQaOIMW3d+edQa83HZWR5p5I7DMq/S+MBthfvrk+GYJFtEDIW+PzFLVPG Y0vTI0Mo4Xg+iRKfTuLmlL/9a5G85OEzOSsDarXrAxtLQIvR9Bm7YwfaVMt0TuxufaGI S9io1P/7ndxl01g1c4hw+AFes1q+sNk6gl4Nw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=JVgfVh3NrPxOGfKyIUAP084/ocYp9DlSLpQxcU76UWQ=; b=bXWOcEJUOnCNS52co5pyOiISjnYDekZ8ympCqZimF8KYGas946ibB5ZXpmWZYNcne8 UdFxIzn6o+5pIzLiiY7CJg7KUTOOtksa9iIFQ4tO/4k/w5PakOh4i1PWxYflvK8FG7F1 uhwQ4H0lkxwxYDg8hLEAXW14J/AxdamL6H3mSrCymn2MEqgr31X9uRoGcK2CEaLz2RFA vm389HTCqaKFTENU0EB9doduw7oUPTEPJiSF0OyA9O1QmkydqccHzpXheOWn3L4lFgKo LRVxoalBTVW52USKCB8B2uqLKxL1yE56WE5PLbV31OZiSG76XJ99izYBkuWjT7TISZaH Py9g== MIME-Version: 1.0 X-Received: by 10.152.28.129 with SMTP id b1mr6650910lah.51.1371849001435; Fri, 21 Jun 2013 14:10:01 -0700 (PDT) Received: by 10.114.79.132 with HTTP; Fri, 21 Jun 2013 14:10:01 -0700 (PDT) In-Reply-To: <51B982A8.10605@bytecamp.net> References: <51B79023.5020109@fsn.hu> <20130612114937.GA13688@icarus.home.lan> <51B982A8.10605@bytecamp.net> Date: Fri, 21 Jun 2013 14:10:01 -0700 Message-ID: Subject: Re: An order of magnitude higher IOPS needed with ZFS than UFS From: Matthew Ahrens To: Robert Schulze X-Gm-Message-State: ALoCoQkcX/3DLmbp5TZ0XmdxZn1qGl7m/7vvLAqQB+VLG/pvm6G5sIFeguuoAFZdfP+PjqI4KB2R Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Jun 2013 21:10:03 -0000 The reason for this behavior is that it's hard to know how much space things will take up on disk until we get to writing them out. For example, because compression changes the space usage, and we don't compress the data until we write it out. (And all metadata, (e.g. indirect blocks) are compressed, and count towards quotas.) I think it would be reasonable to implement something like "quota_enforcement=delayed", which would cause regular quotas, reservations, refquotas, and refreservations to behave like userquota@... quotas, in that they can be exceeded by a small amount (a few seconds worth of change), but they do not impact performance. --matt On Thu, Jun 13, 2013 at 1:28 AM, Robert Schulze wrote: > Hi, > > Am 12.06.2013 13:49, schrieb Jeremy Chadwick: > >> On Wed, Jun 12, 2013 at 06:40:32AM -0500, Mark Felder wrote: >> >>> On Tue, 11 Jun 2013 16:01:23 -0500, Attila Nagy wrote: >>> >>> ZFS write performance can begin to drop pretty badly when you get >>> around 80% full. I've not seen any benchmarks showing an improvement >>> with a very fast and large ZIL or tons of memory, but I'd expect >>> that would help significantly. Just note that you're right at the >>> edge where performance gets impacted. >>> >> >> Mark, do you have any references for this? I'd love to learn/read more >> about this engineering/design aspect (I won't say flaw, I'll just say >> aspect) to ZFS, as it's the first I've heard of it. >> > > this is even true when getting near a quota limit on a zfs, although there > are e.g. 10/16 TB free in the pool. > > Just create a filesystem and set quota=1G, then do sequential invocations > of dd to fill the fs with 100M files. You will see a sharp slowdown when > the last twenty files are beeing created. > > Here are the results from the following short test: > > for i in `jot - 0 99` > do > dd if=/dev/zero of=/pool/quota-test/10M.$i bs=1M count=10 > done > > 0..80: < 0.4 s > 80 0.27 s > 81 0.77 s > 82 0.50 s > 83 0.51 s > 84 0.22 s > 85 0.87 s > 86 0.52 s > 87 1.13 s > 88 0.91 s > 90 0.39 s > 91 1.04 s > 92 0.80 s > 93 1.94 s > 94 1.27 s > 95 1.36 s > 96 1.76 s > 97 2.13 s > 98 3.28 s > 99 4.07 s > > of course, there are some small values beyond 80% utilisation, but I think > the trend is clearly visible. > > In my opinion, hitting a quota limit should not give these results unless > enough free physical disk space is available in the pool. This is a bug or > a design flaw and creating serious problems when exporting quota'ed zfs > over nfs. > > with kind regards, > Robert Schulze > > -- > /7\ bytecamp GmbH > Geschwister-Scholl-Str. 10, 14776 Brandenburg a.d. Havel > HRB15752, Amtsgericht Potsdam, Geschaeftsfuehrer: > Bjoern Barnekow, Frank Rosenbaum, Sirko Zidlewitz > tel +49 3381 79637-0 werktags 10-12,13-17 Uhr, fax +49 3381 79637-20 > mail rs@bytecamp.net, web http://bytecamp.net/ > > ______________________________**_________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@**freebsd.org > " >