From owner-freebsd-fs@freebsd.org Wed Sep 7 08:49:19 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10387B7182B for ; Wed, 7 Sep 2016 08:49:19 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from mx1.internetx.com (mx1.internetx.com [62.116.129.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 733F410B for ; Wed, 7 Sep 2016 08:49:18 +0000 (UTC) (envelope-from juergen.gotteswinter@internetx.com) Received: from localhost (localhost [127.0.0.1]) by mx1.internetx.com (Postfix) with ESMTP id C1BB245FC104; Wed, 7 Sep 2016 10:49:09 +0200 (CEST) X-Virus-Scanned: InterNetX GmbH amavisd-new at ix-mailer.internetx.de Received: from mx1.internetx.com ([62.116.129.39]) by localhost (ix-mailer.internetx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 04-rj857161O; Wed, 7 Sep 2016 10:49:07 +0200 (CEST) Received: from ox-groupware-node02.internetx.de (ox.internetx.com [85.236.36.83]) by mx1.internetx.com (Postfix) with ESMTP id 0F13745FC0FF; Wed, 7 Sep 2016 10:49:07 +0200 (CEST) Received: from ox-groupware-node02.internetx.de (localhost [127.0.0.1]) by ox-groupware-node02.internetx.de (Postfix) with ESMTP id ED7481C4409B; Wed, 7 Sep 2016 10:49:06 +0200 (CEST) Date: Wed, 7 Sep 2016 10:49:06 +0200 (CEST) From: InterNetX - Juergen Gotteswinter To: Ben RUBSON , FreeBSD FS Message-ID: <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> In-Reply-To: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> Subject: Re: [ZFS] refquota is very slow ! MIME-Version: 1.0 X-Priority: 3 Importance: Medium X-Mailer: Open-Xchange Mailer v7.8.0-Rev27 Organization: InterNetX GmbH X-Originating-Client: com.openexchange.ox.gui.dhtml Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 08:49:19 -0000 > Ben RUBSON hat am 7. September 2016 um 10:44 > geschrieben: > > > > OK ! > Well, Juergen you put me on the right way. > I investigated further and further and found that refquota slowness is not due > to the number of files but to the remaining free space. > Let me explain this. > > Let's create a pool > # zpool create -O atime=off test /dev/... > > And do some fio tests with quota on it : > # zfs set quota=500M refquota=none test/test ; cd /test/test > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=499M > WRITE: io=510976KB, aggrb=805955KB/s, minb=805955KB/s, maxb=805955KB/s, > mint=634msec, maxt=634msec > > OK, very fast (nice SSD), even if we have filled the space up to the quota. > > Now, let's do some fio tests with refquota : > # zfs set quota=none refquota=500M test/test ; cd /test/test > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=499M > WRITE: io=510976KB, aggrb=1679KB/s, minb=1679KB/s, maxb=1679KB/s, > mint=304177msec, maxt=304177msec (!!!) > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=490M > WRITE: io=501760KB, aggrb=44010KB/s, minb=44010KB/s, maxb=44010KB/s, > mint=11401msec, maxt=11401msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=480M > WRITE: io=491520KB, aggrb=92774KB/s, minb=92774KB/s, maxb=92774KB/s, > mint=5298msec, maxt=5298msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=450M > WRITE: io=460800KB, aggrb=169038KB/s, minb=169038KB/s, maxb=169038KB/s, > mint=2726msec, maxt=2726msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=400M > WRITE: io=409600KB, aggrb=215126KB/s, minb=215126KB/s, maxb=215126KB/s, > mint=1904msec, maxt=1904msec > > So, refquota is very very slow when filled space is near the limit, as if it > had some (huge) overhead for each block written. > > Now let's do some filling tests with dd, first with quota : > # zfs set quota=500M refquota=none test/test > # dd if=/dev/random of=/test/test/bf conv=sync bs=64k > 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=8K > 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=4K > 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=1K > 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=512 > 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec) > > And now the same filling tests with refquota : > # zfs set quota=none refquota=500M test/test > # dd if=/dev/random of=/test/test/bf conv=sync bs=64k > 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=8K > 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=4K > 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=1K > 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=512 > 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec) > > OK so here, filling remaining space with small blocks, there are of course > more write operations than with big blocks, refquota overhead is then really > impacting. thats a general problem which you will also see on nearly full pools, try to no exceed 90% better 80-85%, its the nature of cow fs that this effect occours > > I think that all these tests reveal that refquota has some overhead compared > to quota to compute a block write when used space is near the limit. > Do you agree ? > > Question would be then, why this overhead, and could it be reduced (as with > quota) ? > (certainly a dev question here) > > Thank you again, > > Ben > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Sep 7 08:53:39 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 742CAB71C69 for ; Wed, 7 Sep 2016 08:53:39 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 03181B5D for ; Wed, 7 Sep 2016 08:53:39 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22b.google.com with SMTP id i204so74222224wma.0 for ; Wed, 07 Sep 2016 01:53:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:message-id:references:to; bh=eDTvCkGaCvtDAuZ3U+dM2nk9yp/S/Zc+680mcHCtdeA=; b=aPoUE5yY50eeVhBw9ipiGsfPvT02A3Uab4cyrAkxrbi3fHnvW8dcu8B+aTDo23M9wr nva27LExXied9EnmxkrYEKJVvJ92FqBV38m1rGhXo0ZV8zssNmpEwSoVgs4YIif0SDRG pN/4mtRg+exmxYFq0Wa31lZyiKpZ0R1lpdWdbLhFKxaTGUiJ8qHM+Mye0/8BuM0ntwX0 eMHi5b1iYcj86cEgAeid1Pn9RnndW8FENap913R7yI+jqS32QYYxN224YTGI3EK26XV1 hlQtBD4gUrzqENiu3hX+KdUt5hWIVUFGp5JPGGnMbtbz2cQY6MOp8VFt+pLz8sihl1UC FQfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :message-id:references:to; bh=eDTvCkGaCvtDAuZ3U+dM2nk9yp/S/Zc+680mcHCtdeA=; b=f82a3XoVwVHcccyyx4HohYj9m4NpuW1fUIy31yrGs5gGMdx6o29YsXDgrd3m749cpW 6XaggAona0pZq3xR88UCW1c2Ttsttn4UIzZE7Y2m7YcLidWXrEUVELlUUU2nR6ZAjqA2 f5hcUWwwVbblIZhq69GHX2Py/Eknm3ZNp2MUqAgoEa/G0p1zvUVp5lQM0aR/3NXsih4y p4TEWf5TqSd2mBWVg9NOyaUWeKUxWTf+mej2r14j2SS1uqPgA8fOr9PK8g+nWDhfvU2Q Q3+pyiGlKUFJpYpW60CRaOsDRySd12eRL4UMOTiaixkQcy8VbpA/ASZ8n/UGm2bp3z2E KOew== X-Gm-Message-State: AE9vXwPHXpsYzqUpZEytwrdFPhsD5M2PprzWhHC30MoTwqp8BNzh4JbTy0hKu6rUBQT98w== X-Received: by 10.28.209.71 with SMTP id i68mr2579802wmg.97.1473238417184; Wed, 07 Sep 2016 01:53:37 -0700 (PDT) Received: from [172.20.10.6] ([80.12.59.77]) by smtp.gmail.com with ESMTPSA id k2sm3039601wmg.23.2016.09.07.01.53.36 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 07 Sep 2016 01:53:36 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON X-Priority: 3 In-Reply-To: <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> Date: Wed, 7 Sep 2016 10:53:37 +0200 Message-Id: <67817F5E-BB61-4DFF-A8E7-B1A190C7CD31@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 08:53:39 -0000 > On 07 Sep 2016, at 10:49, InterNetX - Juergen Gotteswinter = wrote: >=20 > > Ben RUBSON > hat = am 7. September 2016 um 10:44 geschrieben: >=20 > > OK so here, filling remaining space with small blocks, there are of = course more write operations than with big blocks, refquota overhead is = then really impacting. > =20 > thats a general problem which you will also see on nearly full pools, = try to no exceed 90% better 80-85%, its the nature of cow fs that this = effect occours But why quota is not impacted then ? (ref)quota can be used to restrict users, so we should expect filling up = to the (ref)quota. Of course I agree that pool usage should not exceed 80%. Thank you, Ben