Date: Wed, 7 Sep 2016 10:49:06 +0200 (CEST) From: InterNetX - Juergen Gotteswinter <juergen.gotteswinter@internetx.com> To: Ben RUBSON <ben.rubson@gmail.com>, FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: [ZFS] refquota is very slow ! Message-ID: <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> In-Reply-To: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> References: <D559DE69-A535-427C-A401-1458C2AA8C31@gmail.com> <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <EDDE17FC-1B3A-4912-B93C-08E18433A4C9@gmail.com> <f5969fc9-44e2-a8a0-1a7f-9475e65ab93a@internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> Ben RUBSON <ben.rubson@gmail.com> hat am 7. September 2016 um 10:44 > geschrieben: > > > > OK ! > Well, Juergen you put me on the right way. > I investigated further and further and found that refquota slowness is not due > to the number of files but to the remaining free space. > Let me explain this. > > Let's create a pool > # zpool create -O atime=off test /dev/... > > And do some fio tests with quota on it : > # zfs set quota=500M refquota=none test/test ; cd /test/test > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=499M > WRITE: io=510976KB, aggrb=805955KB/s, minb=805955KB/s, maxb=805955KB/s, > mint=634msec, maxt=634msec > > OK, very fast (nice SSD), even if we have filled the space up to the quota. > > Now, let's do some fio tests with refquota : > # zfs set quota=none refquota=500M test/test ; cd /test/test > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=499M > WRITE: io=510976KB, aggrb=1679KB/s, minb=1679KB/s, maxb=1679KB/s, > mint=304177msec, maxt=304177msec (!!!) > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=490M > WRITE: io=501760KB, aggrb=44010KB/s, minb=44010KB/s, maxb=44010KB/s, > mint=11401msec, maxt=11401msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=480M > WRITE: io=491520KB, aggrb=92774KB/s, minb=92774KB/s, maxb=92774KB/s, > mint=5298msec, maxt=5298msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=450M > WRITE: io=460800KB, aggrb=169038KB/s, minb=169038KB/s, maxb=169038KB/s, > mint=2726msec, maxt=2726msec > # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin > --direct=1 --ioengine=sync --size=400M > WRITE: io=409600KB, aggrb=215126KB/s, minb=215126KB/s, maxb=215126KB/s, > mint=1904msec, maxt=1904msec > > So, refquota is very very slow when filled space is near the limit, as if it > had some (huge) overhead for each block written. > > Now let's do some filling tests with dd, first with quota : > # zfs set quota=500M refquota=none test/test > # dd if=/dev/random of=/test/test/bf conv=sync bs=64k > 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=8K > 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=4K > 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=1K > 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=512 > 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec) > > And now the same filling tests with refquota : > # zfs set quota=none refquota=500M test/test > # dd if=/dev/random of=/test/test/bf conv=sync bs=64k > 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=8K > 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=4K > 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=1K > 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec) > # dd if=/dev/random of=/test/test/bf conv=sync bs=512 > 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec) > > OK so here, filling remaining space with small blocks, there are of course > more write operations than with big blocks, refquota overhead is then really > impacting. thats a general problem which you will also see on nearly full pools, try to no exceed 90% better 80-85%, its the nature of cow fs that this effect occours > > I think that all these tests reveal that refquota has some overhead compared > to quota to compute a block write when used space is near the limit. > Do you agree ? > > Question would be then, why this overhead, and could it be reduced (as with > quota) ? > (certainly a dev question here) > > Thank you again, > > Ben > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Wed Sep 7 08:53:39 2016 Return-Path: <owner-freebsd-fs@freebsd.org> Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 742CAB71C69 for <freebsd-fs@mailman.ysv.freebsd.org>; Wed, 7 Sep 2016 08:53:39 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 03181B5D for <freebsd-fs@freebsd.org>; Wed, 7 Sep 2016 08:53:39 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22b.google.com with SMTP id i204so74222224wma.0 for <freebsd-fs@freebsd.org>; Wed, 07 Sep 2016 01:53:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:message-id:references:to; bh=eDTvCkGaCvtDAuZ3U+dM2nk9yp/S/Zc+680mcHCtdeA=; b=aPoUE5yY50eeVhBw9ipiGsfPvT02A3Uab4cyrAkxrbi3fHnvW8dcu8B+aTDo23M9wr nva27LExXied9EnmxkrYEKJVvJ92FqBV38m1rGhXo0ZV8zssNmpEwSoVgs4YIif0SDRG pN/4mtRg+exmxYFq0Wa31lZyiKpZ0R1lpdWdbLhFKxaTGUiJ8qHM+Mye0/8BuM0ntwX0 eMHi5b1iYcj86cEgAeid1Pn9RnndW8FENap913R7yI+jqS32QYYxN224YTGI3EK26XV1 hlQtBD4gUrzqENiu3hX+KdUt5hWIVUFGp5JPGGnMbtbz2cQY6MOp8VFt+pLz8sihl1UC FQfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :message-id:references:to; bh=eDTvCkGaCvtDAuZ3U+dM2nk9yp/S/Zc+680mcHCtdeA=; b=f82a3XoVwVHcccyyx4HohYj9m4NpuW1fUIy31yrGs5gGMdx6o29YsXDgrd3m749cpW 6XaggAona0pZq3xR88UCW1c2Ttsttn4UIzZE7Y2m7YcLidWXrEUVELlUUU2nR6ZAjqA2 f5hcUWwwVbblIZhq69GHX2Py/Eknm3ZNp2MUqAgoEa/G0p1zvUVp5lQM0aR/3NXsih4y p4TEWf5TqSd2mBWVg9NOyaUWeKUxWTf+mej2r14j2SS1uqPgA8fOr9PK8g+nWDhfvU2Q Q3+pyiGlKUFJpYpW60CRaOsDRySd12eRL4UMOTiaixkQcy8VbpA/ASZ8n/UGm2bp3z2E KOew== X-Gm-Message-State: AE9vXwPHXpsYzqUpZEytwrdFPhsD5M2PprzWhHC30MoTwqp8BNzh4JbTy0hKu6rUBQT98w== X-Received: by 10.28.209.71 with SMTP id i68mr2579802wmg.97.1473238417184; Wed, 07 Sep 2016 01:53:37 -0700 (PDT) Received: from [172.20.10.6] ([80.12.59.77]) by smtp.gmail.com with ESMTPSA id k2sm3039601wmg.23.2016.09.07.01.53.36 for <freebsd-fs@freebsd.org> (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 07 Sep 2016 01:53:36 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON <ben.rubson@gmail.com> X-Priority: 3 In-Reply-To: <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> Date: Wed, 7 Sep 2016 10:53:37 +0200 Message-Id: <67817F5E-BB61-4DFF-A8E7-B1A190C7CD31@gmail.com> References: <D559DE69-A535-427C-A401-1458C2AA8C31@gmail.com> <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <EDDE17FC-1B3A-4912-B93C-08E18433A4C9@gmail.com> <f5969fc9-44e2-a8a0-1a7f-9475e65ab93a@internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> <630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange@ox.internetx.com> To: FreeBSD FS <freebsd-fs@freebsd.org> X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems <freebsd-fs.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>; List-Post: <mailto:freebsd-fs@freebsd.org> List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 07 Sep 2016 08:53:39 -0000 > On 07 Sep 2016, at 10:49, InterNetX - Juergen Gotteswinter = <juergen.gotteswinter@internetx.com> wrote: >=20 > > Ben RUBSON <ben.rubson@gmail.com <mailto:ben.rubson@gmail.com>> hat = am 7. September 2016 um 10:44 geschrieben: >=20 > > OK so here, filling remaining space with small blocks, there are of = course more write operations than with big blocks, refquota overhead is = then really impacting. > =20 > thats a general problem which you will also see on nearly full pools, = try to no exceed 90% better 80-85%, its the nature of cow fs that this = effect occours But why quota is not impacted then ? (ref)quota can be used to restrict users, so we should expect filling up = to the (ref)quota. Of course I agree that pool usage should not exceed 80%. Thank you, Ben
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?630576594.1937.7ad461c1-b158-4490-9c7f-fbae4a7c0881.open-xchange>