Date: Wed, 7 Sep 2016 10:44:23 +0200 From: Ben RUBSON <ben.rubson@gmail.com> To: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: [ZFS] refquota is very slow ! Message-ID: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> In-Reply-To: <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> References: <D559DE69-A535-427C-A401-1458C2AA8C31@gmail.com> <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <EDDE17FC-1B3A-4912-B93C-08E18433A4C9@gmail.com> <f5969fc9-44e2-a8a0-1a7f-9475e65ab93a@internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com>
next in thread | previous in thread | raw e-mail | index | archive | help
OK ! Well, Juergen you put me on the right way. I investigated further and further and found that refquota slowness is not due to the number of files but to the remaining free space. Let me explain this. Let's create a pool # zpool create -O atime=off test /dev/... And do some fio tests with quota on it : # zfs set quota=500M refquota=none test/test ; cd /test/test # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=499M WRITE: io=510976KB, aggrb=805955KB/s, minb=805955KB/s, maxb=805955KB/s, mint=634msec, maxt=634msec OK, very fast (nice SSD), even if we have filled the space up to the quota. Now, let's do some fio tests with refquota : # zfs set quota=none refquota=500M test/test ; cd /test/test # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=499M WRITE: io=510976KB, aggrb=1679KB/s, minb=1679KB/s, maxb=1679KB/s, mint=304177msec, maxt=304177msec (!!!) # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=490M WRITE: io=501760KB, aggrb=44010KB/s, minb=44010KB/s, maxb=44010KB/s, mint=11401msec, maxt=11401msec # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=480M WRITE: io=491520KB, aggrb=92774KB/s, minb=92774KB/s, maxb=92774KB/s, mint=5298msec, maxt=5298msec # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=450M WRITE: io=460800KB, aggrb=169038KB/s, minb=169038KB/s, maxb=169038KB/s, mint=2726msec, maxt=2726msec # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=400M WRITE: io=409600KB, aggrb=215126KB/s, minb=215126KB/s, maxb=215126KB/s, mint=1904msec, maxt=1904msec So, refquota is very very slow when filled space is near the limit, as if it had some (huge) overhead for each block written. Now let's do some filling tests with dd, first with quota : # zfs set quota=500M refquota=none test/test # dd if=/dev/random of=/test/test/bf conv=sync bs=64k 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=8K 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=4K 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=1K 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=512 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec) And now the same filling tests with refquota : # zfs set quota=none refquota=500M test/test # dd if=/dev/random of=/test/test/bf conv=sync bs=64k 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=8K 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=4K 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=1K 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec) # dd if=/dev/random of=/test/test/bf conv=sync bs=512 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec) OK so here, filling remaining space with small blocks, there are of course more write operations than with big blocks, refquota overhead is then really impacting. I think that all these tests reveal that refquota has some overhead compared to quota to compute a block write when used space is near the limit. Do you agree ? Question would be then, why this overhead, and could it be reduced (as with quota) ? (certainly a dev question here) Thank you again, Ben
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5E3B106D-99CB-4F25-A363-6419C32FBF57>