Date: Wed, 7 Sep 2016 10:44:23 +0200 From: Ben RUBSON <ben.rubson@gmail.com> To: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: [ZFS] refquota is very slow ! Message-ID: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> In-Reply-To: <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> References: <D559DE69-A535-427C-A401-1458C2AA8C31@gmail.com> <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <EDDE17FC-1B3A-4912-B93C-08E18433A4C9@gmail.com> <f5969fc9-44e2-a8a0-1a7f-9475e65ab93a@internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com>
next in thread | previous in thread | raw e-mail | index | archive | help
OK ! Well, Juergen you put me on the right way. I investigated further and further and found that refquota slowness is = not due to the number of files but to the remaining free space. Let me explain this. Let's create a pool # zpool create -O atime=3Doff test /dev/... And do some fio tests with quota on it : # zfs set quota=3D500M refquota=3Dnone test/test ; cd /test/test # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D499M WRITE: io=3D510976KB, aggrb=3D805955KB/s, minb=3D805955KB/s, = maxb=3D805955KB/s, mint=3D634msec, maxt=3D634msec OK, very fast (nice SSD), even if we have filled the space up to the = quota. Now, let's do some fio tests with refquota : # zfs set quota=3Dnone refquota=3D500M test/test ; cd /test/test # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D499M WRITE: io=3D510976KB, aggrb=3D1679KB/s, minb=3D1679KB/s, = maxb=3D1679KB/s, mint=3D304177msec, maxt=3D304177msec (!!!) # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D490M WRITE: io=3D501760KB, aggrb=3D44010KB/s, minb=3D44010KB/s, = maxb=3D44010KB/s, mint=3D11401msec, maxt=3D11401msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D480M WRITE: io=3D491520KB, aggrb=3D92774KB/s, minb=3D92774KB/s, = maxb=3D92774KB/s, mint=3D5298msec, maxt=3D5298msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D450M WRITE: io=3D460800KB, aggrb=3D169038KB/s, minb=3D169038KB/s, = maxb=3D169038KB/s, mint=3D2726msec, maxt=3D2726msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D400M WRITE: io=3D409600KB, aggrb=3D215126KB/s, minb=3D215126KB/s, = maxb=3D215126KB/s, mint=3D1904msec, maxt=3D1904msec So, refquota is very very slow when filled space is near the limit, as = if it had some (huge) overhead for each block written. Now let's do some filling tests with dd, first with quota : # zfs set quota=3D500M refquota=3Dnone test/test # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D64k 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D8K 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D4K 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D1K 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D512 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec) And now the same filling tests with refquota : # zfs set quota=3Dnone refquota=3D500M test/test # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D64k 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D8K 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D4K 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D1K 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D512 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec) OK so here, filling remaining space with small blocks, there are of = course more write operations than with big blocks, refquota overhead is = then really impacting.=20 I think that all these tests reveal that refquota has some overhead = compared to quota to compute a block write when used space is near the = limit. Do you agree ? Question would be then, why this overhead, and could it be reduced (as = with quota) ? (certainly a dev question here) Thank you again, Ben
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5E3B106D-99CB-4F25-A363-6419C32FBF57>