From owner-freebsd-fs@freebsd.org Wed Sep 7 08:44:29 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A470AB7177C for ; Wed, 7 Sep 2016 08:44:29 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41B2BF5B for ; Wed, 7 Sep 2016 08:44:29 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id i204so73827427wma.0 for ; Wed, 07 Sep 2016 01:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=L2Hpln2Zr0od6RuONKjJAexT++7tMzg9q0nj3bY8Cno=; b=NUEn8RlfwJD5vmPRlKc0Fwe1LqwySgumNSVgbSOokrLOOC+2G+nzmUO4406WZ2Y1lV JmB8cdx6NtyqAwTk3m8O1BM5xWrzJhO7ja/hRW16hmDhZ1+YsXXIu/1iqM72Agb1hiun 60Nu18y8hnrVHHZ4kDTnIRVa806x5rbPrLdQ/n7Nbf+fMuc9fXtnZlVOBc6m62ZhZPhl 1HhIby3zAbEcXLyBvUZvxdi0i5Cn0atcRCp9wKQ7j7nCK7Ft7TwELLPNv4u1DqjS0SgW ZwumSojgZevLPsr7DQn815AmiO2aKFgbIfVt8WrnhjdDb3LwwfUElBHPmnHdP+3+/d77 uFVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=L2Hpln2Zr0od6RuONKjJAexT++7tMzg9q0nj3bY8Cno=; b=Pg+oAMLYcLNuxl9OaAa7bhjrTH6do1nz9hnpB/C5iIYk4c0auUT7V5VV+qBFS60dAI zepEbsaXxn8AzzJhvP+WBLcvl/NCFRUqDcUdeS7SXEuzy8GUhwXo1fgrc4UJLHIMafx9 cNOoU2VfW6lPzOdrKVrQ7hP+iyyxC8VqYnKr5xLDbrYyC0bPI4U0BnUaoPr1+RmcYISb zJE7iYsU/POqjkybrXWGFAxXOETzbEdwEkAedH9AvcNorW6WuoXHRzkoGMIJpEaQ/T/+ 2wTVsCEsmJagMubCsTkpDhMzyOYeQ4pKqbelbEcTxAeLsmFhjAfsjJCP3qdKqnKQN3eM TMJg== X-Gm-Message-State: AE9vXwNUS6kNsk4D7HaKoc6ItcoYzk/akUxzOtcERhwJvHMrQZr/c1V3dcBqRDQXKdpbbw== X-Received: by 10.28.212.140 with SMTP id l134mr2575375wmg.15.1473237867483; Wed, 07 Sep 2016 01:44:27 -0700 (PDT) Received: from [172.20.10.6] ([80.12.59.77]) by smtp.gmail.com with ESMTPSA id ka5sm36921512wjb.7.2016.09.07.01.44.26 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 07 Sep 2016 01:44:26 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON In-Reply-To: <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> Date: Wed, 7 Sep 2016 10:44:23 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <5E3B106D-99CB-4F25-A363-6419C32FBF57@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> <7df8b5ce-d9ae-5b05-0aa5-1de6b06fd29e@internetx.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 08:44:29 -0000 OK ! Well, Juergen you put me on the right way. I investigated further and further and found that refquota slowness is = not due to the number of files but to the remaining free space. Let me explain this. Let's create a pool # zpool create -O atime=3Doff test /dev/... And do some fio tests with quota on it : # zfs set quota=3D500M refquota=3Dnone test/test ; cd /test/test # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D499M WRITE: io=3D510976KB, aggrb=3D805955KB/s, minb=3D805955KB/s, = maxb=3D805955KB/s, mint=3D634msec, maxt=3D634msec OK, very fast (nice SSD), even if we have filled the space up to the = quota. Now, let's do some fio tests with refquota : # zfs set quota=3Dnone refquota=3D500M test/test ; cd /test/test # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D499M WRITE: io=3D510976KB, aggrb=3D1679KB/s, minb=3D1679KB/s, = maxb=3D1679KB/s, mint=3D304177msec, maxt=3D304177msec (!!!) # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D490M WRITE: io=3D501760KB, aggrb=3D44010KB/s, minb=3D44010KB/s, = maxb=3D44010KB/s, mint=3D11401msec, maxt=3D11401msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D480M WRITE: io=3D491520KB, aggrb=3D92774KB/s, minb=3D92774KB/s, = maxb=3D92774KB/s, mint=3D5298msec, maxt=3D5298msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D450M WRITE: io=3D460800KB, aggrb=3D169038KB/s, minb=3D169038KB/s, = maxb=3D169038KB/s, mint=3D2726msec, maxt=3D2726msec # fio --bs=3D8192 --rw=3Dwrite --iodepth=3D24 --name=3Dtest = --filename=3Dtest.bin --direct=3D1 --ioengine=3Dsync --size=3D400M WRITE: io=3D409600KB, aggrb=3D215126KB/s, minb=3D215126KB/s, = maxb=3D215126KB/s, mint=3D1904msec, maxt=3D1904msec So, refquota is very very slow when filled space is near the limit, as = if it had some (huge) overhead for each block written. Now let's do some filling tests with dd, first with quota : # zfs set quota=3D500M refquota=3Dnone test/test # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D64k 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D8K 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D4K 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D1K 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D512 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec) And now the same filling tests with refquota : # zfs set quota=3Dnone refquota=3D500M test/test # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D64k 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D8K 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D4K 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D1K 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec) # dd if=3D/dev/random of=3D/test/test/bf conv=3Dsync bs=3D512 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec) OK so here, filling remaining space with small blocks, there are of = course more write operations than with big blocks, refquota overhead is = then really impacting.=20 I think that all these tests reveal that refquota has some overhead = compared to quota to compute a block write when used space is near the = limit. Do you agree ? Question would be then, why this overhead, and could it be reduced (as = with quota) ? (certainly a dev question here) Thank you again, Ben