From owner-freebsd-fs@freebsd.org Mon Sep 5 18:30:25 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9ADA4B9616B for ; Mon, 5 Sep 2016 18:30:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1FC2BDC9 for ; Mon, 5 Sep 2016 18:30:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id b187so30610188wme.1 for ; Mon, 05 Sep 2016 11:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=0aw2cOVVamznw3mq6Cg2NBuQXTDCxdHylLPqfWkeln4=; b=gDyr/HIyiwkVzBodhr3Ehg0xGOGNoN2usTZ7x2u8vRTBvCxMxOe9TMv1lLBHpd3rZo isqG4KkfQu4Fc23nyeHzdEonOXsbDAEPxCqaJjT9PEorOc4YtRL/fi6/SCa8s6sMLGXW 2pyyuXZ6opkCjVw+tzeAjXzezGFbMiW7wNxof7CkMbbggPUogWkpk6ZbzWAxvTpXQqwS YmLFIRqQAxy64VclNu10pBfnCUsID2+8JKUNXatARucnkz1ume13rRwDNDF8ZqO2RkUC WqliDALM2MYpS4M7ctcWzBnJAcGkJFGRlLphWRBZU7ofDNVg1Kk6C4uam17nZYnzDaUi MA+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=0aw2cOVVamznw3mq6Cg2NBuQXTDCxdHylLPqfWkeln4=; b=NsQ8LeLkGGiuehoZ5seqcW0pYqntzCnq/ueHr/64HYD0406SQpxp6+9pIvki9FEPj3 ThC8P3g3OJC5lbFqSyg56LBuw/EEFR0f9ru7HQmy7JNWUF+AYa3JZZkLxYLY7Cy2LO3Z 0KMnhta3x9pwAphYyUHfautA6mavuNSWPze3+dLcCFPW4r7tLOI8AKB5bTJOdCrmth5V OaVaGsWZzcxk/0ujD2q5kZyPFCMccuFaarVvMY/vwNzAxXp5ddUg9goFy6HQ1Xd1xW9p eKA3WA0uYOb2Aauey9Sb38nLp4fBEgr+io5TnlIAtMVuG0jNW3hDEbcFzVGgn3+PbonR +L3A== X-Gm-Message-State: AE9vXwMQZwr3zJ3fNOOH7qxymq4rrRE7I9nfYnQZ/BkCDiumXluS8imzJNWX+2B2P3TEnA== X-Received: by 10.28.125.80 with SMTP id y77mr17195498wmc.25.1473100223320; Mon, 05 Sep 2016 11:30:23 -0700 (PDT) Received: from [172.20.10.6] ([80.12.63.231]) by smtp.gmail.com with ESMTPSA id g184sm22220342wme.15.2016.09.05.11.30.21 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 05 Sep 2016 11:30:22 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: [ZFS] refquota is very slow ! From: Ben RUBSON In-Reply-To: Date: Mon, 5 Sep 2016 20:30:19 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <67B3E11E-22B7-4719-A7AF-B8479D35A6D2@gmail.com> References: <1472914773423.63807@kuleuven.be> <0E828163-AEAB-4C8C-BFCF-93D42B3DB3B6@gmail.com> <1524067530.1937.a66cb17f-9141-4bef-b758-5bb129d16681.open-xchange@ox.internetx.com> To: FreeBSD FS X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2016 18:30:25 -0000 Yes, I want to dedicate my ARC & L2ARC to my data pool (which anyway = suffers from the same bug...). I followed your advices and created a whole new SSD test pool, let all = options by default, and added lz4 compression. I then create 20.000.000 empty files in this test pool. Then : # zfs set quota=3D7.5G refquota=3Dnone test/test # dd if=3D/dev/random of=3D/test/test/bf dd: bf: Disc quota exceeded 1047553+0 records in 1047552+0 records out 536346624 bytes transferred in 16.769513 secs (31983434 bytes/sec) # rm /test/test/bf # zfs set quota=3Dnone refquota=3D7.5G test/test # dd if=3D/dev/random of=3D/test/test/bf dd: bf: Disc quota exceeded 1047520+0 records in 1047519+0 records out 536329728 bytes transferred in 215.986582 secs (2483162 bytes/sec) # rm /test/test/bf Additional tests give the same results. 6MB before the limit, write IOs are done very slowly and it takes = minutes to fulfil these remaining MB. How many files in the pool you performed this test in Juergen ? Ben > On 05 Sep 2016, at 10:04, InterNetX - Juergen Gotteswinter = wrote: >=20 > any special reason for disabling secondarycache and limiting the > primarycache to metadata? does it change something when you revert it = to > default? >=20 > compression -> lz4, even if its not compressable, it wont hurt >=20 > you probably got several smaller performance issues which end up in = this > mess all together. >=20 > Am 04.09.2016 um 17:16 schrieb Ben RUBSON: >> Same kind of results with a single local (SSD) disk based pool, = refquota takes much more time than quota around the limit. >>=20 >> Here is the output for this single disk based pool : >> zfs get all : http://pastebin.com/raw/TScgy0ps >> zdb : http://pastebin.com/raw/BxmQ4xNx >> zpool get all : http://pastebin.com/raw/XugMbydy >>=20 >> Thank you ! >>=20 >> Ben >>=20 >>=20 >>> On 04 Sep 2016, at 13:42, InterNetX - Juergen Gotteswinter = wrote: >>>=20 >>> Did you try the same in a single local disk based pool? And pls post = output of >>> zfs get all, zdb & zpool get all >>>=20 >>>=20 >>>> Ben RUBSON hat am 4. September 2016 um 11:28 >>>> geschrieben: >>>>=20 >>>>=20 >>>> Juergen & Bram, >>>>=20 >>>> Thank you for your feedback. >>>>=20 >>>> I then investigated further and think I found the root cause. >>>>=20 >>>> No issue with refquota in my zroot pool containing (in this = example) 300.000 >>>> inodes used. >>>>=20 >>>> However, refquota is terribly slow in my data pool containing = around >>>> 12.000.000 inodes used. >>>>=20 >>>> I then created 12.000.000 empty file in my zroot pool, in a test = dataset. >>>> I put a refquota on this dataset and created a dd file to fulfil = empty space. >>>> And around the limit, it began to stall... >>>> I then created an empty dataset in the same pool, refquota is even = slow in >>>> this dataset having no inode used. >>>> The root cause seems then to be the total number of inodes used in = the pool... >>>>=20 >>>> Some numbers : >>>> Time to fulfil 512MB with quota : 17s >>>> Time to fulfil 512MB with refquota : 3m35s >>>>=20 >>>> Very strange. >>>>=20 >>>> Do you experience the same thing ? >>>>=20 >>>> Thank you again, >>>>=20 >>>> Ben >>>>=20 >>>>> On 03 Sep 2016, at 16:59, Bram Vandoren = wrote: >>>>>=20 >>>>> I encountered the same problem over NFS. I didn't manage to = reproduce it not >>>>> using NFS. I think the userquota property works without any = problem though. >>>>>=20 >>>>> Cheers, >>>>> Bram. >>>>=20 >>>>> On 03 Sep 2016, at 12:26, InterNetX - Juergen Gotteswinter >>>>> wrote: >>>>>=20 >>>>> cant confirm this, works like a charm without difference to normal = quota >>>>> setting