From owner-freebsd-fs@FreeBSD.ORG Mon Jul 25 05:42:50 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E201E1065672 for ; Mon, 25 Jul 2011 05:42:50 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id EE3ED8FC0C for ; Mon, 25 Jul 2011 05:42:49 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 304F49478D1; Mon, 25 Jul 2011 07:42:48 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000016, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 14.1348] X-CRM114-CacheID: sfid-20110725_07424_61E03616 X-CRM114-Status: Good ( pR: 14.1348 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Mon Jul 25 07:42:48 2011 X-DSPAM-Confidence: 0.9957 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 4e2d025823651866976671 X-DSPAM-Factors: 27, From*Attila Nagy , 0.00010, >+I, 0.00102, >+I, 0.00102, >+On, 0.00111, com>+wrote, 0.00176, wrote+>, 0.00197, wrote+>>, 0.00315, >+>, 0.00321, >+>, 0.00321, References*mail.gmail.com>, 0.00348, References*mail.gmail.com>, 0.00348, org>+wrote, 0.00352, this+>, 0.00460, On+Thu, 0.00460, >>+>>, 0.00478, >>+>>, 0.00478, wrote, 0.00481, wrote, 0.00481, with+>, 0.00543, In-Reply-To*mail.gmail.com>, 0.00543, files+and, 0.00597, default, 0.00663, for+>, 0.00663, files, 0.00808, files, 0.00808, Is+there, 0.00851, X-Spambayes-Classification: ham; 0.00 Received: from japan.t-online.private (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id 69CD69478C4; Mon, 25 Jul 2011 07:42:47 +0200 (CEST) Message-ID: <4E2D0257.8080608@fsn.hu> Date: Mon, 25 Jul 2011 07:42:47 +0200 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Ivan Voras References: In-Reply-To: X-Stationery: 0.7.1 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS and large directories - caveat report X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2011 05:42:51 -0000 On 07/21/11 18:38, Ivan Voras wrote: > On 21 July 2011 17:50, Freddie Cash wrote: >> On Thu, Jul 21, 2011 at 8:45 AM, Ivan Voras wrote: >>> Is there an equivalent of UFS dirhash memory setting for ZFS? (i.e. the >>> size of the metadata cache) >> vfs.zfs.arc_meta_limit >> >> This sets the amount of ARC that can be used for metadata. The default is >> 1/8th of ARC, I believe. This setting lets you use "primarycache=all" >> (store metadata and file data in ARC) but then tune how much is used for >> each. >> >> Not sure if that will help in your case or not, but it's a sysctl you can >> play with. > I don't think that it works, or at least is not as efficient as dirhash: > > www:~> sysctl -a | grep meta > kern.metadelay: 28 > vfs.zfs.mfu_ghost_metadata_lsize: 129082368 > vfs.zfs.mfu_metadata_lsize: 116224 > vfs.zfs.mru_ghost_metadata_lsize: 113958912 > vfs.zfs.mru_metadata_lsize: 16384 > vfs.zfs.anon_metadata_lsize: 0 > vfs.zfs.arc_meta_limit: 322412800 > vfs.zfs.arc_meta_used: 506907792 > kstat.zfs.misc.arcstats.demand_metadata_hits: 4471705 > kstat.zfs.misc.arcstats.demand_metadata_misses: 2110328 > kstat.zfs.misc.arcstats.prefetch_metadata_hits: 27 > kstat.zfs.misc.arcstats.prefetch_metadata_misses: 51 > > arc_meta_used is nearly 500 MB which should be enough even in this > case. With filenames of 32 characters, all the filenames alone for > 130,000 files in a directory take about 4 MB - I doubt the ZFS > introduces so much extra metadata it doesn't fit in 500 MB. > > I am now deleting the session files, and I hope it will not take days > to complete... > Worse than that, I've seen a similar issue, hashed directories with about 1M+ files. After deleting all those files, even a find on the empty directories took ages...