From owner-freebsd-stable@freebsd.org Mon Oct 24 17:34:13 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D8B2C1ECDD for ; Mon, 24 Oct 2016 17:34:13 +0000 (UTC) (envelope-from jamie@dyslexicfish.net) Received: from dyslexicfish.net (deadcat.mail.dyslexicfish.net [45.63.12.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7804CC9 for ; Mon, 24 Oct 2016 17:34:12 +0000 (UTC) (envelope-from jamie@dyslexicfish.net) Received: from dyslexicfish.net (deadcat.mail.dyslexicfish.net [45.63.12.202]) by dyslexicfish.net (8.14.5/8.14.5) with ESMTP id u9OHGHqG069235; Mon, 24 Oct 2016 18:16:17 +0100 (BST) (envelope-from jamie@dyslexicfish.net) Received: (from jamie@localhost) by dyslexicfish.net (8.14.5/8.14.5/Submit) id u9OHGGUA069234; Mon, 24 Oct 2016 18:16:16 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <201610241716.u9OHGGUA069234@dyslexicfish.net> Date: Mon, 24 Oct 2016 18:16:16 +0100 To: freebsd-stable@freebsd.org, bennett@sdf.org Cc: emz@norma.perm.ru Subject: Re: zfs, a directory that used to hold lot of files and listing pause References: <201610211340.u9LDer6D018453@sdf.org> In-Reply-To: <201610211340.u9LDer6D018453@sdf.org> User-Agent: Heirloom mailx 12.4 7/29/08 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (dyslexicfish.net [45.63.12.202]); Mon, 24 Oct 2016 18:16:17 +0100 (BST) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2016 17:34:13 -0000 Scott Bennett wrote: > thousand blocks allocated. Directories don't shrink. Directory entries do > not get moved around within directories when files are added or deleted. > Directories can remain the same length or they can grow in length. If a > directory once had many tens of thousands of filenames and links to their > primary inodes, then the directory is still that big, even if it now only > contains two [+ 20 to 30 directory], probably widely separated, entries. > IOW, if you want the performance to go back to what it was when the directory > was fresh (and still small), you have to create a new directory and then move > the remaining entries from the old directory into the new (small) directory. Not entirely true. FreeBSD on UFS *will* *truncate* directories where possible, each time a new directory entry is made, and there is contiguous unused space to the end of the directory-file. [ As an aside, tmpfs does 'defragment' and rezise directories in real time, when a file from *any* location is deleted. ] Back to UFS (I don't know about ZFS): So, whilst you are right about directory entries not being moved around, consider the following. Note the size of '.' in each directory listing: | 17:05 [2] (1) "~" jamie@lapcat% md dir | dir | | 17:05 [2] (2) "~" jamie@lapcat% cd dir | | 17:05 [2] (3) "~/dir" jamie@lapcat% l | total 8 | 4 drwxr-xr-x 2 jamie jamie - 512 24 Oct 17:05 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | | 17:05 [2] (4) "~/dir" jamie@lapcat% jot 999 1 999 | awk '{printf "touch %03d\n", $1}' | sh | | 17:05 [2] (5) "~/dir" jamie@lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:05 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:05 [2] (6) "~/dir" jamie@lapcat% rm [678]* | remove 300 files? y | | 17:06 [2] (7) "~/dir" jamie@lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (8) "~/dir" jamie@lapcat% touch x; rm x | | 17:06 [2] (9) "~/dir" jamie@lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (10) "~/dir" jamie@lapcat% rm 9* | remove 100 files? y | | 17:06 [2] (11) "~/dir" jamie@lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (12) "~/dir" jamie@lapcat% touch x ; rm x | | 17:06 [2] (13) "~/dir" jamie@lapcat% l | head | total 12 | 8 drwxr-xr-x 2 jamie jamie - 7680 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (14) "~/dir" jamie@lapcat% rm * | remove 599 files? y | | 17:06 [2] (15) "~/dir" jamie@lapcat% l | total 12 | 8 drwxr-xr-x 2 jamie jamie - 7680 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | | 17:06 [2] (16) "~/dir" jamie@lapcat% touch x ; rm x | | 17:06 [2] (17) "~/dir" jamie@lapcat% l | total 8 | 4 drwxr-xr-x 2 jamie jamie - 512 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | | 17:07 [2] (18) "~/dir" jamie@lapcat% exit