Date: Mon, 24 Oct 2016 18:16:16 +0100 From: Jamie Landeg-Jones <jamie@dyslexicfish.net> To: freebsd-stable@freebsd.org, bennett@sdf.org Cc: emz@norma.perm.ru Subject: Re: zfs, a directory that used to hold lot of files and listing pause Message-ID: <201610241716.u9OHGGUA069234@dyslexicfish.net> In-Reply-To: <201610211340.u9LDer6D018453@sdf.org> References: <201610211340.u9LDer6D018453@sdf.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Scott Bennett <bennett@sdf.org> wrote:
> thousand blocks allocated. Directories don't shrink. Directory entries do
> not get moved around within directories when files are added or deleted.
> Directories can remain the same length or they can grow in length. If a
> directory once had many tens of thousands of filenames and links to their
> primary inodes, then the directory is still that big, even if it now only
> contains two [+ 20 to 30 directory], probably widely separated, entries.
> IOW, if you want the performance to go back to what it was when the directory
> was fresh (and still small), you have to create a new directory and then move
> the remaining entries from the old directory into the new (small) directory.
Not entirely true. FreeBSD on UFS *will* *truncate* directories where possible,
each time a new directory entry is made, and there is contiguous unused space
to the end of the directory-file.
[ As an aside, tmpfs does 'defragment' and rezise directories
in real time, when a file from *any* location is deleted. ]
Back to UFS (I don't know about ZFS):
So, whilst you are right about directory entries not being moved around,
consider the following. Note the size of '.' in each directory listing:
| 17:05 [2] (1) "~" jamie@lapcat% md dir
| dir
|
| 17:05 [2] (2) "~" jamie@lapcat% cd dir
|
| 17:05 [2] (3) "~/dir" jamie@lapcat% l
| total 8
| 4 drwxr-xr-x 2 jamie jamie - 512 24 Oct 17:05 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
|
| 17:05 [2] (4) "~/dir" jamie@lapcat% jot 999 1 999 | awk '{printf "touch %03d\n", $1}' | sh
|
| 17:05 [2] (5) "~/dir" jamie@lapcat% l | head
| total 16
| 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:05 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007
|
| *141* 17:05 [2] (6) "~/dir" jamie@lapcat% rm [678]*
| remove 300 files? y
|
| 17:06 [2] (7) "~/dir" jamie@lapcat% l | head
| total 16
| 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007
|
| *141* 17:06 [2] (8) "~/dir" jamie@lapcat% touch x; rm x
|
| 17:06 [2] (9) "~/dir" jamie@lapcat% l | head
| total 16
| 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007
|
| *141* 17:06 [2] (10) "~/dir" jamie@lapcat% rm 9*
| remove 100 files? y
|
| 17:06 [2] (11) "~/dir" jamie@lapcat% l | head
| total 16
| 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007
|
| *141* 17:06 [2] (12) "~/dir" jamie@lapcat% touch x ; rm x
|
| 17:06 [2] (13) "~/dir" jamie@lapcat% l | head
| total 12
| 8 drwxr-xr-x 2 jamie jamie - 7680 24 Oct 17:06 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006
| 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007
|
| *141* 17:06 [2] (14) "~/dir" jamie@lapcat% rm *
| remove 599 files? y
|
| 17:06 [2] (15) "~/dir" jamie@lapcat% l
| total 12
| 8 drwxr-xr-x 2 jamie jamie - 7680 24 Oct 17:06 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
|
| 17:06 [2] (16) "~/dir" jamie@lapcat% touch x ; rm x
|
| 17:06 [2] (17) "~/dir" jamie@lapcat% l
| total 8
| 4 drwxr-xr-x 2 jamie jamie - 512 24 Oct 17:06 ./
| 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../
|
| 17:07 [2] (18) "~/dir" jamie@lapcat% exit
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201610241716.u9OHGGUA069234>