Date: Tue, 1 Aug 2000 00:32:16 +0000 (GMT) From: Terry Lambert <tlambert@primenet.com> To: zzhang@cs.binghamton.edu (Zhihui Zhang) Cc: stevec@nbci.com (Steve Carlson), freebsd-fs@FreeBSD.ORG Subject: Re: FFS performance for large directories? Message-ID: <200008010033.RAA15725@usr07.primenet.com> In-Reply-To: <Pine.SOL.4.21.0007311534090.1395-100000@sol.cs.binghamton.edu> from "Zhihui Zhang" at Jul 31, 2000 03:44:57 PM
next in thread | previous in thread | raw e-mail | index | archive | help
> A third thing is that FFS performs poor accessing /usr/ports. This has > something to do with how FFS layout directory inode (not file inode). The > book 4.4 BSD design and implementation explains this well. If fact, read > that book carefully, you can have better idea than you can get from a > mailing list. Good luck! This is because the tarball is packed up in the wrong order; change the packing order (breadth-first vs. depth-first), and the "ports problem" goes away. I have done this with the -T option to tar, and it works fine, so long as you have an accurate file. This ensures that there is no cache-busting on the dearchive, which is the source of the problem. In general, your performance will vary most based on your file name length. A given directory entry block is 512b, and a given directory entry will take 8 bytes, plus however many bytes in a name, rounded to a 4 byte boundary. So for an average 8 character file name length, this works out to 32 entries per block, at 16 bytes per entry. When searching directories, the getdirentries(2) system call is used. This is normally called with a buffer capable of handling a full directory entry block worth of entries in a single system call, so with a small average file length (e.g. as with a mail queue directory), this is fairly inexpensive. NB: Mail queue directories pay heavy penalties for the MTA scanning the queue contents, much more so than the overhead of traversing the directory, so the considerations of what you _do_ with the information once you know what files are in the directory is of great importance. Normally, this is not an issue for most applications, and only specific applications have issues with lots of entries (e.g. those like MTAs that must _do_ something with the data, and those which are creating new files, since you must traverse the entire directory to create a new file, whereas doing a lookup will on average only traverse 50% of the directory -- hence negative caches are generally twice as valuable as positive caches). If you would provide more information about why you are asking the question (i.e. what is your intended application?), we could probably give you advice on how to implement it such a way as to ensure the directory depth issue does not impact you. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200008010033.RAA15725>