Date: Sun, 27 Mar 2005 22:30:42 -0800 (PST) From: Don Lewis <truckman@FreeBSD.org> To: rwatson@FreeBSD.org Cc: freebsd-fs@FreeBSD.org Subject: Re: UFS Subdirectory limit. Message-ID: <200503280630.j2S6Ug0U093684@gw.catspoiler.org> In-Reply-To: <Pine.NEB.3.96L.1050327152512.49800C-100000@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 27 Mar, Robert Watson wrote: > FWIW, I regularly use directories with several hundred thousand files in > them, and it works quite well for the set of operations I perform > (typically, I only append new entries to the directory). This is with a > cyrus server hosting fairly large shared folders -- in Cyrus, a > maildir-like format is used. For example, the lists.linux.kernel > directory references 430,000 individual files. Between UFS_DIRHASH and > Cyrus's use of a cache file, opening the folder primarily consists of > mmap'ing the cache file and then doing lookups, which occur quite quickly. I'm doing the same here and performance is OK for day to day operations, but cloning the mail store is glacially slow. I had to move my mail store from a dieing disk to a replacement disk, and migrating about 5GB of data took the better part of a day. As the destination disk started to fill up, the I/O rates dropped to very low levels and the CPU was pegged at close to 100%, mostly system time. The problem is that the allocation strategy of locating file inodes and their data blocks in the same cylinder group as their parent directory fails badly with directories of this size. Once the cylinder group fills, the search for free inodes and blocks gets very slow. Using bigger cylinder groups would help a lot, especially considering disk sizes these days, but both UFS and UFS2 have inconveniently small maximum cylinder group sizes. I think it would make sense to be able to cluster groups of cylinder groups together for allocation purposes. Something else that would seem to make a lot of sense would be to implement something like the maxbpg limit for large directories that would force the inode allocator to start allocating from another cylinder group after a directory grows past a certain size. > My workload doesn't currently require large numbers of directories > referenced by a similar directory, but based on the results I've had with > large numbers of files, I can say it likely would work fine subject to the > ability for UFS to express it.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200503280630.j2S6Ug0U093684>