Date: Tue, 29 May 2012 01:06:20 -0700 (PDT) From: Don Lewis <truckman@FreeBSD.org> To: brde@optusnet.com.au Cc: freebsd-fs@FreeBSD.org, dougb@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options Message-ID: <201205290806.q4T86K8M007099@gw.catspoiler.org> In-Reply-To: <20120529161802.N975@besplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 29 May, Bruce Evans wrote: > On Mon, 28 May 2012, Doug Barton wrote: > >> On 5/28/2012 10:01 AM, Alessio Focardi wrote: >>> So in my case I would have to use -b 4096 -f 512 >>> >>> It's an improvement, but still is not ideal: still a big waste with 200 bytes files! >> >> Are all of the files exactly 200 bytes? If so that's likely the best you >> can do. > > It is easy to do better by using a file system that supports small block > sizes. This might be slow, but it reduces the wastage. Possible file > systems: > - it is easy to fix ffs to support a minimum block size of 512 (by > reducing its gratuitous limit of MINBSIZE and fixing the few things > that break: That shouldn't be necessary, especially if you newfs with the "-o space" option to force the fragments for multiple files to be allocated out of the same block right from the start unstead of waiting to do this once the filesystem starts getting full. I ran a Usenet server this way for quite a while with fairly good results, though the average file size was a bit bigger, about 2K or so. I found that if I didn't use "-o space" that space optimization wouldn't kick in soon enough and I'd tend to run out of full blocks that would be needed for larger files. The biggest performance problem that I ran into was that as the directories shrank and grew, they would tend to get badly fragmented, causing lookups to get slow. This was in the days before dirhash ...
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201205290806.q4T86K8M007099>