Date: Tue, 29 May 2012 17:35:18 +1000 (EST) From: Bruce Evans <brde@optusnet.com.au> To: Doug Barton <dougb@FreeBSD.org> Cc: freebsd-fs@FreeBSD.org Subject: Re: Millions of small files: best filesystem / best options Message-ID: <20120529161802.N975@besplex.bde.org> In-Reply-To: <4FC457F7.9000800@FreeBSD.org> References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org>
index | next in thread | previous in thread | raw e-mail
On Mon, 28 May 2012, Doug Barton wrote: > On 5/28/2012 10:01 AM, Alessio Focardi wrote: >> So in my case I would have to use -b 4096 -f 512 >> >> It's an improvement, but still is not ideal: still a big waste with 200 bytes files! > > Are all of the files exactly 200 bytes? If so that's likely the best you > can do. It is easy to do better by using a file system that supports small block sizes. This might be slow, but it reduces the wastage. Possible file systems: - msdosfs has a minimum block size of 512 and handles caching for this fairly well for a small number of files, but is probably even slower than ffs for a large number of files. Especially when directories are involved. - ext2fs has a minimum block size of 1024 and handles caching for this fairly poorly. - it is easy to fix ffs to support a minimum block size of 512 (by reducing its gratuitous limit of MINBSIZE and fixing the few things that break: % magic 19540119 (UFS2) time Tue May 29 16:16:20 2012 % superblock location 65536 id [ 4fc46886 2007c27b ] % ncg 4 size 1200 blocks 947 % bsize 512 shift 9 mask 0xfffffe00 % fsize 512 shift 9 mask 0xfffffe00 % frag 1 shift 0 fsbtodb 0 % minfree 8% optim time symlinklen 120 % maxbsize 512 maxbpg 64 maxcontig 256 contigsumsize 16 % nbfree 944 ndir 2 nifree 75 nffree 0 % bpg 301 fpg 301 ipg 20 % nindir 64 inopb 2 maxfilesize 136353791 % sbsize 1536 cgsize 512 csaddr 171 cssize 512 % sblkno 144 cblkno 160 iblkno 161 dblkno 171 % cgrotor 0 fmod 0 ronly 0 clean 1 % avgfpdir 64 avgfilesize 16384 % flags none % fsmnt /mnt % volname swuid 0 Note that sbsize is now larger than bsize. Most of the things that break involve wrong checks that sbsize <= bsize. sbsize is not limited by bsize in either direction, since the super block is accessed in DEV_BSIZE-blocks, not bsize-blocks and the upper limit on its size is not the same as upper limit on bsize. > The good news is that it's a big improvement (I've done similar > stuff in the past). You'll also want to tweak the -i (inode) value to > insure that you have sufficient inodes for the number of files you plan > to store. The default is not likely to be adequate for your needs. Big is relative. 4K-blocks with 200-byte files gives a wastage factor of 20. Metadata alone will be 256 bytes for the inode alone with ffs2. Only 128 bytes with ffs1. Only 32 bytes with msdosfs. > ... But I expect using a file system would be so slow for lots of really small files that I wouldn't try it. Caching is already poor for 4K-files, and a factor of 20 loss won't improve it. If you don't want to use a database, maybe you can use tar.[gz] files. These at least reduce the wastage (but still waste about twice as much as msdosfs with 512 byte blocks), unless they are compressed. I think there are ways to treat tar files as file systems and to avoid reading the whole file to find files in it (zip format is better for this). Brucehome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120529161802.N975>