Date: Mon, 28 May 2012 22:00:39 -0700 From: Doug Barton <dougb@FreeBSD.org> To: Alessio Focardi <alessiof@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options Message-ID: <4FC457F7.9000800@FreeBSD.org> In-Reply-To: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On 5/28/2012 10:01 AM, Alessio Focardi wrote: > So in my case I would have to use -b 4096 -f 512 > > It's an improvement, but still is not ideal: still a big waste with 200 bytes files! Are all of the files exactly 200 bytes? If so that's likely the best you can do. The good news is that it's a big improvement (I've done similar stuff in the past). You'll also want to tweak the -i (inode) value to insure that you have sufficient inodes for the number of files you plan to store. The default is not likely to be adequate for your needs. Don't use journaling of any kind. If your primary access is going to be reads, you should be Ok with softupdates. If you're going to be doing a lot of writing consider no softupdates and async. Of course, all of this assumes that you have a reasonable disk array setup (RAID or similar, power backup, etc.) From the rest of your thread it seems that you have this under control. I think you already referenced this, but you also want to shard the file system so that you don't have any one directory with more than 10k files. Even with dirhash that's still a limitation. You don't have to worry too much if you go a few files over 10k, but you don't want to stretch it much more than that. Make sure you plan for growth over time. hth, Doug -- This .signature sanitized for your protection
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4FC457F7.9000800>