Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 29 May 2012 11:22:01 +0300
From:      Daniel Kalchev <daniel@digsys.bg>
To:        freebsd-fs@freebsd.org
Subject:   Re: Millions of small files: best filesystem / best options
Message-ID:  <4FC48729.5050302@digsys.bg>
In-Reply-To: <20120529175504.K1291@besplex.bde.org>
References:  <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> <20120529161802.N975@besplex.bde.org> <20120529175504.K1291@besplex.bde.org>

next in thread | previous in thread | raw e-mail | index | archive | help


On 29.05.12 11:00, Bruce Evans wrote:
> On Tue, 29 May 2012, Bruce Evans wrote:
>
>> On Mon, 28 May 2012, Doug Barton wrote:
>>> The good news is that it's a big improvement (I've done similar
>>> stuff in the past). You'll also want to tweak the -i (inode) value to
>>> insure that you have sufficient inodes for the number of files you plan
>>> to store. The default is not likely to be adequate for your needs.
>>
>> Big is relative.  4K-blocks with 200-byte files gives a wastage factor
>> of 20.  Metadata alone will be 256 bytes for the inode alone with ffs2.
>> Only 128 bytes with ffs1.  Only 32 bytes with msdosfs.
>
> Oops, only a wastage factor of 2.5 with the 512-byte fragments that are
> normally used with 4K-blocks by ffs.  512-byte blocks with ffs only
> give a small reduction in metadata size and better block allocation.
>

But how big the entire filesystem is going to be, anyway?

Say, 10 million 200 byte files is some 2GB of real data.
Let's say we have 4x waste and with UFS this will take some 8GB.
Let's even say with ZFS there will be 20x waste and it grows to 40GB. 
(with data validation, no need to wait eons for fsck etc).
Grow it to 100 million and it will eat say 400GB on ZFS.

These are trivial file system sizes today, unless the data needs to fit 
on a thumb drive or is for an embedded system.

Otherwise, the discussion is good reading :)

Daniel



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4FC48729.5050302>