From owner-freebsd-fs@FreeBSD.ORG Tue May 29 08:22:06 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D8731065673 for ; Tue, 29 May 2012 08:22:06 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 5438D8FC0A for ; Tue, 29 May 2012 08:22:04 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id q4T8M1PZ015018 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 29 May 2012 11:22:01 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <4FC48729.5050302@digsys.bg> Date: Tue, 29 May 2012 11:22:01 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.4) Gecko/20120528 Thunderbird/10.0.4 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> <4FC457F7.9000800@FreeBSD.org> <20120529161802.N975@besplex.bde.org> <20120529175504.K1291@besplex.bde.org> In-Reply-To: <20120529175504.K1291@besplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 08:22:06 -0000 On 29.05.12 11:00, Bruce Evans wrote: > On Tue, 29 May 2012, Bruce Evans wrote: > >> On Mon, 28 May 2012, Doug Barton wrote: >>> The good news is that it's a big improvement (I've done similar >>> stuff in the past). You'll also want to tweak the -i (inode) value to >>> insure that you have sufficient inodes for the number of files you plan >>> to store. The default is not likely to be adequate for your needs. >> >> Big is relative. 4K-blocks with 200-byte files gives a wastage factor >> of 20. Metadata alone will be 256 bytes for the inode alone with ffs2. >> Only 128 bytes with ffs1. Only 32 bytes with msdosfs. > > Oops, only a wastage factor of 2.5 with the 512-byte fragments that are > normally used with 4K-blocks by ffs. 512-byte blocks with ffs only > give a small reduction in metadata size and better block allocation. > But how big the entire filesystem is going to be, anyway? Say, 10 million 200 byte files is some 2GB of real data. Let's say we have 4x waste and with UFS this will take some 8GB. Let's even say with ZFS there will be 20x waste and it grows to 40GB. (with data validation, no need to wait eons for fsck etc). Grow it to 100 million and it will eat say 400GB on ZFS. These are trivial file system sizes today, unless the data needs to fit on a thumb drive or is for an embedded system. Otherwise, the discussion is good reading :) Daniel