Date: Sun, 1 Jul 2007 20:13:53 +0200 (CEST) From: Wojciech Puchar <wojtek@wojtek.tensor.gdynia.pl> To: Nguyen Tam Chinh <unixvn@gmail.com> Cc: freebsd-stable@freebsd.org, FreeBSD-Questions <freebsd-questions@freebsd.org> Subject: Re: UFS2 optimization for many small files Message-ID: <20070701200639.N64116@wojtek.tensor.gdynia.pl> In-Reply-To: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com> References: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> > We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well. From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works. I know It's better to read the exactly as a block/cluster in windows. fragment is the smallest allocation block. "block" is a group of 8 fragments to make allocation faster and smarter. > full ufs2 specification, but hope that someone here can give a hint. > Please advice with optimizations or tricks. please DO NOT make single partition like that. try to divide it to 3-4 partitions. it will work on a single one but waiting for fsck will kill you ;) AFAIK fsck time grows nonlinearly with fs size to some extent.. options for newfs will be like that newfs -m <A> -i <B> -b 4096 -f 512 -U /dev/partition where A is space left. with mostly small files and huge partition don't worry to set it 1 or even 0. B - size of disk(bytes)/amount of inodes default is probably 2048, you may use 1024 or 4096 for your case - make rough estimate how much files will you have (you told between 4 and 0.5k, but what average?). making too much inodes=wasted space (128 bytes/inode), making too little=big problem :) another question - HOW do you plan to make backups of such data? with dump rsync tar etc. it's clearly "mission impossible". feel free to mail me i had such cases not 5E8 but over 1E8 files :)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070701200639.N64116>