From owner-freebsd-questions@FreeBSD.ORG Sun Jul 1 18:14:08 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 48FBD16A400 for ; Sun, 1 Jul 2007 18:14:08 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (chylonia.3miasto.net [83.12.228.78]) by mx1.freebsd.org (Postfix) with ESMTP id B2C3413C45A for ; Sun, 1 Jul 2007 18:14:04 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (localhost [IPv6:::1]) by wojtek.tensor.gdynia.pl (8.13.8/8.13.8) with ESMTP id l61IDssg064189; Sun, 1 Jul 2007 20:13:54 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.13.8/8.13.8/Submit) with ESMTP id l61IDr6V064186; Sun, 1 Jul 2007 20:13:53 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Sun, 1 Jul 2007 20:13:53 +0200 (CEST) From: Wojciech Puchar To: Nguyen Tam Chinh In-Reply-To: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com> Message-ID: <20070701200639.N64116@wojtek.tensor.gdynia.pl> References: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-stable@freebsd.org, FreeBSD-Questions Subject: Re: UFS2 optimization for many small files X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Jul 2007 18:14:08 -0000 > > We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well. From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works. I know It's better to read the exactly as a block/cluster in windows. fragment is the smallest allocation block. "block" is a group of 8 fragments to make allocation faster and smarter. > full ufs2 specification, but hope that someone here can give a hint. > Please advice with optimizations or tricks. please DO NOT make single partition like that. try to divide it to 3-4 partitions. it will work on a single one but waiting for fsck will kill you ;) AFAIK fsck time grows nonlinearly with fs size to some extent.. options for newfs will be like that newfs -m -i -b 4096 -f 512 -U /dev/partition where A is space left. with mostly small files and huge partition don't worry to set it 1 or even 0. B - size of disk(bytes)/amount of inodes default is probably 2048, you may use 1024 or 4096 for your case - make rough estimate how much files will you have (you told between 4 and 0.5k, but what average?). making too much inodes=wasted space (128 bytes/inode), making too little=big problem :) another question - HOW do you plan to make backups of such data? with dump rsync tar etc. it's clearly "mission impossible". feel free to mail me i had such cases not 5E8 but over 1E8 files :)