From owner-freebsd-questions@FreeBSD.ORG Wed Jun 27 12:33:45 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1BC0D16A46D for ; Wed, 27 Jun 2007 12:33:45 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from mail.potentialtech.com (internet.potentialtech.com [66.167.251.6]) by mx1.freebsd.org (Postfix) with ESMTP id E381D13C465 for ; Wed, 27 Jun 2007 12:33:44 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from vanquish.pgh.priv.collaborativefusion.com (pr40.pitbpa0.pub.collaborativefusion.com [206.210.89.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.potentialtech.com (Postfix) with ESMTP id D0750EBC78; Wed, 27 Jun 2007 08:33:43 -0400 (EDT) Date: Wed, 27 Jun 2007 08:33:42 -0400 From: Bill Moran To: "Nguyen Tam Chinh" Message-Id: <20070627083342.f965009b.wmoran@potentialtech.com> In-Reply-To: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com> References: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com> X-Mailer: Sylpheed 2.3.1 (GTK+ 2.10.11; i386-portbld-freebsd6.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: FreeBSD-Questions Subject: Re: UFS2 optimization for many small files X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2007 12:33:45 -0000 In response to "Nguyen Tam Chinh" : > > We're going to build a server with some 1Tb of over 500 million small > files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle > this kind of system well. From newfs(8) the min block size is 4k. This > is not optimal in our case, a 1k or 0,5k block is more effective IMHO. > I'd be happy if anyone can suggest what does fragment (block/8) in the > ufs2 mean and how this parameter works. I know It's better to read the > full ufs2 specification, but hope that someone here can give a hint. > Please advice with optimizations or tricks. > Thank you very much. Read the newfs man page. Based on your assessment of your files, I'd go with a block size of 4K and a frag size of 500 bytes. Blocks are broken in to frags when a file doesn't fill an entire block. Make sure to set -i to about 250 or so. An inode is needed for each file or directory on the filesystem, so you're liable to run out of inodes with the default values. Make sure your files are organized in a directory hierarchy. No filesytem that I know of performs well with huge numbers of files in a single directory. Please don't cross-post. I see no reason to copy stable@ with this message. -- Bill Moran http://www.potentialtech.com