From owner-freebsd-stable@FreeBSD.ORG  Sun Jul  1 10:35:48 2007
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
X-Original-To: freebsd-stable@freebsd.org
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id C9D7716A400
	for <freebsd-stable@freebsd.org>; Sun,  1 Jul 2007 10:35:48 +0000 (UTC)
	(envelope-from junics-fbsdstable@atlantis.maniacs.se)
Received: from mammoth.unixsh.net (mammoth.unixsh.net [195.35.83.67])
	by mx1.freebsd.org (Postfix) with SMTP id 214B513C48A
	for <freebsd-stable@freebsd.org>; Sun,  1 Jul 2007 10:35:47 +0000 (UTC)
	(envelope-from junics-fbsdstable@atlantis.maniacs.se)
Received: (qmail 90463 invoked from network); 1 Jul 2007 10:09:05 -0000
Received: from localhost.maniacs.se (HELO ?192.168.0.34?) (127.0.0.1)
	by localhost.maniacs.se with SMTP; 1 Jul 2007 10:09:05 -0000
Message-ID: <46877D42.8010606@atlantis.maniacs.se>
Date: Sun, 01 Jul 2007 12:09:06 +0200
From: Thomas Herrlin <junics-fbsdstable@atlantis.maniacs.se>
User-Agent: Thunderbird 1.5.0.9 (Windows/20061207)
MIME-Version: 1.0
To: Nguyen Tam Chinh <unixvn@gmail.com>
References: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com>
In-Reply-To: <64b284310706270311j2a6af2f6i6766b483a4b66a5c@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Cc: freebsd-stable@freebsd.org
Subject: Re: UFS2 optimization for many small files
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 01 Jul 2007 10:35:48 -0000

Nguyen Tam Chinh wrote:
> Greetings,
>
> We're going to build a server with some 1Tb of over 500 million small
> files with size from 0,5k to 4k.  I'm wonder if the ufs2 can handle
> this kind of system well. From newfs(8) the min block size is 4k. This
> is not optimal in our case, a 1k or 0,5k block is more effective IMHO.
> I'd be happy if anyone can suggest what does fragment (block/8) in the
> ufs2 mean and how this parameter works. I know It's better to read the
> full ufs2 specification, but hope that someone here can give a hint.
> Please advice with optimizations or tricks.
> Thank you very much.
>
If all else fails; try "divide and conquer" by having one filesystem per 
subdirectory. Unless you plan on having all files in a single dir?! Also 
look at how the squid proxy stores its files using a hashed dir structure.
Another alternative is storing the data in a database if you don't need 
direct random RW file access..