From owner-freebsd-current@FreeBSD.ORG Tue Jun 15 14:30:46 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6160F16A4CE for ; Tue, 15 Jun 2004 14:30:46 +0000 (GMT) Received: from out012.verizon.net (out012pub.verizon.net [206.46.170.137]) by mx1.FreeBSD.org (Postfix) with ESMTP id E89BA43D46 for ; Tue, 15 Jun 2004 14:30:45 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from [192.168.1.3] ([68.161.84.3]) by out012.verizon.net (InterMail vM.5.01.06.06 201-253-122-130-106-20030910) with ESMTP id <20040615143023.BHXO2198.out012.verizon.net@[192.168.1.3]>; Tue, 15 Jun 2004 09:30:23 -0500 Message-ID: <40CF07F8.8020808@mac.com> Date: Tue, 15 Jun 2004 10:30:16 -0400 From: Chuck Swiger Organization: The Courts of Chaos User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040608 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Eugene References: <40CEFAA8.00B9A6.08476@e-post02.e-se.ru> In-Reply-To: <40CEFAA8.00B9A6.08476@e-post02.e-se.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Authentication-Info: Submitted using SMTP AUTH at out012.verizon.net from [68.161.84.3] at Tue, 15 Jun 2004 09:30:22 -0500 cc: freebsd-current@freebsd.org Subject: Re: Storing a lot of little files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 14:30:46 -0000 Eugene wrote: > I need to store a lot (hundreds of millions) of very little files (from 8 bytes > to 50K) in my filesystem. What's the best way to optimize it? Which newfs options can you > recommend me? Eric already provided an answer to this. The type of problem you are considering is similiar to the way a Usenet news server using tradspool works, or the way a large Squid proxy server would want to layout it's cache. The techniques and advice used by INN and Squid for configuring filesystems and laying out datafiles using multi-layer hierarchy with perhaps a few hundred top-level directories, each containing a few hundred second-level directories, etc are topics worth your consideration. > Can UFS2 optiomize storage of little files? For example, put some > files in one cluster - so don't let the 8 byte file to occupy a > whole cluster of about 16K? Good question. UFS breaks up 8K "clusters" into 1K fragments, so the amount of wasted space due to instrinsic fragmentation is significantly less. That being said, 8-byte files are too small to be kept individually with much efficiency in any block-sized filesystem available to you, given the common 512-byte sector size of devices tends to constrain the minimum "cluster" size one can set. There are some useful ideas about how to deal with that from the IronDoc project (Dave McCuster? [sp?] from Apple), but your requirements are difficult to meet without taking some complex issues into consideration. > Is it possible to allocate inodes dynamically (like in ReiserFS?), > bacause I'm not sure I can predict accuratly number of little files, > and inodes not to finish before disk get full. No. Allocate many more inodes that you need if you can't guess accurately, or else be prepared to backup, newfs using different parameters, and restore your data when you run out. > Does FreeBSD support file systems except of UFS? FreeBSD supports FAT, ISO-9660, NTFS, ext2, probably others, although the degree of support decreases in order of my listing. > What about ReiserFS, XFS, JFS? Nope. Partially licensing, partially lack of developer resources. -- -Chuck