From owner-freebsd-current@FreeBSD.ORG Tue Jun 15 23:10:38 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7A41116A4CE for ; Tue, 15 Jun 2004 23:10:38 +0000 (GMT) Received: from sccrmhc12.comcast.net (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2EAB943D46 for ; Tue, 15 Jun 2004 23:10:38 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc12) with ESMTP id <20040615231011012002fv9je>; Tue, 15 Jun 2004 23:10:12 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA50624; Tue, 15 Jun 2004 16:10:10 -0700 (PDT) Date: Tue, 15 Jun 2004 16:10:08 -0700 (PDT) From: Julian Elischer To: Cyrille Lefevre In-Reply-To: <005501c45302$08d2a8c0$7890a8c0@dyndns.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: Eugene Subject: Re: Storing a lot of little files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 23:10:38 -0000 On Tue, 15 Jun 2004, Cyrille Lefevre wrote: > "Eugene" wrote: > > Hello freebsd-current, > > > > I need to store a lot (hundreds of millions) of very little files (from 8 > bytes > > to 50K) in my filesystem. > > some times ago, there where something called "inode fs" (aka IFS). > unfortunatelly, this was killed from -current (5.x) two years ago. > > more details here (google "freebsd +ifs +inode"): > > http://www.squid-cache.org/mail-archive/squid-dev/200101/0432.html > http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/ufs/ifs/Attic/README > http://lists.freebsd.org/pipermail/freebsd-fs/2003-June/000129.html It's "supposed" to be coming back some time.. there are also "microfiles" (< 64 bytes) which store the data in the inode.. You can simulate this by using a symlink and using readlink(2) to instead of read(2) when ever you encounter a symlink. (if you have source). There were patches for microfiles floating around somewhere but they were "pre-softupdates". I still think it would be a good idea. /etc/malloc.conf uses this approach. > > > What's the best way to optimize it? Which newfs options can you > > recommend me? > > so, the best way would be to have multi-level directories to reduce > the number of entries in one directory whatever the underlying file > system is (except, maybe, database-like filesystems). > > something like : > > /a/b/c/cfile > /a/b/d/dfile > /a/c/e/efile > etc. > > using google "million +files +directory +fs" : > > http://aa11.cjb.net/sun_managers/2000/01/msg00303.html > > Cyrille Lefevre. > -- > home: mailto:cyrille.lefevre@laposte.net > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >