From owner-freebsd-hackers Mon Jun 26 8: 5:14 2000 Delivered-To: freebsd-hackers@freebsd.org Received: from Samizdat.uucom.com (samizdat.uucom.com [198.202.217.54]) by hub.freebsd.org (Postfix) with ESMTP id 1495A37B889 for ; Mon, 26 Jun 2000 08:05:11 -0700 (PDT) (envelope-from cshenton@uucom.com) Received: (from cshenton@localhost) by Samizdat.uucom.com (8.9.3/8.9.3) id LAA19865; Mon, 26 Jun 2000 11:04:51 -0400 (EDT) To: "Daniel O'Connor" Cc: "Nicole Harrington." , Luigi Rizzo , hackers@FreeBSD.ORG Subject: Re: How many files can I put in one diretory? References: From: Chris Shenton Date: 26 Jun 2000 11:04:51 -0400 In-Reply-To: "Daniel O'Connor"'s message of "Fri, 23 Jun 2000 13:12:48 +0930 (CST)" Message-ID: Lines: 37 User-Agent: Gnus/5.0807 (Gnus v5.8.7) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On 23-Jun-00 Nicole Harrington. wrote: >> > Yeah.. This is why databases where invented :) Hey I >> agree... However even if the html was databased.. (working on that >> now) the custom graphics cannot be. (yet) On Fri, 23 Jun 2000 13:12:48 +0930 (CST), "Daniel O'Connor" said: Daniel> Hmm.. can't you do binary blobs in a DB and change the image Daniel> URL's to be cgi requests? I was considering this for a project I developed: web up/download of lots of large files. I was using MySQL and some of the folks on that list recommended not storing large files in the DB: even though the disk consumption is the same, if it's in a DB you can't spread it across partitions as space requirements grow. So I store the file path in the DB and the actual file on the UNIX filesystem. To reduce search time I use a two-level directory hierarchy, each of which has 256 subdirectories. To distribute files evenly, I store the file under a name which is the MD5 hash of the filename, time, etc, etc. This gives me a 32-char name of [0-9a-f]. So if file foo.tar.gz hashes to name cafebabedeadbeef0123456789abcdef it is stored under /filestore/ca/fe/cafebabedeadbeef0123456789abcdef This gives me 256 * 256 = 65536 directories. My requirement was to store at least 10 Million files, and this works out to about 150 files per directory -- easy for UNIX to get to quickly. It's been working very well for me. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message