Date: 26 Jun 2000 11:04:51 -0400 From: Chris Shenton <cshenton@uucom.com> To: "Daniel O'Connor" <doconnor@gsoft.com.au> Cc: "Nicole Harrington." <nicole@unixgirl.com>, Luigi Rizzo <luigi@info.iet.unipi.it>, hackers@FreeBSD.ORG Subject: Re: How many files can I put in one diretory? Message-ID: <lf8zvs5vqk.fsf@Samizdat.uucom.com> In-Reply-To: "Daniel O'Connor"'s message of "Fri, 23 Jun 2000 13:12:48 %2B0930 (CST)" References: <XFMail.000623131248.doconnor@gsoft.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23-Jun-00 Nicole Harrington. wrote:
>> > Yeah.. This is why databases where invented :) Hey I
>> agree... However even if the html was databased.. (working on that
>> now) the custom graphics cannot be. (yet)
On Fri, 23 Jun 2000 13:12:48 +0930 (CST), "Daniel O'Connor" <doconnor@gsoft.com.au> said:
Daniel> Hmm.. can't you do binary blobs in a DB and change the image
Daniel> URL's to be cgi requests?
I was considering this for a project I developed: web up/download of
lots of large files. I was using MySQL and some of the folks on that
list recommended not storing large files in the DB: even though the
disk consumption is the same, if it's in a DB you can't spread it
across partitions as space requirements grow.
So I store the file path in the DB and the actual file on the UNIX
filesystem. To reduce search time I use a two-level directory
hierarchy, each of which has 256 subdirectories. To distribute files
evenly, I store the file under a name which is the MD5 hash of the
filename, time, etc, etc. This gives me a 32-char name of [0-9a-f].
So if file foo.tar.gz hashes to name
cafebabedeadbeef0123456789abcdef
it is stored under
/filestore/ca/fe/cafebabedeadbeef0123456789abcdef
This gives me 256 * 256 = 65536 directories. My requirement was to
store at least 10 Million files, and this works out to about 150 files
per directory -- easy for UNIX to get to quickly. It's been working
very well for me.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?lf8zvs5vqk.fsf>