Date: Thu, 22 Oct 1998 23:19:28 +1000 From: "Hallam Oaks" <mlnn4@oaks.com.au> To: "freebsd-chat@FreeBSD.ORG" <freebsd-hackers@FreeBSD.ORG> Subject: Multi-terabyte disk farm Message-ID: <199810221320.XAA07482@mail.aussie.org>
index | next in thread | raw e-mail
G'Day;
I've a few things I'd like, If I may, to get opinions on.
My current client (I'm a contractor) hired me to provide them with some
assistance on a project that involves the digitising and non-real-time
satellite transmission of television commercials. As part of this project,
they want to store about 30,000 ads (they call them 'dubs') online for
retrieval in a digital form.
We estimate that we will need approximately 3-4 terabytes of storage to
achieve this aim.
Prior to my arrival two weeks ago, they had been planning to use a
StorageTek Timberwolf library. This box has two DLT drives, a robot arm,
and about 8 terabytes worth of tape slots.
The requirement is for 24/7 availability, but speed is not an issue.
Provided a dub can be retrieved onto a cache disk within an hour or so of
it being requested from archive they'd be happy.
Now, I know that this is a top-quality library (or so I have gathered from
scanning the net) but when I heard how much the suppliers wanted for it I
had to stop and think if there was a better way.
I can't give exact figures but the box itself won't leave much change out
of about US$100k, and that's with no software. Add the software on a Sun-
based server with 100gb of cache and it goes to somewhere in the vicinity
of US$375k (!). Needless to say my clients don't want to buy the software,
but to write it internally.
With current hard drive prices, I estimate we can put together a one
terabyte disk farm for about US$60k (cost of media only), spread across
several machines using hot swappable drive bays and dual SCSI buses per
machine. We don't intend to use RAID unless there's strong advantages to
it (I don't know a whole lot about it). We don't need striping or
replication.
One advantage of doing it via a distributed farm (I theorise) is that if
one drive fries or one machine self-destructs, at least the rest of the
system will still be working. A fried drive can be restored from one of
the 25gb AIT backup tapes made of all the dubs.
Secondly (and this is a major call I'm making), it won't work out cheaper
for our estimated need of three terabytes unless the cost of HDD's keep on
falling. We won't need full capacity until about two years has passed,
meaning that we can start out with only a few hundred gig and scale it as
we go, taking advantage of (hopefully) falling disk prices and increasing
drive sizes.
But, I'm stepping on one toe because one of the permanents (a hardware
person) strongly favours using the TW. So I have to make darn sure I'm not
blowing smoke out of my nether regions before I stick my neck out even
further.
My desire is to push for the farm because I believe it's better. (Going
with the TW would actually be more profitable for me since one of the main
reasons they hired me was to write the software to drive the flipping
thing:).
A farm ought to work out cheaper, be easier to work with (for the rest of
the system software on a API level, rather than a hardware level), and
could possibly be more reliable overall (being distributed). Additionally,
we don't have to spend I-don't-know-how-long writing software to control
it like we would with the library.
OTOH the down sides are power, heat, extra administration costs, and
probably a bunch of stuff I don't know yet (which is why I'm writing this
;).
Needless to say I'm going to put my preferred solution as FreeBSD-based.
Some of the criteria that I can't yet answer and would like feedback on
are these -
o is FreeBSD able to be made to recognise a new SCSI drive that wasn't
present on boot ? i.e. a new drive is plugged into the hot bays. can
it be recognised, formatted, and mounted by manual intervention ?
o ditto if a drive fries. can it be taken out without the kernel getting
too upset ?
o is it feasable to automatically umount and spin down drives that
haven't been accessed for a day or so ? typically, the older data
(> 6 months) will be rarely, if ever, accessed before its two-year
span expires and it's erased.
o would the boot time of a system be dramatically prolonged by it having
500 or so gigabytes of SCSI drives hanging off its backside ? (I'm
referring to a normal boot, with the drives having been properly
unmounted. I don't even want to THINK about waiting on an fsck of 500
gigs of unclean disk ;). OTOH the size of the files is quite large, so
it'd be feasable to use huge nodes.
I've no particular objection to a reboot if need be to add/remove a
drive, but if it took more than, say, 10 minutes, it'd be an issue I'd
have to tackle with management.
o we're thinking of using Seagate Elite 47gb drives. These are 5400 RPM
units (speed isn't an issue to us). Does anyone have any opinions
about these (good/bad/indifferent) or of previous members of that
drive family ?
o does anyone have an opinion as to whether it's safe to assume that
drive prices will continue to fall as they have done over the past
two years ?
I guess my main interest, despite the above questions, is really to hear
if others think this is a realistic goal to be aiming for, and if others-
who-have-gone-before-me have struck major problems doing the same (or a
similar) thing.
regards,
-- Chris
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199810221320.XAA07482>