Date: Thu, 16 Sep 2004 21:18:37 +0000 From: Kris Kennaway <kris@FreeBSD.org> To: Sam <sah@softcardsystems.com> Cc: freebsd-hackers@freebsd.org Subject: Re: ZFS Message-ID: <20040916211837.GE70401@hub.freebsd.org> In-Reply-To: <Pine.LNX.4.60.0409161031280.28550@athena> References: <Pine.LNX.4.60.0409161031280.28550@athena>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Sep 16, 2004 at 10:31:57AM -0500, Sam wrote:
> >CERN's LHC is expected to produce 10-15 PB/year. e-science ("the grid")
> >is capable of producing whopping huge data sets, and people already are.
> >Many aspects of data custodianship are still open questions, but there's
> >little doubt that what's cutting-edge storage today will be in
> >filesystems between now and 10 years' time. Filesystem views on data
> >sets that are physically stored and replicated at disparate locations
> >around the planet are the kind of things that potentially need larger
> >than 64-bit quantities.
> >
>
> Let's suppose you generate an exabyte of storage per year. Filling a
> 64-bit filesystem would take you approximately 8 million years.
>
> I'm not saying we'll never get there, just that doing it now is nothing
> more than a "look at us, ain't we forward thinking" ploy. It's a
> _single filesystem_. If you want another 8192 ZB, just make another.
The detectors in the particle accelerator at Fermilab produce raw data
at a rate of 100 TB/sec (yes, 100 terabytes per second). They have to
use a three-tiered system of hardware filters to throw away most of
this and try to pick out the events that might actually be
interesting, to get it down to a "slow" data rate of 100 MB/sec that
can actually be written out to storage. If the hardware and software
was up to it I'm sure they'd want to keep much more of the data than
this.
Now, over a year of runtime, the raw data amounts to (according to
Google Calculator):
(100 (terabytes / sec)) * 1 year = 3.4697207 10^21 bytes
or just over 2^71 bytes in a year.
Kris
--
In God we Trust -- all others must submit an X.509 certificate.
-- Charles Forsythe <forsythe@alum.mit.edu>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040916211837.GE70401>