From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 16 21:18:37 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 758) id 5E8E716A4CF; Thu, 16 Sep 2004 21:18:37 +0000 (GMT) Date: Thu, 16 Sep 2004 21:18:37 +0000 From: Kris Kennaway To: Sam Message-ID: <20040916211837.GE70401@hub.freebsd.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.1i cc: freebsd-hackers@freebsd.org Subject: Re: ZFS X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Sep 2004 21:18:37 -0000 On Thu, Sep 16, 2004 at 10:31:57AM -0500, Sam wrote: > >CERN's LHC is expected to produce 10-15 PB/year. e-science ("the grid") > >is capable of producing whopping huge data sets, and people already are. > >Many aspects of data custodianship are still open questions, but there's > >little doubt that what's cutting-edge storage today will be in > >filesystems between now and 10 years' time. Filesystem views on data > >sets that are physically stored and replicated at disparate locations > >around the planet are the kind of things that potentially need larger > >than 64-bit quantities. > > > > Let's suppose you generate an exabyte of storage per year. Filling a > 64-bit filesystem would take you approximately 8 million years. > > I'm not saying we'll never get there, just that doing it now is nothing > more than a "look at us, ain't we forward thinking" ploy. It's a > _single filesystem_. If you want another 8192 ZB, just make another. The detectors in the particle accelerator at Fermilab produce raw data at a rate of 100 TB/sec (yes, 100 terabytes per second). They have to use a three-tiered system of hardware filters to throw away most of this and try to pick out the events that might actually be interesting, to get it down to a "slow" data rate of 100 MB/sec that can actually be written out to storage. If the hardware and software was up to it I'm sure they'd want to keep much more of the data than this. Now, over a year of runtime, the raw data amounts to (according to Google Calculator): (100 (terabytes / sec)) * 1 year = 3.4697207 10^21 bytes or just over 2^71 bytes in a year. Kris -- In God we Trust -- all others must submit an X.509 certificate. -- Charles Forsythe