From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 16 21:46:50 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 758) id 3661A16A4D8; Thu, 16 Sep 2004 21:46:50 +0000 (GMT) Date: Thu, 16 Sep 2004 21:46:50 +0000 From: Kris Kennaway To: Wilko Bulte Message-ID: <20040916214650.GA73372@hub.freebsd.org> References: <20040916211837.GE70401@hub.freebsd.org> <20040916212233.GA64634@freebie.xs4all.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040916212233.GA64634@freebie.xs4all.nl> User-Agent: Mutt/1.4.1i cc: freebsd-hackers@freebsd.org cc: Sam Subject: Re: ZFS X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Sep 2004 21:46:50 -0000 On Thu, Sep 16, 2004 at 11:22:33PM +0200, Wilko Bulte wrote: > On Thu, Sep 16, 2004 at 09:18:37PM +0000, Kris Kennaway wrote.. > > On Thu, Sep 16, 2004 at 10:31:57AM -0500, Sam wrote: > > > > > >CERN's LHC is expected to produce 10-15 PB/year. e-science ("the grid") > > > >is capable of producing whopping huge data sets, and people already are. > > > >Many aspects of data custodianship are still open questions, but there's > > > >little doubt that what's cutting-edge storage today will be in > > > >filesystems between now and 10 years' time. Filesystem views on data > > > >sets that are physically stored and replicated at disparate locations > > > >around the planet are the kind of things that potentially need larger > > > >than 64-bit quantities. > > > > > > > > > > Let's suppose you generate an exabyte of storage per year. Filling a > > > 64-bit filesystem would take you approximately 8 million years. > > > > > > I'm not saying we'll never get there, just that doing it now is nothing > > > more than a "look at us, ain't we forward thinking" ploy. It's a > > > _single filesystem_. If you want another 8192 ZB, just make another. > > > > The detectors in the particle accelerator at Fermilab produce raw data > > at a rate of 100 TB/sec (yes, 100 terabytes per second). They have to > > use a three-tiered system of hardware filters to throw away most of > > this and try to pick out the events that might actually be > > interesting, to get it down to a "slow" data rate of 100 MB/sec that > > can actually be written out to storage. If the hardware and software > > 100MB/s is slow, I think this number is wrong. I think they do heavier software processing in the third stage, so it might have been CPU-bound instead of storage-bound. The figures I quoted were from http://humanresources.web.cern.ch/humanresources/external/training/acad/sphicas.pdf which I now see was from 1998, so this might have improved somewhat. It's also for LHC, although I have the Fermilab stats from 2001 somewhere, which I remember being of comparable magnitudes. Kris -- In God we Trust -- all others must submit an X.509 certificate. -- Charles Forsythe