From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 16 21:22:36 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6568E16A4CE; Thu, 16 Sep 2004 21:22:36 +0000 (GMT) Received: from smtp-vbr12.xs4all.nl (smtp-vbr12.xs4all.nl [194.109.24.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id BA1F643D2F; Thu, 16 Sep 2004 21:22:35 +0000 (GMT) (envelope-from wb@freebie.xs4all.nl) Received: from freebie.xs4all.nl (freebie.xs4all.nl [213.84.32.253]) i8GLMYlI045548; Thu, 16 Sep 2004 23:22:34 +0200 (CEST) (envelope-from wb@freebie.xs4all.nl) Received: from freebie.xs4all.nl (localhost [127.0.0.1]) by freebie.xs4all.nl (8.12.11/8.12.9) with ESMTP id i8GLMXm3064657; Thu, 16 Sep 2004 23:22:33 +0200 (CEST) (envelope-from wb@freebie.xs4all.nl) Received: (from wb@localhost) by freebie.xs4all.nl (8.12.11/8.12.11/Submit) id i8GLMXl5064656; Thu, 16 Sep 2004 23:22:33 +0200 (CEST) (envelope-from wb) Date: Thu, 16 Sep 2004 23:22:33 +0200 From: Wilko Bulte To: Kris Kennaway Message-ID: <20040916212233.GA64634@freebie.xs4all.nl> References: <20040916211837.GE70401@hub.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040916211837.GE70401@hub.freebsd.org> User-Agent: Mutt/1.4.1i X-OS: FreeBSD 4.10-STABLE X-PGP: finger wilko@freebsd.org X-Virus-Scanned: by XS4ALL Virus Scanner cc: freebsd-hackers@freebsd.org cc: Sam Subject: Re: ZFS X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Sep 2004 21:22:36 -0000 On Thu, Sep 16, 2004 at 09:18:37PM +0000, Kris Kennaway wrote.. > On Thu, Sep 16, 2004 at 10:31:57AM -0500, Sam wrote: > > > >CERN's LHC is expected to produce 10-15 PB/year. e-science ("the grid") > > >is capable of producing whopping huge data sets, and people already are. > > >Many aspects of data custodianship are still open questions, but there's > > >little doubt that what's cutting-edge storage today will be in > > >filesystems between now and 10 years' time. Filesystem views on data > > >sets that are physically stored and replicated at disparate locations > > >around the planet are the kind of things that potentially need larger > > >than 64-bit quantities. > > > > > > > Let's suppose you generate an exabyte of storage per year. Filling a > > 64-bit filesystem would take you approximately 8 million years. > > > > I'm not saying we'll never get there, just that doing it now is nothing > > more than a "look at us, ain't we forward thinking" ploy. It's a > > _single filesystem_. If you want another 8192 ZB, just make another. > > The detectors in the particle accelerator at Fermilab produce raw data > at a rate of 100 TB/sec (yes, 100 terabytes per second). They have to > use a three-tiered system of hardware filters to throw away most of > this and try to pick out the events that might actually be > interesting, to get it down to a "slow" data rate of 100 MB/sec that > can actually be written out to storage. If the hardware and software 100MB/s is slow, I think this number is wrong. -- Wilko Bulte wilko@FreeBSD.org