From owner-freebsd-fs@FreeBSD.ORG  Wed Feb  8 19:28:20 2012
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 12E11106564A
	for <freebsd-fs@freebsd.org>; Wed,  8 Feb 2012 19:28:20 +0000 (UTC)
	(envelope-from ronald-freebsd8@klop.yi.org)
Received: from smtp-out0.tiscali.nl (smtp-out0.tiscali.nl [195.241.79.175])
	by mx1.freebsd.org (Postfix) with ESMTP id C779E8FC14
	for <freebsd-fs@freebsd.org>; Wed,  8 Feb 2012 19:28:19 +0000 (UTC)
Received: from [212.182.167.131] (helo=sjakie.klop.ws)
	by smtp-out0.tiscali.nl with esmtp (Exim)
	(envelope-from <ronald-freebsd8@klop.yi.org>)
	id 1RvCqJ-000805-44; Wed, 08 Feb 2012 20:06:11 +0100
Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1])
	by sjakie.klop.ws (Postfix) with ESMTP id AF6BA1006C;
	Wed,  8 Feb 2012 20:05:53 +0100 (CET)
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
To: freebsd-fs@freebsd.org, =?utf-8?Q?Peter_Ankerst=C3=A5l?= <peter@pean.org>
References: <4F2FF72B.6000509@pean.org>
Date: Wed, 08 Feb 2012 20:05:53 +0100
MIME-Version: 1.0
From: "Ronald Klop" <ronald-freebsd8@klop.yi.org>
Message-ID: <op.v9djb3iw8527sy@212-182-167-131.ip.telfort.nl>
In-Reply-To: <4F2FF72B.6000509@pean.org>
User-Agent: Opera Mail/11.61 (FreeBSD)
Content-Transfer-Encoding: quoted-printable
Cc: 
Subject: Re: HPC and zfs.
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 08 Feb 2012 19:28:20 -0000

On Mon, 06 Feb 2012 16:52:11 +0100, Peter Ankerst=C3=A5l <peter@pean.org>=
 wrote:

> Hi,
>
> I want to investigate if it is possible to create your own usable HPC =20
> storage using zfs and some
> network filesystem like nfs.
>
> Just a thought experiment..
> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more)
> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD =20
> deives for  cache.
> Preferrably in  mirror where applicable.
>
> Connected to this machine we will have about 410 3TB drives to give =20
> approx
> 1PB of usable storage in a 8+2 raidz configuration.
>
> Connected to this will be a ~800 nodes big HPC cluster that will access=
 =20
> the storage in parallell
> is this even possible or do we need to distribute the meta data load =20
> over many servers? If that is the case,
> does it exist any software for FreeBSD that could  accomplish this =20
> distribution (pNFS  dosent seem to be
> anywhere close to usable in FreeBSD) or do I need to call NetApp or =20
> Panasas right away? It would be
> really nice if I could build my own storage solution.
>
> Other possible solutions to this problem is extremley welcome.
>
> Best Regards
> Peter Ankerst=C3=A5l

You might make a call to Backblaze.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-c=
heap-cloud-storage/

Ronald.