From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 16:22:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1623106568A for ; Mon, 6 Feb 2012 16:22:07 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.emeryville.ca.mail.comcast.net (qmta09.emeryville.ca.mail.comcast.net [76.96.30.96]) by mx1.freebsd.org (Postfix) with ESMTP id CA0618FC08 for ; Mon, 6 Feb 2012 16:22:07 +0000 (UTC) Received: from omta15.emeryville.ca.mail.comcast.net ([76.96.30.71]) by qmta09.emeryville.ca.mail.comcast.net with comcast id Wf9d1i0061Y3wxoA9gN7Ym; Mon, 06 Feb 2012 16:22:07 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta15.emeryville.ca.mail.comcast.net with comcast id WgN61i00E1t3BNj8bgN6mL; Mon, 06 Feb 2012 16:22:07 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 415CD102C19; Mon, 6 Feb 2012 08:22:06 -0800 (PST) Date: Mon, 6 Feb 2012 08:22:06 -0800 From: Jeremy Chadwick To: Peter Ankerst?l Message-ID: <20120206162206.GA541@icarus.home.lan> References: <4F2FF72B.6000509@pean.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F2FF72B.6000509@pean.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:22:08 -0000 On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: > I want to investigate if it is possible to create your own usable > HPC storage using zfs and some > network filesystem like nfs. > > Just a thought experiment.. > A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) > I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD > deives for cache. > Preferrably in mirror where applicable. > > Connected to this machine we will have about 410 3TB drives to give approx > 1PB of usable storage in a 8+2 raidz configuration. > > Connected to this will be a ~800 nodes big HPC cluster that will > access the storage in parallell > is this even possible or do we need to distribute the meta data load > over many servers? If that is the case, > does it exist any software for FreeBSD that could accomplish this > distribution (pNFS dosent seem to be > anywhere close to usable in FreeBSD) or do I need to call NetApp or > Panasas right away? It would be > really nice if I could build my own storage solution. > > Other possible solutions to this problem is extremley welcome. For starters I'd love to know: - What single motherboard supports up to 192GB of RAM - How you plan on getting roughly 410 hard disks (or 422 assuming an additional 12 SSDs) hooked up to a single machine If you are considering investing the time and especially money (the cost here is almost unfathomable, IMO) into this, I strongly recommend you consider an actual hardware filer (e.g. NetApp). Your performance and reliability will be much greater, plus you will get overall better support from NetApp in the case something goes wrong. In the case you run into problems with FreeBSD (and I can assure you in this kind of setup you will) with this kind of extensive setup, you will be at the mercy of developers' time/schedules with absolutely no guarantee that your problem will be solved. You definitely want a support contract. Thus, go NetApp. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |