Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 6 Feb 2012 18:43:59 +0100
From:      =?iso-8859-1?Q?Peter_Ankerst=E5l?= <peter@pean.org>
To:        Freddie Cash <fjwcash@gmail.com>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: HPC and zfs.
Message-ID:  <7321E3F2-35E2-43F4-9932-EC55F5AA9D0B@pean.org>
In-Reply-To: <CAOjFWZ44nP5MVPgvux=Y-x%2BT%2BBy-WWGVyuAegJYrv6mLmmaN-w@mail.gmail.com>
References:  <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <CAOjFWZ44nP5MVPgvux=Y-x%2BT%2BBy-WWGVyuAegJYrv6mLmmaN-w@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

--
Peter Ankerst=E5l
peter@pean.org
http://www.pean.org/

On 6 feb 2012, at 17:41, Freddie Cash wrote:

> On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick
> <freebsd@jdc.parodius.com> wrote:
>> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote:
>>> I want to investigate if it is possible to create your own usable
>>> HPC storage using zfs and some
>>> network filesystem like nfs.
>>>=20
>>> Just a thought experiment..
>>> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or =
more)
>>> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD
>>> deives for  cache.
>>> Preferrably in  mirror where applicable.
>>>=20
>>> Connected to this machine we will have about 410 3TB drives to give =
approx
>>> 1PB of usable storage in a 8+2 raidz configuration.
>>>=20
>>> Connected to this will be a ~800 nodes big HPC cluster that will
>>> access the storage in parallell
>>> is this even possible or do we need to distribute the meta data load
>>> over many servers? If that is the case,
>>> does it exist any software for FreeBSD that could  accomplish this
>>> distribution (pNFS  dosent seem to be
>>> anywhere close to usable in FreeBSD) or do I need to call NetApp or
>>> Panasas right away? It would be
>>> really nice if I could build my own storage solution.
>>>=20
>>> Other possible solutions to this problem is extremley welcome.
>>=20
>> For starters I'd love to know:
>>=20
>> - What single motherboard supports up to 192GB of RAM
>=20
> SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM
> slots).  It's an AMD board, but there should be variants that support
> Intel CPUs.  It's not uncommon to support 256 GB of RAM these days,
> although 128 GB boards are much more common.
Yeah, the one I was looking at was SuperMicro X8DTU-F, but yeah, the =
more
money RAM the better.
>=20
>> - How you plan on getting roughly 410 hard disks (or 422 assuming
>>  an additional 12 SSDs) hooked up to a single machine
>=20
> In a "head node" + "JBOD" setup?  Where the head node has a mobo that
> supports multiple PCIe x8 and PCIe x16 slots, and is stuffed full of
> 16-24 port multi-lane SAS/SATA controllers with external ports that
> are cabled up to external JBOD boxes.  The SSDs would be connected to
> the mobo SAS/SATA ports.
>=20
> Each JBOD box contains nothing but power, SAS/SATA backplane, and
> harddrives.  Possibly using SAS expanders.
>=20
> We're considering doing the same for our SAN/NAS setup for
> centralising storage for our VM hosts, although not quite to the same
> scale as the OP.  :)

Yep, NetApp has disk-shelves that can be configured JBOD that fits 60 =
drives
into 4U. :D

>=20
>> If you are considering investing the time and especially money (the =
cost
>> here is almost unfathomable, IMO) into this, I strongly recommend you
>> consider an actual hardware filer (e.g. NetApp).  Your performance =
and
>> reliability will be much greater, plus you will get overall better
>> support from NetApp in the case something goes wrong.  In the case =
you
>> run into problems with FreeBSD (and I can assure you in this kind of
>> setup you will) with this kind of extensive setup, you will be at the
>> mercy of developers' time/schedules with absolutely no guarantee that
>> your problem will be solved.  You definitely want a support contract.
>> Thus, go NetApp.
>=20
> For an HPC setup like the OP wants, where performance and uptime are
> critical, I agree. You don't want to be skimping on the hardware and
> software.
>=20
A big consideration for us is also the installation. If we go with =
something like
NetApp they can install the system and we don't need to put in the extra =
hours
(probably a lot) the get the thing running. But being a huge fan of BSD =
I wanted
to at least look up the possibility to build our own system.

> However, if you have the money for a NetApp setup like this ($
> 500,000+ US I'm guessing), then you also have the money to hire a
> FreeBSD developer(s) to work on the parts of the system that are
> critical to this (NFS, ZFS, CAM, drivers, scheduler, GEOM, etc).
> Then, you could go with a white-box, custom build and have the support
> in-house.
>=20
> --=20
> Freddie Cash
> fjwcash@gmail.com
>=20




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7321E3F2-35E2-43F4-9932-EC55F5AA9D0B>