Date: Mon, 06 Feb 2012 18:24:58 +0100 From: Michael Fuckner <michael@fuckner.net> To: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. Message-ID: <4F300CEA.5000901@fuckner.net> In-Reply-To: <CAOjFWZ44nP5MVPgvux=Y-x%2BT%2BBy-WWGVyuAegJYrv6mLmmaN-w@mail.gmail.com> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <CAOjFWZ44nP5MVPgvux=Y-x%2BT%2BBy-WWGVyuAegJYrv6mLmmaN-w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 02/06/2012 05:41 PM, Freddie Cash wrote: Hi all, > On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick > <freebsd@jdc.parodius.com> wrote: >> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>> I want to investigate if it is possible to create your own usable >>> HPC storage using zfs and some network filesystem like nfs. especially HPS sounds interesting to me- but for HPC you typicially need fast r/w-access for all nodes in the cluster. That's why Lustre uses several storages for concurring access over a fast link (typicially Infiniband) Another thing to think about is CPU: you probably need weeks for a rebuild of a single disk in a Petabyte Filesystem- I haven't tried this with ZFS yet, but I'm really interested if anyone already did this. The whole setup sounds a little bit like the system shown by aberdeen: http://www.aberdeeninc.com/abcatg/petabyte-storage.htm schematics at tomshardware: http://www.tomshardware.de/fotoreportage/137-Aberdeen-petarack-petabyte-sas.html The Problem with Aberdeen is they don't use Zil/ L2Arc. >>> Just a thought experiment.. >>> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) >>> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD >>> deives for cache. >>> Preferrably in mirror where applicable. >>> >>> Connected to this machine we will have about 410 3TB drives to give approx >>> 1PB of usable storage in a 8+2 raidz configuration. I don't know what the situation is for the rest of the world, but 3TB currently is still hard to buy in Europe/ Germany. >>> Connected to this will be a ~800 nodes big HPC cluster that will >>> access the storage in parallell what is your typical load pattern? >>> is this even possible or do we need to distribute the meta data load >>> over many servers? It is a good idea to have >>> If that is the case, >>> does it exist any software for FreeBSD that could accomplish this >>> distribution (pNFS dosent seem to be >>> anywhere close to usable in FreeBSD) or do I need to call NetApp or >>> Panasas right away? not that I know of > SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM > slots). It's an AMD board, but there should be variants that support > Intel CPUs. It's not uncommon to support 256 GB of RAM these days, > although 128 GB boards are much more common. Currently Intel CPUs have 3 Memory Channels. If you have 2 Sockets, 2 Dimms per Channel, 3 Channels- 12 Dimms with cheap 16GB Modules is 192GB. 32GB are also available today ;-) >> - How you plan on getting roughly 410 hard disks (or 422 assuming >> an additional 12 SSDs) hooked up to a single machine > > In a "head node" + "JBOD" setup? Where the head node has a mobo that > supports multiple PCIe x8 and PCIe x16 slots, and is stuffed full of > 16-24 port multi-lane SAS/SATA controllers with external ports that > are cabled up to external JBOD boxes. The SSDs would be connected to > the mobo SAS/SATA ports. > > Each JBOD box contains nothing but power, SAS/SATA backplane, and > harddrives. Possibly using SAS expanders. If you use Supermicro I would use X8DTH-iF, some LSI HBA (9200-8e, 2x Multilane external) and some JBOD-Chassis (like SUpermicro 847E16-RJBOD1) Regards, Michael!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F300CEA.5000901>