Date: Mon, 06 Feb 2012 09:26:42 -0800 From: Julian Elischer <julian@freebsd.org> To: Freddie Cash <fjwcash@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. Message-ID: <4F300D52.1050502@freebsd.org> In-Reply-To: <CAOjFWZ44nP5MVPgvux=Y-x%2BT%2BBy-WWGVyuAegJYrv6mLmmaN-w@mail.gmail.com> References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> <CAOjFWZ44nP5MVPgvux=Y-x%2BT%2BBy-WWGVyuAegJYrv6mLmmaN-w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2/6/12 8:41 AM, Freddie Cash wrote: > On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick > <freebsd@jdc.parodius.com> wrote: >> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>> I want to investigate if it is possible to create your own usable >>> HPC storage using zfs and some >>> network filesystem like nfs. >>> >>> Just a thought experiment.. >>> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) >>> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD >>> deives for cache. >>> Preferrably in mirror where applicable. >>> >>> Connected to this machine we will have about 410 3TB drives to give approx >>> 1PB of usable storage in a 8+2 raidz configuration. >>> >>> Connected to this will be a ~800 nodes big HPC cluster that will >>> access the storage in parallell >>> is this even possible or do we need to distribute the meta data load >>> over many servers? If that is the case, >>> does it exist any software for FreeBSD that could accomplish this >>> distribution (pNFS dosent seem to be >>> anywhere close to usable in FreeBSD) or do I need to call NetApp or >>> Panasas right away? It would be >>> really nice if I could build my own storage solution. >>> >>> Other possible solutions to this problem is extremley welcome. >> For starters I'd love to know: >> >> - What single motherboard supports up to 192GB of RAM > SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM > slots). It's an AMD board, but there should be variants that support > Intel CPUs. It's not uncommon to support 256 GB of RAM these days, > although 128 GB boards are much more common. > common wisdom for ZFS is 1GB of RAM per TB of storage.. so 256GB might not be enough. people who have actually tried ZFS more than me may want to comment more on this..
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F300D52.1050502>