From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 17:58:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA3BA106566C for ; Mon, 6 Feb 2012 17:58:39 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 77FAD8FC14 for ; Mon, 6 Feb 2012 17:58:39 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id q16HPLnD040930 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 6 Feb 2012 09:25:24 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4F300D52.1050502@freebsd.org> Date: Mon, 06 Feb 2012 09:26:42 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.26) Gecko/20120129 Thunderbird/3.1.18 MIME-Version: 1.0 To: Freddie Cash References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 17:58:39 -0000 On 2/6/12 8:41 AM, Freddie Cash wrote: > On Mon, Feb 6, 2012 at 8:22 AM, Jeremy Chadwick > wrote: >> On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote: >>> I want to investigate if it is possible to create your own usable >>> HPC storage using zfs and some >>> network filesystem like nfs. >>> >>> Just a thought experiment.. >>> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) >>> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD >>> deives for cache. >>> Preferrably in mirror where applicable. >>> >>> Connected to this machine we will have about 410 3TB drives to give approx >>> 1PB of usable storage in a 8+2 raidz configuration. >>> >>> Connected to this will be a ~800 nodes big HPC cluster that will >>> access the storage in parallell >>> is this even possible or do we need to distribute the meta data load >>> over many servers? If that is the case, >>> does it exist any software for FreeBSD that could accomplish this >>> distribution (pNFS dosent seem to be >>> anywhere close to usable in FreeBSD) or do I need to call NetApp or >>> Panasas right away? It would be >>> really nice if I could build my own storage solution. >>> >>> Other possible solutions to this problem is extremley welcome. >> For starters I'd love to know: >> >> - What single motherboard supports up to 192GB of RAM > SuperMicro H8DGi-F supports 256 GB of RAM using 16 GB modules (16 RAM > slots). It's an AMD board, but there should be variants that support > Intel CPUs. It's not uncommon to support 256 GB of RAM these days, > although 128 GB boards are much more common. > common wisdom for ZFS is 1GB of RAM per TB of storage.. so 256GB might not be enough. people who have actually tried ZFS more than me may want to comment more on this..