From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 16:11:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA5F1106566B for ; Mon, 6 Feb 2012 16:11:31 +0000 (UTC) (envelope-from peter@pean.org) Received: from velox.its.uu.se (velox.its.uu.se [130.238.7.74]) by mx1.freebsd.org (Postfix) with ESMTP id 8AE8C8FC21 for ; Mon, 6 Feb 2012 16:11:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at uu.se Received: from nyx.uppmax.uu.se (nyx.uppmax.uu.se [130.238.137.40]) by velox.its.uu.se (Postfix) with ESMTP id 2A17735434 for ; Mon, 6 Feb 2012 16:52:11 +0100 (CET) Message-ID: <4F2FF72B.6000509@pean.org> Date: Mon, 06 Feb 2012 16:52:11 +0100 From: =?ISO-8859-1?Q?Peter_Ankerst=E5l?= User-Agent: Thunderbird 2.0.0.24 (X11/20111108) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 16:11:31 -0000 Hi, I want to investigate if it is possible to create your own usable HPC storage using zfs and some network filesystem like nfs. Just a thought experiment.. A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD deives for cache. Preferrably in mirror where applicable. Connected to this machine we will have about 410 3TB drives to give approx 1PB of usable storage in a 8+2 raidz configuration. Connected to this will be a ~800 nodes big HPC cluster that will access the storage in parallell is this even possible or do we need to distribute the meta data load over many servers? If that is the case, does it exist any software for FreeBSD that could accomplish this distribution (pNFS dosent seem to be anywhere close to usable in FreeBSD) or do I need to call NetApp or Panasas right away? It would be really nice if I could build my own storage solution. Other possible solutions to this problem is extremley welcome. Best Regards Peter Ankerstål