From owner-freebsd-fs@FreeBSD.ORG Mon Feb 6 20:08:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2B891065670 for ; Mon, 6 Feb 2012 20:08:07 +0000 (UTC) (envelope-from erik@tefre.com) Received: from mta1-filtered.netlife.no (mail.netlife.no [62.92.26.226]) by mx1.freebsd.org (Postfix) with ESMTP id 681E78FC21 for ; Mon, 6 Feb 2012 20:08:07 +0000 (UTC) Received: from amavis2.netlife.no (amavishost [10.115.1.12]) by mta1-filtered.netlife.no (Postfix) with ESMTP id 11F3C3307C0 for ; Mon, 6 Feb 2012 20:51:42 +0100 (CET) X-Virus-Scanned: amavisd-new at netlife.no Received: from mta1-submission.netlife.no ([62.92.26.226]) by amavis2.netlife.no (amavis2.netlife.no [10.115.1.12]) (amavisd-new, port 10026) with ESMTP id luzycQzy_FcJ for ; Mon, 6 Feb 2012 20:51:41 +0100 (CET) Received: from [10.0.0.78] (203.84-49-43.nextgentel.com [84.49.43.203]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: erik@tefre.com) by mta1-submission.netlife.no (Postfix) with ESMTPSA id CC2BC3307BA for ; Mon, 6 Feb 2012 20:51:41 +0100 (CET) Message-ID: <4F302F60.3010302@tefre.com> Date: Mon, 06 Feb 2012 20:52:00 +0100 From: Erik Stian Tefre User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:9.0) Gecko/20111229 Thunderbird/9.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F2FF72B.6000509@pean.org> <20120206162206.GA541@icarus.home.lan> In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: HPC and zfs. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2012 20:08:07 -0000 On 02/06/2012 05:49 PM, Michael Aronsen wrote: >> - How you plan on getting roughly 410 hard disks (or 422 assuming >> an additional 12 SSDs) hooked up to a single machine > > Use LSI SAS92XX 4 (x4) port external controllers, and SuperMicro SC847E26-RJBOD1 disk shelves. > Each disk shelf needs 2 ports on the LSI controller, which means you get 90 disks per LSI card. > The DL580/585's have 11 PCIe slots, so you'd end up with 990 disks per server using this setup. The backplanes/expanders in the SC847E16-RJBOD1 (E16 = SATA/single expander version, E26 = SAS/dual expander version) can be daisy chained if you want to lower the number of controllers/ports. I have no E26 to test on but I guess it may support daisy chaining too. The LSI 9205-8e controller claims to support 1024 devices being connected to its two 4x6gb ports. Does anyone happen to know many daisy chain jumps are supported by SAS? Would it be possible to daisy chain 10 shelves with 450 drives onto 1 of those controllers? That would require 10 jumps on two chains or 20 jumps on one chain... (Yes, 6000*8/450 = 107 Mbit per drive, but ignore that part. ;) -- Erik