From owner-freebsd-fs@FreeBSD.ORG Sat Sep 17 01:36:02 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42222106566B for ; Sat, 17 Sep 2011 01:36:02 +0000 (UTC) (envelope-from jusher71@yahoo.com) Received: from nm23.bullet.mail.ne1.yahoo.com (nm23.bullet.mail.ne1.yahoo.com [98.138.90.86]) by mx1.freebsd.org (Postfix) with SMTP id 0B6A38FC12 for ; Sat, 17 Sep 2011 01:36:01 +0000 (UTC) Received: from [98.138.90.53] by nm23.bullet.mail.ne1.yahoo.com with NNFMP; 17 Sep 2011 01:22:07 -0000 Received: from [98.138.89.193] by tm6.bullet.mail.ne1.yahoo.com with NNFMP; 17 Sep 2011 01:22:07 -0000 Received: from [127.0.0.1] by omp1051.mail.ne1.yahoo.com with NNFMP; 17 Sep 2011 01:22:07 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 44608.31226.bm@omp1051.mail.ne1.yahoo.com Received: (qmail 34818 invoked by uid 60001); 17 Sep 2011 01:22:07 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1316222526; bh=+gg/+DHfWZXrSnFpnohdpiiH0z8BWFYWfwrsFhEx2ec=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=So/xgs0ks21kp0AXNLC3QpFAPaRV66BHmjCTG6sUVV9yQRuXxxSdjm/03YNZki8AsTNkfxA77r1cyCDe+eNIW+0jGYr7zCd3YFRXDZPmfeFQ7kP5SWObh2ceBFf6Woh2uwPhNZObCZp9hqoDB0fuJuSQ8S/OlTKlHkLr2He4KVA= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=W8p1osUGtVAwPZD7//Xlyzw+AKhInL3G3joJo4rReJ6T5Vcqp8kvFmakH8SnN2pxYhHQLOTvDijHJFGBGKSR2vTfaOC9OI8V69jifRL4dDVD8wRepV9KV+jkZjigKN/tf9KA52wvzmhVPlgwwMhOogJcD2aDvcJgdtGzlyGIoHM=; X-YMail-OSG: 0YXLavAVM1lVIh5q9UR8Kqg6rLlXDPhztOpqcy09JADDO5k WgC39LDwP_OhG87QhGPtURic4DzKMo1X7gUomRXEiktxuojUD5VOW0Lgvina LZIZrxveMiesFnqdwq4raN5ryZUDe8yJKldyLQL5F1ZNV4eRSHQErOnsk2pE 1BUVnctC0BWX54H0ikoajwiBoOictAgB0G_41olKtWr9Hq_hWXXZ_5K7eEpP reEqSTMDuAboboqY2xC0HjbgBo.0hQAlSklhOpuz.axXt63YFkDyKpCqKgdO 2BlycbDrDbT46krsPgJmmD5LQG9vfj8wYvX.YMwk7VlCU6B.FHlZSc00UHbw gUmnUis5af28gyeOqnFH3N1QazBltWOTR_1p7KP4X Received: from [199.48.147.46] by web121205.mail.ne1.yahoo.com via HTTP; Fri, 16 Sep 2011 18:22:06 PDT X-Mailer: YahooMailWebService/0.8.114.317681 Message-ID: <1316222526.31565.YahooMailNeo@web121205.mail.ne1.yahoo.com> Date: Fri, 16 Sep 2011 18:22:06 -0700 (PDT) From: Jason Usher To: "freebsd-fs@freebsd.org" MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Subject: ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Jason Usher List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Sep 2011 01:36:02 -0000 =0AHello,=0A=0AI am building my first FreeBSD based ZFS system and am decid= ing on a hardware model.=A0 The overriding requirement is:=0A=0A1) immediat= ely support 48 internal sata3 drives at full bandwidth - every drive has in= dependent path to CPU=0A=0A2) future expansion to support another 48 drives= on an attached JBOD, all of which ALSO have their own independent path to = CPU=0A=0AThe first question is:=A0 how many pcie 2.0 lanes does a motherboa= rd need to run 96 independent sata3 connections ?=A0 Am I correct that this= is extremely important ?=0A=0ANext, I see a lot of implementations done wi= th LSI adaptors - is this as simple as choosing (3) LSI SAS 9201-16i for th= e 48 internal drives and (3) LSI SAS 9201-16e for the external drives ?=A0 = I remember that these cards are supported with mps(4) in FreeBSD, but only = in 9.x (?) - is that still the case, or is that support in 8.2 or later in = 8.3 ?=0A=0ASo I will boot of a pair of mirrored SSDs formatted UFS2 - easy.= =A0 But I would also like to spec and use a ZIL+L2ARC and am not sure where= to go ... the system will be VERY write-biased and use a LOT of inodes - s= o lots of scanning of large dirs with lots of inodes and writing data.=A0 S= omething like 400 million inodes on a filesystem with an average file size = of 150 KB.=0A=0A- can I just skip the l2arc and just add more RAM ?=A0 Woul= dn't the RAM always be faster/better ?=A0 Or do folks build such large L2ar= cs (4x200 GB SSD ?) that it outweighs an extra 32 GB of RAM ?=0A=0A- provid= ed I maintain the free pcie slot(s) and/or free 2.5" drive slots, can I alw= ays just add a ZIL after the fact ?=A0 I'd prefer to skip it for now and sa= ve that complexity for later...=0A=0AThanks very much for any comments/sugg= estions.=0A