From owner-freebsd-fs@FreeBSD.ORG Mon Sep 19 19:11:42 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2682106564A for ; Mon, 19 Sep 2011 19:11:42 +0000 (UTC) (envelope-from jusher71@yahoo.com) Received: from nm30-vm4.bullet.mail.ne1.yahoo.com (nm30-vm4.bullet.mail.ne1.yahoo.com [98.138.91.190]) by mx1.freebsd.org (Postfix) with SMTP id 912108FC1A for ; Mon, 19 Sep 2011 19:11:42 +0000 (UTC) Received: from [98.138.90.50] by nm30.bullet.mail.ne1.yahoo.com with NNFMP; 19 Sep 2011 19:11:42 -0000 Received: from [98.138.86.156] by tm3.bullet.mail.ne1.yahoo.com with NNFMP; 19 Sep 2011 19:11:42 -0000 Received: from [127.0.0.1] by omp1014.mail.ne1.yahoo.com with NNFMP; 19 Sep 2011 19:11:42 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 181598.53278.bm@omp1014.mail.ne1.yahoo.com Received: (qmail 28096 invoked by uid 60001); 19 Sep 2011 19:11:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1316459502; bh=fy5bAV3TSNPwP773KqlcWXxF1qe4cfZc28sYUs1N9dQ=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=cWeTWMcp2AKnp58R6k8+84crLhTFBOWjlb8tvLvWk+JfsyLJlT5HsNH55zjIr02U5hIni9QGjPLLWIkcu7TPJXhAGbs5BFaya1Z8HyjHPOX6aV9VMewXG1MY8A7vE7W6Hp8WQGGCyNPqWbRQuCAchjxBccfAt6RAh1hohk+HpZM= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=Lyan3dAdaUciB2VgGE8ezrw3wXj1sNRBqx8if1dla4EOWyPhArjUSZF7JwMuh0EkffPerZhdXJkHeTappTrPjtg5KYijp9oqKWCWuaNX/HSiAE4F5XtwT8OsMAWfnV/C/1ItsH/wU/67+dxp/awTyTLSO4OV/mECab+/YpSIU3c=; X-YMail-OSG: ostRMcMVM1ksu6q2usRjpLaTDHbsroUzMsD1_k2gaFjuJyl N6iygmj6CsEt.YbF9G3DXSsVBhVIC1oV.eeCUK.HuqZHetLHEMvmEoM4cVT5 ffWebec51QCSkERk_pN3x74oUxcJCJz0bMPWYEWbq8YudPrLMlvV6ueq4INO 9jY7g04vp2Dckwp6QW32YQ0fv1XqffGJhYzy2Sn_WcvHVkhHJb337jFfIpF3 b3rFxTC9EaRy2j_d1TZODCPyyWC1Ou7kwpj4fTGpCe2_dJZncows5DSuuyJJ V7p6ZcKBKR3o6Lf.u68qj71Ona9gurVccsdjd7qyl8OAczE1oCZ3BF1b4gzc D7Jud4ZQS7lqxlljkyM05D5UWxjfkE3fX7IfVHKaWwejQLlZREqYsRLufYWs X4DIT Received: from [77.247.181.164] by web121212.mail.ne1.yahoo.com via HTTP; Mon, 19 Sep 2011 12:11:42 PDT X-Mailer: YahooMailClassic/14.0.5 YahooMailWebService/0.8.114.317681 Message-ID: <1316459502.23423.YahooMailClassic@web121212.mail.ne1.yahoo.com> Date: Mon, 19 Sep 2011 12:11:42 -0700 (PDT) From: Jason Usher To: freebsd-fs@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Re: ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2011 19:11:42 -0000 =0A=0A--- On Mon, 9/19/11, Bob Friesenhahn w= rote:=0A=0A=0A> > Hmmm... I understand this, but is there not any data=0A> = that might transfer from multiple magnetic disks,=0A> simultaneously, at 6G= B, that could periodically max out the=0A> card bandwidth ?=A0 As in, all d= rives in a 12 drive array=0A> perform an operation on their built-in cache = simultaneously=0A> ?=0A> =0A> The best way to deal with this is by careful = zfs pool=0A> design so that disks that can be expected to perform related= =0A> operations (e.g. in same vdev) are carefully split across=0A> interfac= e cards and I/O channels. This also helps with=0A> reliability.=0A=0A=0AUnd= erstood.=0A=0ABut again, can't that all be dismissed completely by having a= one drive / one path build ? And since that does not add extra cost per d= rive, or per card ... only per motherboard ... it seems an easy cost to swa= llow - even if it's a very edge case that it might ever be useful.=0A=0APre= suming I can *find* a 112+ lane mobo, I assume the cost would be at worst d= ouble ($800ish instead of $400ish) a mobo with fewer pcie lanes...