From owner-freebsd-fs@FreeBSD.ORG Mon Sep 19 19:27:21 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89EFE106564A for ; Mon, 19 Sep 2011 19:27:21 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from noop.in-addr.com (mail.in-addr.com [IPv6:2001:470:8:162::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5AD828FC0A for ; Mon, 19 Sep 2011 19:27:21 +0000 (UTC) Received: from gjp by noop.in-addr.com with local (Exim 4.76 (FreeBSD)) (envelope-from ) id 1R5jUu-0002uA-3x; Mon, 19 Sep 2011 15:27:20 -0400 Date: Mon, 19 Sep 2011 15:27:20 -0400 From: Gary Palmer To: Jason Usher Message-ID: <20110919192720.GD10165@in-addr.com> References: <1316459502.23423.YahooMailClassic@web121212.mail.ne1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1316459502.23423.YahooMailClassic@web121212.mail.ne1.yahoo.com> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on noop.in-addr.com); SAEximRunCond expanded to false Cc: freebsd-fs@freebsd.org Subject: Re: ZFS obn FreeBSD hardware model for 48 or 96 sata3 paths... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2011 19:27:21 -0000 On Mon, Sep 19, 2011 at 12:11:42PM -0700, Jason Usher wrote: > > > --- On Mon, 9/19/11, Bob Friesenhahn wrote: > > > > > Hmmm... I understand this, but is there not any data > > that might transfer from multiple magnetic disks, > > simultaneously, at 6GB, that could periodically max out the > > card bandwidth ?? As in, all drives in a 12 drive array > > perform an operation on their built-in cache simultaneously > > ? > > > > The best way to deal with this is by careful zfs pool > > design so that disks that can be expected to perform related > > operations (e.g. in same vdev) are carefully split across > > interface cards and I/O channels. This also helps with > > reliability. > > > Understood. > > But again, can't that all be dismissed completely by having a one drive / one path build ? And since that does not add extra cost per drive, or per card ... only per motherboard ... it seems an easy cost to swallow - even if it's a very edge case that it might ever be useful. The message you quoted said to split the load across interface cards and I/O channels (PCIE lanes I presume). Unless you are going to somehow cram 30+ interface cards into a motherboard and chassis, I cannot see how your query can relate back to the statement unless you are talking about configurations with SAS/SATA port multipiers, which you are determined to avoid. You *cannot* avoid having multiple disks on a single controller card and it is definitely Best Practice to split drives in any array across controllers so that any controller failure at most knocks a single component out of a redundant RAID configuration. Losing multiple disks in a single RAID group (or whatever the ZFS name is) normally results in data loss unless you are extremely lucky. I also think you are going to be pushed to find a motherboard with your requirements and will have to use port multipliers, and somewhat suspect that with the right architecture that the performance hit is not nearly as bad as you expect. Gary