From owner-freebsd-stable@FreeBSD.ORG Fri Sep 3 04:08:44 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 795211065782 for ; Fri, 3 Sep 2010 04:08:44 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.westchester.pa.mail.comcast.net (qmta13.westchester.pa.mail.comcast.net [76.96.59.243]) by mx1.freebsd.org (Postfix) with ESMTP id 21B628FC08 for ; Fri, 3 Sep 2010 04:08:43 +0000 (UTC) Received: from omta18.westchester.pa.mail.comcast.net ([76.96.62.90]) by qmta13.westchester.pa.mail.comcast.net with comcast id 23l81f0061wpRvQ5D48k0v; Fri, 03 Sep 2010 04:08:44 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta18.westchester.pa.mail.comcast.net with comcast id 248i1f0063LrwQ23e48j6r; Fri, 03 Sep 2010 04:08:44 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 59DAE9B425; Thu, 2 Sep 2010 21:08:41 -0700 (PDT) Date: Thu, 2 Sep 2010 21:08:41 -0700 From: Jeremy Chadwick To: Zaphod Beeblebrox Message-ID: <20100903040841.GA59175@icarus.home.lan> References: <4C7FA50D.4000409@sharescope.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: Michal , Joshua Boyd , freebsd-stable@freebsd.org Subject: Re: Extending your zfs pool with multiple devices X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2010 04:08:44 -0000 On Thu, Sep 02, 2010 at 04:56:04PM -0400, Zaphod Beeblebrox wrote: > [regarding getting more disks in a machine] > > On Thu, Sep 2, 2010 at 12:47 PM, Joshua Boyd wrote: > > You need an HP SAS expander card in the new box, and an HBA in your primary > > box with external ports to hook it into. > > > > Then the drives in the other box will show up as local drives on your > > primary box. > > > > You don't even need an operating system on the second box, it just needs > > enough hardware in it to supply power to the SAS expander. > > An inexpensive option are SATA port replicators. Think SATA switch or > hub. 1:4 is common and cheap. > > I have a motherboard with intel ICH10 chipset. It commonly provides 6 > ports. This chipset is happy to configure port replicators. Meaning > you can put 24 drives on this motherboard. > > ... > > With 1.5T disks, I find that the 4 to 1 multipliers have a small > effect on speed. The 4 drives I have on the multipler are saturated > at 100% a little bit more than the drives directly connected. > Essentially you have 3 gigabit for 4 drives instead of 3 gigabit for 1 > drive. 1:4 SATA replicators impose a bottleneck on the overall bandwidth available between the replicator and the disks attached, as you stated. Diagram: ICH10 |||___ (SATA300) Port 0, Disk 0 ||____ (SATA300) Port 1, Disk 1 |_____ (SATA300) Port 2, eSATA Replicator ||||________ (SATA300) Port 0, Disk 2 |||_________ (SATA300) Port 1, Disk 3 ||__________ (SATA300) Port 2, Disk 4 |___________ (SATA300) Port 3, Disk 5 If Disks 2 through 5 are decent disks (pushing 100MB/sec), essentially you have 100*4 = 400MB/sec worth of bandwidth being shoved across a 300MB/sec link. That's making the assumption the disks attached are magnetic and not SSD, and not taking into consideration protocol overhead. Given the evolutionary rate of hard disks and SSDs, replicators are (in my opinion) not a viable solution mid or long-term. Even Silicon Image's products at this point are starting to force a 1:2 ratio on the replicators, probably to address the bottleneck issue: http://www.siliconimage.com/products/product.aspx?pid=154 http://www.siliconimage.com/products/product.aspx?pid=155 A better choice is a SATA multilane HBA, which are usually PCIe-based with a single connector on the back of the HBA which splits out to multiple disks (usually 4, but sometimes more). An ideal choice is ane Areca ARC-1300 series SAS-based PCIe x4 multilane adapters, which provides SATA300 to each individual disk and uses PCIe x4 (which can handle about 1GByte/sec in each direction, so 2GByte/sec total)... http://www.areca.com.tw/products/sasnoneraid.htm ...but there doesn't appear to be driver support for FreeBSD for this series of controller (arcmsr(4) doesn't mention the ARC-1300 series). I also don't know what Areca means on their site when they say "BSD/FreeBSD (will be available with 6Gb/s Host Adapter"), given that none of the ARC-1300 series cards are SATA600. If people are more focused on total number of devices (disks) that are available, then they should probably be looking at dropping a pretty penny on a low-end filer. Otherwise, consider replacing the actual hard disks themselves with drives of a higher capacity. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |