From owner-freebsd-stable@FreeBSD.ORG Fri Sep 3 04:39:03 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8713A106586B for ; Fri, 3 Sep 2010 04:39:03 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3ED298FC12 for ; Fri, 3 Sep 2010 04:39:02 +0000 (UTC) Received: by gwb15 with SMTP id 15so562046gwb.13 for ; Thu, 02 Sep 2010 21:39:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=VlUVH08o63ZYnfHNdPa7UjDClLd284b7CuYodRtZB1I=; b=Brk472hFR2xzuE5zC0OcqBfhXePVO1tAhpwb4iZkRyv7p8lvDYhKWcmluCu268t/u8 ACUWkMVed4X6FGJzwaUyytCnbClEYqXnB7RyFSe8u35wYEfyPh3yaOcBpEV1/njBb8Bu Fl1tSiHOgOAylO2PucaHPuExlzIQeDt4zwLn0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=Uv4uMYNS91b2BZbRuEwSp74GmuVI8fhfh+CE3qG+1yyOLcZzXkyaLHvnbopODUGzvj IJHNOBzZfDxKftkrWmlaqPP1xI9mZBP0KkApRzVXgdydS4b5lye1nfyDiJY4KpSSJaKi h200lAvPgznPwHMc7ihOt4CCOlUt6B4Kb7Zko= MIME-Version: 1.0 Received: by 10.150.181.10 with SMTP id d10mr540949ybf.263.1283488742041; Thu, 02 Sep 2010 21:39:02 -0700 (PDT) Received: by 10.150.186.5 with HTTP; Thu, 2 Sep 2010 21:39:01 -0700 (PDT) In-Reply-To: <20100903040841.GA59175@icarus.home.lan> References: <4C7FA50D.4000409@sharescope.co.uk> <20100903040841.GA59175@icarus.home.lan> Date: Fri, 3 Sep 2010 00:39:01 -0400 Message-ID: From: Zaphod Beeblebrox To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Michal , Joshua Boyd , freebsd-stable@freebsd.org Subject: Re: Extending your zfs pool with multiple devices X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Sep 2010 04:39:03 -0000 On Fri, Sep 3, 2010 at 12:08 AM, Jeremy Chadwick wrote: > On Thu, Sep 02, 2010 at 04:56:04PM -0400, Zaphod Beeblebrox wrote: >> With 1.5T disks, I find that the 4 to 1 multipliers have a small >> effect on speed. =A0The 4 drives I have on the multipler are saturated >> at 100% a little bit more than the drives directly connected. >> Essentially you have 3 gigabit for 4 drives instead of 3 gigabit for 1 >> drive. > > 1:4 SATA replicators impose a bottleneck on the overall bandwidth > available between the replicator and the disks attached, as you stated. > Diagram: > > ICH10 > =A0|||___ (SATA300) Port 0, Disk 0 > =A0||____ (SATA300) Port 1, Disk 1 > =A0|_____ (SATA300) Port 2, eSATA Replicator > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ||||________ (SATA300= ) Port 0, Disk 2 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 |||_________ (SATA300= ) Port 1, Disk 3 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ||__________ (SATA300= ) Port 2, Disk 4 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 |___________ (SATA300= ) Port 3, Disk 5 > > If Disks 2 through 5 are decent disks (pushing 100MB/sec), essentially > you have 100*4 =3D 400MB/sec worth of bandwidth being shoved across a > 300MB/sec link. =A0That's making the assumption the disks attached are > magnetic and not SSD, and not taking into consideration protocol > overhead. > A better choice is a SATA multilane HBA, which are usually PCIe-based > with a single connector on the back of the HBA which splits out to > multiple disks (usually 4, but sometimes more). That's just connector-foo. The cards are still very expensive. Many ZFS loads don't saturate disks ... or don't saturate them consistently. I just built several systems with one port per disk --- and those cards tended towards $100/port. 1:4 replicators are less than $10/port and the six port motherboards don't seem to add any cost (4 or 6 SATA ports seem standard now). My point is: if you're building a database server and speed is all you care about, then one port per disk makes sense. If you are building a pile of disk and you're on a budget, port replicators are a good solution.