From owner-freebsd-stable@FreeBSD.ORG Sun Jun 12 07:33:52 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D04E4106566B for ; Sun, 12 Jun 2011 07:33:52 +0000 (UTC) (envelope-from matt.thyer@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 55F028FC12 for ; Sun, 12 Jun 2011 07:33:51 +0000 (UTC) Received: by wyf23 with SMTP id 23so3866916wyf.13 for ; Sun, 12 Jun 2011 00:33:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=8b+DOLKq19NMVjcGSKppjiUHkp5prW0b5vQEsKh8+rE=; b=i0P0VKayNGxkdME+IV7WjrqLS2SncHJo8Fcb+jiOXJwt9afMFbt1NgMoBWRRevjbpb +77wmmeV9CrD0N9MUo8FFZgXT+YH62SMaT7bmMa0PeHL83g/wKLUx0MOovXRc7MJNWve 8zW/MONk6w+OBYMcyc/YD+wmKP6HC5k/1hhig= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=p3WepIYR8BEiZ92iVbyGTm9HE1dqutogX+HsyZRqd8YWU3wnaZdjh1ijkRM5qqucX3 1BwFbZKNaDT0lkSPDoNnLCeNkfdd2hSvaLyCFFD8cEC9hqLzyFoI6lCqbubkU9ni78yX M39NaLnAyzpwJm/CL3dTuzLiVsyi+7tPpyMQs= MIME-Version: 1.0 Received: by 10.216.221.158 with SMTP id r30mr1395604wep.50.1307864030885; Sun, 12 Jun 2011 00:33:50 -0700 (PDT) Received: by 10.216.157.201 with HTTP; Sun, 12 Jun 2011 00:33:50 -0700 (PDT) In-Reply-To: References: <201106070233.p572XI8T056329@apollo.backplane.com> Date: Sun, 12 Jun 2011 17:03:50 +0930 Message-ID: From: Matt Thyer To: Matthew Dillon Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: stable@freebsd.org Subject: Re: PCIe SATA HBA for ZFS on -STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jun 2011 07:33:52 -0000 On 8 June 2011 20:55, Matt Thyer wrote: > On 7 June 2011 12:03, Matthew Dillon wrote: > >> The absolute cheapest solution is to buy a Sil-3132 PCIe card >> (providing 2 E-SATA ports), and then connect an external port >> multiplier >> to each port. External port multiplier enclosures typically support >> 5 drives each so that would give you your 10 drives. >> > > I've decided to avoid issues with port multiplication by going for a > Supermicro AOC-USAS2-L8i and then to flash it to IT (as opposed to IR) mode > to make it run as a standard non-RAID HBA. > > As I've got 8 x 2 TB drives for the ZFS raidz2 I'll put them all on the > AOC-USAS2-L8i and save my onboard SATA-II ports for my 2 x 1TB drives for > the FreeBSD O.S. and any eSATA use. > > Now the only remaining issue is whether to go with the Supermicro firmware > or the generic Lsi Logic firmware as some have reported better performance > with the version 9 Lsi Logic firmware. > > I'll report on my experiences (as I keep a record of the revision of my > -STABLE build this should actually be useful!). > > I could not be happier with the result. As planned, I've put all 8 2TB Western Digital WD20EARS drives on the 8 lane PCIe 2.0 AOC-USAS2-L8i (flashed with the latest SuperMicro firmware to behave as an AOC-USAS2-L8e). Many of you are now screaming "Why is he using the dreaded WD20EARS ?". The answer is that I bought the first 4 drives before I knew their issues and later decided to continue with them once I knew how to mitigate their issues by: - Using the DOS based WDIDLE3.EXE utility to change the default "park the heads after 8 seconds of idle time" to the maximum of 5 minutes - Avoiding alignment problems by putting ZFS on the whole disks (with no GPT or MBR partition table) - Convincing ZFS to use 4KiB transfers by creating the pool on top of 4KiB block sized devices created with "gnop -S 4096" And the resulting performance... Previously I could sustain about 45 MiB/s writing from high powered Windows 7 machines (via Samba using asynchronous I/O) on to my old arrangement of 4 drives in raidz1 (using only the Intel on-board SATA ports on the H55 chipset motherboard). I can now sustain 90 MiB/s over the network with the 8 drive raidz2 but that's only because the Windows 7 machines can't feed the data fast enough. I can see from "zpool iostat pool 1" that ZFS is idle for about 5-7 seconds and then it will write at around 390 - 410 MB/s for about 3 seconds. dd tests are: > sudo dd if=/dev/zero of=/export/1G bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 3.218604 secs (333604841 bytes/sec) > sudo dd if=/dev/random of=/export/1G bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 15.531615 secs (69132658 bytes/sec) So I'm very happy that I can keep my home network users happy with the limits now being due to my gigabit ethernet network (and I'm not going 10 GbE any time soon!). This is on 8-STABLE at r220359 (~ 5th of April 2011).