Date: Sun, 12 Jun 2011 17:03:50 +0930 From: Matt Thyer <matt.thyer@gmail.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: stable@freebsd.org Subject: Re: PCIe SATA HBA for ZFS on -STABLE Message-ID: <BANLkTikcZa=uQ2ecJy65mmxnhdOZmPOOYw@mail.gmail.com> In-Reply-To: <BANLkTimNrggvPsk3XFBcahB7L7Zojq0fSQ@mail.gmail.com> References: <BANLkTimmjrhnO9fAnFW_HmW%2BVQTrvMYYiA@mail.gmail.com> <201106070233.p572XI8T056329@apollo.backplane.com> <BANLkTimNrggvPsk3XFBcahB7L7Zojq0fSQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8 June 2011 20:55, Matt Thyer <matt.thyer@gmail.com> wrote: > On 7 June 2011 12:03, Matthew Dillon <dillon@apollo.backplane.com> wrote: > >> The absolute cheapest solution is to buy a Sil-3132 PCIe card >> (providing 2 E-SATA ports), and then connect an external port >> multiplier >> to each port. External port multiplier enclosures typically support >> 5 drives each so that would give you your 10 drives. >> > > I've decided to avoid issues with port multiplication by going for a > Supermicro AOC-USAS2-L8i and then to flash it to IT (as opposed to IR) mode > to make it run as a standard non-RAID HBA. > > As I've got 8 x 2 TB drives for the ZFS raidz2 I'll put them all on the > AOC-USAS2-L8i and save my onboard SATA-II ports for my 2 x 1TB drives for > the FreeBSD O.S. and any eSATA use. > > Now the only remaining issue is whether to go with the Supermicro firmware > or the generic Lsi Logic firmware as some have reported better performance > with the version 9 Lsi Logic firmware. > > I'll report on my experiences (as I keep a record of the revision of my > -STABLE build this should actually be useful!). > > I could not be happier with the result. As planned, I've put all 8 2TB Western Digital WD20EARS drives on the 8 lane PCIe 2.0 AOC-USAS2-L8i (flashed with the latest SuperMicro firmware to behave as an AOC-USAS2-L8e). Many of you are now screaming "Why is he using the dreaded WD20EARS ?". The answer is that I bought the first 4 drives before I knew their issues and later decided to continue with them once I knew how to mitigate their issues by: - Using the DOS based WDIDLE3.EXE utility to change the default "park the heads after 8 seconds of idle time" to the maximum of 5 minutes - Avoiding alignment problems by putting ZFS on the whole disks (with no GPT or MBR partition table) - Convincing ZFS to use 4KiB transfers by creating the pool on top of 4KiB block sized devices created with "gnop -S 4096" And the resulting performance... Previously I could sustain about 45 MiB/s writing from high powered Windows 7 machines (via Samba using asynchronous I/O) on to my old arrangement of 4 drives in raidz1 (using only the Intel on-board SATA ports on the H55 chipset motherboard). I can now sustain 90 MiB/s over the network with the 8 drive raidz2 but that's only because the Windows 7 machines can't feed the data fast enough. I can see from "zpool iostat pool 1" that ZFS is idle for about 5-7 seconds and then it will write at around 390 - 410 MB/s for about 3 seconds. dd tests are: > sudo dd if=/dev/zero of=/export/1G bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 3.218604 secs (333604841 bytes/sec) > sudo dd if=/dev/random of=/export/1G bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 15.531615 secs (69132658 bytes/sec) So I'm very happy that I can keep my home network users happy with the limits now being due to my gigabit ethernet network (and I'm not going 10 GbE any time soon!). This is on 8-STABLE at r220359 (~ 5th of April 2011).
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BANLkTikcZa=uQ2ecJy65mmxnhdOZmPOOYw>