From owner-freebsd-stable@FreeBSD.ORG Tue Jun 7 04:33:07 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B17F0106564A for ; Tue, 7 Jun 2011 04:33:07 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [69.163.100.197]) by mx1.freebsd.org (Postfix) with ESMTP id 7D97F8FC12 for ; Tue, 7 Jun 2011 04:33:07 +0000 (UTC) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.4/8.14.1) with ESMTP id p572XIgG056330; Mon, 6 Jun 2011 19:33:18 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.4/8.13.4/Submit) id p572XI8T056329; Mon, 6 Jun 2011 19:33:18 -0700 (PDT) Date: Mon, 6 Jun 2011 19:33:18 -0700 (PDT) From: Matthew Dillon Message-Id: <201106070233.p572XI8T056329@apollo.backplane.com> To: Matt Thyer References: Cc: stable@freebsd.org Subject: Re: PCIe SATA HBA for ZFS on -STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jun 2011 04:33:07 -0000 :I'm not on the -STABLE list so please reply to me. : :I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with 8 x :2TB drives & 2 x 1TB drives. :The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2. : :Now the Intel chipset has only 6 on board SATA II ports so ideally I'm :looking for a non RAID SATA II HBA to give me 6 extra ports (4 min). :Why 6 extra ? :Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK, and 4 :the minimum I need to do the job. : :So... : :What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone :using ZFS ? : :Not wanting to break the bank. :Not interested in SATA III 6GB at this time... though it could be useful if :I add an SSD for... (is it ZIL ?). :Can this be added at any time ? : :The main issue is I need at least 10 ports total for all existing drives... :ZIL would require 11 so ideally we are talking a 6 port HBA. The absolute cheapest solution is to buy a Sil-3132 PCIe card (providing 2 E-SATA ports), and then connect an external port multiplier to each port. External port multiplier enclosures typically support 5 drives each so that would give you your 10 drives. Even the 3132 is a piss-ant little card it does support FIS-Based switching so performance will be very good... it will just be limited to SATA-II speeds is all. Motherboard AHCI-based SATA ports typically do NOT have FIS-Based switching support (this would be the FBSS capability flag when the AHCI driver probes the chipset). This means that while you can attach an external port multiplier enclosure to mobo SATA ports (see later on E-SATA vs SATA), read performance from multiple drives concurrently will be horrible. Write performance will still be decent due to drive write caches despite being serialized. On E-SATA vs SATA. Essentially there are only two differences between E-SATA and SATA. One is the cable and connector format. The other is hot swap detection. Most mobo SATA ports can be strung out to E-SATA with an appropriate adapter. High-end Intel ASUS mobos often come with such adapters (this is why they usually don't sport an actual E-Sata port on the backplane) and the BIOS has setup features to specify E-SATA on a port-by-port basis. -- For SSDs you want to directly connect the SSD to a mobo SATA port and then either mount the SSD in the case or mount it in a hot-swap gadget that you can screw into a PCI slot (it doesn't actually use the PCI connector, just the slot). A SATA-III port with a SATA-III SSD really shines here and 400-500 MBytes/sec random read performance from a single SSD is possible, but it isn't an absolute requirement. A SATA-II port will still work fine as long as you don't mind maxing out the bandwidth at 250 MBytes/sec. -- I can't help with any of the other questions. Someone also suggested the MPS driver for FreeBSD, with caveats. I'll add a caveat on the port multiplier enclosures. Nearly all such enclosures use another SIL chipset internally and it works pretty well EXCEPT that it isn't 100% dependable if you try to hot-swap drives in the enclosure while other drives in the enclosure are active. So with that caveat, I recommend the port multiplier enclosure as the cheapest solution. To get robust hot-swap enclosures you either need to go with SAS or you need to go with discrete SATA ports (no port multiplication), and the ports have to support hot-swap. The best hot-swap support for an AHCI port is if the AHCI chipset supports cold-presence-detect (CPD), and again Mobo AHCI chipsets usually don't. Hot-swap is a bit hit or miss without CPD because power savings modes can effectively prevent hot-swap detect from working properly. Drive disconnects will always be detected but drive connects might not be. And even with discrete SATA ports the AHCI firmware on mobos does not necessarily handle hot-swap properly. For example my Intel-I7 ASUS mobo will generate spurious interrupts and status on a DIFFERENT discrete SATA port when I hot swap on some other discrete SATA port, in addition to generating the status interrupt on the correct port. So then it comes down to the driver in the operating system properly handling the spurious status and properly stopping and restarting pending commands when necessary. So, again, it is best for the machine to be idle before attempting a hot-swap. Lots of caveats. Sorry... you can blame Intel for all the blasted issues with AHCI and SATA. Intel didn't produce a very good chipset spec and vendors took all sorts of liberties. -Matt Matthew Dillon