From owner-freebsd-hackers@FreeBSD.ORG Tue Nov 18 15:48:41 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4C70935E for ; Tue, 18 Nov 2014 15:48:41 +0000 (UTC) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "smarthost.sentex.ca", Issuer "smarthost.sentex.ca" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 228588B6 for ; Tue, 18 Nov 2014 15:48:40 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.9/8.14.9) with ESMTP id sAIFmdMd027131 for ; Tue, 18 Nov 2014 10:48:39 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <546B6A54.1060403@sentex.net> Date: Tue, 18 Nov 2014 10:48:36 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: FreeBSD Hackers Subject: Re: OT-ish SATA port replicators vs. SAS "expanders" References: <54693866.9040009@sentex.net> <20141117225044.GS24601@funkthat.com> In-Reply-To: <20141117225044.GS24601@funkthat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.75 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2014 15:48:41 -0000 On 11/17/2014 5:50 PM, John-Mark Gurney wrote: > Brandon Vincent wrote this message on Sun, Nov 16, 2014 at 17:26 -0700: >> On Sun, Nov 16, 2014 at 4:51 PM, Mike Tancsa wrote: >>> I don think a port multiplier is the same as a SAS expander and a SAS/SATA >>> controller does not necessarily support port multipliers. >> >> You're absolutely correct. The SATA specifications if I recall >> correctly, outline that providing support for port multipliers is >> optional. Not to mention, the vast majority of chipsets that do have >> support for port multipliers only provide the slower command-based >> switching method [1], The LSI SAS 2008 chipset is not what you >> typically find on the cheap consumer "fake RAID" cards which most >> people have [2]. >> >> Most people who purchase this card are purchasing it for SAS, true >> RAID, performance, and reliability. Thus, LSI is only going to provide >> support for the far superior SAS expander technology. > > If you go w/ SATA PMs, make sure you get a controller that supports > FIS-base switching instead of command based switching... You'll get > much better performance w/ NCQ support... As for if the LSI SAS 2008 > will support FBS, probably not, as it'll require the driver and/or > firmware to be aware of how to handle things... Performance was pretty good, but reliability was really poor for us. Not sure if it was the cages, cables, or a firmware issue in the card (SiI3132) or cages or a driver issue. But we would regularly see errors such as kernel: siisch0: Timeout on slot 18 kernel: siisch0: siis_timeout is 00040000 ss 001ff800 rs 001ff800 es 00000000 sts 80112000 serr 00000000 kernel: siisch0: ... waiting for slots 001bf800 kernel: siisch0: Timeout on slot 19 kernel: siisch2: Error while READ LOG EXT kernel: (ada10:siisch2:0:2:0): WRITE_FPDMA_QUEUED. ACB: 61 ab 95 98 71 40 b6 00 00 00 00 00 kernel: (ada10:siisch2:0:2:0): CAM status: ATA Status Error kernel: (ada10:siisch2:0:2:0): ATA status: 00 () kernel: (ada10:siisch2:0:2:0): RES: 00 00 00 00 00 00 00 00 00 00 00 kernel: (ada10:siisch2:0:2:0): Retrying command kernel: siisch2: Error while READ LOG EXT But after having swapped things far too many times, we switched to a pair of LSI 9240-8i, and using the same disks (wiped and started fresh), we have yet to get any of the old errors prior to re-using the disks (there were no SMART errors logged), we did a long test as well as wrote to every sector (no errors). Two weeks later, still no errors. Even better performance too. pool: zbackup1 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zbackup1 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da0p1 ONLINE 0 0 0 da1p1 ONLINE 0 0 0 da2p1 ONLINE 0 0 0 da3p1 ONLINE 0 0 0 da4p1 ONLINE 0 0 0 da5p1 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 logs ada1p2 ONLINE 0 0 0 errors: No known data errors In the end, the price really is not that much different for the hardware. -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/