From owner-freebsd-fs@freebsd.org Tue Sep 29 18:25:58 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76878A0C10C for ; Tue, 29 Sep 2015 18:25:58 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-oi0-x233.google.com (mail-oi0-x233.google.com [IPv6:2607:f8b0:4003:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BA8E1D99 for ; Tue, 29 Sep 2015 18:25:58 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by oiev17 with SMTP id v17so8989293oie.1 for ; Tue, 29 Sep 2015 11:25:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CAialjvqXyCiICDujTrvDOavbRyswRa5dsdVYdZMnNw=; b=ijGmeX5EqDau4tafzNTi1ilPe/SGh4H7cyqELRIxxcX6gjbt3SNgGBhBx4YtGcDtA4 NUeCsIcfA2maN8Pag0NsVbNEC4HFoc+5S5BX/7Ofq8VM7rjVC3LYYtCNVLY8caC9+Br0 /lRgJoG8kifYzG0WdDmyXtoTik2/mvBE8oMWLnK5bQaPZ8f2H2lXbSMtDMSNlUEWuQKb WHq7stKVpkhneKAI1aOnM6b1dm2lW6XFQpDpLRoqd70cSVvjFLlZAojktHinPLf4ctsI M4RMXCQfRlgSL6+o2igAUyt73qhRoCBICnYK0iYgYmMeFWMAu+jw+DdegV+Fu9xtBuSd u1JQ== MIME-Version: 1.0 X-Received: by 10.202.51.87 with SMTP id z84mr3829853oiz.35.1443551157507; Tue, 29 Sep 2015 11:25:57 -0700 (PDT) Received: by 10.76.68.38 with HTTP; Tue, 29 Sep 2015 11:25:57 -0700 (PDT) In-Reply-To: <560AD2B9.5040706@fuckner.net> References: <1443447383.5271.66.camel@data-b104.adm.slu.se> <5609578E.1050606@physics.umn.edu> <560A4640.3030200@internetx.com> <560A9461.8090300@physics.umn.edu> <560A977C.1070102@internetx.com> <560AD2B9.5040706@fuckner.net> Date: Tue, 29 Sep 2015 11:25:57 -0700 Message-ID: Subject: Re: Cannot replace broken hard drive with LSI HBA From: Freddie Cash To: Michael Fuckner Cc: jg@internetx.com, Graham Allan , =?UTF-8?Q?Karli_Sj=C3=B6berg?= , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Sep 2015 18:25:58 -0000 On Tue, Sep 29, 2015 at 11:04 AM, Michael Fuckner wrote: > On 9/29/2015 3:51 PM, InterNetX - Juergen Gotteswinter wrote: > >> From my Experience using SATA Disks on SAS Controllers, no matter if >> theres an Expander between or not or mixed, those Setups keep on beeing >> flakey / unreliable. I might work under certain conditions, but its >> nothing you can bet on. >> >> Garret Damore (Illumos Project) describes the problem more detailed here >> >> http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.ht= ml >> >> > come on, the article is 5 years old, some things changed since then! > > - MUX Boards are unreliable and expensive- long time since I last saw > those boards > - SAS Disks are not just 10/15k high performance Disks anymore, most > Nearline Disks are available with native SAS interface as well > - if you pick the right disk there is no trouble using SATA Disks on SAS > Expanders or SAS Controllers (they should have R/V sensors, optimized > FW...). > - if you use desktop drives in a shelf with lets say 24 slots you should > not expect it to work ;-) > =E2=80=8BWhy not? ;) We use desktop-class drives in our backups storage servers without any issues. Even the monster boxes with 90 drives in them (2 JBODs of 45 drives each) run without issues using desktop-class drives. We're using a mix of WD Black (1, 2, 4 TB), Toshiba (2 TB), and Seagate (1, 2 TB). 2 systems using 24 drive bays. 2 systems using 90 drive bays. Plugged into SuperMicro SAS expanders and LSI 9211-8i or 9211-8e (I think that's the model number) controllers.=E2=80=8B All SAS2008 chipsets using mps(4) = drivers. We're not looking for uber-performance and millions of IOps from these systems, as the gigabit NIC is the bottleneck (rsync and zfs send both saturate that link, but all operations still complete within the allotted 8 hours window). We replace maybe 6-8 drives per year across all 4 systems; a little more than that this year due to overheating in one location, but that's been fixed. When a 2 TB desktop-class harddrive is $ 80 CDN in bulk, and we're only replacing 8 drives per year (under warranty, of course), it just doesn't make sense to spend the extra money on server-class, RAID-aware, nearline, or SAS drives. :) =E2=80=8BIf you =E2=80=8Bare building a storage server that requires millio= ns of IOps with multiple 10 Gbps connections, then sure, desktop-class drives won't cut it. But for everything else, they're fine. --=20 Freddie Cash fjwcash@gmail.com