Date: Thu, 17 Jan 2019 12:50:22 +0000 (UTC) From: Paul Pathiakis <pathiaki2@yahoo.com> To: freebsd-questions@freebsd.org, Julien Cigar <julien@perdition.city> Cc: freebsd-fs@freebsd.org Subject: Re: Supermicro HBA Message-ID: <1443717602.966565.1547729422870@mail.yahoo.com> In-Reply-To: <20190117103043.GA44618@mordor.lan> References: <20190117103043.GA44618@mordor.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
I've used the LSI 3008.=C2=A0 I haven't had any issues with it.=C2=A0 It i= s supported by that driver.=C2=A0 (I haven't built a server with one lately= ) Yes, avoid all the hardware RAID cards they are unnecessary and a JBOD cont= roller with ZFS is a good choice.=C2=A0 Make sure it supports the SAS3 spec= of 12 Gb/s.=C2=A0 That's where the speed is.=C2=A0 I found the following a= s an FYI. Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews |=20 |=20 |=20 | | | | | |=20 | |=20 Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews The Supermicro LSI SAS3008 HBAs (which share the same controller as the LSI= 9300-8i HBAs) are engineered to deli... | | | I looked at it as more of a specs education.=C2=A0 It looks solid.=C2=A0 I'= ve used Supermicro without issue in the past.=C2=A0 (I just decommissioned = my home server which was Supermicro and has that very card in it.) There is a lot to consider when using ZFS beyond just the hardware.=C2=A0 D= on't get me wrong.... I want to have ZFS' baby.=C2=A0 :D=C2=A0 Just be sure= of all the nuances of HDD, SSD, Hybrids, how much memory you have to dedic= ate to ZFS and CPU cores you have.=C2=A0 You have to take into consideratio= n all the ZFS features you're planning to make use of now and in the future= . Also, are you ever planning on expanding the storage to have an additional = JBOD shelf?=C2=A0 If so, you may want a card with some external connectors.= =C2=A0=C2=A0 Some of the people at=C2=A0http://www.ixsystems.com=C2=A0have done some ser= ious research on application specific throughput of ZFS and I believe they = also spec out SuperMicro servers too.=C2=A0 It comes down to IOPS, raw thro= ughput, etc.=C2=A0 (I'm actually talking to them right now about some very = large backup servers that can handle 0.75 PB.... The consideration I have i= s space and using RAIDZ2 and multiple streams from 10Gb interfaces and seri= ous compression and deduplication.=C2=A0 SO,=C2=A0 IOPS not so much, but he= avy raw I/O and RAID checksum computation and dedup. There's also things like dedicated SSDs as ZIL and cache to be thought abou= t. So:=C2=A0 go up the theoretical OSI layer model and optimize each layer rig= ht through the application layer. :D=C2=A0 (I actually find it fun) I hope this all helps. P On Thursday, January 17, 2019, 4:33:38 AM CST, Julien Cigar <julien@per= dition.city> wrote: =20 =20 Hello, We are planning to replace some (web) applications servers (currently running HPE) with Supermicro and the vendor offers the following=20 choices for the Hardware Raid Controller/HBA 4P part: 1) Supermicro AOC-S3008L-L8E, LSI 3008 8 x SATA/SAS III JBOD controller, up to 122 hard drives via expander backplane, PCI-E, ideal for Nexenta/ZFS + =E2=82=AC 205,6 2) LSI 9300-4I, SATA/SAS III JBOD controller, tot 256 harde schijven via expander backplanes, PCI-E, ideal for Nexenta/ZFS + =E2=82=AC 214,5 3) LSI MegaRAID 9341-4i bulk, 4 x SATA/SAS 12Gbs internal entry level hardware RAID, no cache/BBU possible, PCI-e + =E2=82=AC 177,97 4) LSI MegaRAID 9361-4i 1GB cache, 4 x SATA/SAS 12Gbs internal hardware RAID, max. 240 hdd using expander backplanes + =E2=82=AC 401,7 5) LSI 9300-4i4e, SATA/SAS III JBOD controller, 4 x internal, 4 x external, up to 256 hard drives via expander backplane, Nexenta Certified, ideal for ZFS, PCI-E + =E2=82=AC 273,95 6) LSI MegaRAID 9380-4i4e bulk, 8 x SATA/SAS 12Gbs, 4 x external and 4 x internal hardware RAID, 1024MB cache, up to 128 hard drives via expander backplane, support for SSD CacheCade 2.0 write and read caching, CacheVault support (advised), ideal for high en + =E2=82=AC 676 As the plan is to use ZFS, I was planning to choose the=20 "Supermicro AOC-S3008L-L8E, LSI 3008" (it looks like it is supported by mpr) and I was wondering if anyone has any feedback on it ? Would another option be a better choice ? Thank you! Julien --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11=C2=A0 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. =20 From owner-freebsd-fs@freebsd.org Thu Jan 17 14:28:35 2019 Return-Path: <owner-freebsd-fs@freebsd.org> Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35EDB149840C; Thu, 17 Jan 2019 14:28:35 +0000 (UTC) (envelope-from tenzin.lhakhang@gmail.com) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DBC8B69F85; Thu, 17 Jan 2019 14:28:33 +0000 (UTC) (envelope-from tenzin.lhakhang@gmail.com) Received: by mail-lf1-x12a.google.com with SMTP id p6so7979937lfc.1; Thu, 17 Jan 2019 06:28:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xcSamRnxJ5mCx4pHfNQf3p/f43LG3kj8W7WvVqf5kZg=; b=BiJOv4EpScLBe8vG6Nwgagb1cz1jo6/HmPE4vuwHcnUxSDzO0qZ5TFEh3kpGZl0EJ8 MS98WkgAPCdZoNb5HEmvav9kLf956QawHHJdUIprnvQvRGAPscrGesWxBZw9yUj4NxVX 1EJWYwXQUpQSLPOAtyh/02gk4pzcR2q9cuSQhofSBtPMZ6YOLPOQr1a8UduJVjhqYLce 5QtHbcxnfVd4mAHRMLLSoBuNtPfW6GrZHwZ7nl3DNPJv4hXxaXJxlNn4yQSfzCnpfqFF cIEThodTzqy6WxS8x4v9B5ZQiOtoCO/J7bM1SbEBTBunjERI4gG4admoUM7iBheZpQFV K1wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xcSamRnxJ5mCx4pHfNQf3p/f43LG3kj8W7WvVqf5kZg=; b=a74E22u5F6ZPdGcOK5lr/AXLQ7CQDQATJzttBzz8WeyU0Q3GZinawSdAJlEcmofuef BuBZte1oNJF0r+prAfM/5AUXRdXQ2dyTUJHuT1HiBY55N7RT6fpLfB4qyXy/vIPITOhO VXkFwEJETheJjuwvn4lszDfaDFDOZs0nqrbmVJlofJRPqvEXJg4tpiVCkmWt5c3/LGvi TzJcAbdRudP3ICwJfIlDtnkQ37IlAyqJhgKCgJ1S65wIfZW42dMbVMlayNitpce/gf6Q Fohj3Wpo70XoD0+CpF6X8iZ96UToDiBhVPd01xlhb6zlSSYnXEAn3oDE/KAx5xIAQwK1 s9xQ== X-Gm-Message-State: AJcUukdp+9sp1tqwlQKAtW+9DDGHgvacGt/p5SCQBO9cJyZm6KURkfGU IOsw8yCV3EaiNw6vl8QNuOkVqtsgp7Ebd7bCfzXh7+Zs X-Google-Smtp-Source: ALg8bN6pRyhnvI1HrPuBlkqJttA2tfWQG9kCqXhrqYbA/aNSqROTAitaQXvnJN74ogQClC2XZ90EYEXvJTreOzMw8sE= X-Received: by 2002:a19:518d:: with SMTP id g13mr10123601lfl.143.1547735312156; Thu, 17 Jan 2019 06:28:32 -0800 (PST) MIME-Version: 1.0 References: <20190117103043.GA44618@mordor.lan> <1443717602.966565.1547729422870@mail.yahoo.com> In-Reply-To: <1443717602.966565.1547729422870@mail.yahoo.com> From: Tenzin Lhakhang <tenzin.lhakhang@gmail.com> Date: Thu, 17 Jan 2019 09:28:20 -0500 Message-ID: <CALcn87yM3cDrY9Nup-QA21MGZDvetKbEm4nz71BkP5fB9V5HnQ@mail.gmail.com> Subject: Re: Supermicro HBA To: Paul Pathiakis <pathiaki2@yahoo.com> Cc: freebsd-questions@freebsd.org, Julien Cigar <julien@perdition.city>, "freebsd-fs@FreeBSD.org" <freebsd-fs@freebsd.org> X-Rspamd-Queue-Id: DBC8B69F85 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=BiJOv4Ep; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of tenzinlhakhang@gmail.com designates 2a00:1450:4864:20::12a as permitted sender) smtp.mailfrom=tenzinlhakhang@gmail.com X-Spamd-Result: default: False [-6.55 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.93)[-0.925,0]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-2.61)[ip: (-9.11), ipnet: 2a00:1450::/32(-2.07), asn: 15169(-1.81), country: US(-0.08)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[a.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems <freebsd-fs.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>; List-Post: <mailto:freebsd-fs@freebsd.org> List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 17 Jan 2019 14:28:35 -0000 If cost is negligible between Supermicro HBA using 3008 controller versus Broadcom/Avago/LSI card. I'd say opt for the Broadcom card, their support + drivers + firmware downloads is much better than Supermicro's FTP site. Broadcom keeps whitepapers, archive of all their past released firmwares. Must dos: - Make sure to do hot-add hot-remove disks (with workload). - Make sure to do disk led blink light tests. You will have to use the sas3ircu utility, unless you have OS specific utilities to find them. - Perform an fio disk benchmark to get baseline system performance. https://github.com/axboe/fio Issues I've seen: - SuperMicro's FTP site for their firmware doesn't include archive copies, only latest. Sometimes latest FW doesn't work nicely and hopefully you have an old FW downloaded prior. - SuperMicro FTP site: ftp://ftp.supermicro.com/Driver/SAS/LSI/ - Sometimes SM HBA's FW and LSI HBA's FW can be used interchangeable and sometimes not. If you flashed a LSI HBA with SMC firmware, then you have to add -nossid when flashing using the sas3flash when going back to LSI firmware. Sometimes it flashes fine and everything looks ok, you see all the disks, but when you blink disk 4, it blinks disk 1. When you physically remove disk 2, ZFS says disk 3 is gone. It's best to thoroughly qualify the system after FW flash. Broadcom page for 9300-8i - https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#ove= rview - It's a bit hard to understand all the different downloads, but they're helpful when you understand the workflow to flash FW and use the sas3ircu utility. In the latest release, they've bundled all the required downloads into the Firmware download. In the past you would have to individually find the different ROM and BIN files. All said, Supermicro HBA when it's working, it works pretty well. I've seen HBA failures, but the systems were usually 5+ years old. - If you have multiple HBAs in a system, record which physical controller is on PCIe slot is the logical controller reported by OS. Otherwise when a single HBA fails, it's a bit hard to trace which software HBA is which physical HBA. Thanks, Tenzin On Thu, Jan 17, 2019 at 7:51 AM Paul Pathiakis via freebsd-fs < freebsd-fs@freebsd.org> wrote: > I've used the LSI 3008. I haven't had any issues with it. It is > supported by that driver. (I haven't built a server with one lately) > Yes, avoid all the hardware RAID cards they are unnecessary and a JBOD > controller with ZFS is a good choice. Make sure it supports the SAS3 spe= c > of 12 Gb/s. That's where the speed is. I found the following as an FYI. > Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews > > > | > | > | > | | | > > | > > | > | > | | > Supermicro LSI SAS3008 HBA Review | StorageReview.com - Storage Reviews > > The Supermicro LSI SAS3008 HBAs (which share the same controller as the > LSI 9300-8i HBAs) are engineered to deli... > | > > | > > | > > > I looked at it as more of a specs education. It looks solid. I've used > Supermicro without issue in the past. (I just decommissioned my home > server which was Supermicro and has that very card in it.) > There is a lot to consider when using ZFS beyond just the hardware. Don'= t > get me wrong.... I want to have ZFS' baby. :D Just be sure of all the > nuances of HDD, SSD, Hybrids, how much memory you have to dedicate to ZFS > and CPU cores you have. You have to take into consideration all the ZFS > features you're planning to make use of now and in the future. > Also, are you ever planning on expanding the storage to have an additiona= l > JBOD shelf? If so, you may want a card with some external connectors. > Some of the people at http://www.ixsystems.com have done some serious > research on application specific throughput of ZFS and I believe they als= o > spec out SuperMicro servers too. It comes down to IOPS, raw throughput, > etc. (I'm actually talking to them right now about some very large backu= p > servers that can handle 0.75 PB.... The consideration I have is space and > using RAIDZ2 and multiple streams from 10Gb interfaces and serious > compression and deduplication. SO, IOPS not so much, but heavy raw I/O > and RAID checksum computation and dedup. > There's also things like dedicated SSDs as ZIL and cache to be thought > about. > So: go up the theoretical OSI layer model and optimize each layer right > through the application layer. :D (I actually find it fun) > I hope this all helps. > P > > On Thursday, January 17, 2019, 4:33:38 AM CST, Julien Cigar > <julien@perdition.city> wrote: > > Hello, > > We are planning to replace some (web) applications servers (currently > running HPE) with Supermicro and the vendor offers the following > choices for the Hardware Raid Controller/HBA 4P part: > > 1) Supermicro AOC-S3008L-L8E, LSI 3008 8 x SATA/SAS III JBOD controller, = up > to 122 hard drives via expander backplane, PCI-E, ideal for Nexenta/ZFS > + =E2=82=AC 205,6 > > 2) LSI 9300-4I, SATA/SAS III JBOD controller, tot 256 harde schijven via > expander backplanes, PCI-E, ideal for Nexenta/ZFS > + =E2=82=AC 214,5 > > 3) LSI MegaRAID 9341-4i bulk, 4 x SATA/SAS 12Gbs internal entry level > hardware RAID, no cache/BBU possible, PCI-e > + =E2=82=AC 177,97 > > 4) LSI MegaRAID 9361-4i 1GB cache, 4 x SATA/SAS 12Gbs internal > hardware RAID, max. 240 hdd using expander backplanes > + =E2=82=AC 401,7 > > 5) LSI 9300-4i4e, SATA/SAS III JBOD controller, 4 x internal, 4 x externa= l, > up to 256 hard drives via expander backplane, Nexenta Certified, ideal > for ZFS, PCI-E > + =E2=82=AC 273,95 > > 6) LSI MegaRAID 9380-4i4e bulk, 8 x SATA/SAS 12Gbs, 4 x external and 4 x > internal hardware RAID, 1024MB cache, up to 128 hard drives via expander > backplane, support for SSD CacheCade 2.0 write and read caching, > CacheVault support (advised), ideal for high en > + =E2=82=AC 676 > > As the plan is to use ZFS, I was planning to choose the > "Supermicro AOC-S3008L-L8E, LSI 3008" (it looks like it is supported by > mpr) and I was wondering if anyone has any feedback on it ? Would > another option be a better choice ? > > Thank you! > Julien > > -- > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1443717602.966565.1547729422870>