Date: Tue, 12 May 2020 19:29:46 +0200 From: Peter Eriksson <pen@lysator.liu.se> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: RAID - hardware vs. ZFS Message-ID: <6D707E39-7213-44CE-96DA-060C3CB65843@lysator.liu.se> In-Reply-To: <5782dcc1-3375-c693-246c-65eda7aebc29@umn.edu> References: <e2ae7253-732b-ea37-f6d8-b93a5d532da2@druid.net> <5782dcc1-3375-c693-246c-65eda7aebc29@umn.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
We are running a number of HP DL380G9 servers with HP H241 =E2=80=9CSmart = HBA=E2=80=9D controllers in "JBOD mode=E2=80=9D and HP D6020 external = SAS disk cabinets (70 drives per cabinet (10 & 12TB drives)) using = FreeBSD 11.3-12.1 and ZFS. I would (and do) definitely stay away from RAID mode and use single disk = JBODs instead (if possible). ZFS handles errors in a much better/more = secure way than typical hardware RAID controllers since it has = =E2=80=9Cknowledge=E2=80=9D of the data that is stored - something a = RAID controller never can have. As other have stated the HP =E2=80=9CSmart=E2=80=9D controllers (using = the FreeBSD =E2=80=9Cciss=E2=80=9D device driver) are a bit = =E2=80=9Cfinicky=E2=80=9D=E2=80=A6 They (the HP H241 controller at = least) behave a bit better when put into a =E2=80=9Cpure=E2=80=9D JBOD = mode (a setting you can set from the BIOS setup menu). Before we found = that setting they were a pain to use. I _think_ you can put the modern HP =E2=80=9CSmart RAID=E2=80=9D = controllers (Pxxx) also into that JBOD mode but I haven=E2=80=99t tried = it. (I also run couple of old DL380G5 with a HP P400 controller but that = one doesn=E2=80=99t support the JBOD mode so there I have to fake it = with single-disk logical volumes). The only thing to watch out for (that I=E2=80=99ve ran into so far) with = many disks on HP controllers is a couple of bugs in the =E2=80=9Cciss=E2=80= =9D device driver which prevents it from detecting more than about 48 = drives per controller when in JBOD mode (at least with the H241 = controller). (The =E2=80=9Cciss=E2=80=9D device driver incorrectly puts a cap on = =E2=80=9Cmax_target=E2=80=9D (drive =E2=80=9Cid:s=E2=80=9D to probe to = the number of logical volumes the controller supports. For the H241 that is 64. And it seems to start to enumerate physical = disks from target 16 so 64-16 =3D a 48 drives limit=E2=80=A6 It also incorrectly sets the =E2=80=9CInitiator_id=E2=80=9D to = =E2=80=9Cmax_logical_volumes=E2=80=9D so any physical drive that happens = to get the same target number is silently skipped. I=E2=80=99ve patched the ciss driver source code and there is a bug = report so I=E2=80=99m hoping someone comes around to it and applies it = to the normal release kernel eventually (we try to stay away from custom = kernels if possible)=E2=80=A6 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246279 = <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246279>; (This is not a problem unless you have many physical drives behind a = single controller in JBOD mode. We didn=E2=80=99t really notice this = until we got our second fully stocked D6020 cabinet and tried to use = both at the same time=E2=80=A6 ) We=E2=80=99=E2=80=99ve been using these servers for a couple of years = now and it works fine. That said - if I could choose I=E2=80=99d go for = an LSI HBA controller instead of the HP ones, but HP apparently = doesn=E2=80=99t sell those. (We use the Dell-branded LSI SAS3008 (Dell = HBA330) controllers in our Dell servers and they are really nice). - Peter > On 12 May 2020, at 18:07, Graham Allan via freebsd-fs = <freebsd-fs@freebsd.org> wrote: >=20 > I have implemented ZFS on HPE SmartArray controllers, but not really = out of choice - it was the hardware available. I'd prefer just to use a = JBOD controller. You lose all the benefits of ZFS knowledge of hardware = and drive state, but of course there are useful benefits to ZFS outside = of that (volume management, snapshots etc) so it's not a total loss. >=20 > When I did create a ZFS pool on the hardware RAID, I just used a = single large hardware RAID volume (I didn't try to expose the individual = drives either as single-drive RAID-0 volumes, or drive pass-through). I = found the drive pass-through on Gen9 SmartArray controllers to be very = flaky, though the Gen10 ones look like they might be better. >=20 > On 5/12/2020 10:46 AM, D'Arcy Cain wrote: >> I have to purchase new servers soon. I am planning on getting = Proliant >> DL360 servers. These come with hardware RAID. I was wondering what >> opinions people had about hardware RAID vs. using ZFS for RAID. Is = one >> safer than the other? What about performance? What about hot = swapping? >> All opinions welcome. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6D707E39-7213-44CE-96DA-060C3CB65843>