Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 23 Jan 2015 21:57:38 +1000
From:      David Gwynne <david@gwynne.id.au>
To:        Ahmed Kamal <email.ahmedkamal@googlemail.com>
Cc:        freebsd-scsi@freebsd.org
Subject:   Re: Dell branded storage controller for ZFS file-server | Advice requested
Message-ID:  <A427E3EC-2732-4295-B303-31910B8C1762@gwynne.id.au>
In-Reply-To: <CANzjMX4eyuPJ1qFvye6ECHDbJq8OZdoE7F2PzpaAFiME06fX5A@mail.gmail.com>
References:  <CANzjMX4GXf_jMMKZVSE3kf4NCvOUR0anjda3KSvFTNn6-%2BWmGg@mail.gmail.com> <EDB1FFA8-5311-4846-BCFF-127DF9A71453@gwynne.id.au> <CANzjMX4eyuPJ1qFvye6ECHDbJq8OZdoE7F2PzpaAFiME06fX5A@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

> On 23 Jan 2015, at 9:41 pm, Ahmed Kamal =
<email.ahmedkamal@googlemail.com> wrote:
>=20
> Thanks a lot for your reply .. I've got a couple of questions inline
>=20
> On Fri, Jan 23, 2015 at 12:21 PM, David Gwynne <david@gwynne.id.au> =
wrote:
>=20
> > On 23 Jan 2015, at 7:16 pm, Ahmed Kamal =
<email.ahmedkamal@googlemail.com> wrote:
> >
> > Hi everyone,
> >
> > I'm looking to build a ZFS file-server, unfortunately I'm locked =
into Dell
> > branded hardware only. I know Dell sometimes ships LSI branded =
cards, but
> > that too probably won't work for me, as I live outside the US and =
the local
> > Dell rep is clueless (so whatever I want to buy, has to be on Dell's
> > website). I am considering Dell R720xd or R730xd. I will use Intel =
i350
> > NICs, and ECC RAM, but the biggest questions are around the SAS HBA
> >
> > Per my research, I have the following notes and questions. =
Appreciating
> > your comments on them
> >
> > * H310 Card supports syspd (jbod) mode per
> > http://svnweb.freebsd.org/base?view=3Drevision&revision=3D254906 so =
it is
> > probably the best card for what I want to do, right?
>=20
> no.
>=20
> > * H310 is a low end card, it has very limited queue depth (25 I =
believe) ..
> > Is this actually a problem when using the disks (20 x SAS 10k disks) =
in
> > syspd/jbod mode? what about if I attach a couple of SSD disks?
>=20
> there are two queue limits here, one for the controller and one for =
each disk attached to it.
>=20
> h310s have a controller queue depth of 31, ie, they can only support =
31 commands at a time. if you have 24 disks and distribute the =
controllers command slots between them, thats about 1 command per disk.
>=20
> My initial line of thought was that in pass-through (jbod) mode, the =
HBA does *not* buffer commands and proxy disk responses internally. =
Which meant that in jbod mode, the HBA basically becomes a =
communications channel connecting the OS to disks, making its internal =
buffering limitations not-applicable. Is this not correct ?

the h310 physically doesnt have cache, so it cant buffer your I/O even =
if it wanted to.

so yes, in your use case it would just be a hba, but a very constrained =
one. even most sata controllers (eg, ahci or the silicon image ones) can =
keep more commands up in the air since they can have up 32 command slots =
per port instead of 31 for the whole controller.

im also not sure if dell will let you order an r720xd with a h310. i =
dont think they consider that a supported config.

> =20
>=20
> > * H310 can be flashed to IT mode (LSI firmware), and is supported by
> > Illumos kernel (if I'll ever need that) .. so overall it looks like =
a good
> > option. Any "watch-outs" I should be aware of ?
>=20
> yes. if you flash the h310 to become an sas hba, the pci =
product/vendor and subproduct/subvendor ids on the controller change. =
the dell bios will refuse to boot if it detects an unsupported =
controller in its storage slot. you'll brick the machine until you =
remove the h310. because it is on a non standard mezzanine connector you =
will not be able to flash it back to a h310, and you'll have a useless =
chunk of silicon to put on your desk as a monument to how annoying =
vendors are.
>=20
> pff nightmare scenario .. thanks for pointing this out. I stumbled =
across web posts where people successfully flashed h310, but meh, now =
I'm scared to buy that configuration

there are h310s that are normal pci-e cards that go into dells lower end =
servers. the ones in the r720s sit on the custom mezzanine connector.

> > * H710/p have no way of exposing JBOD/syspd in any way .. If using =
those
> > cards, the best I can do, is to create a raid-0 per disk, and put =
zfs on
> > top of that. How bad is this? Is the only problem that I'm locked =
into H710
> > firmware? Would the nvram and bbu on H710 improve performance =
significantly
> > vs H310?
>=20
> i wouldnt run zfs on that.
>=20
> > * H330 and H730, seem supported by mrsas driver. Is this driver able =
to
> > expose jbod/syspd mode? If yes, what's the overall advise, should I =
go with
> > one of those, or the "trusty" H310 ?
>=20
> thats a workable option. h330 should be fine if all you want to do is =
jbod.
>=20
> Now that you're pointing me to H330 being my best option given the =
situation, got some more questions:
> * "if all you want is jbod" .. with zfs, that's all we need right ? =
just wondering if I'm missing something important

nope.

> * Any idea if jbod configuration on h330 is done on bios level, or =
using some freebsd tool (how?)

you can do it in the bios.

> * mrsas man page, mentions the driver appeared in 10.1 .. Does that =
mean if I decide to use FreeNAS (9.x based) it won't be there ?

i have no idea :)

> * If I go the h330 path, it'll be a r730xd server. Worst case if the =
h330 gives me trouble, can I throw it out and plug a LSI 9207-8i in its =
place (same pci slot, cables ..etc) ?

the h330 sits on a mezzanine connector. you probably can get that card =
and drive the disks with it, but i havent done it with an r730 or 730xd =
so i cant say for sure. i have done that with an r720 though, and it =
mostly works.

we had to order those systems with a h310 and add the lsi card =
ourselves, and then we discovered that the disks they shipped in the =
system do not spin up when theyre powered on. the raid controller (h310) =
explicitly spins them up before issuing io to them.

if you attach those disks to a straight sas hba or use them via syspd on =
the h310, you cant boot off them because the machines bios (not the raid =
controller) blindly issues io to them without spinning them up first.

fun times.

dlg



> =20
>=20
> Thanks!
> =20
>=20
> >
> > Your help and advice is most appreciated
> > _______________________________________________
> > freebsd-scsi@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > To unsubscribe, send any mail to =
"freebsd-scsi-unsubscribe@freebsd.org"
>=20
>=20




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A427E3EC-2732-4295-B303-31910B8C1762>