From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 23 11:57:52 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DBA96CA9 for ; Fri, 23 Jan 2015 11:57:52 +0000 (UTC) Received: from mail-pa0-f51.google.com (mail-pa0-f51.google.com [209.85.220.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AD4F299C for ; Fri, 23 Jan 2015 11:57:52 +0000 (UTC) Received: by mail-pa0-f51.google.com with SMTP id fb1so8216252pad.10 for ; Fri, 23 Jan 2015 03:57:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=7ellI1AyIPrEp8zjd+ETOKORrWXVc73y0UhKskZGios=; b=I2DynzhDMPtctFouhEi21bEdlcUxQwQOYD4MnK6lyZBH3goQ2UUa4zhzkiQXFIRNIZ rQyTALBKNIVpfK04jJkwzDASPQPXfNt+9EZe4QV2fGjvgt7h5Xs4gUuBGqJIL5Y9HtlI D361Dse1Ap6yia40WSNfsTLt6CVPX/kEEy8rK9nxv8f7N6/YGWI04G88WrB0aHuK3VYN LSo/pA/VDVc+KH+WmNrOfkTuPEIj0yhW3aO4CHntWeOSV7lY4WlOLm7QZE0Y4oY9nbSW gxXgP/i/5D3CS8ssk7nsjx2kp1/TPMzbDRkA7QFhxUgdmyKpWtxMkKjM7SV6NkjEy0Ig WeYg== X-Gm-Message-State: ALoCoQkVAeDi+b+5NizIzQ+6CSAdnceb9a8U2GtQb1VQGWxmuqvDqv73+8yaavT4VgbcJFnOIJjJ X-Received: by 10.70.100.202 with SMTP id fa10mr5105002pdb.131.1422014266561; Fri, 23 Jan 2015 03:57:46 -0800 (PST) Received: from ?IPv6:2001:388:e000:ba00:86a:91f9:4155:c482? ([2001:388:e000:ba00:86a:91f9:4155:c482]) by mx.google.com with ESMTPSA id w9sm1752157pbt.10.2015.01.23.03.57.43 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 23 Jan 2015 03:57:45 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Subject: Re: Dell branded storage controller for ZFS file-server | Advice requested From: David Gwynne In-Reply-To: Date: Fri, 23 Jan 2015 21:57:38 +1000 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Ahmed Kamal X-Mailer: Apple Mail (2.1993) Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jan 2015 11:57:52 -0000 > On 23 Jan 2015, at 9:41 pm, Ahmed Kamal = wrote: >=20 > Thanks a lot for your reply .. I've got a couple of questions inline >=20 > On Fri, Jan 23, 2015 at 12:21 PM, David Gwynne = wrote: >=20 > > On 23 Jan 2015, at 7:16 pm, Ahmed Kamal = wrote: > > > > Hi everyone, > > > > I'm looking to build a ZFS file-server, unfortunately I'm locked = into Dell > > branded hardware only. I know Dell sometimes ships LSI branded = cards, but > > that too probably won't work for me, as I live outside the US and = the local > > Dell rep is clueless (so whatever I want to buy, has to be on Dell's > > website). I am considering Dell R720xd or R730xd. I will use Intel = i350 > > NICs, and ECC RAM, but the biggest questions are around the SAS HBA > > > > Per my research, I have the following notes and questions. = Appreciating > > your comments on them > > > > * H310 Card supports syspd (jbod) mode per > > http://svnweb.freebsd.org/base?view=3Drevision&revision=3D254906 so = it is > > probably the best card for what I want to do, right? >=20 > no. >=20 > > * H310 is a low end card, it has very limited queue depth (25 I = believe) .. > > Is this actually a problem when using the disks (20 x SAS 10k disks) = in > > syspd/jbod mode? what about if I attach a couple of SSD disks? >=20 > there are two queue limits here, one for the controller and one for = each disk attached to it. >=20 > h310s have a controller queue depth of 31, ie, they can only support = 31 commands at a time. if you have 24 disks and distribute the = controllers command slots between them, thats about 1 command per disk. >=20 > My initial line of thought was that in pass-through (jbod) mode, the = HBA does *not* buffer commands and proxy disk responses internally. = Which meant that in jbod mode, the HBA basically becomes a = communications channel connecting the OS to disks, making its internal = buffering limitations not-applicable. Is this not correct ? the h310 physically doesnt have cache, so it cant buffer your I/O even = if it wanted to. so yes, in your use case it would just be a hba, but a very constrained = one. even most sata controllers (eg, ahci or the silicon image ones) can = keep more commands up in the air since they can have up 32 command slots = per port instead of 31 for the whole controller. im also not sure if dell will let you order an r720xd with a h310. i = dont think they consider that a supported config. > =20 >=20 > > * H310 can be flashed to IT mode (LSI firmware), and is supported by > > Illumos kernel (if I'll ever need that) .. so overall it looks like = a good > > option. Any "watch-outs" I should be aware of ? >=20 > yes. if you flash the h310 to become an sas hba, the pci = product/vendor and subproduct/subvendor ids on the controller change. = the dell bios will refuse to boot if it detects an unsupported = controller in its storage slot. you'll brick the machine until you = remove the h310. because it is on a non standard mezzanine connector you = will not be able to flash it back to a h310, and you'll have a useless = chunk of silicon to put on your desk as a monument to how annoying = vendors are. >=20 > pff nightmare scenario .. thanks for pointing this out. I stumbled = across web posts where people successfully flashed h310, but meh, now = I'm scared to buy that configuration there are h310s that are normal pci-e cards that go into dells lower end = servers. the ones in the r720s sit on the custom mezzanine connector. > > * H710/p have no way of exposing JBOD/syspd in any way .. If using = those > > cards, the best I can do, is to create a raid-0 per disk, and put = zfs on > > top of that. How bad is this? Is the only problem that I'm locked = into H710 > > firmware? Would the nvram and bbu on H710 improve performance = significantly > > vs H310? >=20 > i wouldnt run zfs on that. >=20 > > * H330 and H730, seem supported by mrsas driver. Is this driver able = to > > expose jbod/syspd mode? If yes, what's the overall advise, should I = go with > > one of those, or the "trusty" H310 ? >=20 > thats a workable option. h330 should be fine if all you want to do is = jbod. >=20 > Now that you're pointing me to H330 being my best option given the = situation, got some more questions: > * "if all you want is jbod" .. with zfs, that's all we need right ? = just wondering if I'm missing something important nope. > * Any idea if jbod configuration on h330 is done on bios level, or = using some freebsd tool (how?) you can do it in the bios. > * mrsas man page, mentions the driver appeared in 10.1 .. Does that = mean if I decide to use FreeNAS (9.x based) it won't be there ? i have no idea :) > * If I go the h330 path, it'll be a r730xd server. Worst case if the = h330 gives me trouble, can I throw it out and plug a LSI 9207-8i in its = place (same pci slot, cables ..etc) ? the h330 sits on a mezzanine connector. you probably can get that card = and drive the disks with it, but i havent done it with an r730 or 730xd = so i cant say for sure. i have done that with an r720 though, and it = mostly works. we had to order those systems with a h310 and add the lsi card = ourselves, and then we discovered that the disks they shipped in the = system do not spin up when theyre powered on. the raid controller (h310) = explicitly spins them up before issuing io to them. if you attach those disks to a straight sas hba or use them via syspd on = the h310, you cant boot off them because the machines bios (not the raid = controller) blindly issues io to them without spinning them up first. fun times. dlg > =20 >=20 > Thanks! > =20 >=20 > > > > Your help and advice is most appreciated > > _______________________________________________ > > freebsd-scsi@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > To unsubscribe, send any mail to = "freebsd-scsi-unsubscribe@freebsd.org" >=20 >=20