From owner-freebsd-scsi@FreeBSD.ORG Fri Jan 23 13:02:54 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9990DFC for ; Fri, 23 Jan 2015 13:02:54 +0000 (UTC) Received: from mail-la0-x234.google.com (mail-la0-x234.google.com [IPv6:2a00:1450:4010:c03::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0994DEF for ; Fri, 23 Jan 2015 13:02:54 +0000 (UTC) Received: by mail-la0-f52.google.com with SMTP id ge10so2982494lab.11 for ; Fri, 23 Jan 2015 05:02:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=6nxbQY+Va2Rs8SXj/+biSvkhc+M2XP4UVOyjz/2U+Tw=; b=u7v1pEjt230P2gfqFEkz2R9GSihI7LrjEENbO5W0cwEAkg1wjYx/JBcBlk60Pp1y5X +sMz6Rxm2Z7MD4pxSo5a9OCad72VuNy4wQW+NzxQ/hRG8w9zn48GMrOdLTHKAkfzyPo3 /eZS2uSvcuhjgz1IHv03D52FgtJMgs7ZfAhJ9REFHeg7T3mmz5qVFj1KPcu73KVS+9zx fCXxF2glAKg9Z58DZdnV6EEAQAIE8XvSK+GYijBNeHrDXiU2PC5zL1Oe5SaWSUNya10O iIw5EYWs8ARv3Xd9zbOSuwHP21vUB9GXTx92YK32F+aS2v9kESBkSiXyBsZrPbvdDjqy x63A== X-Received: by 10.112.38.4 with SMTP id c4mr7261116lbk.46.1422018171927; Fri, 23 Jan 2015 05:02:51 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.163.73 with HTTP; Fri, 23 Jan 2015 05:02:31 -0800 (PST) In-Reply-To: References: From: Ahmed Kamal Date: Fri, 23 Jan 2015 15:02:31 +0200 Message-ID: Subject: Re: Dell branded storage controller for ZFS file-server | Advice requested To: David Gwynne Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jan 2015 13:02:54 -0000 comments down below On Fri, Jan 23, 2015 at 1:57 PM, David Gwynne wrote: > > > On 23 Jan 2015, at 9:41 pm, Ahmed Kamal > wrote: > > > > Thanks a lot for your reply .. I've got a couple of questions inline > > > > On Fri, Jan 23, 2015 at 12:21 PM, David Gwynne > wrote: > > > > > On 23 Jan 2015, at 7:16 pm, Ahmed Kamal < > email.ahmedkamal@googlemail.com> wrote: > > > > > > Hi everyone, > > > > > > I'm looking to build a ZFS file-server, unfortunately I'm locked into > Dell > > > branded hardware only. I know Dell sometimes ships LSI branded cards, > but > > > that too probably won't work for me, as I live outside the US and the > local > > > Dell rep is clueless (so whatever I want to buy, has to be on Dell's > > > website). I am considering Dell R720xd or R730xd. I will use Intel i350 > > > NICs, and ECC RAM, but the biggest questions are around the SAS HBA > > > > > > Per my research, I have the following notes and questions. Appreciating > > > your comments on them > > > > > > * H310 Card supports syspd (jbod) mode per > > > http://svnweb.freebsd.org/base?view=revision&revision=254906 so it is > > > probably the best card for what I want to do, right? > > > > no. > > > > > * H310 is a low end card, it has very limited queue depth (25 I > believe) .. > > > Is this actually a problem when using the disks (20 x SAS 10k disks) in > > > syspd/jbod mode? what about if I attach a couple of SSD disks? > > > > there are two queue limits here, one for the controller and one for each > disk attached to it. > > > > h310s have a controller queue depth of 31, ie, they can only support 31 > commands at a time. if you have 24 disks and distribute the controllers > command slots between them, thats about 1 command per disk. > > > > My initial line of thought was that in pass-through (jbod) mode, the HBA > does *not* buffer commands and proxy disk responses internally. Which meant > that in jbod mode, the HBA basically becomes a communications channel > connecting the OS to disks, making its internal buffering limitations > not-applicable. Is this not correct ? > > the h310 physically doesnt have cache, so it cant buffer your I/O even if > it wanted to. > > so yes, in your use case it would just be a hba, but a very constrained > one. even most sata controllers (eg, ahci or the silicon image ones) can > keep more commands up in the air since they can have up 32 command slots > per port instead of 31 for the whole controller. > > im also not sure if dell will let you order an r720xd with a h310. i dont > think they consider that a supported config. > > > > > > > > * H310 can be flashed to IT mode (LSI firmware), and is supported by > > > Illumos kernel (if I'll ever need that) .. so overall it looks like a > good > > > option. Any "watch-outs" I should be aware of ? > > > > yes. if you flash the h310 to become an sas hba, the pci product/vendor > and subproduct/subvendor ids on the controller change. the dell bios will > refuse to boot if it detects an unsupported controller in its storage slot. > you'll brick the machine until you remove the h310. because it is on a non > standard mezzanine connector you will not be able to flash it back to a > h310, and you'll have a useless chunk of silicon to put on your desk as a > monument to how annoying vendors are. > > > > pff nightmare scenario .. thanks for pointing this out. I stumbled > across web posts where people successfully flashed h310, but meh, now I'm > scared to buy that configuration > > there are h310s that are normal pci-e cards that go into dells lower end > servers. the ones in the r720s sit on the custom mezzanine connector. > > > > * H710/p have no way of exposing JBOD/syspd in any way .. If using > those > > > cards, the best I can do, is to create a raid-0 per disk, and put zfs > on > > > top of that. How bad is this? Is the only problem that I'm locked into > H710 > > > firmware? Would the nvram and bbu on H710 improve performance > significantly > > > vs H310? > > > > i wouldnt run zfs on that. > > > > > * H330 and H730, seem supported by mrsas driver. Is this driver able to > > > expose jbod/syspd mode? If yes, what's the overall advise, should I go > with > > > one of those, or the "trusty" H310 ? > > > > thats a workable option. h330 should be fine if all you want to do is > jbod. > > > > Now that you're pointing me to H330 being my best option given the > situation, got some more questions: > > * "if all you want is jbod" .. with zfs, that's all we need right ? just > wondering if I'm missing something important > > nope. > > > * Any idea if jbod configuration on h330 is done on bios level, or using > some freebsd tool (how?) > > you can do it in the bios. > > > * mrsas man page, mentions the driver appeared in 10.1 .. Does that mean > if I decide to use FreeNAS (9.x based) it won't be there ? > > i have no idea :) > > > * If I go the h330 path, it'll be a r730xd server. Worst case if the > h330 gives me trouble, can I throw it out and plug a LSI 9207-8i in its > place (same pci slot, cables ..etc) ? > > the h330 sits on a mezzanine connector. you probably can get that card and > drive the disks with it, but i havent done it with an r730 or 730xd so i > cant say for sure. i have done that with an r720 though, and it mostly > works. > > we had to order those systems with a h310 and add the lsi card ourselves, > and then we discovered that the disks they shipped in the system do not > spin up when theyre powered on. the raid controller (h310) explicitly spins > them up before issuing io to them. > > if you attach those disks to a straight sas hba or use them via syspd on > the h310, you cant boot off them because the machines bios (not the raid > controller) blindly issues io to them without spinning them up first. > scary stuff .. so just to be clear: - LSI SAS hba => disks DONT spin up - h310 in jbod mode => disks DONT spin up - h330 in jbod mode => disks work fine ? Note that currently dell's web configurator, only sells h330 with r730(xd) not r720(xd) .. The r720 comes with h310 or h710 afaict Note 2, Illumos people are able to pass a driver parameter to spin up crappy dell disks .. check http://www.listbox.com/member/archive/182179/2013/06/sort/from/page/4/entry/15:430/20130617155143:52F0F7EA-D787-11E2-84A9-87C6F7D228D8/ > > fun times. > > dlg > > > > > > > > > Thanks! > > > > > > > > > > Your help and advice is most appreciated > > > _______________________________________________ > > > freebsd-scsi@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org > " > > > > > >