From owner-freebsd-stable@freebsd.org Tue Aug 2 03:22:39 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C906BAC19A; Tue, 2 Aug 2016 03:22:39 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: from mail-yw0-x22e.google.com (mail-yw0-x22e.google.com [IPv6:2607:f8b0:4002:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1487F19F7; Tue, 2 Aug 2016 03:22:39 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: by mail-yw0-x22e.google.com with SMTP id r9so190568492ywg.0; Mon, 01 Aug 2016 20:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=EhBNUcs8lgqhpBnZ5R5QKsQsfkBSkZLt4s92Cr+/9VQ=; b=h8/1FO4JPGOd+vLjNn4QO7CdoT5S2QiLxwnh7WrLzBEAa7dJYWsu9zpb//sbon9XKw n4yRbYKeOkbV+aYMqNXBZKdob8L7VYEiXpA0ZmDOhfkYq/aaDQB406i89nyex+AO1euN cLXtzW85+S9izeALjMWBayd09JRlg+kA0AgQvzZhtXsSzWBkneNvbjoix3CDlC4jabtr +2cHPZZkSSXLKyLS+tXXekyfteYcgBC4NPGRaEy+mcAlBiNukgEMs0x9tmODmuK8ukyr 86uFcRJU5Ven8S3zygEgIifI9eMAQO1SmosrcHm8mPaaec2+6eEb7oZgIvZU0pwZ+d7Y 1+jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=EhBNUcs8lgqhpBnZ5R5QKsQsfkBSkZLt4s92Cr+/9VQ=; b=OW6HfflMb8+jvPf9JZTOHhMf25nkyHdgqiPxEVELPz3HUxbVXXJD9qs9khC6gbZ7DD /h+Vc7aj+Z2xENhKZUZhfbcRUnqAieLUWnzQVdgo8PdBxNHD00Wav9GpZJTL8wzw11Wt gB+icUDGGTDIc5UZFqW9gsa8gs6EVxPYTaZ9srJinwC/etl0V/9tx3flDdQV1EzJULH9 xyppIgQVe29ZlqCIYL912f/34d4VmAV41x4YZMGU7O5oI8sSoDgTy8wnD6v3BpbTHNBq xlkIXiOTgCT9FL/3XaKHmzdk3YgYql8w/TnCI73e62zc6np7k6dyRwQ1WzmMfo8r+7Fw G6ZA== X-Gm-Message-State: AEkoouulBy9374uIPj/TnlS4RZ9vZ+R/mPKnGIzZpAOA4eD7nZt4XUXBrh62cy8BY8Nt5dXcE+COzQ3QOMfQpw== X-Received: by 10.37.201.131 with SMTP id z125mr43845470ybf.183.1470108158170; Mon, 01 Aug 2016 20:22:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.51.150 with HTTP; Mon, 1 Aug 2016 20:22:37 -0700 (PDT) In-Reply-To: <579F8743.8030104@sorbs.net> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> <579F8743.8030104@sorbs.net> From: Ultima Date: Mon, 1 Aug 2016 23:22:37 -0400 Message-ID: Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i To: Michelle Sullivan Cc: Borja Marcos , "O. Hartmann" , Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 03:22:39 -0000 If anyone is interested, as Michelle Sullivan just mentioned. One problem I found when looking for an HBA is that they are not so easy to find. Scoured the internet for a backup HBA I came across these - http://www.avagotech.com/products/server-storage/host-bus-adapters/#tab-12G= b1 Can only speak for sas-9305-24i. All 24 bays are occupied and quite pleased with the performance compared to its predecessor. It was originally going to be a backup unit, however that changed after running a scrub and the amount of hours to complete cut in half (around 30ish to 15 for 35T). And of course, the reason for this post, it replaced a raid card in passthrough mode. Another note, because it is an HBA, the ability to flash firmware is once again possible! (yay!) +1 to HBA's + ZFS, if possible replace it for an HBA. On Mon, Aug 1, 2016 at 1:30 PM, Michelle Sullivan wrote: > Borja Marcos wrote: > >> On 01 Aug 2016, at 15:12, O. Hartmann >>> wrote: >>> >>> First, thanks for responding so quickly. >>> >>> - The third option is to make the driver expose the SAS devices like a >>>> HBA >>>> would do, so that they are visible to the CAM layer, and disks are >>>> handled by >>>> the stock =E2=80=9Cda=E2=80=9D driver, which is the ideal solution. >>>> >>> I didn't find any switch which offers me the opportunity to put the PRA= ID >>> CP400i into a simple HBA mode. >>> >> The switch is in the FreeBSD mfi driver, the loader tunable I mentioned, >> regardless of what the card >> firmware does or pretends to do. >> >> It=E2=80=99s not visible doing a "sysctl -a=E2=80=9D, but it exists and = it=E2=80=99s unique even. >> It=E2=80=99s defined here: >> >> >> https://svnweb.freebsd.org/base/stable/10/sys/dev/mfi/mfi_cam.c?revision= =3D267084&view=3Dmarkup >> (line 93) >> >> In order to do it you need a couple of things. You need to set the >>>> variable >>>> hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. >>>> >>>> When booting installation media, enter command mode and use these >>>> commands: >>>> >>>> ----- >>>> set hw.mfi.allow_cam_disk_passthrough=3D1 >>>> load mfip >>>> boot >>>> =E2=80=94=E2=80=94=E2=80=94 >>>> >>> Well, I'm truly aware of this problemacy and solution (now), but I run >>> into a >>> henn-egg-problem, literally. As long as I can boot off of the >>> installation >>> medium, I have a kernel which deals with the setting. But the boot >>> medium is >>> supposed to be a SSD sitting with the PRAID CP400i controller itself! >>> So, I >>> never be able to boot off the system without crippling the ability to >>> have a >>> fullspeed ZFS configuration which I suppose to have with HBA mode, but >>> not >>> with any of the forced RAID modes offered by the controller. >>> >> Been there plenty of times, even argued quite strongly about the >> advantages of ZFS against hardware based RAID >> 5 cards. :) I remember when the Dell salesmen couldn=E2=80=99t possibly >> understand why I wanted a =E2=80=9Csoftware based RAID rather than a >> robust, hardware based solution=E2=80=9D :D >> > > There are reasons for using either... > > Nowadays its seems the conversations have degenerated into those like > Windows vs Linux vs Mac where everyone thinks their answer is the right o= ne > (just as you suggested you (Borja Marcos) did with the Dell salesman), > where in reality each has its own advantages and disadvantages. Eg: I'm > running 2 zfs servers on 'LSI 9260-16i's... big mistake! (the ZFS, not > LSI's)... one is a 'movie server' the other a 'postgresql database' > server... The latter most would agree is a bad use of zfs, the die-hards > won't but then they don't understand database servers and how they work o= n > disk. The former has mixed views, some argue that zfs is the only way to > ensure the movies will always work, personally I think of all the years > before zfs when my data on disk worked without failure until the disks > themselves failed... and RAID stopped that happening... what suddenly > changed, are disks and ram suddenly not reliable at transferring data? .. > anyhow back to the issue there is another part with this particular > hardware that people just throw away... > > The LSI 9260-* controllers have been designed to provide on hardware > RAID. The caching whether using the Cachecade SSD or just oneboard ECC > memory is *ONLY* used when running some sort of RAID set and LVs... this = is > why LSI recommend 'MegaCli -CfgEachDskRaid0' because it does enable > caching.. A good read on how to setup something similar is here: > https://calomel.org/megacli_lsi_commands.html (disclaimer, I haven't > parsed it all so the author could be clueless, but it seems to give > generally good advice.) Going the way of 'JBOD' is a bad thing to do, ju= st > don't, performance sucks. As for the recommended command above, can't > comment because currently I don't use it nor will I need to in the near > future... but... > > If you (O Hartmann) want to use or need to use ZFS with any OS including > FreeBSD don't go with the LSI 92xx series controllers, its just the wrong > thing to do.. Pick an HBA that is designed to give you direct access to > the drives not one you have to kludge and cajole.. Including LSI > controllers with caches that use the mfi driver, just not those that are > not designed to work in a non RAID mode (with or without the passthru > command/mode above.) > > > > >> At worst, you can set up a simple boot from a thumb drive or, even >> better, a SATADOM installed inside the server. I guess it will >> have SATA ports on the mainboard. That=E2=80=99s what I use to do. FreeN= AS uses a >> similar approach as well. And some modern servers >> also can boot from a SD card which you can use just to load the kernel. >> >> Depending on the number of disks you have, you can also sacrifice two to >> set up a mirror with a =E2=80=9Cnomal=E2=80=9D boot system, and using >> the rest of the disks for ZFS. Actually I=E2=80=99ve got an old server I= set up >> in 2012. It has 16 disks, and I created a logical volume (mirror) >> with 2 disks for boot, the other 14 disks for ZFS. >> >> If I installed this server now I would do it different, booting off a >> thumb drive. But I was younger and naiver :) >> >> >> > If I installed mine now I would do them differently as well... neither > would run ZFS, both would use their on card RAID kernels and UFS on top o= f > them... ZFS would be reserved for the multi-user NFS file servers. (and > trust me here, when it comes to media servers - where the media is just > stored not changed/updated/edited - the 16i with a good highspeed SSD as > 'Cachecade' really performs well... and on a moderately powerful MB/CPU > combo with good RAM and several gigabit interfaces it's surprising how ma= ny > unicast transcoded media streams it can handle... (read: my twin fibres a= re > saturated before the machine reaches anywhere near full load, and I can > still write at 13MBps from my old Mac Mini over NFS... which is about all > it can do without any load either.) > > So moral of the story/choices. Don't go with ZFS because people tell you > its best, because it isn't, go with ZFS if it suits your hardware and > application, and if ZFS suits your application, get hardware for it. > > Regards, > > -- > Michelle Sullivan > http://www.mhix.org/ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"