From owner-freebsd-performance@freebsd.org Sun Jul 31 16:51:54 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B718BAAFE9; Sun, 31 Jul 2016 16:51:54 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0A6111772; Sun, 31 Jul 2016 16:51:53 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id u6VGpS1i058864; Sun, 31 Jul 2016 18:51:37 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id F41ADB18; Sun, 31 Jul 2016 18:51:27 +0200 (CEST) Message-ID: <579E2C8F.8090306@omnilan.de> Date: Sun, 31 Jul 2016 18:51:27 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Jason Zhang CC: freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, freebsd-hardware@freebsd.org Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i References: In-Reply-To: Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Sun, 31 Jul 2016 18:51:37 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-Mailman-Approved-At: Sun, 31 Jul 2016 22:45:12 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jul 2016 16:51:54 -0000 Bez=A8=B9glich Jason Zhang's Nachricht vom 17.06.2016 09:16 (localtime):= > Hi, > > I am working on storage service based on FreeBSD. I look forward to a = good result because many professional storage company use FreeBSD as its = OS. But I am disappointed with the Bad performance. I tested the the pe= rformance of LSI MegaRAID 9260-8i and had the following bad result: > > 1. Test environment: > (1) OS: FreeBSD 10.0 release > (2) Memory: 16G > (3) RAID adapter: LSI MegaRAID 9260-8i > (4) Disks: 9 SAS hard drives (10000 rpm), performance is expe= cted for each hard drive =20 Were the drives completely initialized? I remember that at least one vendor had implemented read-past-write for every sector when written first. It was with 15k 3.5" spindles and I'm really not sure which vendor it was, so I won't name any. But "slow init" had solved a similir problem for me back then... -Harry From owner-freebsd-performance@freebsd.org Mon Aug 1 06:45:23 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67342BACD35; Mon, 1 Aug 2016 06:45:23 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 281C81A8F; Mon, 1 Aug 2016 06:45:22 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.85) with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (envelope-from ) id <1bU6yM-000mWX-2P>; Mon, 01 Aug 2016 08:45:10 +0200 Received: from p578a69f9.dip0.t-ipconnect.de ([87.138.105.249] helo=freyja.zeit4.iv.bundesimmobilien.de) by inpost2.zedat.fu-berlin.de (Exim 4.85) with esmtpsa (TLSv1.2:AES256-GCM-SHA384:256) (envelope-from ) id <1bU6yL-001051-PK>; Mon, 01 Aug 2016 08:45:09 +0200 Date: Mon, 1 Aug 2016 08:45:04 +0200 From: "O. Hartmann" To: Borja Marcos Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-hardware@freebsd.org, freebsd-stable@freebsd.org Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i Message-ID: <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> Organization: FU Berlin X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: 87.138.105.249 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 06:45:23 -0000 On Wed, 22 Jun 2016 08:58:08 +0200 Borja Marcos wrote: > > On 22 Jun 2016, at 04:08, Jason Zhang wrote: > >=20 > > Mark, > >=20 > > Thanks > >=20 > > We have same RAID setting both on FreeBSD and CentOS including cache > > setting. In FreeBSD, I enabled the write cache but the performance is = the > > same. =20 > >=20 > > We don=E2=80=99t use ZFS or UFS, and test the performance on the RAW GE= OM disk > > =E2=80=9Cmfidx=E2=80=9D exported by mfi driver. We observed the =E2=80= =9Cgstat=E2=80=9D result and found > > that the write latency is too high. When we =E2=80=9Cdd" the disk with= 8k, it is > > lower than 1ms, but it is 6ms on 64kb write. It seems that each single > > write operation is very slow. But I don=E2=80=99t know whether it is a = driver > > problem or not. =20 >=20 > There is an option you can use (I do it all the time!) to make the card > behave as a plain HBA so that the disks are handled by the =E2=80=9Cda=E2= =80=9D driver.=20 >=20 > Add this to /boot/loader.conf >=20 > hw.mfi.allow_cam_disk_passthrough=3D1 > mfip_load=3D=E2=80=9CYES" >=20 > And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To avoid co= nfusions, it=E2=80=99s > better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D or= logical volume > configuration. >=20 >=20 >=20 >=20 > Borja. [...] How is this supposed to work when ALL disks (including boot device) are set= tled with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 and detected within FreeBSD 11-BETA and 12-CURRENT) controller itself? I did not find any solution to force the CP400i into a mode making itself acting as a HBA (we intend to use all drives with ZFS and let FreeBSD kernel/ZFS control everything). The boot device is a 256 GB Samsung SSD for enterprise use and putting the = UEFI load onto a EFI partition from 11-CURRENT-ALPHA4 is worse: dd takes up to almost a minute to put the image onto the SSD. The SSD active LED is blinki= ng alle the time indicating activity. Caches are off. I tried to enable the ca= che via the mfiutil command by 'mfiutil cache mfid0 enable', but it failed ... = It failed also on all other attached drives. I didn't further go into more investigations right now, since the experience with the EFI boot loader makes me suspect bad performance and that is harsh= so to speak. Glad to have found this thread anyway. I cross post this also to CURRENT as it might be an issue with CURRENT ... Kind regards, Oliver Hartmann From owner-freebsd-performance@freebsd.org Mon Aug 1 09:56:44 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD56EBAA2C5; Mon, 1 Aug 2016 09:56:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3AE8E1812; Mon, 1 Aug 2016 09:56:43 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id 5B5319DC575; Mon, 1 Aug 2016 11:48:31 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> Date: Mon, 1 Aug 2016 11:48:30 +0200 Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> To: "O. Hartmann" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 09:56:44 -0000 > On 01 Aug 2016, at 08:45, O. Hartmann = wrote: >=20 > On Wed, 22 Jun 2016 08:58:08 +0200 > Borja Marcos wrote: >=20 >> There is an option you can use (I do it all the time!) to make the = card >> behave as a plain HBA so that the disks are handled by the =E2=80=9Cda=E2= =80=9D driver.=20 >>=20 >> Add this to /boot/loader.conf >>=20 >> hw.mfi.allow_cam_disk_passthrough=3D1 >> mfip_load=3D=E2=80=9CYES" >>=20 >> And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To = avoid confusions, it=E2=80=99s >> better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D = or logical volume >> configuration. >>=20 >>=20 >>=20 >>=20 >> Borja. > [...] >=20 > How is this supposed to work when ALL disks (including boot device) = are settled > with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 = and > detected within FreeBSD 11-BETA and 12-CURRENT) controller itself? >=20 > I did not find any solution to force the CP400i into a mode making = itself > acting as a HBA (we intend to use all drives with ZFS and let FreeBSD > kernel/ZFS control everything). Have you tried that particular option?=20 With kinda recent LSI based cards you have three options: - The most usual and definitely NOT RECOMMENDED option is to define a = logical volume per disk which actually LSI Logic called before JBOD mode. It=E2=80=99s not = recommended at all if you want to run ZFS. - Recent cards, I think I saw this first on the LSI3008, have a JBOD = mode that exposes the drives as =E2=80=9Cmfisyspd=E2=80=9D devices. I don=E2=80=99t recommend it either, because the syspd drives are a sort = of limited version of a disk device. With SSDs, especially, you don=E2=80=99t have access to the TRIM command. - The third option is to make the driver expose the SAS devices like a = HBA would do, so that they are visible to the CAM layer, and disks are handled by the stock =E2=80=9Cda=E2=80=9D = driver, which is the ideal solution.=20 However, this third option might not be available in some custom = firmware versions for certain manufacturers? I don=C2=B4t know. And I would hesitate to make the conversion on a production = machine unless you have a complete and reliable full backup of all the data in case you need to rebuild it. In order to do it you need a couple of things. You need to set the = variable hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. When booting installation media, enter command mode and use these = commands: ----- set hw.mfi.allow_cam_disk_passthrough=3D1 load mfip boot =E2=80=94=E2=80=94=E2=80=94 Remember that after installation you need to update /boot/loader.conf in = the system you just installed with the following contents: hw.mfi.allow_cam_disk_passthrough=3D1 mfip_load=3D=E2=80=9CYES=E2=80=9D A note regarding CAM and MFI visibility: On some old firmware versions = for the LSI2008 I=E2=80=99ve even seen the disks available both as =E2=80=9Cmfi=E2=80=9D and =E2=80=9Cda=E2=80=9D = drivers. If possible, you should try to set them up as =E2=80=9Cunconfigur= ed good=E2=80=9D on the RAID firmware. Use the RAID firmware set up or maybe mfiutil(8) Also, make sure you don=E2=80=99t create any logical volumes on the = disks you want exposed to CAM. You should delete the logical volumes so that the MFI firmware doesn=E2=80=99t do anything = with them.=20 AND BEWARE: Doing these changes to a system in production with valuable = data is dangerous. Make sure you have a full and sound backup before making these changes. As a worst case, the card could expose the devices both as =E2=80=9Csyspd=E2= =80=9D and CAM (i.e., =E2=80=9Cda=E2=80=9D drives) but as long as you = don=E2=80=99t touch the syspd devices the card won=E2=80=99t do anything to them as = far as I know. It could be a serious problem, however, if you=20 access a drive part of a logical volume through CAM, as RAID cards tend = do to =E2=80=9Cpatrol reads=E2=80=9D and other stuff on them.=20 Provided it=E2=80=99s safe to do what I recommended, try it and follow = up by email.=20 Borja. From owner-freebsd-performance@freebsd.org Mon Aug 1 13:12:09 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31D76BA7AFD; Mon, 1 Aug 2016 13:12:09 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E526F15E9; Mon, 1 Aug 2016 13:12:08 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.85) with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (envelope-from ) id <1bUD0m-0037tt-6I>; Mon, 01 Aug 2016 15:12:04 +0200 Received: from p578a69f9.dip0.t-ipconnect.de ([87.138.105.249] helo=freyja.zeit4.iv.bundesimmobilien.de) by inpost2.zedat.fu-berlin.de (Exim 4.85) with esmtpsa (TLSv1.2:AES256-GCM-SHA384:256) (envelope-from ) id <1bUD0l-001Y2q-Rz>; Mon, 01 Aug 2016 15:12:04 +0200 Date: Mon, 1 Aug 2016 15:12:03 +0200 From: "O. Hartmann" To: Borja Marcos Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i Message-ID: <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> Organization: FU Berlin X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: 87.138.105.249 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 13:12:09 -0000 On Mon, 1 Aug 2016 11:48:30 +0200 Borja Marcos wrote: Hello. First, thanks for responding so quickly. > > On 01 Aug 2016, at 08:45, O. Hartmann wro= te: > >=20 > > On Wed, 22 Jun 2016 08:58:08 +0200 > > Borja Marcos wrote: > > =20 > >> There is an option you can use (I do it all the time!) to make the card > >> behave as a plain HBA so that the disks are handled by the =E2=80=9Cda= =E2=80=9D driver.=20 > >>=20 > >> Add this to /boot/loader.conf > >>=20 > >> hw.mfi.allow_cam_disk_passthrough=3D1 > >> mfip_load=3D=E2=80=9CYES" > >>=20 > >> And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To avoid= confusions, it=E2=80=99s > >> better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D= or logical volume > >> configuration. > >>=20 > >>=20 > >>=20 > >>=20 > >> Borja. =20 > > [...] > >=20 > > How is this supposed to work when ALL disks (including boot device) are > > settled with the mfi (in our case, it is a Fujitsu CP400i, based upon > > LSI3008 and detected within FreeBSD 11-BETA and 12-CURRENT) controller > > itself? > >=20 > > I did not find any solution to force the CP400i into a mode making itse= lf > > acting as a HBA (we intend to use all drives with ZFS and let FreeBSD > > kernel/ZFS control everything). =20 >=20 > Have you tried that particular option?=20 I have, indeed, used the "JBOD" function of the PRAID CP400i controller and= the intention of my posting regards to the suspicion, that this is, as mentione= d in many posts concerning RAID controllers and ZFS, the reason for the worse performance. And as I can see, it has been confirmed, sadly. >=20 > With kinda recent LSI based cards you have three options: >=20 > - The most usual and definitely NOT RECOMMENDED option is to define a log= ical > volume per disk which actually LSI Logic called before JBOD mode. It=E2= =80=99s not > recommended at all if you want to run ZFS. This is the only way to expose each disk as it is to the OS with the PRAID CP400i built-in into our RX1330-M2 server (XEON Skylake based). I ordered t= hat specific box with a HBA capable controller. Searching the net reveals that there is another one, called PSAS CP400i, which is also based on LSI/Avago SAS3008 and the possibility to expose drives as-is is explicitely mentioned= . I do not know whether this is a software feature - as I suspect - or something which has been hardwired to the controller. >=20 > - Recent cards, I think I saw this first on the LSI3008, have a JBOD mode > that exposes the drives as =E2=80=9Cmfisyspd=E2=80=9D devices. I don=E2= =80=99t recommend it either, > because the syspd drives are a sort of limited version of a disk device. = With > SSDs, especially, you don=E2=80=99t have access to the TRIM command. They expose the drives as "mfidX" if setup as JBOD. >=20 > - The third option is to make the driver expose the SAS devices like a HBA > would do, so that they are visible to the CAM layer, and disks are handle= d by > the stock =E2=80=9Cda=E2=80=9D driver, which is the ideal solution.=20 I didn't find any switch which offers me the opportunity to put the PRAID CP400i into a simple HBA mode. =20 >=20 > However, this third option might not be available in some custom firmware > versions for certain manufacturers? I don=C2=B4t know. And I would hesita= te to > make the conversion on a production machine unless you have a complete and > reliable full backup of all the data in case you need to rebuild it. The boxes are empty and ready-for-installation, so I do not worry. It is mo= re worrying about this stupid software-based strangulations of options by Fuji= tsu - if any. i do not want to blame them before I haven't double-checked. >=20 > In order to do it you need a couple of things. You need to set the variab= le > hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. >=20 > When booting installation media, enter command mode and use these command= s: >=20 > ----- > set hw.mfi.allow_cam_disk_passthrough=3D1 > load mfip > boot > =E2=80=94=E2=80=94=E2=80=94 Well, I'm truly aware of this problemacy and solution (now), but I run into= a henn-egg-problem, literally. As long as I can boot off of the installation medium, I have a kernel which deals with the setting. But the boot medium is supposed to be a SSD sitting with the PRAID CP400i controller itself! So, I never be able to boot off the system without crippling the ability to have a fullspeed ZFS configuration which I suppose to have with HBA mode, but not with any of the forced RAID modes offered by the controller.=20 I will check with Fujitsu for a solution. Maybe the PRAID CP400i is capable somehow of being a PSAS CP400i also, even if not exposed by the recent/installed firmware. Kind regards, Oliver=20 =20 >=20 >=20 > Remember that after installation you need to update /boot/loader.conf in = the > system you just installed with the following contents: >=20 > hw.mfi.allow_cam_disk_passthrough=3D1 > mfip_load=3D=E2=80=9CYES=E2=80=9D >=20 >=20 > A note regarding CAM and MFI visibility: On some old firmware versions for > the LSI2008 I=E2=80=99ve even seen the disks available both as =E2=80=9Cm= fi=E2=80=9D and =E2=80=9Cda=E2=80=9D > drivers. If possible, you should try to set them up as =E2=80=9Cunconfigu= red good=E2=80=9D on > the RAID firmware. Use the RAID firmware set up or maybe mfiutil(8) >=20 > Also, make sure you don=E2=80=99t create any logical volumes on the disks= you want > exposed to CAM. You should delete the logical volumes so that the MFI > firmware doesn=E2=80=99t do anything with them.=20 >=20 > AND BEWARE: Doing these changes to a system in production with valuable d= ata > is dangerous. Make sure you have a full and sound backup before making th= ese > changes. >=20 > As a worst case, the card could expose the devices both as =E2=80=9Csyspd= =E2=80=9D and CAM > (i.e., =E2=80=9Cda=E2=80=9D drives) but as long as you don=E2=80=99t touc= h the syspd devices the card > won=E2=80=99t do anything to them as far as I know. It could be a serious= problem, > however, if you access a drive part of a logical volume through CAM, as R= AID > cards tend do to =E2=80=9Cpatrol reads=E2=80=9D and other stuff on them.= =20 >=20 > Provided it=E2=80=99s safe to do what I recommended, try it and follow up= by email.=20 >=20 >=20 >=20 >=20 >=20 > Borja. >=20 >=20 >=20 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= .org" From owner-freebsd-performance@freebsd.org Mon Aug 1 13:30:09 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 816A2BA7F48; Mon, 1 Aug 2016 13:30:09 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 06F271D5B; Mon, 1 Aug 2016 13:30:08 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id D14279DCCC2; Mon, 1 Aug 2016 15:29:58 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> Date: Mon, 1 Aug 2016 15:29:58 +0200 Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> To: "O. Hartmann" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 13:30:09 -0000 > On 01 Aug 2016, at 15:12, O. Hartmann = wrote: >=20 > First, thanks for responding so quickly. >=20 >> - The third option is to make the driver expose the SAS devices like = a HBA >> would do, so that they are visible to the CAM layer, and disks are = handled by >> the stock =E2=80=9Cda=E2=80=9D driver, which is the ideal solution.=20= >=20 > I didn't find any switch which offers me the opportunity to put the = PRAID > CP400i into a simple HBA mode. The switch is in the FreeBSD mfi driver, the loader tunable I mentioned, = regardless of what the card firmware does or pretends to do. It=E2=80=99s not visible doing a "sysctl -a=E2=80=9D, but it exists and = it=E2=80=99s unique even. It=E2=80=99s defined here: = https://svnweb.freebsd.org/base/stable/10/sys/dev/mfi/mfi_cam.c?revision=3D= 267084&view=3Dmarkup (line 93) >> In order to do it you need a couple of things. You need to set the = variable >> hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. >>=20 >> When booting installation media, enter command mode and use these = commands: >>=20 >> ----- >> set hw.mfi.allow_cam_disk_passthrough=3D1 >> load mfip >> boot >> =E2=80=94=E2=80=94=E2=80=94 >=20 > Well, I'm truly aware of this problemacy and solution (now), but I run = into a > henn-egg-problem, literally. As long as I can boot off of the = installation > medium, I have a kernel which deals with the setting. But the boot = medium is > supposed to be a SSD sitting with the PRAID CP400i controller itself! = So, I > never be able to boot off the system without crippling the ability to = have a > fullspeed ZFS configuration which I suppose to have with HBA mode, but = not > with any of the forced RAID modes offered by the controller.=20 Been there plenty of times, even argued quite strongly about the = advantages of ZFS against hardware based RAID 5 cards. :) I remember when the Dell salesmen couldn=E2=80=99t possibly = understand why I wanted a =E2=80=9Csoftware based RAID rather than a robust, hardware based solution=E2=80=9D :D=20 At worst, you can set up a simple boot from a thumb drive or, even = better, a SATADOM installed inside the server. I guess it will have SATA ports on the mainboard. That=E2=80=99s what I use to do. = FreeNAS uses a similar approach as well. And some modern servers also can boot from a SD card which you can use just to load the kernel. Depending on the number of disks you have, you can also sacrifice two to = set up a mirror with a =E2=80=9Cnomal=E2=80=9D boot system, and using the rest of the disks for ZFS. Actually I=E2=80=99ve got an old server I = set up in 2012. It has 16 disks, and I created a logical volume (mirror) with 2 disks for boot, the other 14 disks for ZFS. If I installed this server now I would do it different, booting off a = thumb drive. But I was younger and naiver :) Borja. From owner-freebsd-performance@freebsd.org Tue Aug 2 08:27:08 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE388BAC21B; Tue, 2 Aug 2016 08:27:08 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64B0518E3; Tue, 2 Aug 2016 08:27:07 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id B555C9DD613; Tue, 2 Aug 2016 10:27:03 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <579F8743.8030104@sorbs.net> Date: Tue, 2 Aug 2016 10:27:02 +0200 Cc: "O. Hartmann" , Jason Zhang , freebsd-performance@freebsd.org, freebsd-current , FreeBSD-STABLE Mailing List , freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> <579F8743.8030104@sorbs.net> To: Michelle Sullivan X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 08:27:08 -0000 > On 01 Aug 2016, at 19:30, Michelle Sullivan = wrote: >=20 > There are reasons for using either=E2=80=A6 Indeed, but my decision was to run ZFS. And getting a HBA in some = configurations can be difficult because vendors insist on using=20 RAID adapters. After all, that=E2=80=99s what most of their customers = demand. Fortunately, at least some Avago/LSI cards can work as HBAs pretty well. = An example is the now venerable LSI2008. > Nowadays its seems the conversations have degenerated into those like = Windows vs Linux vs Mac where everyone thinks their answer is the right = one (just as you suggested you (Borja Marcos) did with the Dell = salesman), where in reality each has its own advantages and = disadvantages. I know, but this is not the case. But it=E2=80=99s quite frustrating to = try to order a server with a HBA rather than a RAID and receiving an = answer such as =E2=80=9Cthe HBA option is not available=E2=80=9D. That=E2=80=99s why = people are zapping, flashing and, generally, torturing HBA cards rather = cruelly ;) So, in my case, it=E2=80=99s not about what=E2=80=99s better or worse. = It=E2=80=99s just a simpler issue. Customer (myself) has made a = decision, which can be right or wrong. Manufacturer fails to deliver = what I need. If it was only one manufacturer, well, off with them, but = the issue is widespread in industry.=20 > Eg: I'm running 2 zfs servers on 'LSI 9260-16i's... big mistake! (the = ZFS, not LSI's)... one is a 'movie server' the other a 'postgresql = database' server... The latter most would agree is a bad use of zfs, = the die-hards won't but then they don't understand database servers and = how they work on disk. The former has mixed views, some argue that zfs = is the only way to ensure the movies will always work, personally I = think of all the years before zfs when my data on disk worked without = failure until the disks themselves failed... and RAID stopped that = happening... what suddenly changed, are disks and ram suddenly not = reliable at transferring data? .. anyhow back to the issue there is = another part with this particular hardware that people just throw = away=E2=80=A6 Well, silent corruption can happen. I=E2=80=99ve seen it once caused by = a flaky HBA and ZFS saved the cake. Yes. there were reliable replicas. = Still, rebuilding would be a pain in the ass.=20 > The LSI 9260-* controllers have been designed to provide on hardware = RAID. The caching whether using the Cachecade SSD or just oneboard ECC = memory is *ONLY* used when running some sort of RAID set and LVs... this = is why LSI recommend 'MegaCli -CfgEachDskRaid0' because it does enable = caching.. A good read on how to setup something similar is here: = https://calomel.org/megacli_lsi_commands.html (disclaimer, I haven't = parsed it all so the author could be clueless, but it seems to give = generally good advice.) Going the way of 'JBOD' is a bad thing to do, = just don't, performance sucks. As for the recommended command above, = can't comment because currently I don't use it nor will I need to in the = near future... but=E2=80=A6 Actually it=E2=80=99s not a good idea to use heavy disk caching when = running ZFS. Its reliability depends on being able to commit metadata to = disk. So I don=E2=80=99t care about that caching option. Provided you = have enough RAM, ZFS is very effective caching data itself. > If you (O Hartmann) want to use or need to use ZFS with any OS = including FreeBSD don't go with the LSI 92xx series controllers, its = just the wrong thing to do.. Pick an HBA that is designed to give you = direct access to the drives not one you have to kludge and cajole.. = Including LSI controllers with caches that use the mfi driver, just not = those that are not designed to work in a non RAID mode (with or without = the passthru command/mode above.) As I said, the problem is, sometimes it=E2=80=99s not so easy to find = the right HBA.=20 > So moral of the story/choices. Don't go with ZFS because people tell = you its best, because it isn't, go with ZFS if it suits your hardware = and application, and if ZFS suits your application, get hardware for it. Indeed, I second this. But really, "hardware for it" covers a rather = broad cathegory ;) ZFS can even manage to work on hardware _against_ it. Borja. From owner-freebsd-performance@freebsd.org Tue Aug 2 03:22:39 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C906BAC19A; Tue, 2 Aug 2016 03:22:39 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: from mail-yw0-x22e.google.com (mail-yw0-x22e.google.com [IPv6:2607:f8b0:4002:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1487F19F7; Tue, 2 Aug 2016 03:22:39 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: by mail-yw0-x22e.google.com with SMTP id r9so190568492ywg.0; Mon, 01 Aug 2016 20:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=EhBNUcs8lgqhpBnZ5R5QKsQsfkBSkZLt4s92Cr+/9VQ=; b=h8/1FO4JPGOd+vLjNn4QO7CdoT5S2QiLxwnh7WrLzBEAa7dJYWsu9zpb//sbon9XKw n4yRbYKeOkbV+aYMqNXBZKdob8L7VYEiXpA0ZmDOhfkYq/aaDQB406i89nyex+AO1euN cLXtzW85+S9izeALjMWBayd09JRlg+kA0AgQvzZhtXsSzWBkneNvbjoix3CDlC4jabtr +2cHPZZkSSXLKyLS+tXXekyfteYcgBC4NPGRaEy+mcAlBiNukgEMs0x9tmODmuK8ukyr 86uFcRJU5Ven8S3zygEgIifI9eMAQO1SmosrcHm8mPaaec2+6eEb7oZgIvZU0pwZ+d7Y 1+jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=EhBNUcs8lgqhpBnZ5R5QKsQsfkBSkZLt4s92Cr+/9VQ=; b=OW6HfflMb8+jvPf9JZTOHhMf25nkyHdgqiPxEVELPz3HUxbVXXJD9qs9khC6gbZ7DD /h+Vc7aj+Z2xENhKZUZhfbcRUnqAieLUWnzQVdgo8PdBxNHD00Wav9GpZJTL8wzw11Wt gB+icUDGGTDIc5UZFqW9gsa8gs6EVxPYTaZ9srJinwC/etl0V/9tx3flDdQV1EzJULH9 xyppIgQVe29ZlqCIYL912f/34d4VmAV41x4YZMGU7O5oI8sSoDgTy8wnD6v3BpbTHNBq xlkIXiOTgCT9FL/3XaKHmzdk3YgYql8w/TnCI73e62zc6np7k6dyRwQ1WzmMfo8r+7Fw G6ZA== X-Gm-Message-State: AEkoouulBy9374uIPj/TnlS4RZ9vZ+R/mPKnGIzZpAOA4eD7nZt4XUXBrh62cy8BY8Nt5dXcE+COzQ3QOMfQpw== X-Received: by 10.37.201.131 with SMTP id z125mr43845470ybf.183.1470108158170; Mon, 01 Aug 2016 20:22:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.51.150 with HTTP; Mon, 1 Aug 2016 20:22:37 -0700 (PDT) In-Reply-To: <579F8743.8030104@sorbs.net> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> <579F8743.8030104@sorbs.net> From: Ultima Date: Mon, 1 Aug 2016 23:22:37 -0400 Message-ID: Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i To: Michelle Sullivan Cc: Borja Marcos , "O. Hartmann" , Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org X-Mailman-Approved-At: Tue, 02 Aug 2016 10:58:17 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 03:22:39 -0000 If anyone is interested, as Michelle Sullivan just mentioned. One problem I found when looking for an HBA is that they are not so easy to find. Scoured the internet for a backup HBA I came across these - http://www.avagotech.com/products/server-storage/host-bus-adapters/#tab-12G= b1 Can only speak for sas-9305-24i. All 24 bays are occupied and quite pleased with the performance compared to its predecessor. It was originally going to be a backup unit, however that changed after running a scrub and the amount of hours to complete cut in half (around 30ish to 15 for 35T). And of course, the reason for this post, it replaced a raid card in passthrough mode. Another note, because it is an HBA, the ability to flash firmware is once again possible! (yay!) +1 to HBA's + ZFS, if possible replace it for an HBA. On Mon, Aug 1, 2016 at 1:30 PM, Michelle Sullivan wrote: > Borja Marcos wrote: > >> On 01 Aug 2016, at 15:12, O. Hartmann >>> wrote: >>> >>> First, thanks for responding so quickly. >>> >>> - The third option is to make the driver expose the SAS devices like a >>>> HBA >>>> would do, so that they are visible to the CAM layer, and disks are >>>> handled by >>>> the stock =E2=80=9Cda=E2=80=9D driver, which is the ideal solution. >>>> >>> I didn't find any switch which offers me the opportunity to put the PRA= ID >>> CP400i into a simple HBA mode. >>> >> The switch is in the FreeBSD mfi driver, the loader tunable I mentioned, >> regardless of what the card >> firmware does or pretends to do. >> >> It=E2=80=99s not visible doing a "sysctl -a=E2=80=9D, but it exists and = it=E2=80=99s unique even. >> It=E2=80=99s defined here: >> >> >> https://svnweb.freebsd.org/base/stable/10/sys/dev/mfi/mfi_cam.c?revision= =3D267084&view=3Dmarkup >> (line 93) >> >> In order to do it you need a couple of things. You need to set the >>>> variable >>>> hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. >>>> >>>> When booting installation media, enter command mode and use these >>>> commands: >>>> >>>> ----- >>>> set hw.mfi.allow_cam_disk_passthrough=3D1 >>>> load mfip >>>> boot >>>> =E2=80=94=E2=80=94=E2=80=94 >>>> >>> Well, I'm truly aware of this problemacy and solution (now), but I run >>> into a >>> henn-egg-problem, literally. As long as I can boot off of the >>> installation >>> medium, I have a kernel which deals with the setting. But the boot >>> medium is >>> supposed to be a SSD sitting with the PRAID CP400i controller itself! >>> So, I >>> never be able to boot off the system without crippling the ability to >>> have a >>> fullspeed ZFS configuration which I suppose to have with HBA mode, but >>> not >>> with any of the forced RAID modes offered by the controller. >>> >> Been there plenty of times, even argued quite strongly about the >> advantages of ZFS against hardware based RAID >> 5 cards. :) I remember when the Dell salesmen couldn=E2=80=99t possibly >> understand why I wanted a =E2=80=9Csoftware based RAID rather than a >> robust, hardware based solution=E2=80=9D :D >> > > There are reasons for using either... > > Nowadays its seems the conversations have degenerated into those like > Windows vs Linux vs Mac where everyone thinks their answer is the right o= ne > (just as you suggested you (Borja Marcos) did with the Dell salesman), > where in reality each has its own advantages and disadvantages. Eg: I'm > running 2 zfs servers on 'LSI 9260-16i's... big mistake! (the ZFS, not > LSI's)... one is a 'movie server' the other a 'postgresql database' > server... The latter most would agree is a bad use of zfs, the die-hards > won't but then they don't understand database servers and how they work o= n > disk. The former has mixed views, some argue that zfs is the only way to > ensure the movies will always work, personally I think of all the years > before zfs when my data on disk worked without failure until the disks > themselves failed... and RAID stopped that happening... what suddenly > changed, are disks and ram suddenly not reliable at transferring data? .. > anyhow back to the issue there is another part with this particular > hardware that people just throw away... > > The LSI 9260-* controllers have been designed to provide on hardware > RAID. The caching whether using the Cachecade SSD or just oneboard ECC > memory is *ONLY* used when running some sort of RAID set and LVs... this = is > why LSI recommend 'MegaCli -CfgEachDskRaid0' because it does enable > caching.. A good read on how to setup something similar is here: > https://calomel.org/megacli_lsi_commands.html (disclaimer, I haven't > parsed it all so the author could be clueless, but it seems to give > generally good advice.) Going the way of 'JBOD' is a bad thing to do, ju= st > don't, performance sucks. As for the recommended command above, can't > comment because currently I don't use it nor will I need to in the near > future... but... > > If you (O Hartmann) want to use or need to use ZFS with any OS including > FreeBSD don't go with the LSI 92xx series controllers, its just the wrong > thing to do.. Pick an HBA that is designed to give you direct access to > the drives not one you have to kludge and cajole.. Including LSI > controllers with caches that use the mfi driver, just not those that are > not designed to work in a non RAID mode (with or without the passthru > command/mode above.) > > > > >> At worst, you can set up a simple boot from a thumb drive or, even >> better, a SATADOM installed inside the server. I guess it will >> have SATA ports on the mainboard. That=E2=80=99s what I use to do. FreeN= AS uses a >> similar approach as well. And some modern servers >> also can boot from a SD card which you can use just to load the kernel. >> >> Depending on the number of disks you have, you can also sacrifice two to >> set up a mirror with a =E2=80=9Cnomal=E2=80=9D boot system, and using >> the rest of the disks for ZFS. Actually I=E2=80=99ve got an old server I= set up >> in 2012. It has 16 disks, and I created a logical volume (mirror) >> with 2 disks for boot, the other 14 disks for ZFS. >> >> If I installed this server now I would do it different, booting off a >> thumb drive. But I was younger and naiver :) >> >> >> > If I installed mine now I would do them differently as well... neither > would run ZFS, both would use their on card RAID kernels and UFS on top o= f > them... ZFS would be reserved for the multi-user NFS file servers. (and > trust me here, when it comes to media servers - where the media is just > stored not changed/updated/edited - the 16i with a good highspeed SSD as > 'Cachecade' really performs well... and on a moderately powerful MB/CPU > combo with good RAM and several gigabit interfaces it's surprising how ma= ny > unicast transcoded media streams it can handle... (read: my twin fibres a= re > saturated before the machine reaches anywhere near full load, and I can > still write at 13MBps from my old Mac Mini over NFS... which is about all > it can do without any load either.) > > So moral of the story/choices. Don't go with ZFS because people tell you > its best, because it isn't, go with ZFS if it suits your hardware and > application, and if ZFS suits your application, get hardware for it. > > Regards, > > -- > Michelle Sullivan > http://www.mhix.org/ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"