Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 22 Aug 2016 11:46:35 -0500
From:      Karl Denninger <karl@denninger.net>
To:        freebsd-fs@freebsd.org
Subject:   Re: Recommended HBA for ZFS, contemporary
Message-ID:  <580a6ea8-9c84-3e99-5a7d-7b2434c84e67@denninger.net>
In-Reply-To: <alpine.BSF.2.00.1608221926550.82304@woozle.rinet.ru>
References:  <alpine.BSF.2.00.1608221904270.82304@woozle.rinet.ru> <ba972848-140c-31a8-505e-2470ed750fc8@denninger.net> <alpine.BSF.2.00.1608221926550.82304@woozle.rinet.ru>
index | next in thread | previous in thread | raw e-mail 

[-- Attachment #1 --]
On 8/22/2016 11:30, Dmitry Morozovsky wrote:
> Karl,
>
> On Mon, 22 Aug 2016, Karl Denninger wrote:
>
>> I'm still a fan of the LSI SAS-9211 cards along with (for higher
>> density) a SAS port expander.  I've been extremely happy with this
>> combination with one caveat -- most systems must boot from the actual
>> card and not a port on the expander(s) you connect to it due to BIOS
>> constraints.  They're also crazily cost-effective on-balance.
>>
>> In practice this means you have 4 ports on the base card available (2
>> used for mirrored boot drives, 2 for other things) and the rest of the
>> disks go on the expander, or you run more than one card.
>>
>> Whether you can actually saturate this combination depends on what you
>> attach.  In large configurations, especially those stuffed with SSDs,
>> you can -- at which point something faster such as the 93xx series which
> which immediately leads me to the question: what is the driver supporting 9300 
> in HBA mode?  Quick googling does not reveal interoperability, and 9300 is not 
> in hardware list for FreeBSD yet.
So much for LSI's claims eh? :-)
>> are materially faster (but still in "host" mode as opposed to any sort
>> of "smart", "raid" or "buffered" mode) should be considered.
>>
>> The "smarter" the card the dumber the results tend to be when ZFS is in
>> use, in my experience :)
> Yes, I'm well aware of it ;)
>
> Thanks!

Incidentally actually managing to saturate a 9211 (which comes up as a
"SAS-2008") on the mps driver is non-trivial.  If you THINK you are,
make sure you really are.  If you really are and have a second PCIe slot
with the requisite lanes available that's probably the next thing to do.

Yes, it's possible (especially with SSDs) to saturate these but not
easy.  You can get the 2xxx series cards in 16-port versions (4 x 4)
which leaves you three ports that can each be attached to an expander. 
The limiting factor usually winds up being the configuration of said
plugs and the shelves for the drives themselves (2i/2e configurations
work great if you're using expansion chassis with an expander in them,
by the way.)

I have a number of the Intel 6-port expanders in production servers and
have never have had any trouble with them; they just plain work.  The
only note I'll make on the expanders (and these host cards) is to make
sure you have sufficient airflow in the chassis; they're passive
heat-sink cooled so airflow matters.  In any sort of proper server
chassis this is probably not going to be an issue but if you're going
for something that doesn't sound like an old 707 jet on take-off you
need to pay attention to make sure there's sufficient cooling airflow.

I tend to avoid using the motherboard SATA ports; I've had my share of
"fun" with them over time, especially if I need to hot-swap a drive
(e.g. a failed member of the root mirror.)  You only need to have a
machine crash on you unnecessarily in this fashion once to be disabused
of that sort of insanity in the future.

-- 
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

[-- Attachment #2 --]
0	*H
010
	`He0	*H
_0[0C)0
	*H
010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA0
150421022159Z
200419022159Z0Z10	UUS10UFlorida10U
Cuda Systems LLC10UKarl Denninger (OCSP)0"0
	*H
0
X@vkY
Tq/vE]5#֯MX\8LJ/V?5Da+
sJc*/r{ȼnS+w")ąZ^DtdCOZ ~7Q '@a#ijc۴oZdB&!Ӝ-<	?HN5y
5}F|ef゘"Vلio74zn">a1qWuɖbFeGE&3(KhixG3!#e_XƬϜ/,$+;4y'Bz<qT9_?rRUpn5
Jn&Rx/p Jyel*pN8/#9u/YPEC)TY>~/˘N[vyiDKˉ,^" ?$T8v&K%z8C @?K{9f`+@,|Mbia007++0)0'+0http://cudasystems.net:88880	U00	`HB0U0,	`HB
OpenSSL Generated Certificate0U-h\Ff Y0U#0$q}ݽʒm50U0karl@denninger.net0
	*H
Owbabɺx&Uk[(Oj!%pMQ0I!#QH}.>~2&D}<wm_>V6v]f>=Nn+8;q wfΰ/RLyUG#b}n!Dր_up|_ǰc/%ۥ
nN8:d;-UJd/m1~VނיnN I˾$tF1&}|?q?\đXԑ&\4V<lKۮ3%Am_(q-(cAeGX)f}-˥6cv~Kg8m~v;|9:-iAPқ6ېn-.)<[$KJtt/L4ᖣ^Cmu4vb{+BG$M0c\[MR|0FԸP&78"4p#}DZ9;V9#>Sw"[UP7100010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA)0
	`HeM0	*H
	1	*H
0	*H
	1
160822164635Z0O	*H
	1B@,3o%Ix9߷r.>F+
\`ϫǨ`Wo! 'q"CMѯV0l	*H
	1_0]0	`He*0	`He0
*H
0*H
0
*H
@0+0
*H
(0	+710010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA)0*H
	1010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems LLC CA1"0 	*H
	Cuda Systems LLC CA)0
	*H
8AS;VIeQJ$ZE͏ٍ;T
*5耽ozPU~46$cmq8ˬ٥&aIXAi9>WψHW
[?/~O>ھH9+cpRV.wV81_¤Z"sU7ZiT$aX&|Z(c i~8:	sdaf*mLrD 
8ewAoZOH3;ZBK	u6mE3Eb7TAAE]*v
QD\	r-,/NІ@+wH{
T61^6#YTCԟRRb)601N"VB9"hbqz	3fߥֆ<n*G#6/ѿ
WbЉJ'q
home | help

Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?580a6ea8-9c84-3e99-5a7d-7b2434c84e67>