Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 24 Dec 2018 18:02:09 -0600
From:      Karl Denninger <karl@denninger.net>
To:        freebsd-fs@freebsd.org
Subject:   Re: Suggestion for hardware for ZFS fileserver
Message-ID:  <1d76f92c-6665-81ef-1b94-dc1b4b8925d1@denninger.net>
In-Reply-To: <CACpH0Md5y%2BSFTHbRL=OzP9joG60gKStOkoK3GrZqTYHO97k_FA@mail.gmail.com>
References:  <CAEW%2BogZnWC07OCSuzO7E4TeYGr1E9BARKSKEh9ELCL9Zc4YY3w@mail.gmail.com> <C839431D-628C-4C73-8285-2360FE6FFE88@gmail.com> <CAEW%2BogYWKPL5jLW2H_UWEsCOiz=8fzFcSJ9S5k8k7FXMQjywsw@mail.gmail.com> <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> <YQBPR01MB038805DBCCE94383219306E1DDB80@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM> <20181223113031.00005150@Leidinger.net> <YQBPR01MB038868AC3D6BAC5C6FB40C9CDDBB0@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM> <CACpH0Md5y%2BSFTHbRL=OzP9joG60gKStOkoK3GrZqTYHO97k_FA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help 

[-- Attachment #1 --]
On 12/24/2018 17:13, Zaphod Beeblebrox wrote:
> [ regarding ZFS hardware thread ]
>
> There's another type of server --- the "ghetto" or home storage serer.  For
> this server, I like to optimize for not loosing data, not for uptime.
>
> Going back a few years, there were consumer motherboards with 10 or 12 SATA
> onboard.  Mostly, this was at the change of technologies ... so you had
> some of one kind of port and some of another.  Used SAS HBAs are another
> option ... but they have a caviat: many SATA drives will eventually reject
> them under load.  Good SATA drives won't (but again, we're talking a ghetto
> system).  If you're taking WD reds (and not, say, seagate barracudas) ...
> these work well.  On the seagates, however, I've had drives repeatedly fail
> ... only to go on working fine in a workstation with a SATA controller.

I've run "ghetto mode" fileservers with the LSI adapters in IT mode
(that always just seem to work) with one of their SFP ports connected to
a SAS expander, and then fanned THAT out to SATA drives.  The only
constraint is that you can run into problems booting from an expander,
so don't -- use the ports on the HBA (or even the motherboard) for the
boot drives.

Never had a problem doing this with HGST drives, Intel SSDs and most
others.  The Seagates I've had fail actually physically failed; they
didn't throw a protocol hissy fit on the bus.  I don't buy Seagates any
more as I've had too many die out-of-warranty for my taste.  They work
fine with WD drives too.  Never had one of the drives that failed cause
a cascade detach event either.  The last few years (five or so) for
spinning rust HGST seems to sell the most-reliable stuff in my
experience but YMMV on that.

Those adapters and expanders are cheap these days.  The expanders used
to be expensive, but not any more -- there's a ton of them around on the
secondary market for very little money (not much more than the LSI
cards.)  Their only downside is they run hot so you need good fan
coverage in the case.

Older SuperMicro boards (X8DTL- series) that will take the 5600-series
Westmere Xeon processors can be had for almost nothing (note you have to
have the latest BIOS in them, which can be flashed, to run the Westmere
processors), and the CPUs are a literal $25.  The only "gotcha" is you
need ECC memory, but if you can find it at a decent price used you're
golden.  I would NOT run a ZFS filesystem without ECC memory; the risk
of undetected data corruption that you don't catch for months or even
years is material and if it happens you WILL cry since your backup
coverage may have expired at that point.

> Last point.  RAID-Z2 at a minimum.  I could even see the argument for Z3.
> My current array is 16x 4T drvies in to 8 disk Z2 plexes.  Of that, one
> plex is all WD Red on a SAS controller ... and the other (older) plex is
> still largely cheap drives on SATA.  Right now, drives below 4T are
> artificially expensive.  Drives right up to 10T are about the same price
> per G (at least here in Canada).

Yes, RaidZ2 is a (very) good idea.

Oh, and keep a spare power supply in the building, plus a spare disk
adapter and expander.  Hell, for $25 you can keep a spare CPU in the
building! :)  A box like this can trivially saturate more than one GigE
port (the SuperMicros typically have two on board plus the dedicated
management port) all day long.  While there are failure points in such a
ghetto system config they're coverable (with a reboot after swapping the
dead item) at very low cost in backstock in the building.

I've had a couple of these in 24x7 field use for coming up on ten years
now -- they're still running and have been dead-balls reliable --
duplicating them today is disgustingly inexpensive.  If there's one
downside its that modern CPUs are FAR more power-efficient per-cycle
than the older Xeons (they're much faster too, but that's not the
limiting factor for a fileserver.)

-- 
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

[-- Attachment #2 --]
0	*H
010
	`He0	*H

00H^Ōc!5
H0
	*H
010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems CA1!0UCuda Systems LLC 2017 CA0
170817164217Z
270815164217Z0{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CA0"0
	*H
0
h-5B>[;olӴ0~͎O9}9Ye*$g!ukvʶLzN`jL>MD'7U45CB+kY`bd~b*c3Ny-78ju]9HeuέsӬDؽmgwER?&UURj'}9nWD i`XcbGz\gG=u%\Oi13ߝ4
K44pYQr]Ie/r0+eEޝݖ0C15Mݚ@JSZ(zȏNTa(25DD5.l<g[[ZarQQ%Buȴ~~`IohRbʳڟu2MS8EdFUClCMaѳ!}ș+2k/bųE,n当ꖛ\(8WV8	d]b	yXw	܊:I39
00U]^§Q\ӎ0U#0T039N0b010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems CA1!0UCuda Systems LLC 2017 CA	@Ui0U00U0
	*H
:P U!>vJnio-#ן]WyujǑR̀Q
nƇ!GѦFg\yLxgw=OPycehf[}ܷ['4ڝ\[p6\o.B&JF"ZC{;*o*mcCcLY߾`
t*S!񫶭(`]DHP5A~/NPp6=mhk밣'doA$86hm5ӚS@jެEgl
)0JG`%k35PaC?σ
׳HEt}!P㏏%*BxbQwaKG$6h¦Mve;[o-Iی&
I,Tcߎ#t wPA@l0P+KXBպT	zGv;NcI3&JĬUPNa?/%W6G۟N000k#Xd\=0
	*H
0{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CA0
170817212120Z
220816212120Z0W10	UUS10UFlorida10U
Cuda Systems LLC10Ukarl@denninger.net0"0
	*H
0
T[I-ΆϏdn;Å@שy.us~_ZG%<MYd\gvfnsa1'6Egyjs"C [{~_KPn+<*pv#Q+H/7[-vqDV^U>f%GX)H.|l`M(Cr>е͇6#odc"YljҦln8@5SA0&ۖ"OGj?UDWZ5	dDB7k-)9Izs-JAv
J6L$Ն1SmY.Lqw*SH;EF'DĦH]MOgQQ|Mٙג2Z9y@y]}6ٽeY9Y2xˆ$T=eCǺǵbn֛{j|@LLt1[Dk5:$=	`	M00<+00.0,+0 http://ocsp.cudasystems.net:88880	U00	`HB0U0U%0++03	`HB
&$OpenSSL Generated Client Certificate0U%՞V=؁;bzQ0U#0]^§Q\ӎϡ010	UUS10UFlorida10U	Niceville10U
Cuda Systems LLC10UCuda Systems CA1!0UCuda Systems LLC 2017 CAH^Ōc!5
H0U0karl@denninger.net0
	*H
۠A0-j%--$%g2#ޡ1^>{K+uGEv1ş7Af&b&O;.;A5*U)ND2bF|\=]<sˋL!wrw٧>YMÄ3\mWR hSv!_zvl? 3_ xU%\^#O*Gk̍YI_&Fꊛ@&1n”} ͬ:{hTP3B.;bU8:Z=^Gw8!k-@xE@i,+'Iᐚ:fhztX7/(hY` O.1}a`%RW^akǂpCAufgDixUTЩ/7}%=jnVZvcF<M=
2^GKH5魉
_O4ެByʈySkw=5@h.0z>
W1000{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CAk#Xd\=0
	`HeE0	*H
	1	*H
0	*H
	1
181225000209Z0O	*H
	1B@Š>eM!ߪ}kcjU1ۚ@4
[Re&mʑaѧ3x0l	*H
	1_0]0	`He*0	`He0
*H
0*H
0
*H
@0+0
*H
(0	+7100{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CAk#Xd\=0*H
	10{10	UUS10UFlorida10U
Cuda Systems LLC10UCuda Systems CA1%0#UCuda Systems LLC 2017 Int CAk#Xd\=0
	*H
V-Z4}gՕBC#gd	J$D
=tHhyQ()A&cmuO"mfD߻G=SuG4xd(vrdRBhJ4SLOHLئS/R>cgُFlXl(eg;HMA$ŖWD*xmku*#A']*Ps=
2oyf2@sY_BcM#ߋto~Ag7Ɉ.=\Z !-NM1R-cAg)#byz'%G`^YVjdwB4)háIO4}2t!rA$:Gs'[~f5^]q6Ŗ/T`igy~KM=Z~'iEXR[%/OZz< _^Bh̲Vkʊ7ۂ+5GO~*R

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1d76f92c-6665-81ef-1b94-dc1b4b8925d1>