Date: Mon, 24 Dec 2018 18:13:47 -0500 From: Zaphod Beeblebrox <zbeeble@gmail.com> To: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: Suggestion for hardware for ZFS fileserver Message-ID: <CACpH0Md5y%2BSFTHbRL=OzP9joG60gKStOkoK3GrZqTYHO97k_FA@mail.gmail.com> In-Reply-To: <YQBPR01MB038868AC3D6BAC5C6FB40C9CDDBB0@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM> References: <CAEW%2BogZnWC07OCSuzO7E4TeYGr1E9BARKSKEh9ELCL9Zc4YY3w@mail.gmail.com> <C839431D-628C-4C73-8285-2360FE6FFE88@gmail.com> <CAEW%2BogYWKPL5jLW2H_UWEsCOiz=8fzFcSJ9S5k8k7FXMQjywsw@mail.gmail.com> <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> <YQBPR01MB038805DBCCE94383219306E1DDB80@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM> <20181223113031.00005150@Leidinger.net> <YQBPR01MB038868AC3D6BAC5C6FB40C9CDDBB0@YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
[ regarding ZFS hardware thread ] There's another type of server --- the "ghetto" or home storage serer. For this server, I like to optimize for not loosing data, not for uptime. Going back a few years, there were consumer motherboards with 10 or 12 SATA onboard. Mostly, this was at the change of technologies ... so you had some of one kind of port and some of another. Used SAS HBAs are another option ... but they have a caviat: many SATA drives will eventually reject them under load. Good SATA drives won't (but again, we're talking a ghetto system). If you're taking WD reds (and not, say, seagate barracudas) ... these work well. On the seagates, however, I've had drives repeatedly fail ... only to go on working fine in a workstation with a SATA controller. Another source of ghetto ports are 4 port SATA controllers (~$50). Cheaper than SAS, but taking more PCIe ports. I've had a 16 drive system running on 10 motherboard ports and 2x 4 port SATA PCIe. Watch out for the 8 port ones --- the only 8 port SATA I've come accross are using a SATA expander for the last 5 ports !?!. Not a good value. The motherboards with 10 or so ports tend to be high end boards, so a ghetto system will want to get those on eBay used. ASUS ROG and ASUS TUF boards of the 4-ish generation (Intel) or the bulldozer (AMD) are configured this way. Having 10 + 8 gives you enough to have 2 SSD drives for cache/log or 2 boot drives. Get as much RAM as you can stuff in. 32G is often that number. You'll get a good GigE on the motherboard, but 10GE will need to be an addin card. You'll likely be getting short on slots. The next iteration of this hardware is the only one that will have a chance at ECC RAM. The cheapest threadripper motherboards with the basic threadripper CPU will be in this range used soon ... giving you upto 128G of ECC ram and many, many more PCIe lanes. Might make me reverse my use of SAS controllers and push me to all NAS drives. To be clear, with SATA controllers, cheap SATA drives seem to last fine. But the SAS controllers seem to spar with the cheap drive electronics (the drives smart test fine, but the electronics disconnect --- and if the SAS controller is swapped for a SATA one, this goes away). Last point. RAID-Z2 at a minimum. I could even see the argument for Z3. My current array is 16x 4T drvies in to 8 disk Z2 plexes. Of that, one plex is all WD Red on a SAS controller ... and the other (older) plex is still largely cheap drives on SATA. Right now, drives below 4T are artificially expensive. Drives right up to 10T are about the same price per G (at least here in Canada). This server (obviously) has many single points of failure. Watch Netflix when this happens :). It doesn't fail often (I haven't had more than a few hours downtime in 10-or-so-years) and the basic system can be had very cheaply. You don't need a recent CPU to fill GigE. I'm about to upgrade mine to 10GE ... but I realize that I may only get something less than 10GE from it ... some fraction... but more than GigE nonetheless...
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACpH0Md5y%2BSFTHbRL=OzP9joG60gKStOkoK3GrZqTYHO97k_FA>