From owner-freebsd-fs@freebsd.org Mon Dec 24 23:13:59 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B76531351ECE for ; Mon, 24 Dec 2018 23:13:59 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55579701D4 for ; Mon, 24 Dec 2018 23:13:58 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-io1-xd2f.google.com with SMTP id k7so9837985iob.6 for ; Mon, 24 Dec 2018 15:13:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=T39k9a+kJCNuVzMUMCrAIyH97mMg8qkcXnvFjpuX6JU=; b=GjS94Dv9sSCj9DIGLJSF3NWdD7DvRkd9aPy1LRqeGeH4FVm9QAakfpxzDz51G6vt97 u2tosbD/sNqKcj90tSW0TmM+y+LSMFT5VMS+L8U0F5KJMgC9+XilU4gVCQ/OUC6CkXov DzU3JCrchoIgqUUGLyeZRdpcklnXKd2SlOMIDW+bZOA9u35T7kI0mbUL5nRU9VnGW0qk n8O/g5MjWVTqMW11dCfLn3MckWe+I+7mPYgxFBQn8OGKQ6NixNdcIe3gFOYK9VyYh+Tr U8BRxHHPNLKZv/Mc/IYxslGCLbEw/5ZE+DgzbedGkBowoykb+BS+TmsAOz3qDzP4M1Np N3Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=T39k9a+kJCNuVzMUMCrAIyH97mMg8qkcXnvFjpuX6JU=; b=sb3h5WYwJmJpc3tzQYZAo6yMRYkr9jyJB97c7eONqWjGvheyi2urp+YwJyyu6FVQY7 8TwD6PpgAJEOtbiHMHF5P7mv9H8qr8/FD8pTKqJzJBuqf+ST6bBCTooTRPSIiQb4T+IS CI/xsHJyAebiE1UoS+XuQm8hMac8Ws1I64hb5mFbgVt9td6VJGDHPaXc7GOXkttQWmfz 10S2H3QadCkhepXJqqiIDrB2FSbUOvAey2lYicMc5YO9uhLP6afdiXqJQdEnh+tQeE/d Ipfc8AQiPEcG31DfAnGnLC5CjskRdJka9vrnj7Xq375Ns3w3Vjk4fr+sbiUuMCA+Jgdj QC/Q== X-Gm-Message-State: AJcUukcvOboYdkctF/AZuZNmaUhZSLqyfwtUKdc2tU3dW0q+tVqGD/cD EsO7sSFWuhFCfYoCYGpOxhB9mw16hD/Ejzxt6wV/uHo= X-Google-Smtp-Source: ALg8bN7Y3HWEYGrUuLSxrMNem7GMlFYhasvs9wf1HI6f4BAYJ8XuybO3/DDF0V9fEsW+3/pbrys30v+0VKRNGon1g5k= X-Received: by 2002:a6b:3806:: with SMTP id f6mr10536369ioa.194.1545693236812; Mon, 24 Dec 2018 15:13:56 -0800 (PST) MIME-Version: 1.0 References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> <20181223113031.00005150@Leidinger.net> In-Reply-To: From: Zaphod Beeblebrox Date: Mon, 24 Dec 2018 18:13:47 -0500 Message-ID: Subject: Re: Suggestion for hardware for ZFS fileserver To: freebsd-fs X-Rspamd-Queue-Id: 55579701D4 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=GjS94Dv9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2607:f8b0:4864:20::d2f as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-6.41 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[f.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-2.41)[ip: (-8.64), ipnet: 2607:f8b0::/32(-1.85), asn: 15169(-1.48), country: US(-0.08)]; NEURAL_HAM_SHORT(-0.99)[-0.985,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2018 23:14:00 -0000 [ regarding ZFS hardware thread ] There's another type of server --- the "ghetto" or home storage serer. For this server, I like to optimize for not loosing data, not for uptime. Going back a few years, there were consumer motherboards with 10 or 12 SATA onboard. Mostly, this was at the change of technologies ... so you had some of one kind of port and some of another. Used SAS HBAs are another option ... but they have a caviat: many SATA drives will eventually reject them under load. Good SATA drives won't (but again, we're talking a ghetto system). If you're taking WD reds (and not, say, seagate barracudas) ... these work well. On the seagates, however, I've had drives repeatedly fail ... only to go on working fine in a workstation with a SATA controller. Another source of ghetto ports are 4 port SATA controllers (~$50). Cheaper than SAS, but taking more PCIe ports. I've had a 16 drive system running on 10 motherboard ports and 2x 4 port SATA PCIe. Watch out for the 8 port ones --- the only 8 port SATA I've come accross are using a SATA expander for the last 5 ports !?!. Not a good value. The motherboards with 10 or so ports tend to be high end boards, so a ghetto system will want to get those on eBay used. ASUS ROG and ASUS TUF boards of the 4-ish generation (Intel) or the bulldozer (AMD) are configured this way. Having 10 + 8 gives you enough to have 2 SSD drives for cache/log or 2 boot drives. Get as much RAM as you can stuff in. 32G is often that number. You'll get a good GigE on the motherboard, but 10GE will need to be an addin card. You'll likely be getting short on slots. The next iteration of this hardware is the only one that will have a chance at ECC RAM. The cheapest threadripper motherboards with the basic threadripper CPU will be in this range used soon ... giving you upto 128G of ECC ram and many, many more PCIe lanes. Might make me reverse my use of SAS controllers and push me to all NAS drives. To be clear, with SATA controllers, cheap SATA drives seem to last fine. But the SAS controllers seem to spar with the cheap drive electronics (the drives smart test fine, but the electronics disconnect --- and if the SAS controller is swapped for a SATA one, this goes away). Last point. RAID-Z2 at a minimum. I could even see the argument for Z3. My current array is 16x 4T drvies in to 8 disk Z2 plexes. Of that, one plex is all WD Red on a SAS controller ... and the other (older) plex is still largely cheap drives on SATA. Right now, drives below 4T are artificially expensive. Drives right up to 10T are about the same price per G (at least here in Canada). This server (obviously) has many single points of failure. Watch Netflix when this happens :). It doesn't fail often (I haven't had more than a few hours downtime in 10-or-so-years) and the basic system can be had very cheaply. You don't need a recent CPU to fill GigE. I'm about to upgrade mine to 10GE ... but I realize that I may only get something less than 10GE from it ... some fraction... but more than GigE nonetheless...