From owner-freebsd-fs@freebsd.org Thu Dec 13 23:44:38 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21D03133878F for ; Thu, 13 Dec 2018 23:44:38 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from mailout.ifm.liu.se (mailout.ifm.liu.se [130.236.160.13]) by mx1.freebsd.org (Postfix) with ESMTP id BFD7F77A42 for ; Thu, 13 Dec 2018 23:44:36 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from [192.168.1.79] (h-99-23.A785.priv.bahnhof.se [158.174.99.23]) (authenticated bits=0) by mail.ifm.liu.se (8.15.2/8.14.4) with ESMTPSA id wBDNDfA0029234 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 14 Dec 2018 00:13:43 +0100 (CET) From: Peter Eriksson Message-Id: <2CB9CF77-DBC4-4452-8FC1-0A302884E71B@ifm.liu.se> Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: Suggestion for hardware for ZFS fileserver Date: Fri, 14 Dec 2018 00:13:35 +0100 In-Reply-To: To: "freebsd-fs@freebsd.org" References: X-Mailer: Apple Mail (2.3445.101.1) X-IFMLiUSE-MailScanner-Information: Please contact postmaster@ifm.liu.se more information X-IFMLiUSE-MailScanner-ID: wBDNDfA0029234 X-IFMLiUSE-MailScanner: Found to be clean X-IFMLiUSE-MailScanner-From: peter@ifm.liu.se X-Spam-Status: No X-Rspamd-Queue-Id: BFD7F77A42 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of peter@ifm.liu.se designates 130.236.160.13 as permitted sender) smtp.mailfrom=peter@ifm.liu.se X-Spamd-Result: default: False [0.46 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-0.00)[country: SE(-0.02)]; NEURAL_SPAM_MEDIUM(0.46)[0.458,0]; NEURAL_SPAM_SHORT(0.42)[0.416,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[13.160.236.130.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; MX_GOOD(-0.01)[e-mailfilter04.sunet.se,e-mailfilter03.sunet.se]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Dec 2018 23:44:38 -0000 I can give you the specs for the servers we use here for our = FreeBSD-based fileservers - which have been working really well for us = serving Home directors Servers: Dell PowerEdge R730xd Dell PowerEdge R740xd HP ProLiant DL380 G9 CPUs Intel Xeon E5-2620 v4 @ 2.10GHz Intel Xeon E5-2623 v4 @ 2.60GHz Intel Xeon E5-2624 v4 @ 2.60GHz l: Intel Xeon Silver 4110 CPU @ 2.10GHz Intel Xeon Silver 4116 CPU @ 2.10GHz RAM: 256GB RAM HBA: LSI SAS3008 (Dell HBA330 Mini, Dell Storage Cntlr.12GB-SASHBA) HP Smart HBA H241 (configured in =E2=80=9Cdumb=E2=80=9D mode) Network: Intel X710 2 & 4-port 10Gb Ethernet adapters=20 (Dell branded ones & HPE Ethernet 10Gb 2-port 562SFP+ Adapter) We had some trouble getting this to behave well in the beginning so we = had to disable TSO via the =E2=80=9C-tso=E2=80=9D option to ifconfig = causing us to lose a bit performance but since we=E2=80=99re nowhere = near saturating the 10Gb ethernet links anyway=E2=80=A6 If HP/Dell had = some other 10G ethernet controllers we might have considered switching, = but it works =E2=80=9Cwell enough=E2=80=9D for now. Hopefully eventually = Intel will find and fix the last TSO-related firmware bugs sometime... Disks: 10GB HGST HUH721010AL4200 (Dell & HP OEM 7200 rpm disks) (Requires a low-level reformat to get to 4096-byte sectors as they, = when delivered from Dell, have a larger sector size not compatible with = the SAS3008 controller with FreeBSD) + some SSD:s for SLOG and L2ARC. OS: FreeBSD 11.2-RELEASE-p6 (We discussed using FreeNAS at the beginning but went the =E2=80=9Cpure=E2= =80=9D route since we wrote quite a bit of our own management software = anyway) Filesystems setup: ZFS with RAID-Z2 (14 10TB in 4+2 RAID-Z2 config with 2 spares), SSDs for = LOG and L2ARC (Bothe SATA SSDs and M.2 PCIe ones). ZFS with Mirror for boot disks. Sharing software: SMB (SMB v2 or higher only) Samba 4.7.6 (we test 4.9.3 right now - but we=E2=80=99re seeing = problems with anything later that 4.7.6 currently). NFS (NFSv4 only, Kerberos/GSS authentication) More or less the only thing we=E2=80=99ve tuned for NFS so far is: nfsuserd_flags=3D"-manage-gids -domain OURDOMAIN -usertimeout 10 = -usermax 100000 16=E2=80=9D As more clients start using NFS I assume we will have to adjust other = stuff too.. Suggestions are welcome :-) SFTP We are servicing around 500-600 concurrent SMB users per server, and = 1700 users on one (but lighter load), just a few NFS clients right now = but more are connecting every day, some SFTP. System joined to our = Windows AD (Windows Server 2016) system. SAS Expanders:=20 We only use the built-in disk slots on the Dell servers. We use the HP = D6020 external SAS disk shelfs (70 drives per shelf) on the HP servers. We talked about building our own servers but decided to go the =E2=80=9Cve= ndor route=E2=80=9D for warranties and stuff. Costs a bit more money = but=E2=80=A6. - Peter + AdapterOn 13 Dec 2018, at 22:59, Sami Halabi = wrote: >=20 >=20 > Hi, > I'm planning on building a machine for fileserver using ZFS (still > considering vanilla FreeBSD VS FreeNAS). >=20 > I would like to get recommendation for good and working hardware. > I have 2 setups I'm interested in getting your help for each. > 1. Heavy load machine with 10G connection. > 2. Light machine with 1G connection. >=20 > Both setups will have up to 22 SAS hard drives 2T each + 2 SSD for = L2ARC. > For both setups I would appreciate your help in: > 1. For raidz2 best strip of vdevs for maximum capacity with reasonable > performance. > 2. Hardware recommendations in terms of: > CPU > MEMORY > HBA > Sas expander for future connecting Jbod > Server (aka my own build VS brand one like HP/Dell.. Etc). >=20 > Thanks in advance, > Sami > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"