Date: Fri, 14 Dec 2018 00:13:35 +0100 From: Peter Eriksson <peter@ifm.liu.se> To: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Suggestion for hardware for ZFS fileserver Message-ID: <2CB9CF77-DBC4-4452-8FC1-0A302884E71B@ifm.liu.se> In-Reply-To: <CAEW%2BogZnWC07OCSuzO7E4TeYGr1E9BARKSKEh9ELCL9Zc4YY3w@mail.gmail.com> References: <CAEW%2BogZnWC07OCSuzO7E4TeYGr1E9BARKSKEh9ELCL9Zc4YY3w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I can give you the specs for the servers we use here for our =
FreeBSD-based fileservers - which have been working really well for us =
serving Home directors
Servers:
Dell PowerEdge R730xd
Dell PowerEdge R740xd
HP ProLiant DL380 G9
CPUs
Intel Xeon E5-2620 v4 @ 2.10GHz
Intel Xeon E5-2623 v4 @ 2.60GHz
Intel Xeon E5-2624 v4 @ 2.60GHz
l:
Intel Xeon Silver 4110 CPU @ 2.10GHz
Intel Xeon Silver 4116 CPU @ 2.10GHz
RAM:
256GB RAM
HBA:
LSI SAS3008 (Dell HBA330 Mini, Dell Storage Cntlr.12GB-SASHBA)
HP Smart HBA H241 (configured in =E2=80=9Cdumb=E2=80=9D mode)
Network:
Intel X710 2 & 4-port 10Gb Ethernet adapters=20
(Dell branded ones & HPE Ethernet 10Gb 2-port 562SFP+ Adapter)
We had some trouble getting this to behave well in the beginning so we =
had to disable TSO via the =E2=80=9C-tso=E2=80=9D option to ifconfig =
causing us to lose a bit performance but since we=E2=80=99re nowhere =
near saturating the 10Gb ethernet links anyway=E2=80=A6 If HP/Dell had =
some other 10G ethernet controllers we might have considered switching, =
but it works =E2=80=9Cwell enough=E2=80=9D for now. Hopefully eventually =
Intel will find and fix the last TSO-related firmware bugs sometime...
Disks:
10GB HGST HUH721010AL4200 (Dell & HP OEM 7200 rpm disks)
(Requires a low-level reformat to get to 4096-byte sectors as they, =
when delivered from Dell, have a larger sector size not compatible with =
the SAS3008 controller with FreeBSD)
+ some SSD:s for SLOG and L2ARC.
OS:
FreeBSD 11.2-RELEASE-p6
(We discussed using FreeNAS at the beginning but went the =E2=80=9Cpure=E2=
=80=9D route since we wrote quite a bit of our own management software =
anyway)
Filesystems setup:
ZFS with RAID-Z2 (14 10TB in 4+2 RAID-Z2 config with 2 spares), SSDs for =
LOG and L2ARC (Bothe SATA SSDs and M.2 PCIe ones).
ZFS with Mirror for boot disks.
Sharing software:
SMB (SMB v2 or higher only)
Samba 4.7.6 (we test 4.9.3 right now - but we=E2=80=99re seeing =
problems with anything later that 4.7.6 currently).
NFS (NFSv4 only, Kerberos/GSS authentication)
More or less the only thing we=E2=80=99ve tuned for NFS so far is:
nfsuserd_flags=3D"-manage-gids -domain OURDOMAIN -usertimeout 10 =
-usermax 100000 16=E2=80=9D
As more clients start using NFS I assume we will have to adjust other =
stuff too.. Suggestions are welcome :-)
SFTP
We are servicing around 500-600 concurrent SMB users per server, and =
1700 users on one (but lighter load), just a few NFS clients right now =
but more are connecting every day, some SFTP. System joined to our =
Windows AD (Windows Server 2016) system.
SAS Expanders:=20
We only use the built-in disk slots on the Dell servers. We use the HP =
D6020 external SAS disk shelfs (70 drives per shelf) on the HP servers.
We talked about building our own servers but decided to go the =E2=80=9Cve=
ndor route=E2=80=9D for warranties and stuff. Costs a bit more money =
but=E2=80=A6.
- Peter
+ AdapterOn 13 Dec 2018, at 22:59, Sami Halabi <sodynet1@gmail.com> =
wrote:
>=20
>=20
> Hi,
> I'm planning on building a machine for fileserver using ZFS (still
> considering vanilla FreeBSD VS FreeNAS).
>=20
> I would like to get recommendation for good and working hardware.
> I have 2 setups I'm interested in getting your help for each.
> 1. Heavy load machine with 10G connection.
> 2. Light machine with 1G connection.
>=20
> Both setups will have up to 22 SAS hard drives 2T each + 2 SSD for =
L2ARC.
> For both setups I would appreciate your help in:
> 1. For raidz2 best strip of vdevs for maximum capacity with reasonable
> performance.
> 2. Hardware recommendations in terms of:
> CPU
> MEMORY
> HBA
> Sas expander for future connecting Jbod
> Server (aka my own build VS brand one like HP/Dell.. Etc).
>=20
> Thanks in advance,
> Sami
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2CB9CF77-DBC4-4452-8FC1-0A302884E71B>