From owner-freebsd-fs@freebsd.org Thu Dec 20 23:03:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75AE61357269 for ; Thu, 20 Dec 2018 23:03:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670067.outbound.protection.outlook.com [40.107.67.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 399AE727D0 for ; Thu, 20 Dec 2018 23:02:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM (10.169.142.146) by YQBPR01MB0497.CANPRD01.PROD.OUTLOOK.COM (10.169.143.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1446.18; Thu, 20 Dec 2018 23:02:58 +0000 Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c]) by YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c%8]) with mapi id 15.20.1446.022; Thu, 20 Dec 2018 23:02:58 +0000 From: Rick Macklem To: Peter Eriksson , "freebsd-fs@freebsd.org" Subject: Re: Suggestion for hardware for ZFS fileserver Thread-Topic: Suggestion for hardware for ZFS fileserver Thread-Index: AQHUkzl4CLTPDjfxZEms4Ul8AJXDZ6WIQ5Vm Date: Thu, 20 Dec 2018 23:02:58 +0000 Message-ID: References: , <2CB9CF77-DBC4-4452-8FC1-0A302884E71B@ifm.liu.se> In-Reply-To: <2CB9CF77-DBC4-4452-8FC1-0A302884E71B@ifm.liu.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR01MB0497; 6:hBGoSfpE+xJAC/NAuU6+Qt2GkWvV6ySDyX4v4XoHxleRvdkyYG9recJis3G09TcIrO3rwzrAaf5v7CvV2nJrQpWd87LhAT4xs+9/c7vQOO1E7/JpB3JsNF8r2H/nn40dtT7tUuigBI0E8lQB8g1ielsmHp3A3c4913caNnaW7tZlbIEXlk6PaY5BAaU3ZSbQ0G88UN7/dgVDR9DNb1pfu/aF3dpOxMY1GD6fOEK5pLYwrn03k3K/2rIKC077OrzJw151rPAvcKCf6sVAJ3aVo+8ZR5vbTXgwGMtfSiyXltLFkR0jJr0mlsjRLTEcfrv55KcmjNNd5Gb59Qr+CCYxv1gcTAmzgsm/G4kgol0PzEilRFNIwlqTkUpdteokbDD+sxseDJmTgJTyKFzk3fVk4vjUcSVCS6oNxP70Z6eSm9/h96DCIo/t8c07KpGVklwVK38wHcwhtnWFanbKM40UGA==; 5:6I05Xxlbr1CrBibwGy7OrYsd8dFZHXl4G53sqn+ceGvSAd4VaphQ+63RzyWLmqd0yVhvNuvZN5HxFyMWuCQ94yi/WIY96p7rjYUf+6BkgewYCbpfvRRcjhnozjJzmeaJJ8ZYZVhS1kS4hUevx4QhPHUa4j97Otpaya2K72bAseU=; 7:qHD5MhXxPkjeOk77+qUfDsvY6H+RUslhZ2X8SlzCjmAhiBJjem/JEw61RomfVijVBXkxlkgaGejTrHC5uyRwdJAYmMuDD/Prehb0OCxVdNQdd0+WUtWHdjoRHTZHKJdhKgEpKgdN2XftBGbRqWtIMA== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 854fc07a-f022-40d0-ca9a-08d666cf48b6 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YQBPR01MB0497; x-ms-traffictypediagnostic: YQBPR01MB0497: x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(3230021)(999002)(5005026)(6040522)(2401047)(8121501046)(93006095)(93001095)(3231475)(944501520)(52105112)(10201501046)(3002001)(149066)(150057)(6041310)(20161123562045)(20161123558120)(20161123564045)(20161123560045)(201703131423095)(201703031522075)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:YQBPR01MB0497; BCL:0; PCL:0; RULEID:; SRVR:YQBPR01MB0497; x-forefront-prvs: 0892FA9A88 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(346002)(376002)(136003)(396003)(366004)(189003)(199004)(9686003)(25786009)(55016002)(106356001)(105586002)(6246003)(6436002)(316002)(786003)(296002)(8936002)(81156014)(81166006)(8676002)(256004)(11346002)(186003)(476003)(486006)(86362001)(71200400001)(7696005)(76176011)(71190400001)(446003)(74482002)(6506007)(53936002)(229853002)(102836004)(46003)(97736004)(5660300001)(74316002)(33656002)(2501003)(305945005)(14454004)(478600001)(2906002)(110136005)(99286004)(68736007); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR01MB0497; H:YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: z5p1VClxJbeuWQ7eQQye1E2bU6BIBDSnibXmHepszaaP73KhaMx80/QG8K9JZBFkOHCjTskaAuYxw8fCoPCntnWh7ICXzr2J0FNtn+EUJ6tLMzTXTQ3jD6lGJZF40vIlfzHqBRFRUsVQx3I4eZdgF+Uwunztfqotg+yfYZWW3T3kKtBqcRXzpNidjynEBUOKKabOcXiT/tNAsacXCtht2SiKJd23dcebM88O/wL4ge1B2+L3u1bEp4SQp6gV9JxytO4lpNOeXrTiOKKg35jF9mEElYB+YRB1wx/KlzQjLYGq8XM0ReEYhUol/oZxyucL spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 854fc07a-f022-40d0-ca9a-08d666cf48b6 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2018 23:02:58.2675 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR01MB0497 X-Rspamd-Queue-Id: 399AE727D0 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.67 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.02 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[67.67.107.40.list.dnswl.org : 127.0.3.0]; NEURAL_HAM_SHORT(-0.87)[-0.865,0]; IP_SCORE(-0.84)[ipnet: 40.64.0.0/10(-2.08), asn: 8075(-2.04), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 23:03:02 -0000 Peter Eriksson wrote: >I can give you the specs for the servers we use here for our FreeBSD-based= >fileservers - which have been working really well for us serving Home dir= ectors [good stuff snipped] >NFS (NFSv4 only, Kerberos/GSS authentication) > More or less the only thing we=92ve tuned for NFS so far is: > nfsuserd_flags=3D"-manage-gids -domain OURDOMAIN -usertimeout 10 -use= rmax >100000 16=94 > As more clients start using NFS I assume we will have to adjust other st= uff too.. >Suggestions are welcome :-) I am not the best person to suggest values for these tunables because I nev= e run an NFS server under heavy load, but at least I can mention possible val= ues. (I'll assume a 64bit arch with more than a few Gbytes of RAM that can be de= dicated to serving NFS.) For NFSv3 and NFSv4.0 clients: - The DRC (which improves correctness and not performance) is enabled for T= CP. (Some NFS server vendors only use the DRC for UDP.) This can result in si= gnificant CPU overheads and RPC RTT delays. You have two alternatives: 1 - set vfs.nfsd.cachetcp =3D 0 to disable use of the DRC for TCP. 2 - Increase vfs.nfsd.tcphighwater to something like 100000. You can also decrease vfs.nfsd.tcpcachetimeo, but that reduces the effectiveness of the DRC for TCP, since the timeout needs to be larg= er than the longest time it is likely for a client to take to do a TCP = reconnect and retry RPCs after a server crash or network partitioning. For NFSv4.1, you don't need to do the above, because it uses something ca= lled sessions instead of the DRC. For NFSv4.1 clients you will, however, want = to increase vfs.nfsd.sessionhashsize to something like 1000. For NFSv4.0 and NFSv4.1 clients, you will want to increase the state relate= d stuff to something like: vfs.nfsd.fhhashsize=3D10000 vfs.nfsd.statehashsize=3D100 vfs.nfsd.clienthashsize=3D1000 (or 1/10th of the number of client mounts up= to something like 10000) As you can see, it depends upon which NFS version your clients are using. ("nfsstat -m" should tell you that on both FreeBSD and Linux clients.) If your exported file systems are UFS, you might consider increasing your b= uffer cache size, but not for ZFS exports. Most/all of these need to be set in your /boot/loader.conf, since they need to be statically configured. vfs.nfsd.cachetcp can be cleared at any time, = I think? For your case of mostly non-NFS usage, it is hard to say if/when you want t= o do the above, but these changes probably won't hurt when you have 256Gbytes of RAM. Good luck with it, rick [more good stuff snipped]