From owner-freebsd-fs@freebsd.org Wed Mar 17 00:06:45 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1569E573131 for ; Wed, 17 Mar 2021 00:06:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670089.outbound.protection.outlook.com [40.107.67.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Vnc0qRNz3t4t; Wed, 17 Mar 2021 00:06:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mfyUJmofVacVpZyVnjICuLUaut3+vEsd8Qg+OK7y7a/vW4n97U0KrJBnIZXgsupVk7MAVGhshMIUJZcqCKagzTy/PlqLV8RU2VKU/nn3oRAAykFSLRec5BjP0pD86xxt1UCvIaK9cwXvy1rG2fO/Mg0kLFJ9u6RcdSrHO3e1GZaB0GzCok579isK2mlEcnCz0Gi9EGUQveO5NiTTfkbs88/V0QHAJvk7aHMTkIsogBhvcdCU6qgkE4TMUGRfRJvcpyt2aHw96zN+eIVzJBCdsgyFs11LqfXzFJWzk48K4fsLNsZ40VZCOQ5oRzO+kc9HY33fTp8pk3GTXFMMaDMwKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pHvjowwaioWXpHFZsKodIFjuLdc/7Wa3c3oCFpPgMhg=; b=jS65r+jZ5ZUB40dTTaXv5K0mV79Ej087/NBA046qdzcn5fqQSwqDR/F+OabWn1hbKU6uqKTjzOysgaPodiO9JtSIsbLCVtOI0xk3k7iYfQ6VjQGmZzcwiL5ROUNTBfvtfqFYag6SH9H5Wbovato76nYn8z0jeEHcsV2GWJGY05lPtn92FoKxOGCtvEJVPS9XNUrJYHh4e39T1oFAWf6/XoU1aUKcn50m75CfhazcqMtjDFLIhS8MFJpBsNZDR7OnsFHhFf+RHnl35FRiX5fjz04xe6YjCQDlvi2rOhV+jXd/d/Fa48qiVevClG21S0M2ihTy0QKKHgEMgTpfafaQ7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pHvjowwaioWXpHFZsKodIFjuLdc/7Wa3c3oCFpPgMhg=; b=Lakji1Z+1bFmjZOkqetVV7a0wje3V4yYaXNR0xk+6DSMXWo+H7rUexikCvxafv3d+SwMHsMIC9t/b0IiaTRHix6n7C28QPWDnqqX1RflBmDciGIXDsZ20Q9gFVg+r+ISnWt/vQP4ODEpcmDO1o+P0FKshf2P7vuLeWsQM+pL6b4EtPuuTa2zlRSOMxnYCzNzA1MElDnYLaoUlt8eVzTmzltEOtA/R8C+b6Nzgh7C21tkGgRQM0dE+Ujt1D82Uk+ihDmsYbedf1dieQXKHLK3UiTBXEwWHKzZh+j+w8LVf2HBeXlW5yFVu6rYOHTeG5o+h06iyqAtm/VkjMLdyK76/A== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB1764.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:e::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar 2021 00:06:42 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021 00:06:42 +0000 From: Rick Macklem To: Alan Somers , freebsd-fs Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtT Date: Wed, 17 Mar 2021 00:06:42 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0f2d188b-be5a-4991-4a75-08d8e8d88b71 x-ms-traffictypediagnostic: YQBPR0101MB1764: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5797; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: eLNYg7dSW95q2AkE5IjWGzxnkzfOK6PB+mNSjViMvybs5dbNUH/NrdGSEXSaBugGPp79PBH9cLki8R0DemuHxXAM84S5juhecJL8MZrvTsZsrx7i8WGL1cccUVzujpTSDUqQjsMKbTp5gZo4Mb6PX3vsDBmjCg1TF0lyTfgTlrgh1YnOC0ihCqMbSWULZ58PFiFFuNLtt0TG89LnEtfEkivqxCXN6hqsh6v8MVSvQux4olcKglmO4uJGTugKXQvJNRAHs9KuzMGIyBeRDTs0ZQrqbzXAy9Fofml4HAy2ponbqRhVHIH7/kPcgizK2WrBd4XFWDHZ8P9NFKrZQwg3Pwz1ATmQS+AULkjEwzKsujkDQboIiAshhakMzTbhNeW7VGxm3DQlOoNXrQfRIc7aa0tYHd97SZ8jtIn1PRoywE/jQNMfO+KLp2rNRWSC/daUS+wpY9bNQVZGFafvhU1Ct9ZK3ZOGcmojkfwprE32LdHLS/Gsv4DwpArCAx7SB9k6vg6PwKKiAbjT+Mgc2c1qNH2y3xXvB4DmOI0M6WpJL4XzeA65LFuku4aJFbCih2OY2YZSwd9Taw8yr2wSkWfRLcwCyIBaQ74y2jTeLFGYjTBFQ0nrPi3l8YaIlIAVUqga4y9d2VZHjO7Sb9+xQs3F7w== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(39860400002)(376002)(136003)(366004)(346002)(396003)(966005)(71200400001)(66946007)(8676002)(9686003)(52536014)(6506007)(76116006)(7696005)(66476007)(478600001)(64756008)(5660300002)(91956017)(2906002)(316002)(55016002)(786003)(8936002)(66446008)(83380400001)(33656002)(450100002)(186003)(66556008)(86362001)(110136005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?C44i7Z9ehjPF9pzMdKd/hYkkYMycz2gTvdvLVrGdUqn3YJMFWz/uAIgZe0?= =?iso-8859-1?Q?piRuBuB6tjU3H7fh15Z4F1IITVTY55APGJWYMIAdIh+dpOe7lZW2KOTBeD?= =?iso-8859-1?Q?bsqL9lxwNExv2FLpuD/axE4qKL1wSzuYDyqyI3up9LtngBRDxzYVP3f3Rs?= =?iso-8859-1?Q?bfBbOkmaT2FTZbi1H/mSgxVS28YFYCouXX2oEsAQPXNwtFgNFjWvdzDrgw?= =?iso-8859-1?Q?BLgLzsLTXfBOUIm4ihnrE4YFpaTHLv6CE0X4BslY6nx1LRB/TXjv76qnLM?= =?iso-8859-1?Q?yzDposGEFB/QnNFFTFkG0zfcigV3nPtFHtzt5rYOO2ukySoIVibvu7CxBi?= =?iso-8859-1?Q?J6EGTixLv9YZipddhOpS8kNGQt2ewrG8Z10cHOMZxzwpiW0W9ScECnreyD?= =?iso-8859-1?Q?d2PSC6wWiUZ1ixzf3clR+2spl94WrHTLz0gTyhUkGI8J73RKpTP2xaQpsf?= =?iso-8859-1?Q?WKGBYnU+zfVDSxGzHXnc6RhQFrGd0atA1b50tWb8zDOAZZZ4DqpZhdmMUf?= =?iso-8859-1?Q?Lj6kH02T7eZG8+kILFu+J7XTRyVtxZ2D+mpA/gJsdRE2u/nltJIu16v/BO?= =?iso-8859-1?Q?H8tc9kYpuhZAfxtRLfWl5G7Aesmw2p6WDtkbTFnA9didQ+i0frmg+VX7ha?= =?iso-8859-1?Q?9pbA0/7Ie6v/hbgya8ibA3oJ2u0wFgq+x6/Un4ZcPcCbFb/1dKghzGwyK8?= =?iso-8859-1?Q?y6HoAZI3M+Ylk+zmBnLrAsfuMNN7u8wsef7851yD7UUF7NE/GD1aqlSWRz?= =?iso-8859-1?Q?8vPh/i0Z7oSy44s4c0BZWw4CGe+/RSntsfxx0EmJJxiNgQJ5F0bE7sbsM0?= =?iso-8859-1?Q?fYTw+rJgCGUuUi/4DZ7wkeLL0lGY1BXCFzVkavZA2khTx1nVBq7rTYPjDA?= =?iso-8859-1?Q?/Pm3QvqT/3YUGT5iSiZkLLBib6JTKJSIv9zRDhJUsxvGblsmUGvGJqajnk?= =?iso-8859-1?Q?YaoBM2Tpf7/N2yi6fSwM2PJjuHeC/RjJWT5JxZEwSHlGlxvBGpFZ9Kouch?= =?iso-8859-1?Q?rDYmGNgg5t3ILLquA0nSgS/zHK8gAu9PBD0D5qt9du+jlIFlwnTG/V/etY?= =?iso-8859-1?Q?aYVEkKbYTZfKRSkcrQ56Woj8l2dSNqyDx1l4W88DFJUOvsN+zR4j4bT6Sw?= =?iso-8859-1?Q?dZ30ZZdhdOnI+rGumDzaRoa+0iFqVLvdNOTHF+WMtNeGMzRKlgbOwe4rGQ?= =?iso-8859-1?Q?whGhATBfK+fHnmyPpDIYfogKRIfjHYTgpC/myXGxvr/sVfbKnfu4uMW6Rf?= =?iso-8859-1?Q?ghVSAkDmsnAR+V1HF7DMdoVk1umoxQjveXYaTQjtwGKhZUMjKc6uvF/0QS?= =?iso-8859-1?Q?T1ofprLSjqqnPJCW1+gHW7Og/weRTV0I+PNdkQET2mO3YLPhIIUqBvhwxP?= =?iso-8859-1?Q?SazMCe2NS3+lPs6l2xPSmIyo943AM/3v7KDpg7ygW17j1cJboQAupxmL1f?= =?iso-8859-1?Q?nLScc6fTHR51fV8a?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 0f2d188b-be5a-4991-4a75-08d8e8d88b71 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 00:06:42.1943 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: QF3g69QNjNpkb74iVwKpOW0wIbuBJKRXpcfXeBZ7BcHHaYe7AIpPx4xSjlPjo5edtAeo7k5viWYqmQ7HrPOBZg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1764 X-Rspamd-Queue-Id: 4F0Vnc0qRNz3t4t X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=Lakji1Z+; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.89 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.03 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.89:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.97)[0.968]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.89:from]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; SPAMHAUS_ZRD(0.00)[40.107.67.89:from:127.0.2.255]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-fs]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.89:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 00:06:45 -0000 Alan Somers wrote:=0A= >Could somebody help me understand the meaning of the various=0A= >vfs.nfsd.request_space* sysctls, and what implication they have for tuning= =0A= >NFS?=0A= >=0A= >I have several NFS servers. Most are performing well, but one with a=0A= >metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAtt= r=0A= >and Lookup ops) is not. Total throughput is a not-unreasonable several=0A= >hundred MB/s, but some clients are limited to very slow speeds, sometimes= =0A= >writing at < 10 MB/s.=0A= >=0A= >The request_space sysctls show some pretty stark divergence between the=0A= >well-performing and poor-performing servers:=0A= >=0A= >sysctl well-performing poor-performing=0A= >request_space_used < 6 M varies, but=0A= >currently 40 M=0A= >request_space_used_highest < 39 M 24 G=0A= >request_space_throttle_count 0 35=0A= >=0A= >So what does request_space_used measure, anyway?=0A= It's the number of bytes of request message(s) received,=0A= but not yet being processed by nfsd threads.=0A= (They're actually in sys/rpc/svc.c, although named for the=0A= nfs server.)=0A= =0A= Anytime used > high, it throttles, which means it leaves=0A= the RPC messages on the socket receive queue.=0A= This indicates the server is not keeping up with requests.=0A= (ie Overloaded or ???)=0A= =0A= > And how can I either=0A= >increase the available space or decrease the stuff that's using it?=0A= Increasing vfs.nfsd.request_space_high avoids the throttling,=0A= but it is hard to say that is a good idea, since there won't be=0A= backpressure applied to the clients via TCP windows, etc.=0A= =0A= Fixing the server so that it isn't overloaded would be better,=0A= I think?=0A= --> I'm the last guy to take ZFS advice from, but I think there=0A= is a tunable w.r.t. how much arc is used for metadata.=0A= Getattrs will need cached (metadata) to reply quickly.=0A= Lookups also depend on cached attributes for good=0A= perf. as well.=0A= --> Make sure you have lots of nfsd threads. They are=0A= just kernel threads, so they don't use a lot of resources=0A= and having too many is much better than not enough.=0A= -->I can't remember what the upper limit is these days,=0A= but I'd set it to that for a busy nfs server.=0A= For NFSv4.1, each client can send a maximum of=0A= session_slot_table_size concurrent RPCs. FreeBSD=0A= uses a single 64slot table for each mount. I think=0A= Linux uses a 32slot table, but supports trunking.=0A= I don't know if the Linux client uses a separate=0A= session table for each trunk or not?=0A= --> Something like #clients * 32 (or 64) nfsd=0A= threads running on the server.=0A= =0A= rick=0A= =0A= -Alan=0A= _______________________________________________=0A= freebsd-fs@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= =0A=