From owner-freebsd-fs@freebsd.org Wed Mar 17 21:48:44 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A798579B6F for ; Wed, 17 Mar 2021 21:48:44 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670082.outbound.protection.outlook.com [40.107.67.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F13gv0gqQz3Mrt; Wed, 17 Mar 2021 21:48:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XSlJe0vq+wm4TA80eFJoz7zI89mYBhshe9zPKQKz9MOQbnC8ZJehM/+nr/+XXSvWNBq0PGD8WR6u2hSMHqGIe5JnLO2HG/dwvCnYTOc4hJXVXb9pq9vUAIsXFvOwuAOzzICUQ8fyntUU6H9hg9jTy0WUmPkLj8VOGNIlsqQ2x3uKiLE6H7JLCP/esSyAzcMo5CuM9OD/kftO98lk0ANjJMNP0DEnnz9ChnZx/V6NNl5D2vdcFjaVT5XZB16QmIsqTqgGMvO2xVOYCzVxtdccIWk52PZKi2OVx5JyPuZpVh0aDeELvpL+qaKBjsqgOtRg2LmAl2zWkwO4VxkGFGgffA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DPEvm0/qKlhDJlzhuiDcziP+SZAPTJvTqWhnuAVz9SI=; b=hyt39gLoSh4N+JSwCSzEAPQ4CXChBvk4o6XQWjsfglFC7aTYJLxTXPtrU+gRgwKJItDn4pXzU4tdH1atP1YtX1J0wMD4BiiPKYIKLUSNQIBU05cHAwFjT1bF/6we9KSXR9I/3m4Q9hVZdUrKfwCpj6zQO7MixMX/tVPokF1n4rC6UnFXqzE1TWPtTYzfy2UxC6oUqpQoHn4/fCdim8uK27h4jmCzoTKVYs756CM4iSs8YIH5ishNvsFkZ3KRrbfFvksit8k9LsCIVNITctL/8co+pfKTE7vAL/IKwCiWlGpz5r4/9cf01RomfuK2T14hl6ti1LwoHwPIb4Y+v1//XA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DPEvm0/qKlhDJlzhuiDcziP+SZAPTJvTqWhnuAVz9SI=; b=ZpfgDTZQwiEtBEjmpmaDiZlWB9MHKYDGa6D5YxTdFP3ctbgrvA42tgnGhrEqq9T93t4t8g2UXzJk6PMflJReM27xtpmEe8mdX/0pxNRJL3i2wjbCq+5EwQPzjK6padTYojQUV+4ZGmdb3gCEkRkMzJDOAMGSn/tf2o3MtBGNig5tJlRfPEIaadYSxB62hknzWkveCfwx29gp0JeIiTX5n1LBo7fryqCyZR0Xmzn1oBguRVjq2gv9ulbGjDtiJJdL+LQV6zz3ZU99PpmB462tf+Yb0Jy+67X7fbxIrQ/8HQ1Dqe0ZsYL/Lv+y23ORPmwBzmOWUCvwA+JEToBUbYmdRQ== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB3432.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:52::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Wed, 17 Mar 2021 21:48:41 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3933.032; Wed, 17 Mar 2021 21:48:41 +0000 From: Rick Macklem To: Alan Somers CC: freebsd-fs Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Thread-Topic: Understanding the vfs.nfsd.request_space* sysctls Thread-Index: AQHXGom0SMOUlewkdkSbAUM6yshgqaqHSCtTgAAdewCAAVEfeg== Date: Wed, 17 Mar 2021 21:48:41 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1c9e5862-1dc7-4773-db70-08d8e98e6e45 x-ms-traffictypediagnostic: YQXPR01MB3432: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: QgeRAAo+d1w0uz/+9hzSEtkpm/NjyyuW0aPpCfkeshZVe743SG/nN/eJGETzSsTKKflFq3gL4wXM/Ht4LQgFCiJNshm4Q60/S8eQN+5yL2+t9CJGqv/utVRmOqpFIgrhfDZN7FYjj/f8aiwHc2C/l0VK63z/GAtP/4QmZYB7WQVYB+PWHK5M7xJJnTXoI9Qdc4jHqvmKbnmKAF3HuHP0wqbRwF1tI9QVmmiH74QJhQz/RXgbjqu6l5MvCcUPtE2ZMCmuThPSQq/gruTlm2bSbFPI6vBY9rfr9LWnBXRHaS41gZfxxslbAEg9ojPp1dhJnTVBhL83tYOBK6Hvh+VP7qHfi/ObWS07kbD3TSyRb4qlCNTwk6o3et3Dawh5gTaZHwNMhehYZR6BZSE+rmGNTmHAS80dYDjRMDxErhY8q9whPzpjSuufuBIQIOnrtSEv3ulBGB4HX9rjKDfebWznT+jFm5/5dLMRhNHdF6LrUU8zr1u7Xq6GsiY427cUu083/6sCd8WLvNAlU74vjIkg8Rv40lkmAlFIpRjsQZV/9AaKpxXppHLxYpiHqnfBYbwmV2uQhZrq+mxY1747QQ+Q9JmKEatN/En+NKbtp6njC52dYrews4h5BFRDDA90KVbn0iB/jofDbyr/DEanZt+cZQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(136003)(376002)(366004)(396003)(39860400002)(346002)(8936002)(2906002)(33656002)(9686003)(8676002)(86362001)(316002)(52536014)(76116006)(55016002)(186003)(478600001)(83380400001)(6916009)(71200400001)(786003)(6506007)(7696005)(66946007)(66476007)(66446008)(66556008)(64756008)(91956017)(5660300002)(4326008)(450100002)(966005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?JdoC7+FN/ZSrf8SWKJn7fiLnownfmWMV1nNlgULrSEY9EcXM0kw+Q44IEa?= =?iso-8859-1?Q?2eZPXgCNQbkKbhWyrDrreu41emQcpQZaSBo6oUrGmob89+XfrDGuhwMoTY?= =?iso-8859-1?Q?lOzxze8I5GyOrm7VA3Xa6muqNRUPZVf/CWwOW/jIHfVep8ao6eJ/OGd1bR?= =?iso-8859-1?Q?QL0ZayAWvKNMtzQcVTFf8URffJRnwHzj9ME6l69yMnblq8MFCgOX5z/I4b?= =?iso-8859-1?Q?prvcU8UMMuACBqKV2c8ndkUeAJXz58g8JfUl3hgvMhaI/NBf92vuaZ39//?= =?iso-8859-1?Q?clXGqslcfhsHH4nwwgYMY2WQt/8Ge8ElJ63pQh8B8z16dwDEpzEo9k626s?= =?iso-8859-1?Q?JQ1tJhEm4XTa0vIAhCnE4ghzwt3ZLjt6WrBIU96BuCHi2a80WBr4Afkwxe?= =?iso-8859-1?Q?mt64oFYd9DmreErgb4AsrHVoU7uN9ToS/TPSYIRlfNIsqX2LeElSZSk4EZ?= =?iso-8859-1?Q?yZkj8zkAdFHsj06Jm/cELg3RFiaRLe1RCsZnOa9Fc1jYkUGO+SN8F0E265?= =?iso-8859-1?Q?H0gDuS+VIlA/gDOGcWcbGXNsTFsy1HUPW/36jCw1X4xUdVqnWBKOt0L0eB?= =?iso-8859-1?Q?7vJvGg2peUOdOKjiAZeKLpM4q7g8jL/cHLVhJLGUvZs8J3vB3FDm3z88bD?= =?iso-8859-1?Q?VYw6Ea+7v9kwC3l7gdNq9wBqehUAzDH2OqaFjippVQVF2uczZ6r6QpnF6s?= =?iso-8859-1?Q?cMmHqytm1GIWnjOfLln6lv3lXC8ZK201dnUjbnMYjRiR3nVuOxu5sJbGSU?= =?iso-8859-1?Q?tWAAZkNp7a2xrqCHuk+z7ijGef72ulYToKKb7/tZOwJ3aYYv3zzxYDCFhD?= =?iso-8859-1?Q?tqf20v1aeUsvqtc7+ixVLTOoXQyXll1d06VwHpYd74XeY6Ggw0+6p+qrJ6?= =?iso-8859-1?Q?f2xoBgTRM6rFDACXI3UJNoR2j3DLqz0wtdK/DhzIoqov6oov1zKXwqKWgk?= =?iso-8859-1?Q?ya1Dupbi2uv/+Z3F0qYHNx3AsjkzYu/OvG5TPjGs+ViVhf0NqJcr9iHs7Z?= =?iso-8859-1?Q?XXd2AqUfK5CeKJbrwKFwvsc8q1eIIfc6D8qor+5JAHrKqNq6Dyagejq1g7?= =?iso-8859-1?Q?r8jCYy24vVdgHHQz3tJ2DPqjav4o1hx/OBwhxiSVfcfgJj2NwKVhsvWwrF?= =?iso-8859-1?Q?T2QT0MLfitvaHWj3DaNUKt8226QcFOJvuflQ7OMs8yCilohh3jkQGOMfhb?= =?iso-8859-1?Q?GfekHnQrGUZOPsGrrTjRwC4dprzauSnoHgfgDhY0XWb0HlJk6B6VZtzrHt?= =?iso-8859-1?Q?x9M7DAv4/I84TfJUddt2tLZVXdRvZ+8QBRI1pRrqlc131bHWHp/BKSsrly?= =?iso-8859-1?Q?sSL+7AWwyoUE/qgiR0/FjO+ULY1dZInd0t96wEQ59LSfeVH8mpmAZfviXr?= =?iso-8859-1?Q?HKNofDtF2M8ncQh4Tz1LtiTKySPQaCT1I7BC8sTcdXFxLWVdvrv7E=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 1c9e5862-1dc7-4773-db70-08d8e98e6e45 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2021 21:48:41.6975 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +3Jl6bY+r8jMnnWOZVfc8VT/YEjRrXY0ot0S1UQV383wL76VZDq707Ppqe6b1hrOmClV/Qh39ETIX2UKNyubmg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3432 X-Rspamd-Queue-Id: 4F13gv0gqQz3Mrt X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=ZpfgDTZQ; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.82 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.67.82:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[40.107.67.82:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.82:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-fs]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.82:from] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2021 21:48:44 -0000 Alan Somers wrote:=0A= >Rick Macklem wrote:=0A= [stuff snipped]=0A= >>Anytime used > high, it throttles, which means it leaves=0A= >>the RPC messages on the socket receive queue.=0A= >>This indicates the server is not keeping up with requests.=0A= >>(ie Overloaded or ???)=0A= >=0A= >That all makes sense. So having a high request_space_used basically just = means that >my storage is too slow.=0A= One other thing you might want to look at is rcv interrupt moderation=0A= in the net device driver.=0A= --> This *might* result in received RPC requests being "batched", resulting= in=0A= a burst of RPCs that overloads the server. (Except for write requests= , each=0A= RPC is a small message typically in a single small packet.)=0A= --> Disabling rcv interrupt moderation will result in more interrupts= , but=0A= then each RPC request will be handled right away. This might r= esult=0A= in more evenly spaced RPCs (in a temporal sense).=0A= Also, delaying processing of RPC requests increases RPC RTT an= d that=0A= always has a negative impact on NFS performance.=0A= =0A= rick=0A= =0A= > And how can I either=0A= >increase the available space or decrease the stuff that's using it?=0A= Increasing vfs.nfsd.request_space_high avoids the throttling,=0A= but it is hard to say that is a good idea, since there won't be=0A= backpressure applied to the clients via TCP windows, etc.=0A= =0A= Fixing the server so that it isn't overloaded would be better,=0A= I think?=0A= --> I'm the last guy to take ZFS advice from, but I think there=0A= is a tunable w.r.t. how much arc is used for metadata.=0A= Getattrs will need cached (metadata) to reply quickly.=0A= Lookups also depend on cached attributes for good=0A= perf. as well.=0A= =0A= I've added an L2ARC since I sent that original email. We'll see how well i= t works. I expect it to take 48 hours before results are apparent.=0A= =0A= --> Make sure you have lots of nfsd threads. They are=0A= just kernel threads, so they don't use a lot of resources=0A= and having too many is much better than not enough.=0A= -->I can't remember what the upper limit is these days,=0A= but I'd set it to that for a busy nfs server.=0A= For NFSv4.1, each client can send a maximum of=0A= session_slot_table_size concurrent RPCs. FreeBSD=0A= uses a single 64slot table for each mount. I think=0A= Linux uses a 32slot table, but supports trunking.=0A= I don't know if the Linux client uses a separate=0A= session table for each trunk or not?=0A= --> Something like #clients * 32 (or 64) nfsd=0A= threads running on the server.=0A= =0A= Experimentally, I determined that 768 threads is sufficient. But your form= ula would suggest > 2500, which is a lot of threads! I'll keep it in mind = if I ever reach 768, though.=0A= =0A= =0A= rick=0A= =0A= -Alan=0A= _______________________________________________=0A= freebsd-fs@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= =0A=