From nobody Wed Mar 9 14:39:39 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 51E6E1A12B9A for ; Wed, 9 Mar 2022 14:39:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670084.outbound.protection.outlook.com [40.107.67.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KDFGB3MWKz3th4 for ; Wed, 9 Mar 2022 14:39:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EACFuBN64QKGkXUEfMeFxd8XMJOJpNANI1OFk5vzghj1QXodSnL2voUn+NvXgYzOvuNg7abzL35t6o4py3GHGKtU2RYVuuGnjTgcUVcKlqblGuBvbZC4rZ1P8vCZ5oxuoaqAWaFCSAcYsDE3aZ/SNKxFiL97TIGzRcJBlmeZhlvfJwF4JpUgiJXYTUzBa+Bx9iTb5c3g/fl99KlAS8QhZRVY15Z3wwce8NkadexxBIa2SV9IdFiKT2Aq32vClJh4HczVxLf6Y+SmS+eFak8rpyXNFtW4gnbaBHTaTlJSK3w1RjGowcsHIDaaQ1WGuRWsy+qx00sJ6DQi8Gn81gcDJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LRext5S8g6x6Aymb40SveqfMr0b6on89Oim5jqAcZ68=; b=AfN0SX1fPIIw7Nko4qbdU9e8D/IJMDSH9DbxECixTp9Pk7WxQuYyyXEeBR6JMyEXmbWIxZr1TaS0YTC2u4emGwxCX9kiM//i5r4dgvWviwC5867eMKP1vheQ06pUBoLJ58zUW7t4SzSfzpJoUkvoiEXXkdrBqaNuMENUxlKnbNUP71C3OnJM8rzN6QCWZqnGTJccBmwAgydnf1xpA6zJscywVoDESPSfk7PnKDTSyGpYz2vSVHmBZTY5xf2BLG/7Z2AAUxVXyM/dX7G66CTkSIK4RZGEAM6qLrkfQszOPUuKQ9ZKhvIj3Tp3lG1P2s+QY3XJn5H6We/eWvhVN9TI8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LRext5S8g6x6Aymb40SveqfMr0b6on89Oim5jqAcZ68=; b=eMJL1VSr/wZLhJiScQ74tUQYxDKJqBbfFi7vWWWV/NwxlPo/fWnfADwKdJ2beerEvcgb3h+B+6cKTjJ4JIXO5eXl1GbCbavuTJi6WCLORrkkQxpqqat6ARdsuAK1GLmGveP2Rv+Nb8Gwhh1Vzfo8bmgii4SRAXDwvw6QFh9bMmXz6C+p6RAxyin7h6eEuJiOLbhQwok6Bybocm5vRrGMVeE5xLDSK7bRkksaq0TZE6jq3NrfadQM+c9fPVrqlv0QYJ4LyvczONh+oQkj69YnxrBNldVFxjTLZ5pW4fYGuvynQlvV1n/ox1PAESh8YP79slf8iqvaIvjeyDvqPo2FeQ== Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:de::14) by YT1PR01MB3369.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:10::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.17; Wed, 9 Mar 2022 14:39:39 +0000 Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::10d3:2fb2:36ac:b3af]) by YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::10d3:2fb2:36ac:b3af%2]) with mapi id 15.20.5038.027; Wed, 9 Mar 2022 14:39:39 +0000 From: Rick Macklem To: Yoshihiro Ota , freebsd-stable Subject: Re: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Thread-Topic: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Thread-Index: AQHYM7vqHmVJAWfo2UaVD6ISojss76y3HTBU Date: Wed, 9 Mar 2022 14:39:39 +0000 Message-ID: References: <20220309034601.ea3135e31aec3ffb2623f145@j.email.ne.jp> In-Reply-To: <20220309034601.ea3135e31aec3ffb2623f145@j.email.ne.jp> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 22519b47-ee10-dd53-ac5f-05f360e88af2 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1abada3e-c9e2-4c1a-8c62-08da01daa43a x-ms-traffictypediagnostic: YT1PR01MB3369:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: BcFxHiUK0aXJJVTig3jSbYbO4eZ6Jh88sAuU02iWmF5xIJsshLdalucFI+pQq2/Xo1s2DLkA1ncsxLILiEpE4zBsutT1xpFUQBX9hlXT8T0J+OvcQy5WtZwk28Z/NPTJB7gIOMGM67YnFD9efTb2F6uAxakrvcbe+25+7LSwQXYgYqeNOP0BbDm0kmh03P37DOANWlwieoxNpDujI4WWHMVsCfUw/R/p9c85ojfCnm9cPObdA8C/0AsT1lzLS//+cUkYTqefBvzy0ZlaZ+G6CLdzTkPAXkKfFqzFJWdmAa+uXSXtJpYxYAPQncV971V1vezq5NI1Dh4a2Es+QUwO9uSTSvU7O16egoUpayBogztMQS2isKKh3fKMjAmMElN889S9yqeWgC8BoxCyP8KeqsX3qghl2IXhNJgq9EdPFMUe9x4ZL1dLHPTkL0WlMv3Kxg7kBAK3Xeg45spWjZxXOSr/PbB15MauF3+WGlh6wbkXq32+JjQaPztSubjGpgxx1cRkr/kRlMLTtANUIDoIlAf2cDbsym+7NXankj/6Gme/KnVXIgkdlMTsL9Su5uWgDm8Vb/ZTycPlAUYiO/cUKH2OBsv0TItyNaj4R7Vjop32dA772e6zzLNtCc3p6NphMYgsgn4DW7CGu2A7LCj1Aewx3yj11LN48FR6LNTS/pSpQGObbf8ulyfa96c3DNoLZuu1F8DI6P3ekng5JU2OVQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(366004)(86362001)(91956017)(122000001)(55016003)(110136005)(9686003)(52536014)(8936002)(508600001)(2906002)(6506007)(66476007)(64756008)(66446008)(66556008)(8676002)(76116006)(66946007)(5660300002)(83380400001)(38070700005)(7696005)(38100700002)(33656002)(316002)(786003)(186003)(71200400001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?cFusGN0VeH6lAFC6Jg5VYSedIw2mz/3mWCXexgOg8XnVaDhIBhslideHy3?= =?iso-8859-1?Q?cTjfhd7RzkAbuzAcUtdBwMGr3tG/IBadEbm8wG7RVWannLDl0mq+8bvJUi?= =?iso-8859-1?Q?GhPqhJ9XBUQpwa/tq8myWO4di2EnICiVIc54ew2gn6uFSduwhqwt7me1Um?= =?iso-8859-1?Q?Gh+eloJK9voOxOkFyw5bd6q8uOKvF37O8a0atmnDNGcbOC2rGfa3LgKq1R?= =?iso-8859-1?Q?WATTj8t18nTNyciL51JlLdzR2bnZ3I8Q+tssrEi/9OVVMiAebIl823KLCd?= =?iso-8859-1?Q?c82iHNLoG5QRcBJY0H5SonAf5giNhznJ+duiGiJPOqqDi/RC8dV0hkl4zt?= =?iso-8859-1?Q?phITmeTB0sj/XDW7M/vWmvv+tzf9k8AmV3B4qImFhv3u0wzkGniyie0Blz?= =?iso-8859-1?Q?TbSHvbnY6ybVkPIbVrLiwy6tk/vQeiyKre7Lo5GO66ZeWskewChNgIm8r1?= =?iso-8859-1?Q?MK8IJWFGufz9uMxRMf0vZo1uE++v2ff675NxCXtl6m4pEZaYNCmAK8QD0g?= =?iso-8859-1?Q?7uT3yreBysVgyWwMUR1rROvtuTi2iqCF5MeLQdAgn2Xabv2VIeHyXBR0wp?= =?iso-8859-1?Q?lKnVPp4eUhQQdCRXzZKUOsDZ/JccH4xoBZPrXirSFFcIVl9IEXDx+Eo4de?= =?iso-8859-1?Q?qVGyKrT8pQ42LpkYpUYK/uQLD3CbqVp5jYqXEwejN9GtxWDDGI6ysWuMoC?= =?iso-8859-1?Q?rj91+Q0WYWRw3+CS6qq2+6BKMEkdhaGof9OfMQCi/7HBXnlGM9BM5Zg1xv?= =?iso-8859-1?Q?dQ15cI24CSfpzYKkKarLYYew2W+LBF2ffXEiUQiXcaK0B63fkgVadXnATE?= =?iso-8859-1?Q?bqNzFccsIak+HtMSGDNesD8ObsgPSAtEx07RxOZkMCE80P9rUg87H+AyEr?= =?iso-8859-1?Q?q3DS1vXXhTLb7Mo1LsVnNZCKBIRYY1SqChsCKS0lmP+3OJl0sH5KUZMWCW?= =?iso-8859-1?Q?nt14dCmfy0SqebxEQXCVHhG5a900u+aTaTsaqbCSvbJ4Ok38wtnkdJeIjF?= =?iso-8859-1?Q?zlGuy/vifhn4QiF3EGK+Hv3qk/p+YTvMXTfCJYzuvr7V/+afJwIjbxbiVq?= =?iso-8859-1?Q?x5dR2mJWmB373k3KSl7YpaQeyt32spM+kKf0B3VE/jo0n/XW0yj8g5LsfQ?= =?iso-8859-1?Q?u5uKpY48gDnzWpzhuDKmkopNA4fSFj5eZCumAVTjHbjndHzARZHBJbyhWU?= =?iso-8859-1?Q?/VF3rbnJGxVIoH3Q1IyPqZyrzRBijKzVWpBIR3NVIlo/P4wyLAKKswQfNw?= =?iso-8859-1?Q?YrfoWD75QjtJ6uY7R0ISCDu4nN0FgIMNWdctsUFnyWikz8YriT2a+V41Xp?= =?iso-8859-1?Q?IWMYhP8D3PAZ/aHSpJV9R1Am06boum3tSSRNhZr8MSR6pUqNHFxlqM7dpD?= =?iso-8859-1?Q?ANWx+mrEf7ta5FYZgiIODReqMy91wgHOtRAW4ukOqP3ALXXFBxCE5aHjpT?= =?iso-8859-1?Q?EQ6XpvWnTuYm66vaSIX8DNYK+V2wRlONchJHWKs+okK+bAfug6A6FZxhF3?= =?iso-8859-1?Q?IUckC4CXhpDgarleT1vmgXNNZlxZ9aJuCeh0xDg0iAo4DBe6RA9yE03Y8r?= =?iso-8859-1?Q?Mje0AjIqFYJ9dJWjD1dt2iY51wlePoZyoNMWplYX4xjpDO+SXWg9DbMp19?= =?iso-8859-1?Q?UeeRc5+oFqQkz/Hq/5Qf5m6KZr3pTnZRsedvUrqEw+7XVeOJR9ruPFb5rT?= =?iso-8859-1?Q?CzfUCkRilc7QU0JE7XUgoK3/V0W1Zfz5h731F+OkJAuIXwuUKoTDWMz2HP?= =?iso-8859-1?Q?AwwQ=3D=3D?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 1abada3e-c9e2-4c1a-8c62-08da01daa43a X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Mar 2022 14:39:39.5832 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: n9QbF4dgQ37975X+mR7I+MoYnKubAu0nr/ZjuCkQ+2wlaUv72mT7SqrSG/W1SIfx3XrzH3g39WvRF2+uu8uZRw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT1PR01MB3369 X-Rspamd-Queue-Id: 4KDFGB3MWKz3th4 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=eMJL1VSr; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.84 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.08 / 15.00]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_MEDIUM(0.92)[0.917]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.84:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; MLMMJ_DEST(0.00)[freebsd-stable]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.84:from] X-ThisMailContainsUnwantedMimeParts: N Yoshihiro Ota wrote:=0A= > Hi,=0A= >=0A= > I'm on stable/13 with latest code base.=0A= > I started testing pre-13.1 branch.=0A= >=0A= > I noticed major performance degrades with NFS when all CPUs are fully =0A= > utilized.=0A= >=0A= > This happends with stable/13 but not releng/13.0 nor releng/12.3.=0A= NFS performance is sensitive to RPC response time.=0A= Since this only happens when the COUs are busy, I'd suspect:=0A= - Kernel thread scheduling changes=0A= or=0A= - Timing of receive socket upcalls (which wake up the nfsd kernel threads).= =0A= =0A= I suspect bisecting to the actual commit that causes this is the only way= =0A= to find it.=0A= If you know of a working stable/13 that is more recent than 13.0, it would= =0A= help. If not, you start at this commit (which did make socket upcall change= s):=0A= commit 55cc0a478506ee1c2db7b2f9aadb9855e5490af3=0A= which was done on May 21, 2021.=0A= =0A= Maybe others can suggest commits related to thread scheduling (which I=0A= know nothing about).=0A= =0A= If you don't have the time/resources to bisect, I doubt this will get resol= ved.=0A= =0A= Good luck with it, rick=0A= =0A= I had NFS server with above versions and rsynced nfs mount to ufs mount on = NFS clients.=0A= My NFS server has 4 cores.=0A= When I had load average of 3 with make buildworld -j3, NFS server was fine.= =0A= After adding another 1 load, NFS server throughput came down to about 10% o= f before.=0A= After taking back to 3 load avg, performance recovered and down again after= getting over 4.=0A= Disk was fully avaiable for rsync; buildworld was done on another disk.=0A= =0A= =0A= Someone told me his smbfs was also slow and he suspected TCP/IP regression = instead of NFS, by the way.=0A= =0A= Hiro=0A= =0A= =0A=