From nobody Wed May 4 23:23:00 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2ECB31AB6201 for ; Wed, 4 May 2022 23:23:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0606.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::606]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KttDR6zpCz4nLB; Wed, 4 May 2022 23:23:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X89vjecDZVCkbTS9SHYlhi+d26c/aSw6utJIwUesq9iFZAtaNj2/IzwL66kEcZeCebv2wiYdOfyeglt/0ajZw5ZcBigkLpq9qyvLl51jSZ8kmuTgzWL/GWGwByzthFtQDrzeBKwybsPNt/Wp5yzBTtop6HxnesD1ligLACSV7h9kG1bqjwH7mgn9dpIFA4QHuyIQe2CMnRAiiG19v8QBb8GD+jmR7cQNUVZBqicQD8/SRrBAeFuequOn/p4QRp6yphoc2sNNGJj3G1xROepy9ca6ejf49qPYoyKnXwCvb69sxsskar+IGFlD73VdpNcY39swlYCEKHHtJuKz1mZe8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xlfmcEZJCt1FCLHfJ6U3ifiCS8QLrQs7PNV6PSGMBLg=; b=CpAk2VxNC7vA5KazKReXd23VtI7JZ4OdzHHExV9LFhNn+IlksHbWYrjwadzc1yHuIMXiJcTvdRxCVC0y2zXyHrC8dud4F9sWmDrN+Zkx5igt2nfO2Br2ryJk/JDMoAgB769FJ/T2HAOa3RtgwGug4+5X2gXBtx25MeGJdz0uTHlUN5lCLg1JgTTMKO2APdwpzA586+crhf00HDA7bxVDbnOhVO9Q814yN2j4bPl/dsgiX/qBTDp/3YNdmZZVtJaoLdmkFveBC/y6/Pf98+deq+yrCz+HEYDRPHmz2/0HVLBgQGfIPZ78vWWugrbH8K8nvDz0pf0TV+q6AOiBaU6Wlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xlfmcEZJCt1FCLHfJ6U3ifiCS8QLrQs7PNV6PSGMBLg=; b=Qz0zuW9T0ssvp76WEpEoxPyWBQzLp0RWJ4cT+tZaW5wpkN7Wl6Kn6qwHBdsIg00n5UbfPOBlMnmqOPN8XoTkiIxyRHoazzNOyN/KgTtuym2ktGM96mRXLig/MRWOKYPrK7djT1ga/ZO0lRrGe2ZjZdB4uInEmNSULViUvcSiqwaAMDSEz5KxaaobNzkc1yM/cFGA0BZ2WHj2BKsoG9Byyd2VrHvPjHjwsKF/WA55Fe52l45yOCQGoltQ5TA4VvQVJS6jdMSOH7x7rcZnECn2QK0B3lV0M8PbVcH8ll2venFwDLK6UysfAx7bqOqsrwVb5UcQZW7VHDjslg5mptolaA== Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:de::14) by YT3PR01MB5571.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:60::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.13; Wed, 4 May 2022 23:23:05 +0000 Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::fdb1:ada8:7af0:3003]) by YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::fdb1:ada8:7af0:3003%6]) with mapi id 15.20.5206.027; Wed, 4 May 2022 23:23:00 +0000 From: Rick Macklem To: Alan Somers , FreeBSD Stable ML Subject: Re: nfs client's OpenOwner count increases without bounds Thread-Topic: nfs client's OpenOwner count increases without bounds Thread-Index: AQHYX/oEvZae9laYBUaMKYABkmfK960PSuzX Date: Wed, 4 May 2022 23:23:00 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 833e2894-d841-0fe1-5dd5-8c4c78f983e6 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 239855b0-bfbb-4b31-55df-08da2e250800 x-ms-traffictypediagnostic: YT3PR01MB5571:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: VDOgrAMqkU0IcfRYOg9Mp+01//X7WAZ2TnAgLIsCzZAdsCLpxOhVWoSwligY6HXn5AIDTeq8PmcyolvKU19CkWchgolWAZ/ohLRkqJFzh69LRn9wSsQhn2AQU4O4XYBS5DrmmTgLXp8wOGj1LzYGr31PeAr8fZhG9YyW4ZXTq/pby8tE+J2VQG0x0Ws4dhzUtMet19bIzVJ28ny/gNxq+7i402wGz39aNMW9glYowJcK0CMsWsJf7ez7uitF4DlVNzjoI2K2Xaws96e5hVp9LvIYOkajvmtiIMarGEmADt7WsUo5R97Tu9WgtWcHiU1JbO1Ynrc1Uh/HvpdY/ShVzWRu4sfI+yBCURP/En09yzV9+wKHtROcGHQgam8cGF8WLR0PGPJEM3gRkCkfowqggEaNsBl3CdgTG0gfkKlWu0bIthTmBQGMUp2okdaangftQbDRhlbx1Sseud++V1X+rKhHxcVfUGjgGZIC6509IgHX0Ip9mRMtByj+kB7XUnJfGWVn1TC1zTG4q6O0egY5DnYh3b5GVVJEAXiaS1Q8Gv+XcxP8qJRMAKjW1PTJyuOtvS63DPKJQD4gojRi4OZpPHuSpmYQmgFJMyLZfYSoupJE86qTj8LW0okuyqLs0aHvnNGPg+V3/BlIvmHlwdoQtiVFYCbu1l05WRi0wyploPNNri9sXbNTv296Pvp0+y8wnJHRO8w67xz9Czem9+cguw== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(122000001)(83380400001)(6486002)(38100700002)(33656002)(508600001)(5660300002)(71200400001)(186003)(2906002)(64756008)(8936002)(6506007)(9686003)(6512007)(450100002)(786003)(66946007)(316002)(76116006)(52536014)(66556008)(110136005)(8676002)(38070700005)(66446008)(91956017)(66476007)(86362001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 2 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?hJCmoG82s1QK83d+Zj283t/9F8maR6gyXXMueQEaikKmx+iMQzqojyBNZg?= =?iso-8859-1?Q?CiOeHxMI1Je6f8hr7123CxLuWoVLBwC9Gt5C42SuQhuXxASneuSIVD8a+j?= =?iso-8859-1?Q?tgBHIHdYFWONkyIoWU8lmZz9cJ6KO/VN/vU/EL+7KfZQ7Tuc9aIvmp9QgD?= =?iso-8859-1?Q?VW3znuvC0JMz6LD6JUE18fjLJN+l975YZqoJ8bs1DJT60cxdBQFzdE8WSF?= =?iso-8859-1?Q?Bxsy/U2bprsENkXitgjmWyUKiXToh+FZ7Z81oyEPrRvRiBpZz05m2L13GC?= =?iso-8859-1?Q?90pbaY3+UmrLk82LQasRV/w/BBWwWJhzMFBFMviPaer0p3dv2H/YrnYeX7?= =?iso-8859-1?Q?WHJYn7+9dzgHONgzUDWQ3wz0WllTTdIHdVhCcWdQ4dnUraAW2XVIARfZcn?= =?iso-8859-1?Q?yTHx7TA2k/I/YNX1+VA+qGUeKVdGaS7AO0/MPaQ521A6WLCJNSPJ2KL36I?= =?iso-8859-1?Q?ss8nZZMKBxUAzTffYMyeGT+sh7JMyod/wm5EpHppse4FIm6AQHwds0wjFr?= =?iso-8859-1?Q?qag1+Cu0a2hqeRJFMc5m92aLOC5kPB4i/OTGcht+96ozb3lWjDL67991YV?= =?iso-8859-1?Q?O2/pGrR15lg0GRzdfUzx3Gf1IynCBpuX0KUfVQPA8kkyHbn+IjjVLXDRdw?= =?iso-8859-1?Q?1SxXLxpFGWtwJrJknzpUDIkjOVDpWMZXAK2U+wayCajQOcT0RSpU4fE6dG?= =?iso-8859-1?Q?XARMQ8DRHLt88ngw4ChVm4CWZ9fP8Kq6saxG9MQ/fclu8JLOICLvaFa+Bq?= =?iso-8859-1?Q?GaMZGTeFTZC0ZckJEDJhfyiOeh0rdPipCy33EV5VDp4bwC41/jkYLdbClR?= =?iso-8859-1?Q?1wRYILRLExnb5RNvx13sfQbQbgeTCEcajfsQm7ce7wZvFEoOIRQW2deW4F?= =?iso-8859-1?Q?yh+ole+BxYFqhM3mlqDt+UcEtEvy8CzD4Xs4D0/nR9oYsxfZY0Tu6RBylJ?= =?iso-8859-1?Q?QtIHq1dJn7XVSQxleHyH/346I6SfxLTgMHcRofeo1Z2gozIxC76D/OoE1d?= =?iso-8859-1?Q?Y6xq296VDTLp8dlvOoHRV9XyXY+paNfuYVg9K2ajIOeoDWyAlxakNeDkqt?= =?iso-8859-1?Q?/QCU01GgASlNqpaVTD7J33n2jGPkwtwHG+WcQzwea5H/Co0yn6w1+l0Vxi?= =?iso-8859-1?Q?AkxFApEWxvHasfsArqQdRb3Yqh3wZYaNy2VyLHbKbrfrK8OalOl+oOg2Sx?= =?iso-8859-1?Q?hVdGEthwJduiurAeOjhlHmKXzHaS/4u5j6c1LDYHPDl6+8XEJqsM/cr9U4?= =?iso-8859-1?Q?qe6AN6PGMMTpEtfoe/FrxlOl/YUNKO7Jz2H1HdNjxgQNUcZXES/b2IO1AN?= =?iso-8859-1?Q?3hE3BHSxmUIb+2DQy7OXo+KF7XuM84JTfvZ847MjI5MoYVu9C0vNSzi5+3?= =?iso-8859-1?Q?LB6bUpTL9jtciPTt6qT+zaiHhnMkfp3iVpnQ5VvZe8NU4WI8ajv51B82CF?= =?iso-8859-1?Q?98FVAUsv3xTYUQWRKA/loeIBCERFDQfkY+otN+jIHL0m7zNfnQsx2h6DlQ?= =?iso-8859-1?Q?NUpfKdoaYDTfnhQKUi5q0ZjojMGir5uZWjMmeyhsx2R1rfSfx2ncAeZUBv?= =?iso-8859-1?Q?SBY7rO/Hz/TxAmI/Yghn199IKSf7HYWRpkGfzEnUblHVojDm81QBGQ3Yn6?= =?iso-8859-1?Q?7kINzk3WifgLP27rT2/zfqumCF0lhtEJi1c4t3Ro1rWmDNo1GG4P/iucLN?= =?iso-8859-1?Q?CpeLN4/SFnxHxp5qmzih2gpWTdDCXT6YFe5YCHn/CBcuWZ8ZREz8Nf7CLE?= =?iso-8859-1?Q?dB3Bh83oT58glXosjZ7shV15ZiI4p2pd6V+pkp33Ydc4WcNlMfv3TrNZY/?= =?iso-8859-1?Q?DYUZw+7yX20iNdpSTWQIV7UIVdRgu6ZbZ+1T04rOLfpnITlPpjDcXNsbw/?= =?iso-8859-1?Q?YN?= x-ms-exchange-antispam-messagedata-1: cckxwg1MWvTodw== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 239855b0-bfbb-4b31-55df-08da2e250800 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 May 2022 23:23:00.8628 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: /tAL9YolbUnh17UZfKb4oDmuP2GC9yUCQfH/2r03h9shpLmSdpc1dlMW2l3A8q1SZTjvelq+LVd2xkE7mInhUg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT3PR01MB5571 X-Rspamd-Queue-Id: 4KttDR6zpCz4nLB X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=Qz0zuW9T; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5c::606 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.61 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-0.61)[-0.606]; MLMMJ_DEST(0.00)[stable]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N Alan Somers wrote:=0A= > I have a FreeBSD 13 (tested on both 13.0-RELEASE and 13.1-RC5) desktop=0A= > mounting /usr/home over NFS 4.2 from an 13.0-RELEASE server. It=0A= > worked fine until a few weeks ago. Now, the desktop's performance=0A= > slowly degrades. It becomes less and less responsive until I restart=0A= > X after 2-3 days. /var/log/Xorg.0.log shows plenty of entries like=0A= > "AT keyboard: client bug: event processing lagging behind by 112ms,=0A= > your system is too slow". "top -S" shows that the busiest process is=0A= > nfscl. A dtrace profile shows that nfscl is spending most of its time=0A= > in nfscl_cleanup_common, in the loop over all nfsclowner objects.=0A= > Running "nfsdumpstate" on the server shows thousands of OpenOwners for=0A= > that client, and < 10 for any other NFS client. The OpenOwners=0A= > increases by about 3000 per day. And yet, "fstat" shows only a couple=0A= > hundred open files on the NFS file system. Why are OpenOwners so=0A= > high? Killing most of my desktop processes doesn't seem to make a=0A= > difference. Restarting X does improve the perceived responsiveness,=0A= > though it does not change the number of OpenOwners.=0A= >=0A= > How can I figure out which process(es) are responsible for the=0A= > excessive OpenOwners? =0A= An OpenOwner represents a process on the client. The OpenOwner=0A= name is an encoding of pid + process startup time.=0A= However, I can't think of an easy way to get at the OpenOwner name.=0A= =0A= Now, why aren't they going away, hmm..=0A= =0A= I'm assuming the # of Opens is not large?=0A= (Openowners cannot go away until all associated opens=0A= are closed.)=0A= =0A= Commit 1cedb4ea1a79 in main changed the semantics of this=0A= a little, to avoid a use-after-free bug. However, it is dated=0A= Feb. 25, 2022 and is not in 13.0, so I don't think it could=0A= be the culprit.=0A= =0A= Essentially, the function called nfscl_cleanupkext() should call=0A= nfscl_procdoesntexist(), which returns true after the process has=0A= exited and when that is the case, calls nfscl_cleanup_common().=0A= --> nfscl_cleanup_common() will either get rid of the openowner or,=0A= if there are still children with open file descriptors, mark it "defu= nct"=0A= so it can be free'd once the children close the file.=0A= =0A= It could be that X is now somehow creating a long chain of processes=0A= where the children inherit a file descriptor and that delays the cleanup=0A= indefinitely?=0A= Even then, everything should get cleaned up once you kill off X?=0A= (It might take a couple of seconds after killing all the processes off.)=0A= =0A= Another possibility is that the "nfscl" thread is wedged somehow.=0A= It is the one that will call nfscl_cleanupkext() once/sec. If it never=0A= gets called, the openowners will never go away.=0A= =0A= Being old fashioned, I'd probably try to figure this out by adding=0A= some printf()s to nfscl_cleanupkext() and nfscl_cleanup_common().=0A= =0A= To avoid the problem, you can probably just use the "oneopenown"=0A= mount option. With that option, only one openowner is used for=0A= all opens. (Having separate openowners for each process was needed=0A= for NFSv4.0, but not NFSv4.1/4.2.)=0A= =0A= > Or is it just a red herring and I shouldn't=0A= > worry?=0A= Well, you can probably avoid the problem by using the "oneopenown"=0A= mount option.=0A= =0A= Thanks for reporting this, rick=0A= ps: And, yes, large numbers of openowners will slow things down,=0A= since the code ends up doing linear scans of them all in a linked=0A= list in various places.=0A= =0A= -Alan=0A= =0A=