From nobody Sat Jul 16 13:43:11 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LlTvW3y71z4WYqZ for ; Sat, 16 Jul 2022 13:43:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-YT3-obe.outbound.protection.outlook.com (mail-yt3can01on2086.outbound.protection.outlook.com [40.107.115.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LlTvV3v2lz3Q0S for ; Sat, 16 Jul 2022 13:43:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kPHlNq55EuuIKgrrYkmucwPoBkLZE1/YxIu47B7nzX8NM7f1nkWBXKgGQ0isyjYiqr4s3T6FohovRCkIeiydEZOnuf0SxUSjEGn+NE/OWhplC71EjtPvT+rVXHlMO9Ck6htt/ByxGQPw8zqb1oekGSCl2DZRA7/qmnJmX/sHb9X7zBhU3sTCJPnzHBU+sW2UF53/g9C3TsuTBMiGeU4h1Dxbm5oDR9BfbM9tltpxBlzOis8GRP/JWBIgSCWgV9OoQ+qf8fHx1ZxKMOiD7dku34R3cOuZXjwaAaU6R/DtfZd5DRi67Z/IsnUmwIM0zMan+FF/fuhMYHZl1HTKDDYyvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iFHNMtU8imGLKpmbBIy/iQHauCk0Kg0GzV+Q4zs/seE=; b=ZJdehS04/f/Y/VRg3izo5hXlUdcU5uLKLyH7KtyVNdEWxLzVFabxwjNibLHkC7Md7sNZufLnbjMATHD0w9pL8ztciJInQwQPBwM1ow0QUgkKXNh+VbVXjlC8fFgTbmLrWLbMa0GSaHn2pd5Gettk3cpKzIM4EciNaSt1LhBd2rzpqvHpQd0+Kx8RnMszmnw1nvcJU3yEsYfZYZq8mEgTPZTclWGdmZHd3EUDqE8sulr/44S7EI7nQaL+Jgd6pFVVQRRubs4ACGYfdn7kScySogjoZevqObnynaz4hdOzmn+uvipMI0rmyzkf93wQhSirZuEGJgjztTHiE/9Nk1H3vg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iFHNMtU8imGLKpmbBIy/iQHauCk0Kg0GzV+Q4zs/seE=; b=FS8N4FytGclETBqxoZe4NbqyxUJKiapFYbtgl0UeKSwmu2mC6+M0QM5Se8kTeG2aVNZ3yuPNd2oWwitFvyr3Rk6SbHd8CUWw9BVJ0/Zm7W1PvQCsVsObf/GZ/pdAi6CTE2MCFgpwuzMUgePXdWFEpaNfLUAoZiioOvqNHYyRmZ7wvQvg1RgMdxi4EtLZfreH2IpsdDWDLaBrDNK7djLGyXLWIQlEI4xTHkFrgY1cwqGTCP7aG3r0VW2lW1VNSYrJhV08QPQkxYBEWE8qiR8XLadoIKwG366b4xJHL8TtLW4TpyM7evJPYTX7SWEaQr4p1vQki1AqPWtiSnS7MKghjg== Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:81::14) by YT3PR01MB6131.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:68::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5438.12; Sat, 16 Jul 2022 13:43:16 +0000 Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::51a6:243b:f3db:61fe]) by YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::51a6:243b:f3db:61fe%7]) with mapi id 15.20.5438.021; Sat, 16 Jul 2022 13:43:11 +0000 From: Rick Macklem To: Peter , "freebsd-stable@freebsd.org" Subject: Re: nfs stalls client: nfsrv_cache_session: no session Thread-Topic: nfs stalls client: nfsrv_cache_session: no session Thread-Index: AQHYmQ4r/JoiRFCEK06ukXc4DHs0E62A/gBy Date: Sat, 16 Jul 2022 13:43:11 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: aee0be81-6fb5-b591-0fd7-2f6157b4e566 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 50bead64-920e-4402-cae0-08da6731200d x-ms-traffictypediagnostic: YT3PR01MB6131:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: LsFmb7v/LcOmhm50U2Ya3w6jD3CD9S6UoaXNn6qFPDIIZeHPMW3mVp4OBn54IIrKh5LrpIQbXUvoA8N/vEBna06DemeYjR7xSD2VsZrPJvFdDS3qH5TQNu5Tk0oXVwtQyZs8+MZZq9kYmM5TeYZM7TBaiuCRRCfJMWeS+VVhEm4V3X8WB1nKNZNdkGX+ikOhQ42PXqjuD7v7O+HCrnbflolZ/YxP3+0ZpBkGBC76t7wje0yCu6N1MtgXazGnS9m3ecMoXeECO+aJFxLYmxilKtfxCt5MQJAvkrdLHqNZCwSpNwOHAxAVIgmV3mLHjZeSLwfRmPDhYYAoYsk9alNY/AUoyHVIvfJbbfiGv7pL3kSIci5tr7oa57O/T/Gm6VhM/5R3Qqt+YKVUCYpiPEKCHzCv51p235/Ya/onnJDw07aRZp5STII3vH2vB36YcDmShxKuS+d8xqynyw+SHthB0AgHZy/jzsWN2AtFR1vEXBWzB20N7ZDOkTF/wBRH3lSpN/laGG6ExRY47hnx86LORdMn3FSAhfSHNrUOtzPSm052ubCVkpIzMcDyXSiv0i7AnK4w/5VAR0FPF00Cp2vQK6iKrpeK4Kg0G1cAYNgs/Lm6wRe8D1+KQoCiWn2zFB28m0ExTaflKShA66NFEcXF/xcO9u89copssmVbCETsEQxvoLqHk0hEwoANh6H3JxN3ycOb/7ORFMqa7HbZgNq8uMGm+ujiz/AyR+5CICnIX8Uz/QgouUCL9MCkdHUg1vXmKGZFjSueprrT1E9FvoTF10tU8R9dPKv8c+3ApTPzB2QDlO9DyDVTtQudI3MSjDVh x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230016)(4636009)(366004)(136003)(376002)(39860400002)(346002)(396003)(71200400001)(83380400001)(478600001)(38070700005)(41300700001)(6506007)(7696005)(9686003)(110136005)(316002)(786003)(91956017)(186003)(55016003)(76116006)(41320700001)(66946007)(5660300002)(8936002)(66556008)(66476007)(52536014)(8676002)(66446008)(64756008)(122000001)(86362001)(33656002)(2906002)(38100700002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?Da18jG1p+gfsUXbt88FAFlQgqh/FEkGvrT0iRbnN3foAZJaK7S6zxFea2c?= =?iso-8859-1?Q?0/fwSDDHh55c+d8CFZy4dJAwPFCW7mJSZORId5OTdcMFnjMNzGD2Q2hFXy?= =?iso-8859-1?Q?NEBfIa+dPDbes272y7V/VI26FxuIG8YLtbUeYHa1io0B5Z2TKQfsEla6mu?= =?iso-8859-1?Q?91U6kLurF5KgKVCR8mp1ZFsRUpIpmCZhicSIa2vBmkygei0kMSv6iSvttU?= =?iso-8859-1?Q?2NeFVCexuuvLo1HyGVGh/wwxGqprto4WUQ79oxuE6HiEhBN6nRWZMfwDTV?= =?iso-8859-1?Q?MVM4pMAzhMKqlb9o3Y/O3GpmzIW8FyTLErIxYcmTxWSvNOvx6BnsVMu0U7?= =?iso-8859-1?Q?eXLyOGItCiEa++WAOx0fHp2fHwSbGagzf2FPxQtyfamkvLUCJ78f5EQxSM?= =?iso-8859-1?Q?g5VIbXb+u41OodttaIoDzB2E1ZL+MbIWkhUgRxWCdYFRPt7IY6gZgHxW+G?= =?iso-8859-1?Q?Q79iv3wyEcvOIetktf4GYq9sQJbIysKvWPmgvAFHecKeE9KKw8+/hRLbJ4?= =?iso-8859-1?Q?WNo4AfrcxKRp4KzYAdxihp7gyrhs0wrDQeJrjUVWUPLtxtASZcNA/7FMOD?= =?iso-8859-1?Q?oDY7zyDc4qAIplq7wKNHvKjleN1g2+PeJ0N28r5IW4D+7t0Gwr2aE/ouRf?= =?iso-8859-1?Q?Zqtfayvw3Df5e0GkaUYGYkYj9VifKon26VYK1o4ACoi0NFowVp7+DlWHMk?= =?iso-8859-1?Q?c2Ghkj+rWFaGGoHO/gBDsHKFiixTzzotFEAxzIVTdqMTAlct7j0B3B/tn0?= =?iso-8859-1?Q?D9u7z0VRnKK3b2LHLdQMkSQC2dlPcO8y9adqVkUeCRYceHvjs/wWe3dDH8?= =?iso-8859-1?Q?XlyeaOCDknm1E5R7FNCjXxonWrc2A/gg959lX8jJ/mYNj42+slm/4Ohhin?= =?iso-8859-1?Q?AQ6zKzKKij0tFUOj+4rSyyfxK0Ec5gsl9E1eefLdMxf+YNT/NvejaZnw18?= =?iso-8859-1?Q?2sgxmxwcslE6a7nE6BYle55DKWpDQsqOFZXT6ge8PN7HVLZfChv1UUqOYt?= =?iso-8859-1?Q?i+o1UPmjL4J26V0RUE1slxDP+4OusFZtwhegsjUuXR8cuuhPPEVDos8+/m?= =?iso-8859-1?Q?OyiUJiqMT196utbejKPfNxjxKTZxVkuxcnwwNBakQ5xEets72qo5V9bkTi?= =?iso-8859-1?Q?X+XS3s3mzp9vDqimb17fnKw/bxKeoaHqCZiUKBzZuoxegZX2jjzwHuQNcN?= =?iso-8859-1?Q?J0J9qFA1a7NVXQ2AR0HPUmne/qlcqQ96COlLcMCTWknUpRSU2hJezzTcd9?= =?iso-8859-1?Q?mt5cAwiPJiSe3cLlQh5Ka3LfjlIYIjU5MvuovPRN2yWBVKRpcehKZok8gi?= =?iso-8859-1?Q?EHu+RZfVEoZ6XH/VgEIIWosOIGme1aB5P1Ag1hm3RkkELTzlk16wawJ3Uc?= =?iso-8859-1?Q?ikmYOFc54c9igLY1xasV+tZ7jwARPcCLYNd0x29LfC5RZohnOTWfZyPVNN?= =?iso-8859-1?Q?2fHmm8CXAbQWZtghlUXD0dD1T7qJ/GOuJOPbirkDJPCCUNEfB/sFPSG8Jp?= =?iso-8859-1?Q?8+h0uPiXqfEYgR/drqDRGGw3mbXTwEqG/JXec8vpg92FLykaoOsV2bg6Fj?= =?iso-8859-1?Q?/cVvPjKt/yEP4fEq+2Ki9WBT9RNbOu2t0oQ73C+t+kz2aa+vLc6A/hJr+u?= =?iso-8859-1?Q?8Qi3QNN1/SNbkqdVqUhFHSHxAAtLhppWnl0IqOw3K+KW347QCKlFSj628E?= =?iso-8859-1?Q?pg1jV8A4fSEQvD6vPvwxv3ymuZgbV08yPyNdoeKM?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 50bead64-920e-4402-cae0-08da6731200d X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jul 2022 13:43:11.4541 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: iqATWWnRNfXP173/bONUx8AX2FCCbXoNDLPQYNTgJ752NqMzMt7gLsWy5jNzjre1z7mm9k3MH3Bpr/ET/h9zZg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT3PR01MB6131 X-Rspamd-Queue-Id: 4LlTvV3v2lz3Q0S X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=FS8N4Fyt; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.115.86 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-6.00 / 15.00]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[rmacklem]; RCVD_IN_DNSWL_NONE(0.00)[40.107.115.86:from]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCVD_COUNT_THREE(0.00)[3]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.115.86:from] X-ThisMailContainsUnwantedMimeParts: N Peter wrote:=0A= > Hija,=0A= > I have a problem with NFSv4:=0A= >=0A= > The configuration:=0A= > Server Rel. 13.1-RC2=0A= > nfs_server_enable=3D"YES"=0A= > nfs_server_flags=3D"-u -t --minthreads 2 --maxthreads 20 -h ..."=0A= Allowing it to go down to 2 threads is very low. I've never even=0A= tried to run a server with less than 4 threads. Since kernel threads=0A= don't generate much overhead, I'd suggest replacing the=0A= minthreads/maxthreads with "-n 32" for a very small server.=0A= (I didn't write the code that allows number of threads to vary and=0A= never use that either.)=0A= =0A= > mountd_enable=3D"YES"=0A= > mountd_flags=3D"-S -p 803 -h ..."=0A= > rpc_lockd_enable=3D"YES"=0A= > rpc_lockd_flags=3D"-h ..."=0A= > rpc_statd_enable=3D"YES"=0A= > rpc_statd_flags=3D"-h ..."=0A= > rpcbind_enable=3D"YES"=0A= > rpcbind_flags=3D"-h ..."=0A= > nfsv4_server_enable=3D"YES"=0A= > sysctl vfs.nfs.enable_uidtostring=3D1=0A= > sysctl vfs.nfsd.enable_stringtouid=3D1=0A= > =0A= > Client bhyve Rel. 13.1-RELEASE on the same system=0A= > nfs_client_enable=3D"YES"=0A= > nfs_access_cache=3D"600"=0A= > nfs_bufpackets=3D"32"=0A= > nfscbd_enable=3D"YES"=0A= > =0A= > Mount-options: nfsv4,readahead=3D1,rw,async=0A= I would expect the behaviour you are seeing for "intr" and/or "soft"=0A= mounts, but since you are not using those, I don't know how you=0A= broke the session? (10052 is NFSERR_BADSESSION)=0A= You might want to do "nfsstat -m" on the client to see what options=0A= were actually negotiated for the mount and then check that neither=0A= "soft" nor "intr" are there.=0A= =0A= I suspect that the recovery thread in the client (called "nfscl") is=0A= somehow wedged and cannot do the recovery from the bad session,=0A= as well.=0A= A "ps axHl" on the client would be useful to see what the=0A= processes/threads are up to on the client when it is hung.=0A= =0A= If increasing the number of nfsd threads in the server doesn't resolve=0A= the problem, I'd guess it is some network weirdness caused by how=0A= the bhyve instance is networked to its host. (I always use bridging=0A= for bhyve instances and do NFS mounts, but I don't work those=0A= mounts hard.)=0A= =0A= Btw, "umount -N " on the client will normally get rid=0A= of a hung mount, although it can take a couple of minutes to complete.=0A= =0A= rick=0A= =0A= =0A= Access to the share suddenly stalled. Server reports this in messages,=0A= every second:=0A= nfsrv_cache_session: no session IPaddr=3D192.168...=0A= =0A= Restarting nfsd and mountd didn't help, only now the client started to=0A= also report in messages, every second:=0A= nfs server 192.168...:/var/sysup/mnt/tmp.6.56160: is alive again=0A= =0A= Mounting the same share anew to a different place works fine.=0A= =0A= The network babble is this, every second:=0A= NFS request xid 1678997001 212 getattr fh 0,6/2=0A= NFS reply xid 1678997001 reply ok 52 getattr ERROR: unk 10052=0A= =0A= Forensics: I tried to build openoffice on that share, a couple of=0A= times. So there was a bit of traffic, and some things may have=0A= overflown.=0A= =0A= There seems to be no way to recover, only crashing the client.=0A= =0A= =0A= =0A=