From nobody Sat May 28 15:30:18 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 577F51B5C33D for ; Sat, 28 May 2022 15:30:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on2063.outbound.protection.outlook.com [40.107.66.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L9Qbv371Xz3h2G; Sat, 28 May 2022 15:30:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bsOOvkkeg8/VvNEn4fA7uJq4HrOpM/GrjL+CoVET8E3DS/XJ8xfZjqvTDTnesKA5QF2xsZphBsXAc0uOibNPgVb+iUArMtT/AqeOYKwJHtkPYAtZGRlYXhh2/9JVJ3qr6NfpziFcYwYXLK5ohjFQl+VJnLSR4NoZUlsuw4CxruX62gw4FHPAapGrw2vGT3iv2vBVM4MvMpKXRYmLL+mgAAhl5fjXaL6y38GeaShS21L9T/PKZiaWpnEFuNsD9URYkc3aZcaOvENM2lIvTZxD3k1e9zKqnDEVpQ1XfFXUI2hvMSu1HoShB1N0ENrU9vnUbsdpos08mvxXdsh1Xe4rRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iayVepgpgUjkeRpb1SuhuBzBb7ZW9lImrV+rHE+NWo4=; b=Ey7PZGvFqBiNzAY8D4DetvHqD8UaB1BEQcfeAdui53BB/pC3MZ6YzMTRk+Z4JYwp0r3aQz4adw0FBkYyI+RJuQ97zByATcHfSzoJEcMp3yFGtFjtDo1EVQkLyNuVrllwG7CUex9irdVdkKQb5ZSuFe+jtu4Jh3IPtTfpoYJykzIkasME5cd2rxyHL7D2dUwtJSKb+X+UqWqyxPjntN73TItWWzLb9A1R1ovGydZLZG9ih9yoOh1WUaY/ZGJ5lzNil/bsI2lKhc8gdyXLOzzPdwUHXnqOQyPnNIJvSpnU27qUC4mkG1zSnrVjLG0/MDZXbOS9cF2qxpvb7GK9zfS0wA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iayVepgpgUjkeRpb1SuhuBzBb7ZW9lImrV+rHE+NWo4=; b=ZonOn1qffKwDoXVs9zZqc2X8zYf7nTuQROA7osXdT7NofJ8lfWuGE+Et7XOv7pQBTL66ESQjke/0WdLiou3PxuBxo8ug0dWcaEH6omMq6zWsnrv9ojUWpV84Xaa5ZvNNOPp9VLD51fxFizfNK9f4Pd8prS84xILqFLWrHFaoICbIU1Jj/2iP4+O5Z0TzIhcA6AOxFm1lPKf3DcBNtM/NDW2xKuVXByJZKtHNjOBlR4/UNuVxiHBQ1F5G8PMLXkzaq8W7gQuDV9LCoop1Wy2PbEabm19dqy/17KCVj7K1Y+sJb0ZU3swILznxM6qNl8aVwdCJuHek0la8/LT0TvYTmA== Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:81::14) by YT2PR01MB8309.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:b0::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Sat, 28 May 2022 15:30:27 +0000 Received: from YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc]) by YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM ([fe80::b921:251e:4a0b:54fc%6]) with mapi id 15.20.5293.018; Sat, 28 May 2022 15:30:18 +0000 From: Rick Macklem To: Kurt Jaeger CC: "freebsd-fs@freebsd.org" Subject: Re: FreeBSD 12.3/13.1 NFS client hang Thread-Topic: FreeBSD 12.3/13.1 NFS client hang Thread-Index: AQHYcgY+3LPS/CtmUk+ZVTnbRrY6qK0zLx2fgADCMYCAAHUPJw== Date: Sat, 28 May 2022 15:30:18 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: fe4cbff8-9a52-c762-042b-7efb69b2fd08 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4f92cfe1-2f7f-40d0-f0f3-08da40bef8af x-ms-traffictypediagnostic: YT2PR01MB8309:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: jPXyJC35i6v7LWZ0JomJxxLyH3rHgucQ9vl+XbKZ0EPuMMEg5hrtEK0GOJIJNZxE4T6WQpFat57EsKj2vIVvODhYRWko2vyI9MOx8g7biug65bnJXllY1g5obR+fOD5ySaRvAjVfFdLO3axpkn25b2CerTcg87sW5aC6PEWj7rXb2k0GEC/9seOrGk8g0RDJfs/zDDXRv3wtTH97Zux6exAKkIjKRo7TvpQ/+A2vEcf2yuKM+DL85JkSvVfWwi3Q3608z3+xg/1642MBpQ/g4vtJ68MwoinL2p1DgXkRYcrO2OloNKO7qMoiuSnIPaX0wc43FV++8RxH2zOlNmOo2WpUg8ydNa5QvAx0rBuxe4/NsH70QBMm/ON3UThkJuJ+fDnPTLBo1yi5A89pqfFZeYt9pGwX3Lfbkahz1Z+bET8mpnf+tWQbM/EufiE1S75q8VFuhh9X2YarjI2uvBY4AFc8bIEywhhUdnfoNsSwNccSYlUUEVZPdLNvT86+IyC5ZM+Zk0cUOWfr69O45B3pWnGCFMsxP+wTtyouHS31UbPqVYn5an3Pq4Egmiy54eDxgxEgO3uDPU1VCLfGpnEHxkVKnsOaOdqyx7B80EQ+DskpMzT/thW1k9PuyPwy2H2X6PCuDjJB4FP62Nr//gw8QaJosJnTA4pxAc0bZjuem1bIIvfT/gp0iP3UH69bY3IFE7KsuRJJyvKy3u7+Jq1x7UslXHDlTsxV/+W4fGmHdjw6TkNIkuhtd6Rn19hnzgSPfc3vcwgm2VC1jSoK+HzkmhS6veka/bfyRXSHSURVY+Q= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(83380400001)(966005)(71200400001)(186003)(2906002)(52536014)(5660300002)(508600001)(55016003)(8936002)(6916009)(786003)(38070700005)(38100700002)(91956017)(316002)(33656002)(86362001)(76116006)(450100002)(64756008)(66446008)(66556008)(8676002)(66946007)(4326008)(66476007)(9686003)(7696005)(122000001)(6506007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 2 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?9kfDK6PO7DhOKDGE4bHRC/ZRQw9gVHnKgEQ7n01NeS5JV01qdn5uspKKI2?= =?iso-8859-1?Q?WVYvePo5FkecCxg5nXadvRXyiit1oOqpaLnswpjDf6rsbvGmGXQfekqxTf?= =?iso-8859-1?Q?FpCUOWWqgUstVg2Nc2v2RoNi78hxFfHezdLaAkYdWiVr6EQyUyxHM1OZVv?= =?iso-8859-1?Q?GyUBJAVbA11UkHHuCoxzzxzjRbLKk8VXpac+bRkUG6hrzcQ4YDPLBszC2X?= =?iso-8859-1?Q?Qs8mrq+qzoF1WlpxxWUejk7TYf8GqVetXsekhHefGeco0Gz9D3513WralR?= =?iso-8859-1?Q?rh6sbjD8nCvgmnmoNekfDusvrbZ4Jp2pbMF0mptzoB5tnXxBfH11yvYuva?= =?iso-8859-1?Q?x1W63/RaAMSYxJme+VHDJ1vs4OV767ii1r3Gp2DPxb72KlJV1CA3dwf+F8?= =?iso-8859-1?Q?vncq8ZyqzDT+/Yh1rv7XkMd8fE1sFxCBIlPuHNsnEPkUfsQEu/K4YSF5me?= =?iso-8859-1?Q?CLAL/ml0qkkaQhtNhVDQuOgMfEnK4kESoBaUS8dmcjtI+9VZhcGYCpR6ER?= =?iso-8859-1?Q?zkhpl7Aiq5qE20RL/Ryg+UuPlhKQKpeCqJxMc+xAe8f/ZdaRO5LSFeF+hh?= =?iso-8859-1?Q?JGHNgueUQCpKJU9BllRoRjGTAMrue3x+8JcYZpLCAWcgSV96fgU1Grf0yc?= =?iso-8859-1?Q?nRCr0ul1u/xLGkufDGTnVtKJt7JlGJHIsrRmF9BcG05Bsn63GBVHeTGk8M?= =?iso-8859-1?Q?5mWYshSiEpVc428HKsMOy/7OUEGwJIV6Zu8YcoYdq+cleWM06AT78EQvWW?= =?iso-8859-1?Q?3JcL3sWfOtBUZyb4ffyCEUJ0wcICyi/lvtJP3YrFrQzzOo2t+vBoThDbul?= =?iso-8859-1?Q?Lkz/9BgUnOeCkhPIKSh/vxZskmrLABuAEfed2ylxjJ7ZxepaKFsjoMPrJx?= =?iso-8859-1?Q?ol9qvwQ4v391r1Lnfup465Ur7Ipz9RnKpjbfDyfuoYMEIwBljpdkCYULkx?= =?iso-8859-1?Q?DdMPakog9XYjIvWth9CH0kppokESuVO0Cvbxlbk9AWyIDfTxEmy20zAw6c?= =?iso-8859-1?Q?cUvgaNJnXQsl2NDIUnaBRq4iJSBbkyldAccPYSTwn5dDSOCMrKjDdfJxMG?= =?iso-8859-1?Q?eO84LuykBgR+dW/6vW223UDK1ply7LMYelqRp3vVQU0NvVdwUtRaQIIj/d?= =?iso-8859-1?Q?4j8ZzdAdz/hgYmNCEytC/iOkldwrFLBfLJARWk6A/2HnM5O+WgSVllRasx?= =?iso-8859-1?Q?3LRRYrTvtx6DO6CebjhVfSN/RRFZjIDU+1XIqey06Jbz1ejueR5X99/WoB?= =?iso-8859-1?Q?f2/vRN7lFyVxTBChBPpwByRO/2xty67VEkwUudpZndIB6ceHwJrNu3xrSP?= =?iso-8859-1?Q?8egCsDfhQCmyo9WXDZReZ/OevbBVemJFn9XD5evOeKGqael+XwmwRb7TO+?= =?iso-8859-1?Q?pVExnZlrNOMSSV7JAt3saxvBsubhRGv9dbnYJbTuYx1g/KdRre1VihBTce?= =?iso-8859-1?Q?sEUVNQNp9ZhH77+FXcgw4aJdVMhXdZuoSV5rpMQSOsCQE2noiv7o8Q6hzm?= =?iso-8859-1?Q?r8XwSt5VuxEVN1NCgWdkqDN0Qxqc3ZqHChuxPyNLtPgvCdDX5MP5c1ADXW?= =?iso-8859-1?Q?QMetAbtt4kqJ9U9tZJsVHdXp//NjTIxvB5e7OpgcrMdiITyH2EHNjIasW8?= =?iso-8859-1?Q?Dm4kE6pjGIAmXVBLkLEEwZp7jHh3QJaaz+pj15m4esRgIRg8sZfqF7aKP7?= =?iso-8859-1?Q?NTmyolqjmo6CNisHxDoqPdnfX/UluGsvFEbanlMzMiMtm50MuDBuwXzBVr?= =?iso-8859-1?Q?SIG/+FIjBRJoj3g1ZoKtX8GdH485pS2TRGEBMTQR6eN6c11rWPmIWy2bwL?= =?iso-8859-1?Q?gM2sFzJlmvgfZhOQfamS2W5KHqhY9eMcFggULlITOe+DY5sxKPDXS2Px2N?= =?iso-8859-1?Q?Zs?= x-ms-exchange-antispam-messagedata-1: Keplqfqi4z4SjQ== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 4f92cfe1-2f7f-40d0-f0f3-08da40bef8af X-MS-Exchange-CrossTenant-originalarrivaltime: 28 May 2022 15:30:18.6434 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: EPgUPHVF20WnL1UjdMQd0DhbZcFujFacIQ2bi6d5CqjiYk3E1hBlmOcELvyOHA3iQ25tioKkZZ42GqR590D42A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT2PR01MB8309 X-Rspamd-Queue-Id: 4L9Qbv371Xz3h2G X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=ZonOn1qf; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.63 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.97 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.63:from]; NEURAL_HAM_SHORT(-0.97)[-0.972]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.63:from] X-ThisMailContainsUnwantedMimeParts: N Kurt Jaeger wrote:=0A= > > > I'm having issues with the NFS clients on FreeBSD 12.3 and 13.1=0A= >=0A= > I have it with an 13.0p7 client against an 13.1 server with=0A= > a hanging soft-mount (I tried unmount to change it to a hard mount).=0A= > =0A= > 61585 93- D+ 0:00.00 umount /office/serv=0A= > 61635 133 D 0:00.00 umount -f /office/serv=0A= > 7784 138 D 0:00.00 umount -N /office/serv=0A= The first umount must be "-N". Once you've hung a non "-N" umount,=0A= rebooting is the only option.=0A= (I have thought of doing a "umount -N -A" (for all NFS mounts), which=0A= would allow it to kill off all NFS activity without even finding the pathna= me=0A= for the mountpoint, but I have not done so.)=0A= =0A= > and procstat:=0A= >=0A= > # procstat -kk 7784=0A= > PID TID COMM TDNAME KSTACK=0A= > 7784 107226 umount - mi_switch+0xc1 sleepl= k+0xec lockmgr_xlock_hard+0x345 _vn_lock+0x48 vget_finish+0x21 cache_lookup= +0x299 vfs_cache_lookup+0x7b lookup+0x68c namei+0x487 kern_unmount+0x164 am= d64_syscall+0x10c fast_syscall_common+0xf8=0A= > # procstat -kk 61635=0A= > PID TID COMM TDNAME KSTACK=0A= > 61635 775458 umount - mi_switch+0xc1 sleep= lk+0xec lockmgr_slock_hard+0x382 _vn_lock+0x48 vget_finish+0x21 cache_looku= p+0x299 vfs_cache_lookup+0x7b lookup+0x68c namei+0x487 sys_statfs+0xc3 amd6= 4_syscall+0x10c fast_syscall_common+0xf8=0A= > # procstat -kk 61585=0A= > PID TID COMM TDNAME KSTACK=0A= > 61585 516164 umount - mi_switch+0xc1 sleep= lk+0xec lockmgr_xlock_hard+0x345 nfs_lock+0x2c vop_sigdefer+0x2b _vn_lock+0= x48 vflush+0x151 nfs_unmount+0xc3 vfs_unmount_sigdefer+0x2e dounmount+0x437= kern_unmount+0x332 amd64_syscall+0x10c fast_syscall_common+0xf8=0A= These just show that they are waiting for NFS vnodes. In the "ps" there are= =0A= threads waiting on zfs vnodes as well.=0A= =0A= > ps-axHl can be found at=0A= > =0A= > https://people.freebsd.org/~pi/logs/ps-axHl.txt=0A= I suspect your problem might be related to wired pages. Note that=0A= several threads are sleeping on "vmopar". I'm no vm guy, but I=0A= think that might mean too many pages have become wired.=0A= =0A= rick=0A= =0A= > > systems hanging when using a CentOS 7 server.=0A= > First, make sure you are using hard mounts. "soft" or "intr" mounts won't= =0A= > work and will mess up the session sooner or later. (A messed up session c= ould=0A= > result in no free slots on the session and that will wedge threads in=0A= > nfsv4_sequencelookup() as you describe.=0A= > (This is briefly described in the BUGS section of "man mount_nfs".)=0A= >=0A= > Do a:=0A= > # nfsstat -m=0A= > on the clients and look for "hard".=0A= =0A= No output at all for that 8-(=0A= =0A= --=0A= pi@FreeBSD.org +49 171 3101372 Now what ?=0A= =0A=