From nobody Sun May 29 19:27:53 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AA4BC1B64EBE for ; Sun, 29 May 2022 19:28:02 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from fc.opsec.eu (fc.opsec.eu [IPv6:2001:14f8:200:4::4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LB7qP6vK3z4nMZ for ; Sun, 29 May 2022 19:28:01 +0000 (UTC) (envelope-from pi@freebsd.org) Received: from pi by fc.opsec.eu with local (Exim 4.95 (FreeBSD)) (envelope-from ) id 1nvOZt-000Aob-VB; Sun, 29 May 2022 21:27:53 +0200 Date: Sun, 29 May 2022 21:27:53 +0200 From: Kurt Jaeger To: Rick Macklem Cc: "freebsd-fs@freebsd.org" Subject: Re: FreeBSD 12.3/13.1 NFS client hang Message-ID: References: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4LB7qP6vK3z4nMZ X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=softfail (mx1.freebsd.org: 2001:14f8:200:4::4 is neither permitted nor denied by domain of pi@freebsd.org) smtp.mailfrom=pi@freebsd.org X-Spamd-Result: default: False [1.72 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[pi]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.51)[0.508]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; R_SPF_SOFTFAIL(0.00)[~all]; NEURAL_SPAM_MEDIUM(0.31)[0.310]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(1.00)[0.999]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:12502, ipnet:2001:14f8::/32, country:DE]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Hi! > > Kurt Jaeger wrote: > > > > > I'm having issues with the NFS clients on FreeBSD 12.3 and 13.1 > > > I have it with an 13.0p7 client against an 13.1 server with > > > a hanging soft-mount (I tried unmount to change it to a hard mount). [...] > I take this back. I just did a fairly trivial test of this and it worked. > Looking at the "ps" output, I don't think your case is a "NFS protocol hang". Today I upgraded the client to 13.1, and changed the nfs mount from soft to hard,intr. nfsstat -M now show this: # nfsstat -m :/serv on /office/serv nfsv3,tcp,resvport,nconnect=1,hard,intr,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 We use RCS for single files, quite a lot. Sometimes the co -l or the ci -u would not return and hang as well. We still have rpc.lockd and rpc.statd running on the server, are they still used or not ? What happens. We also use vi to edit files, and in the past they reliably showed us when someone on nfsclient A had a file in vi and someone on nfsclient B tried to edit the same file. Let's see if the problems re-occurs with hard mounts. I'll investigate how to move to nfsv4 with a test box as well, if this helps ? > I suspect that some process/thread is hung for something non-NFS > while holding a lock on a NFS vnode. "umount -N" won't know how > to unhang this process/thread. > Just a hunch, but I'd suspect one of the threads sleeping on > "vmopar", although I'm not a vm guy. > What I don't know how to do is figure out what thread(s) are > holding vnode locks? Given that simple NFS mounts and editing with vi on nfs mounts should be very basic use-cases, it's time to find ways to debug this 8-( > This also implies that switching from soft->hard won't fix the problem. Thanks for the encouragement 8-} > In summary, I don't think your hang is anything like Andreas's, rick I'll report as soon as I have a another hang. -- pi@FreeBSD.org +49 171 3101372 Now what ?