Date: Mon, 30 May 2022 03:07:59 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Kurt Jaeger <pi@freebsd.org> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: FreeBSD 12.3/13.1 NFS client hang Message-ID: <YQBPR0101MB9742F83F3A983D151C9BAE1BDDDD9@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <YpPJOchbcMvKUfed@fc.opsec.eu> References: <YpEwxdGCouUUFHiE@shipon.lysator.liu.se> <YQBPR0101MB9742280313FC17543132A61CDDD89@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM> <YpHZb0fgsmbBrxD8@fc.opsec.eu> <YQBPR0101MB9742B91118878E58691DB94CDDDB9@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM> <YQBPR0101MB9742D1CC394CB37C135A94E3DDDB9@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM> <YpPJOchbcMvKUfed@fc.opsec.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
Kurt Jaeger <pi@freebsd.org> wrote:=0A= >=0A= > Hi!=0A= >=0A= > > > Kurt Jaeger <pi@freebsd.org> wrote:=0A= > > > > > > I'm having issues with the NFS clients on FreeBSD 12.3 and 13.1= =0A= =0A= > > > > I have it with an 13.0p7 client against an 13.1 server with=0A= > > > > a hanging soft-mount (I tried unmount to change it to a hard mount)= .=0A= [...]=0A= > > I take this back. I just did a fairly trivial test of this and it worke= d.=0A= >=0A= > > Looking at the "ps" output, I don't think your case is a "NFS protocol = hang".=0A= >=0A= > Today I upgraded the client to 13.1, and changed the nfs mount=0A= > from soft to hard,intr.=0A= >=0A= > nfsstat -M now show this:=0A= >=0A= > # nfsstat -m=0A= > <myhost>:/serv on /office/serv=0A= >nfsv3,tcp,resvport,nconnect=3D1,hard,intr,cto,lockd,sec=3Dsys,acdirmin=3D3= ,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negnametimeo=3D60,= rsize=3D65536,wsize=3D65536,readdirsize=3D65536,readahead=3D1,wcommitsize= =3D16777216,timeout=3D120,retrans=3D2=0A= >=0A= Since you are using NFSv3 mounts, the comments w.r.t. hard vs soft are=0A= not relevant.=0A= =0A= > > We use RCS for single files, quite a lot. Sometimes the co -l or the=0A= > > ci -u would not return and hang as well.=0A= >=0A= > We still have rpc.lockd and rpc.statd running on the server,=0A= > are they still used or not ? What happens.=0A= Since you are using NFSv3, if you want the locks to be visible across=0A= multiple clients then, yes, you need to run these.=0A= Alternately, you can use the "nolockd" mount option, which avoids=0A= using rpc.lockd and creates file locks visible within the client only.=0A= (NFSv4 does not use rpc.lockd, rpc.statd.)=0A= =0A= > We also use vi to edit files, and in the past they reliably showed=0A= > us when someone on nfsclient A had a file in vi and someone on=0A= > nfsclient B tried to edit the same file.=0A= > =0A= > Let's see if the problems re-occurs with hard mounts.=0A= It probably will, since you are using NFSv3 and, as I said, the hang you=0A= posted "ps" output for was not even an NFS problem.=0A= --> It appears to be a VM problem.=0A= =0A= > I'll investigate how to move to nfsv4 with a test box as well, if=0A= > this helps ?=0A= Again, I do not think this is an NFS problem, so switching to NFSv4=0A= will not resolve this.=0A= =0A= > > I suspect that some process/thread is hung for something non-NFS=0A= > > while holding a lock on a NFS vnode. "umount -N" won't know how=0A= > > to unhang this process/thread.=0A= > =0A= > > Just a hunch, but I'd suspect one of the threads sleeping on=0A= > > "vmopar", although I'm not a vm guy.=0A= > > What I don't know how to do is figure out what thread(s) are=0A= > > holding vnode locks?=0A= >=0A= > Given that simple NFS mounts and editing with vi on nfs mounts should=0A= > be very basic use-cases, it's time to find ways to debug this 8-(=0A= Look for a VM problem...=0A= =0A= > This also implies that switching from soft->hard won't fix the problem.= =0A= >=0A= > Thanks for the encouragement 8-}=0A= >=0A= > > In summary, I don't think your hang is anything like Andreas's, rick=0A= >=0A= > I'll report as soon as I have a another hang.=0A= I suggest you report this to a mailing list where the VM guys hang=0A= out, with a subject line like "processes hung sleeping on vmopar".=0A= Don't even mention NFS in the subject line. Reporting it here with=0A= NFS in the subject line won't get it seem by the VM people.=0A= (I could be wrong w.r.t. it being a VM problem, but I am pretty=0A= sure it is not an NFS problem. It just happens to result in NFS=0A= vnode being locked, which results in the hangs you observe.)=0A= =0A= rick=0A= =0A= --=0A= pi@FreeBSD.org +49 171 3101372 Now what ?=0A= =0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQBPR0101MB9742F83F3A983D151C9BAE1BDDDD9>