Date: Mon, 30 May 2022 14:51:32 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Andreas Kempe <kempe@lysator.liu.se> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: FreeBSD 12.3/13.1 NFS client hang Message-ID: <YQBPR0101MB974292C2178FD27714BAB00ADDDD9@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <YpS5TBH/h1m39uwk@shipon.lysator.liu.se> References: <YpEwxdGCouUUFHiE@shipon.lysator.liu.se> <YQBPR0101MB9742280313FC17543132A61CDDD89@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM> <YpFM2bSMscG4ekc9@shipon.lysator.liu.se> <YQBPR0101MB9742BDE2175F07CF23A7CD5ADDD89@YQBPR0101MB9742.CANPRD01.PROD.OUTLOOK.COM> <YpS5TBH/h1m39uwk@shipon.lysator.liu.se>
next in thread | previous in thread | raw e-mail | index | archive | help
Andreas Kempe <kempe@lysator.liu.se> wrote:=0A= [lots of stuff snipped]=0A= >=0A= > I guess this means you think the error is at a protocol handling level=0A= > and the issues aren't caused by locking issues in the code? I was=0A= > wondering whether the hangs that were not slot related could possibly=0A= > be due to some race condition when locking since it happens so=0A= > seemingly randomly.=0A= Anything is possible, but the locking is pretty straightforward and no one= =0A= has found a bug in it for ages. (You can certainly run a kernel with=0A= WITNESS, DEBUG_VFS_LOCKS, etc., but there will be a performance=0A= penalty.=0A= =0A= My experience is that most hangs (other than the business with sessions=0A= for soft or intr mounts) are caused by network fabric issues.=0A= A couple of examples:=0A= As I noted, having TSO fail for some specific segment. Then retransmit of= =0A= the segment fails again, and again... =0A= =0A= In 13.0, there was a bug in TCP that=0A= caused the receive socket upcall to not happen under certain circumstances= =0A= and that could cause a hang. The bug is not in 12.n or 13.1 and the hang=0A= was normally observed when a Linux client had a FreeBSD server mounted,=0A= not vise versa.=0A= =0A= After a network partitioning healed, Linux and FreeBSD would get into=0A= what I might call an "RST storm". Every time one end would try to=0A= establish a new TCP connection, the other end would RST it.=0A= (Sorry, but it has been a while and I cannot remember exactly how to cause = it=0A= or if it even got resolved?)=0A= =0A= > > If you can reproduce it for a hard mount, you could capture packets via= :=0A= > > # tcpdump -s 0 -w out.pcap host <nfs-server>=0A= > > Tcpdump is useless at decoding NFS, but wireshark can decode the out.pc= ap=0A= > > quite nicely. I can look at the out.pcap or, if you do so, you start by= looking for=0A= > > NFSv4 specific errors.=0A= > > --> The client will usually log if it gets one of these. It will be an = error # > 10000.=0A= > >=0A= > =0A= > With us not knowing the NFSv4 protocol, we were holding off on even=0A= > trying to get Wireshark dumps since we wouldn't know what to look for=0A= > and would have to learn the protocol first. You having a look would be=0A= > greatly appreciated! As I wrote above, I'll try to get dumps if we can=0A= > find a reproducer.=0A= I certainly don't mind looking, but you might be surprised at how good=0A= wireshark is at this stuff.=0A= It not onlt decodes the RPCs for you, it flags anything that looks "sketchy= "=0A= in yellow and anything obviously broken in red.=0A= It was wireshark that spotted and flagged the RSTs I mentioned above.=0A= Beyond that, you just try and get to the place where things broke (a hang= =0A= might be at the end of the capture, for example) and then work backwards.= =0A= It is true that you need to know the protocol to spot things other than=0A= server error returns that are not going as planned.=0A= =0A= The big challenge is getting the packet capture that is less than petabytes= =0A= in size. Although starting a packet capture after a hang has occurred can= =0A= be useful, it is usually too late, since the breakage has already happened.= =0A= =0A= rick=0A= =0A= > Good luck with it, rick=0A= > > rick=0A= > >=0A= =0A= Cordially,=0A= Andreas Kempe=0A= =0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQBPR0101MB974292C2178FD27714BAB00ADDDD9>