Date: Fri, 19 Mar 2021 21:17:35 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Jason Breitman <jbreitman@tildenparkcapital.com> Cc: "tuexen@freebsd.org" <tuexen@freebsd.org>, "Scheffenegger, Richard" <Richard.Scheffenegger@netapp.com>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Alexander Motin <mav@FreeBSD.org> Subject: Re: NFS Mount Hangs Message-ID: <YQXPR0101MB096861283CE48450784B7766DD689@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <BC4B2749-F92D-4F47-BA3F-6C77AF015AF8@tildenparkcapital.com> References: <C643BB9C-6B61-4DAC-8CF9-CE04EA7292D0@tildenparkcapital.com> <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> <YQXPR0101MB0968DC18E00833DE2969C636DD6A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <SN4PR0601MB3728780CE9ADAB144B3B681486699@SN4PR0601MB3728.namprd06.prod.outlook.com> <2890D243-AF46-43A4-A1AD-CB0C3481511D@lurchi.franken.de> <YQXPR0101MB0968D2362456D43DF528A7E9DD699@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <9EE3DFAC-72B0-4256-B57C-DE6AA811413C@freebsd.org> <YQXPR0101MB0968E1537E26CDBDC31C58E5DD689@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>, <BC4B2749-F92D-4F47-BA3F-6C77AF015AF8@tildenparkcapital.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Jason Breitman wrote:=0A=
>Thank you for your focus on the issue I am having and I look forward to se=
eing your >patch ported to FreeBSD 12.X.=0A=
I'll only be committing the patch I am convinced it actually fixes somethin=
g.=0A=
I'll be looking more closely at it and seeing what mav@ thinks aboutm it.=
=0A=
=0A=
>I also appreciate that you understand the difficulties in testing changes =
on a core >piece of infrastructure.=0A=
>=0A=
>I will let the group know if the issue occurs following the change that di=
sabled TSO >and LRO on my FreeBSD NFS Server NICs.=0A=
>Those types of settings are less disruptive and pose less risk to apply in=
production, >so I am happy to do so. That change was made on 3/17/2021.=
=0A=
>My hope is that this issue is a race condition specific to my NIC handling=
certain TCP >operations.=0A=
>=0A=
>I am happy that you are approaching the issue from multiple angles.=0A=
Well, another angle would be a nfsd thread that is "stuck" while doing=0A=
an RPC. If that happens, it won't call svc_freereq(), which derefs the=0A=
structure.=0A=
If it happens again and you can do, please go onto the FreeBSD server=0A=
and perform the following commands a few times, a few minutes apart,=0A=
capturing the output:=0A=
# ps axHl=0A=
# procstat -kk <pid of nfsd (server) as listed by "ps ax")=0A=
If the output doesn't have proprietary or security stuff, email me the resu=
lts.=0A=
(Or if you can sanitize it.)=0A=
--> I'll be looking for an nfsd thread that is sleeping on the same thing=
=0A=
without cpu increasing (except for rpcsvc, which is just a thread wait=
ing=0A=
for an RPC to work on).=0A=
--> The procstat -kk gives more information w.r.t what the thread is up to.=
=0A=
=0A=
Good luck with it, rick=0A=
=0A=
Thanks.=0A=
=0A=
Jason Breitman=0A=
=0A=
=0A=
On Mar 19, 2021, at 11:58 AM, Rick Macklem <rmacklem@uoguelph.ca> wrote:=0A=
=0A=
Michael Tuexen wrote:=0A=
>> On 18. Mar 2021, at 21:55, Rick Macklem <rmacklem@uoguelph.ca> wrote:=0A=
>>=0A=
>> Michael Tuexen wrote:=0A=
>>>> On 18. Mar 2021, at 13:42, Scheffenegger, Richard <Richard.Scheffenegg=
er@netapp.com> wrote:=0A=
>>>>=0A=
>>>>>> Output from the NFS Client when the issue occurs # netstat -an | gre=
p=0A=
>>>>>> NFS.Server.IP.X=0A=
>>>>>> tcp 0 0 NFS.Client.IP.X:46896 NFS.Server.IP.X:2049 FIN_WAIT2=0A=
>>>>> I'm no TCP guy. Hopefully others might know why the client would be s=
tuck in FIN_WAIT2 (I vaguely recall this means it is waiting for a fin/ack,=
but could be wrong?)=0A=
>>>>=0A=
>>>> When the client is in Fin-Wait2 this is the state you end up when the =
Client side actively close() the tcp session, and then the server also ACKe=
d the FIN.=0A=
>> Jason noted:=0A=
>>=0A=
>>> When the issue occurs, this is what I see on the NFS Server.=0A=
>>> tcp4 0 0 NFS.Server.IP.X.2049 NFS.Client.IP.X.51550 CLOSE_WAIT=0A=
>>>=0A=
>>> which corresponds to the state on the client side. The server received =
the FIN=0A=
>>> from the client and acked it.=0A=
>>> The server is waiting for a close call to happen.=0A=
>>> So the question is: Is the server also closing the connection?=0A=
>> Did you mean to say "client closing the connection here?"=0A=
>Yes.=0A=
>>=0A=
>> The server should call soclose() { it never calls soshutdown() } when=0A=
>> soreceive(with MSG_WAIT) returns 0 bytes or an error that indicates=0A=
>> the socket is broken.=0A=
Btw, I looked and the soreceive() is done with MSG_DONTWAIT, but the=0A=
EWOULDBLOCK is handled appropriately.=0A=
=0A=
>> --> The soreceive() call is triggered by an upcall for the rcv side of t=
he socket.=0A=
>> So, are you saying the FreeBSD NFS server did not call soclose() for thi=
s case?=0A=
>Yes. If the state at the server side is CLOSE_WAIT, no close call has happ=
ened yet.=0A=
>The FIN from the client was received, it was ACKED, but no close() call=0A=
>(or shutdown(..., SHUT_WR) or shutdown(..., SHUT_RDWR)) was issued. Theref=
ore,=0A=
>no FIN was sent and the client should be in the FINWAIT-2 state. This was =
also=0A=
>reported. So the reported states are consistent.=0A=
For a test, I commented out the soclose() call in the server side krpc and,=
when I=0A=
dismounted, it did leave the server socket in CLOSE_WAIT.=0A=
For the FreeBSD client, it did the dismount and the socket was in FIN_WAIT2=
=0A=
for a little while and then disappeared (someone mentioned a short timeout=
=0A=
and that seems to be the case).=0A=
I might argue that the Linux client should not get hung when this occurs,=
=0A=
but there does appear to be an issue on the FreeBSD end.=0A=
=0A=
So it does appear you have a case where the soclose() call is not happening=
=0A=
on the FreeBSD NFS server. I am a little surprised since I don't think I've=
=0A=
heard of this before and the code is at least 10years old (at least the par=
ts=0A=
related to this).=0A=
=0A=
For the soclose() to not happen, the reference count on the socket=0A=
structure cannot have gone to zero. (ie a SVC_RELEASE() was missed)=0A=
Upon code inspection, I was not able to spot a reference counting bug.=0A=
(Not too surprising, since a reference counting bug should have shown=0A=
up long ago.)=0A=
=0A=
The only thing I spotted that could conceivably explain this is that the=0A=
function svc_vc_stat() which returns the indication that the socket has=0A=
been closed at the other end did not bother to do any locking when=0A=
it checked the status. (I am not yet sure if this could result in the=0A=
status of XPRT_DIED being missed by the call, but if so, that would=0A=
result in the soclose() call not happening.)=0A=
=0A=
I have attached a small patch, which I think is safe, that adds locking=0A=
to svc_vc_stat(),which I am hoping you can try at some point.=0A=
(I realize this is difficult for a production server, but...)=0A=
I have tested it a little and will test it some more, to try and ensure=0A=
it does not break anything.=0A=
=0A=
I have also cc'd mav@, since he's the guy who last worked on this=0A=
code, in case he has any insight w.r.t. how the soclose() might get=0A=
missed (or any other way the server socket gets stuck in CLOSE_WAIT).=0A=
=0A=
rick=0A=
ps: I'll create a PR for this, so that it doesn't get forgotten.=0A=
=0A=
Best regards=0A=
Michael=0A=
=0A=
>=0A=
> rick=0A=
>=0A=
> Best regards=0A=
> Michael=0A=
>> This will last for ~2 min or so, but is asynchronous. However, the same =
4-tuple can not be reused during this time.=0A=
>>=0A=
>> With other words, from the socket / TCP, a properly executed active clos=
e() will end up in this state. (If the other side initiated the close, a pa=
ssive close, will not end in this state)=0A=
>>=0A=
>>=0A=
>> _______________________________________________=0A=
>> freebsd-net@freebsd.org mailing list=0A=
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A=
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=
=0A=
>=0A=
>=0A=
> _______________________________________________=0A=
> freebsd-net@freebsd.org mailing list=0A=
> https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A=
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A=
=0A=
_______________________________________________=0A=
freebsd-net@freebsd.org mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A=
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A=
=0A=
<xprtdied.patch>_______________________________________________=0A=
freebsd-net@freebsd.org mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A=
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A=
=0A=
=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB096861283CE48450784B7766DD689>