Date: Fri, 19 Mar 2021 15:58:02 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: "tuexen@freebsd.org" <tuexen@freebsd.org> Cc: "Scheffenegger, Richard" <Richard.Scheffenegger@netapp.com>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Alexander Motin <mav@FreeBSD.org> Subject: Re: NFS Mount Hangs Message-ID: <YQXPR0101MB0968E1537E26CDBDC31C58E5DD689@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <9EE3DFAC-72B0-4256-B57C-DE6AA811413C@freebsd.org> References: <C643BB9C-6B61-4DAC-8CF9-CE04EA7292D0@tildenparkcapital.com> <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> <YQXPR0101MB0968DC18E00833DE2969C636DD6A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <SN4PR0601MB3728780CE9ADAB144B3B681486699@SN4PR0601MB3728.namprd06.prod.outlook.com> <2890D243-AF46-43A4-A1AD-CB0C3481511D@lurchi.franken.de> <YQXPR0101MB0968D2362456D43DF528A7E9DD699@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>, <9EE3DFAC-72B0-4256-B57C-DE6AA811413C@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--_002_YQXPR0101MB0968E1537E26CDBDC31C58E5DD689YQXPR0101MB0968_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Michael Tuexen wrote:=0A=
>> On 18. Mar 2021, at 21:55, Rick Macklem <rmacklem@uoguelph.ca> wrote:=0A=
>>=0A=
>> Michael Tuexen wrote:=0A=
>>>> On 18. Mar 2021, at 13:42, Scheffenegger, Richard <Richard.Scheffenegg=
er@netapp.com> wrote:=0A=
>>>>=0A=
>>>>>> Output from the NFS Client when the issue occurs # netstat -an | gre=
p=0A=
>>>>>> NFS.Server.IP.X=0A=
>>>>>> tcp 0 0 NFS.Client.IP.X:46896 NFS.Server.IP.X:2049 =
FIN_WAIT2=0A=
>>>>> I'm no TCP guy. Hopefully others might know why the client would be s=
tuck in FIN_WAIT2 (I vaguely recall this means it is waiting for a fin/ack,=
but could be wrong?)=0A=
>>>>=0A=
>>>> When the client is in Fin-Wait2 this is the state you end up when the =
Client side actively close() the tcp session, and then the server also ACKe=
d the FIN.=0A=
>> Jason noted:=0A=
>>=0A=
>>> When the issue occurs, this is what I see on the NFS Server.=0A=
>>> tcp4 0 0 NFS.Server.IP.X.2049 NFS.Client.IP.X.51550 =
CLOSE_WAIT=0A=
>>>=0A=
>>> which corresponds to the state on the client side. The server received =
the FIN=0A=
>>> from the client and acked it.=0A=
>>> The server is waiting for a close call to happen.=0A=
>>> So the question is: Is the server also closing the connection?=0A=
>> Did you mean to say "client closing the connection here?"=0A=
>Yes.=0A=
>>=0A=
>> The server should call soclose() { it never calls soshutdown() } when=0A=
>> soreceive(with MSG_WAIT) returns 0 bytes or an error that indicates=0A=
>> the socket is broken.=0A=
Btw, I looked and the soreceive() is done with MSG_DONTWAIT, but the=0A=
EWOULDBLOCK is handled appropriately.=0A=
=0A=
>> --> The soreceive() call is triggered by an upcall for the rcv side of t=
he socket.=0A=
>> So, are you saying the FreeBSD NFS server did not call soclose() for thi=
s case?=0A=
>Yes. If the state at the server side is CLOSE_WAIT, no close call has happ=
ened yet.=0A=
>The FIN from the client was received, it was ACKED, but no close() call=0A=
>(or shutdown(..., SHUT_WR) or shutdown(..., SHUT_RDWR)) was issued. Theref=
ore,=0A=
>no FIN was sent and the client should be in the FINWAIT-2 state. This was =
also=0A=
>reported. So the reported states are consistent.=0A=
For a test, I commented out the soclose() call in the server side krpc and,=
when I=0A=
dismounted, it did leave the server socket in CLOSE_WAIT.=0A=
For the FreeBSD client, it did the dismount and the socket was in FIN_WAIT2=
=0A=
for a little while and then disappeared (someone mentioned a short timeout=
=0A=
and that seems to be the case).=0A=
I might argue that the Linux client should not get hung when this occurs,=
=0A=
but there does appear to be an issue on the FreeBSD end.=0A=
=0A=
So it does appear you have a case where the soclose() call is not happening=
=0A=
on the FreeBSD NFS server. I am a little surprised since I don't think I've=
=0A=
heard of this before and the code is at least 10years old (at least the par=
ts=0A=
related to this).=0A=
=0A=
For the soclose() to not happen, the reference count on the socket=0A=
structure cannot have gone to zero. (ie a SVC_RELEASE() was missed)=0A=
Upon code inspection, I was not able to spot a reference counting bug.=0A=
(Not too surprising, since a reference counting bug should have shown=0A=
up long ago.)=0A=
=0A=
The only thing I spotted that could conceivably explain this is that the=0A=
function svc_vc_stat() which returns the indication that the socket has=0A=
been closed at the other end did not bother to do any locking when=0A=
it checked the status. (I am not yet sure if this could result in the=0A=
status of XPRT_DIED being missed by the call, but if so, that would=0A=
result in the soclose() call not happening.)=0A=
=0A=
I have attached a small patch, which I think is safe, that adds locking=0A=
to svc_vc_stat(),which I am hoping you can try at some point.=0A=
(I realize this is difficult for a production server, but...)=0A=
I have tested it a little and will test it some more, to try and ensure=0A=
it does not break anything.=0A=
=0A=
I have also cc'd mav@, since he's the guy who last worked on this=0A=
code, in case he has any insight w.r.t. how the soclose() might get=0A=
missed (or any other way the server socket gets stuck in CLOSE_WAIT).=0A=
=0A=
rick=0A=
ps: I'll create a PR for this, so that it doesn't get forgotten.=0A=
=0A=
Best regards=0A=
Michael=0A=
=0A=
>=0A=
> rick=0A=
>=0A=
> Best regards=0A=
> Michael=0A=
>> This will last for ~2 min or so, but is asynchronous. However, the same =
4-tuple can not be reused during this time.=0A=
>>=0A=
>> With other words, from the socket / TCP, a properly executed active clos=
e() will end up in this state. (If the other side initiated the close, a pa=
ssive close, will not end in this state)=0A=
>>=0A=
>>=0A=
>> _______________________________________________=0A=
>> freebsd-net@freebsd.org mailing list=0A=
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A=
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=
=0A=
>=0A=
>=0A=
> _______________________________________________=0A=
> freebsd-net@freebsd.org mailing list=0A=
> https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A=
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A=
=0A=
_______________________________________________=0A=
freebsd-net@freebsd.org mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-net=0A=
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A=
=0A=
--_002_YQXPR0101MB0968E1537E26CDBDC31C58E5DD689YQXPR0101MB0968_
Content-Type: application/octet-stream; name="xprtdied.patch"
Content-Description: xprtdied.patch
Content-Disposition: attachment; filename="xprtdied.patch"; size=921;
creation-date="Fri, 19 Mar 2021 15:50:41 GMT";
modification-date="Fri, 19 Mar 2021 15:50:41 GMT"
Content-Transfer-Encoding: base64
LS0tIHN5cy9ycGMvc3ZjX3ZjLmMueHBydGRpZWQJMjAyMS0wMy0xOSAwNjozNDowNi4yOTc1NDUw
MDAgLTA3MDAKKysrIHN5cy9ycGMvc3ZjX3ZjLmMJMjAyMS0wMy0xOSAwNzowMjozOS45OTcyMTIw
MDAgLTA3MDAKQEAgLTU1OSwxOSArNTU5LDI2IEBAIHN0YXRpYyBlbnVtIHhwcnRfc3RhdAogc3Zj
X3ZjX3N0YXQoU1ZDWFBSVCAqeHBydCkKIHsKIAlzdHJ1Y3QgY2ZfY29ubiAqY2Q7CisJZW51bSB4
cHJ0X3N0YXQgcnN0YXQ7CiAKIAljZCA9IChzdHJ1Y3QgY2ZfY29ubiAqKSh4cHJ0LT54cF9wMSk7
CiAKKwlyc3RhdCA9IFhQUlRfSURMRTsKKwlzeF94bG9jaygmeHBydC0+eHBfbG9jayk7CiAJaWYg
KGNkLT5zdHJtX3N0YXQgPT0gWFBSVF9ESUVEKQotCQlyZXR1cm4gKFhQUlRfRElFRCk7CisJCXJz
dGF0ID0gWFBSVF9ESUVEOworCWVsc2UgaWYgKGNkLT5tcmVxICE9IE5VTEwgJiYgY2QtPnJlc2lk
ID09IDAgJiYgY2QtPmVvcikKKwkJcnN0YXQgPSBYUFJUX01PUkVSRVFTOwogCi0JaWYgKGNkLT5t
cmVxICE9IE5VTEwgJiYgY2QtPnJlc2lkID09IDAgJiYgY2QtPmVvcikKLQkJcmV0dXJuIChYUFJU
X01PUkVSRVFTKTsKKwlpZiAocnN0YXQgPT0gWFBSVF9JRExFKSB7CisJCVNPQ0tCVUZfTE9DSygm
eHBydC0+eHBfc29ja2V0LT5zb19yY3YpOworCQlpZiAoc29yZWFkYWJsZSh4cHJ0LT54cF9zb2Nr
ZXQpKQorCQkJcnN0YXQgPSBYUFJUX01PUkVSRVFTOworCQlTT0NLQlVGX1VOTE9DSygmeHBydC0+
eHBfc29ja2V0LT5zb19yY3YpOworCX0KKwlzeF94dW5sb2NrKCZ4cHJ0LT54cF9sb2NrKTsKIAot
CWlmIChzb3JlYWRhYmxlKHhwcnQtPnhwX3NvY2tldCkpCi0JCXJldHVybiAoWFBSVF9NT1JFUkVR
Uyk7Ci0KLQlyZXR1cm4gKFhQUlRfSURMRSk7CisJcmV0dXJuIChyc3RhdCk7CiB9CiAKIHN0YXRp
YyBib29sX3QK
--_002_YQXPR0101MB0968E1537E26CDBDC31C58E5DD689YQXPR0101MB0968_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968E1537E26CDBDC31C58E5DD689>