Date: Sun, 4 Apr 2021 11:50:47 +0000 From: "Scheffenegger, Richard" <Richard.Scheffenegger@netapp.com> To: Rick Macklem <rmacklem@uoguelph.ca>, "tuexen@freebsd.org" <tuexen@freebsd.org> Cc: Youssef GHORBAL <youssef.ghorbal@pasteur.fr>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: NFS Mount Hangs Message-ID: <SN4PR0601MB3728E392BCA494EAD49605FE86789@SN4PR0601MB3728.namprd06.prod.outlook.com> In-Reply-To: <YQXPR0101MB0968E0A17D8BCACFAF132225DD7A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> References: <C643BB9C-6B61-4DAC-8CF9-CE04EA7292D0@tildenparkcapital.com> <3750001D-3F1C-4D9A-A9D9-98BCA6CA65A4@tildenparkcapital.com> <33693DE3-7FF8-4FAB-9A75-75576B88A566@tildenparkcapital.com> <D67AF317-D238-4EC0-8C7F-22D54AD5144C@pasteur.fr> <YQXPR0101MB09684AB7BEFA911213604467DD669@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <C87066D3-BBF1-44E1-8398-E4EB6903B0F2@tildenparkcapital.com> <8E745920-1092-4312-B251-B49D11FE8028@pasteur.fr> <YQXPR0101MB0968C44C7C82A3EB64F384D0DD7B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>, <DEF8564D-0FE9-4C2C-9F3B-9BCDD423377C@freebsd.org>, <YQXPR0101MB0968E0A17D8BCACFAF132225DD7A9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
For what it=91s worth, suse found two bugs in the linux nfconntrack (statef=
ul firewall), and pfifo-fast scheduler, which could conspire to make tcp se=
ssions hang forever.
One is a missed updaten when the c=F6ient is not using the noresvport moint=
option, which makes tje firewall think rsts are illegal (and drop them);
The fast scheduler can run into an issue if only a single packet should be =
forwarded (note that this is not the default scheduler, but often recommend=
ed for perf, as it runs lockless and lower cpu cost that pfq (default). If =
no other/additional packet pushes out that last packet of a flow, it can be=
come stuck forever...
I can try getting the relevant bug info next week...
________________________________
Von: owner-freebsd-net@freebsd.org <owner-freebsd-net@freebsd.org> im Auftr=
ag von Rick Macklem <rmacklem@uoguelph.ca>
Gesendet: Friday, April 2, 2021 11:31:01 PM
An: tuexen@freebsd.org <tuexen@freebsd.org>
Cc: Youssef GHORBAL <youssef.ghorbal@pasteur.fr>; freebsd-net@freebsd.org <=
freebsd-net@freebsd.org>
Betreff: Re: NFS Mount Hangs
NetApp Security WARNING: This is an external email. Do not click links or o=
pen attachments unless you recognize the sender and know the content is saf=
e.
tuexen@freebsd.org wrote:
>> On 2. Apr 2021, at 02:07, Rick Macklem <rmacklem@uoguelph.ca> wrote:
>>
>> I hope you don't mind a top post...
>> I've been testing network partitioning between the only Linux client
>> I have (5.2 kernel) and a FreeBSD server with the xprtdied.patch
>> (does soshutdown(..SHUT_WR) when it knows the socket is broken)
>> applied to it.
>>
>> I'm not enough of a TCP guy to know if this is useful, but here's what
>> I see...
>>
>> While partitioned:
>> On the FreeBSD server end, the socket either goes to CLOSED during
>> the network partition or stays ESTABLISHED.
>If it goes to CLOSED you called shutdown(, SHUT_WR) and the peer also
>sent a FIN, but you never called close() on the socket.
>If the socket stays in ESTABLISHED, there is no communication ongoing,
>I guess, and therefore the server does not even detect that the peer
>is not reachable.
>> On the Linux end, the socket seems to remain ESTABLISHED for a
>> little while, and then disappears.
>So how does Linux detect the peer is not reachable?
Well, here's what I see in a packet capture in the Linux client once
I partition it (just unplug the net cable):
- lots of retransmits of the same segment (with ACK) for 54sec
- then only ARP queries
Once I plug the net cable back in:
- ARP works
- one more retransmit of the same segement
- receives RST from FreeBSD
** So, is this now a "new" TCP connection, despite
using the same port#.
--> It matters for NFS, since "new connection"
implies "must retry all outstanding RPCs".
- sends SYN
- receives SYN, ACK from FreeBSD
--> connection starts working again
Always uses same port#.
On the FreeBSD server end:
- receives the last retransmit of the segment (with ACK)
- sends RST
- receives SYN
- sends SYN, ACK
I thought that there was no RST in the capture I looked at
yesterday, so I'm not sure if FreeBSD always sends an RST,
but the Linux client behaviour was the same. (Sent a SYN, etc).
The socket disappears from the Linux "netstat -a" and I
suspect that happens after about 54sec, but I am not sure
about the timing.
>>
>> After unpartitioning:
>> On the FreeBSD server end, you get another socket showing up at
>> the same port#
>> Active Internet connections (including servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address (state=
)
>> tcp4 0 0 nfsv4-new3.nfsd nfsv4-linux.678 ESTABL=
ISHED
>> tcp4 0 0 nfsv4-new3.nfsd nfsv4-linux.678 CLOSED
>>
>> The Linux client shows the same connection ESTABLISHED.
But disappears from "netstat -a" for a while during the partitioning.
>> (The mount sometimes reports an error. I haven't looked at packet
>> traces to see if it retries RPCs or why the errors occur.)
I have now done so, as above.
>> --> However I never get hangs.
>> Sometimes it goes to SYN_SENT for a while and the FreeBSD server
>> shows FIN_WAIT_1, but then both ends go to ESTABLISHED and the
>> mount starts working again.
>>
>> The most obvious thing is that the Linux client always keeps using
>> the same port#. (The FreeBSD client will use a different port# when
>> it does a TCP reconnect after no response from the NFS server for
>> a little while.)
>>
>> What do those TCP conversant think?
>I guess you are you are never calling close() on the socket, for with
>the connection state is CLOSED.
Ok, that makes sense. For this case the Linux client has not done a
BindConnectionToSession to re-assign the back channel.
I'll have to bug them about this. However, I'll bet they'll answer
that I have to tell them the back channel needs re-assignment
or something like that.
I am pretty certain they are broken, in that the client needs to
retry all outstanding RPCs.
For others, here's the long winded version of this that I just
put on the phabricator review:
In the server side kernel RPC, the socket (struct socket *) is in a
structure called SVCXPRT (normally pointed to by "xprt").
These structures a ref counted and the soclose() is done
when the ref. cnt goes to zero. My understanding is that
"struct socket *" is free'd by soclose() so this cannot be done
before the xprt ref. cnt goes to zero.
For NFSv4.1/4.2 there is something called a back channel
which means that a "xprt" is used for server->client RPCs,
although the TCP connection is established by the client
to the server.
--> This back channel holds a ref cnt on "xprt" until the
client re-assigns it to a different TCP connection
via an operation called BindConnectionToSession
and the Linux client is not doing this soon enough,
it appears.
So, the soclose() is delayed, which is why I think the
TCP connection gets stuck in CLOSE_WAIT and that is
why I've added the soshutdown(..SHUT_WR) calls,
which can happen before the client gets around to
re-assigning the back channel.
Thanks for your help with this Michael, rick
Best regards
Michael
>
> rick
> ps: I can capture packets while doing this, if anyone has a use
> for them.
>
>
>
>
>
>
> ________________________________________
> From: owner-freebsd-net@freebsd.org <owner-freebsd-net@freebsd.org> on be=
half of Youssef GHORBAL <youssef.ghorbal@pasteur.fr>
> Sent: Saturday, March 27, 2021 6:57 PM
> To: Jason Breitman
> Cc: Rick Macklem; freebsd-net@freebsd.org
> Subject: Re: NFS Mount Hangs
>
> CAUTION: This email originated from outside of the University of Guelph. =
Do not click links or open attachments unless you recognize the sender and =
know the content is safe. If in doubt, forward suspicious emails to IThelp@=
uoguelph.ca
>
>
>
>
> On 27 Mar 2021, at 13:20, Jason Breitman <jbreitman@tildenparkcapital.com=
<mailto:jbreitman@tildenparkcapital.com>> wrote:
>
> The issue happened again so we can say that disabling TSO and LRO on the =
NIC did not resolve this issue.
> # ifconfig lagg0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso
> # ifconfig lagg0
> lagg0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metri=
c 0 mtu 1500
> options=3D8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLA=
N_HWFILTER>
>
> We can also say that the sysctl settings did not resolve this issue.
>
> # sysctl net.inet.tcp.fast_finwait2_recycle=3D1
> net.inet.tcp.fast_finwait2_recycle: 0 -> 1
>
> # sysctl net.inet.tcp.finwait2_timeout=3D1000
> net.inet.tcp.finwait2_timeout: 60000 -> 1000
>
> I don=92t think those will do anything in your case since the FIN_WAIT2 a=
re on the client side and those sysctls are for BSD.
> By the way it seems that Linux recycles automatically TCP sessions in FIN=
_WAIT2 after 60 seconds (sysctl net.ipv4.tcp_fin_timeout)
>
> tcp_fin_timeout (integer; default: 60; since Linux 2.2)
> This specifies how many seconds to wait for a final FIN
> packet before the socket is forcibly closed. This is
> strictly a violation of the TCP specification, but
> required to prevent denial-of-service attacks. In Linux
> 2.2, the default value was 180.
>
> So I don=92t get why it stucks in the FIN_WAIT2 state anyway.
>
> You really need to have a packet capture during the outage (client and se=
rver side) so you=92ll get over the wire chat and start speculating from th=
ere.
> No need to capture the beginning of the outage for now. All you have to d=
o, is run a tcpdump for 10 minutes or so when you notice a client stuck.
>
> * I have not rebooted the NFS Server nor have I restarted nfsd, but do no=
t believe that is required as these settings are at the TCP level and I wou=
ld expect new sessions to use the updated settings.
>
> The issue occurred after 5 days following a reboot of the client machines=
.
> I ran the capture information again to make use of the situation.
>
> #!/bin/sh
>
> while true
> do
> /bin/date >> /tmp/nfs-hang.log
> /bin/ps axHl | grep nfsd | grep -v grep >> /tmp/nfs-hang.log
> /usr/bin/procstat -kk 2947 >> /tmp/nfs-hang.log
> /usr/bin/procstat -kk 2944 >> /tmp/nfs-hang.log
> /bin/sleep 60
> done
>
>
> On the NFS Server
> Active Internet connections (including servers)
> Proto Recv-Q Send-Q Local Address Foreign Address (state)
> tcp4 0 0 NFS.Server.IP.X.2049 NFS.Client.IP.X.48286 C=
LOSE_WAIT
>
> On the NFS Client
> tcp 0 0 NFS.Client.IP.X:48286 NFS.Server.IP.X:2049 =
FIN_WAIT2
>
>
>
> You had also asked for the output below.
>
> # nfsstat -E -s
> BackChannelCtBindConnToSes
> 0 0
>
> # sysctl vfs.nfsd.request_space_throttle_count
> vfs.nfsd.request_space_throttle_count: 0
>
> I see that you are testing a patch and I look forward to seeing the resul=
ts.
>
>
> Jason Breitman
>
>
> On Mar 21, 2021, at 6:21 PM, Rick Macklem <rmacklem@uoguelph.ca<mailto:rm=
acklem@uoguelph.ca>> wrote:
>
> Youssef GHORBAL <youssef.ghorbal@pasteur.fr<mailto:youssef.ghorbal@pasteu=
r.fr>> wrote:
>> Hi Jason,
>>
>>> On 17 Mar 2021, at 18:17, Jason Breitman <jbreitman@tildenparkcapital.c=
om<mailto:jbreitman@tildenparkcapital.com>> wrote:
>>>
>>> Please review the details below and let me know if there is a setting t=
hat I should apply to my FreeBSD NFS Server or if there is a bug fix that I=
can apply to resolve my issue.
>>> I shared this information with the linux-nfs mailing list and they beli=
eve the issue is on the server side.
>>>
>>> Issue
>>> NFSv4 mounts periodically hang on the NFS Client.
>>>
>>> During this time, it is possible to manually mount from another NFS Ser=
ver on the NFS Client having issues.
>>> Also, other NFS Clients are successfully mounting from the NFS Server i=
n question.
>>> Rebooting the NFS Client appears to be the only solution.
>>
>> I had experienced a similar weird situation with periodically stuck Linu=
x NFS clients >mounting Isilon NFS servers (Isilon is FreeBSD based but the=
y seem to have there >own nfsd)
> Yes, my understanding is that Isilon uses a proprietary user space nfsd a=
nd
> not the kernel based RPC and nfsd in FreeBSD.
>
>> We=92ve had better luck and we did manage to have packet captures on bot=
h sides >during the issue. The gist of it goes like follows:
>>
>> - Data flows correctly between SERVER and the CLIENT
>> - At some point SERVER starts decreasing it's TCP Receive Window until i=
t reachs 0
>> - The client (eager to send data) can only ack data sent by SERVER.
>> - When SERVER was done sending data, the client starts sending TCP Windo=
w >Probes hoping that the TCP Window opens again so he can flush its buffer=
s.
>> - SERVER responds with a TCP Zero Window to those probes.
> Having the window size drop to zero is not necessarily incorrect.
> If the server is overloaded (has a backlog of NFS requests), it can stop =
doing
> soreceive() on the socket (so the socket rcv buffer can fill up and the T=
CP window
> closes). This results in "backpressure" to stop the NFS client from flood=
ing the
> NFS server with requests.
> --> However, once the backlog is handled, the nfsd should start to sorece=
ive()
> again and this shouls cause the window to open back up.
> --> Maybe this is broken in the socket/TCP code. I quickly got lost in
> tcp_output() when it decides what to do about the rcvwin.
>
>> - After 6 minutes (the NFS server default Idle timeout) SERVER racefully=
closes the >TCP connection sending a FIN Packet (and still a TCP Window 0)
> This probably does not happen for Jason's case, since the 6minute timeout
> is disabled when the TCP connection is assigned as a backchannel (most li=
kely
> the case for NFSv4.1).
>
>> - CLIENT ACK that FIN.
>> - SERVER goes in FIN_WAIT_2 state
>> - CLIENT closes its half part part of the socket and goes in LAST_ACK st=
ate.
>> - FIN is never sent by the client since there still data in its SendQ an=
d receiver TCP >Window is still 0. At this stage the client starts sending =
TCP Window Probes again >and again hoping that the server opens its TCP Win=
dow so it can flush it's buffers >and terminate its side of the socket.
>> - SERVER keeps responding with a TCP Zero Window to those probes.
>> =3D> The last two steps goes on and on for hours/days freezing the NFS m=
ount bound >to that TCP session.
>>
>> If we had a situation where CLIENT was responsible for closing the TCP W=
indow (and >initiating the TCP FIN first) and server wanting to send data w=
e=92ll end up in the same >state as you I think.
>>
>> We=92ve never had the root cause of why the SERVER decided to close the =
TCP >Window and no more acccept data, the fix on the Isilon part was to rec=
ycle more >aggressively the FIN_WAIT_2 sockets (net.inet.tcp.fast_finwait2_=
recycle=3D1 & >net.inet.tcp.finwait2_timeout=3D5000). Once the socket recyc=
led and at the next >occurence of CLIENT TCP Window probe, SERVER sends a R=
ST, triggering the >teardown of the session on the client side, a new TCP h=
andchake, etc and traffic >flows again (NFS starts responding)
>>
>> To avoid rebooting the client (and before the aggressive FIN_WAIT_2 was =
>implemented on the Isilon side) we=92ve added a check script on the client=
that detects >LAST_ACK sockets on the client and through iptables rule enf=
orces a TCP RST, >Something like: -A OUTPUT -p tcp -d $nfs_server_addr --sp=
ort $local_port -j REJECT >--reject-with tcp-reset (the script removes this=
iptables rule as soon as the LAST_ACK >disappears)
>>
>> The bottom line would be to have a packet capture during the outage (cli=
ent and/or >server side), it will show you at least the shape of the TCP ex=
change when NFS is >stuck.
> Interesting story and good work w.r.t. sluething, Youssef, thanks.
>
> I looked at Jason's log and it shows everything is ok w.r.t the nfsd thre=
ads.
> (They're just waiting for RPC requests.)
> However, I do now think I know why the soclose() does not happen.
> When the TCP connection is assigned as a backchannel, that takes a refere=
nce
> cnt on the structure. This refcnt won't be released until the connection =
is
> replaced by a BindConnectiotoSession operation from the client. But that =
won't
> happen until the client creates a new TCP connection.
> --> No refcnt release-->no refcnt of 0-->no soclose().
>
> I've created the attached patch (completely different from the previous o=
ne)
> that adds soshutdown(SHUT_WR) calls in the three places where the TCP
> connection is going away. This seems to get it past CLOSE_WAIT without a
> soclose().
> --> I know you are not comfortable with patching your server, but I do th=
ink
> this change will get the socket shutdown to complete.
>
> There are a couple more things you can check on the server...
> # nfsstat -E -s
> --> Look for the count under "BindConnToSes".
> --> If non-zero, backchannels have been assigned
> # sysctl -a | fgrep request_space_throttle_count
> --> If non-zero, the server has been overloaded at some point.
>
> I think the attached patch might work around the problem.
> The code that should open up the receive window needs to be checked.
> I am also looking at enabling the 6minute timeout when a backchannel is
> assigned.
>
> rick
>
> Youssef
>
> _______________________________________________
> freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> mailing list
> https://urldefense.com/v3/__https://lists.freebsd.org/mailman/listinfo/fr=
eebsd-net__;!!JFdNOqOXpB6UZW0!_c2MFNbir59GXudWPVdE5bNBm-qqjXeBuJ2UEmFv5OZci=
Lj4ObR_drJNv5yryaERfIbhKR2d$
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org<mai=
lto:freebsd-net-unsubscribe@freebsd.org>"
> <xprtdied.patch>
>
> <nfs-hang.log.gz>
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?SN4PR0601MB3728E392BCA494EAD49605FE86789>