Date: Mon, 19 Apr 2021 15:03:35 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Olav Gjerde <olav@backupbay.com> Cc: Allan Jude <allanjude@freebsd.org>, "freebsd-current@freebsd.org" <freebsd-current@freebsd.org> Subject: Re: NFS issues since upgrading to 13-RELEASE Message-ID: <YQXPR0101MB0968924C8AEAB808888D75B1DD499@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CAJ7kQyFBoSePYA%2Bz60ZfNpOGn6wspy-xPh4wOypw5U5RL9VUJw@mail.gmail.com> References: <902a3c81-2ce8-49c0-b163-5ffa4b90afe5@www.fastmail.com> <e8f585eb-a2a8-ae9d-7f33-526e412ec462@freebsd.org> <CAJ7kQyGQrxe7wJs%2BMezErdNUoLE1HEvD6ixiu2W5=irDNg2mOA@mail.gmail.com> <CAJ7kQyExd6LLg5mb4V_o9uR1fmEqz3P5JkUFuQ6XYjegba1toA@mail.gmail.com> <YQXPR0101MB096888C9CDFA3BC26833AFB2DD4C9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>, <CAJ7kQyFBoSePYA%2Bz60ZfNpOGn6wspy-xPh4wOypw5U5RL9VUJw@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
Olav Gjerde wrote:
>I have tried D29690 patch and reverting back to r367492 this weekend. Neither made any difference for my system.
Just to clarify it, I meant "revert the patch in r367492" and not
"revert to revision r367492". I've attached the patch that
backs out the changes made by the patch in r367492, which
should apply to a fairly recent main/13 kernel.
This should be done instead of applying D29690, not combined with it.
My testing of D29690 has suggested it is not yet mature, so
I would not recommend choosing that alternative yet.
If you have tried a kernel with the attached patch applied to it, but not
D29690 applied to it, then please:
Let us know if you still have Linux clients "hanging" with this kernel.
If still "hanging", try the following to see if they help:
- Use the "minorversion=1" mount option on the Linux clients,
to ensure that they are not using NFSv4.2, to see if it is a
NFSv4.2 specific issue.
- Try disabling tso and lro and avoid jumbo frames for drivers
that use jumbo mbufs when handling jumbo frames.
Collect the following info when it happens:
- "netstat -a", to see what the TCP connection is up to.
- "tcpdump -s 0 -w hang.pcap host <nfs-client>"
run for several minutes on the server, to see what is going on the
wire. I use wireshark to look at hang.pcap, since it
knows NFS as well as TCP.
You can also do the above with "host <nfs-server>" instead
of "host <nfs-client>" run on the client.
- "ps axHl" on the server, to see what the nfsd threads
are up to.
If none of the above contains confidential info, please
send it to me, if not the list.
Good luck with it, rick
ps: Yea, I started this post and then realized I had hit
reply instead of reply all.
There is also a reddit thread about this https://www.reddit.com/r/freebsd/comments/mqol4o/nfs_issues_since_upgrading_to_13release/
On Sat, Apr 17, 2021 at 1:10 AM Rick Macklem <rmacklem@uoguelph.ca<mailto:rmacklem@uoguelph.ca>> wrote:
Just fyi, I just got a "recursed on non-recursed mutex" panic in
socantrcvmore() with the D29690 patch, so you might not
want to test with that one yet.
rick
________________________________________
From: owner-freebsd-current@freebsd.org<mailto:owner-freebsd-current@freebsd.org> <owner-freebsd-current@freebsd.org<mailto:owner-freebsd-current@freebsd.org>> on behalf of Olav Gjerde <olav@backupbay.com<mailto:olav@backupbay.com>>
Sent: Thursday, April 15, 2021 3:21 PM
To: Allan Jude
Cc: freebsd-current@freebsd.org<mailto:freebsd-current@freebsd.org>
Subject: Re: NFS issues since upgrading to 13-RELEASE
CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca<mailto:IThelp@uoguelph.ca>
Well something do happen if I restart NFS Service on FreeBSD , it works for
like 10 seconds then it gets unresponsive again.
This is my output from `nfsstat -d 1`
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
8.00 1025 8.00 8.02 17170 134.54 2.01 72716 142.54 0.07 51 34
8.00 2273 17.76 7.99 31273 244.07 2.01 133267 261.83 0.14 20 82
8.03 4889 38.33 7.99 25885 202.07 2.06 119340 240.40 0.13 21 81
[===== Read =====] [===== Write ====] [=========== Total ============]
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s ms ql %b
7.98 8811 68.64 8.00 12997 101.54 2.22 78396 170.18 0.15 1 80
7.99 922 7.20 8.00 3798 29.68 2.10 17965 36.87 0.09 0 11
8.07 2959 23.31 0.00 0 0.00 2.67 8938 23.31 0.86 32 72
7.97 7088 55.18 0.00 0 0.00 2.66 21233 55.18 1.05 16 98
7.98 4666 36.38 0.00 0 0.00 2.66 13986 36.38 0.36 9 29
8.00 4513 35.24 8.00 7662 59.86 2.20 44188 95.10 0.27 10 49
7.98 4799 37.40 8.00 11422 89.23 2.16 60076 126.63 0.19 0 51
8.00 4322 33.76 0.00 0 0.00 2.67 12967 33.76 0.89 0 42
8.02 4839 37.91 0.00 0 0.00 2.67 14550 37.91 0.54 17 41
8.01 4516 35.32 0.00 0 0.00 2.67 13569 35.32 0.57 27 38
7.95 4459 34.62 8.00 1195 9.34 2.49 18109 43.96 0.55 0 45
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0
On Thu, Apr 15, 2021 at 9:07 PM Olav Gjerde <olav@backupbay.com<mailto:olav@backupbay.com>> wrote:
> I have the same issue, using Ubuntu 20.10 with Linux 5.8 kernel. The Linux
> NFS client will get unresponsive and it does not recover in my case, even
> if I restart NFS on FreeBSD. I upgraded from FreeBSD 12.1-RELEASE though.
>
> On Thu, Apr 15, 2021 at 8:36 PM Allan Jude <allanjude@freebsd.org<mailto:allanjude@freebsd.org>> wrote:
>
>> On 4/15/2021 9:22 AM, Chris Roose wrote:
>> > I posted this in -questions and someone suggested I post here as well.
>> >
>> > I'm having NFS availability issues between my Proxmox client and
>> FreeBSD server (10G link) since upgrading to 13-RELEASE. And unfortunately
>> I upgraded my ZFS pool to v2.0.0 before I noticed the issue, so I'm kind of
>> stuck.
>> >
>> > Periodically, the NFS server (I've tried both v3 and v4.2 clients) will
>> go unresponsive for several minutes. I never had this problem on 12.2, and
>> as far as I can tell it's not a disk or network I/O issue. I'll get several
>> "nfs: server not responding, still trying" messages on the client and a few
>> minutes later it usually recovers. It's not clear to me yet what's causing
>> the block. Restarting nfsd on the server will resolve the issue if it
>> doesn't clear itself.
>> >
>> > Any pointers for troubleshooting this? I've been looking through
>> vmstat, gstat, top, etc. when the problem occurs, but I haven't been able
>> to pinpoint the issue. I can get pcap, but it would be from the hosts,
>> because I don't have a 10G tap or managed switch.
>> >
>>
>> run `nfsstat -d 1` and try to capture a few lines from before, during,
>> and after the stall, and that may provide some insight.
>>
>> Specifically, does the queue length grow, suggesting it is waiting on
>> the I/O subsystem, or does it just stop getting traffic all together.
>>
>>
>> --
>> Allan Jude
>> _______________________________________________
>> freebsd-current@freebsd.org<mailto:freebsd-current@freebsd.org> mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org<mailto:freebsd-current-unsubscribe@freebsd.org>
>> "
>>
>
>
> --
> Kind Regards / Med Vennlig Hilsen
>
> Olav Grønås Gjerde
>
> BackupBay Gjerde
> Madlaforen 35
> 4042 HAFRSFJORD
> Norway
> Phone: +47 918 000 59
>
--
Kind Regards / Med Vennlig Hilsen
Olav Grønås Gjerde
BackupBay Gjerde
Madlaforen 35
4042 HAFRSFJORD
Norway
Phone: +47 918 000 59
_______________________________________________
freebsd-current@freebsd.org<mailto:freebsd-current@freebsd.org> mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org<mailto:freebsd-current-unsubscribe@freebsd.org>"
--
Kind Regards / Med Vennlig Hilsen
Olav Grønås Gjerde
BackupBay Gjerde
Madlaforen 35
4042 HAFRSFJORD
Norway
Phone: +47 918 000 59
[-- Attachment #2 --]
--- sys/netinet/tcp_stacks/bbr.c.orig 2021-04-09 08:24:05.364212000 -0700
+++ sys/netinet/tcp_stacks/bbr.c 2021-04-09 08:33:49.799902000 -0700
@@ -7876,8 +7876,7 @@ bbr_process_ack(struct mbuf *m, struct tcphdr *th, str
acked_amount = min(acked, (int)sbavail(&so->so_snd));
tp->snd_wnd -= acked_amount;
mfree = sbcut_locked(&so->so_snd, acked_amount);
- SOCKBUF_UNLOCK(&so->so_snd);
- tp->t_flags |= TF_WAKESOW;
+ sowwakeup_locked(so);
m_freem(mfree);
if (SEQ_GT(th->th_ack, tp->snd_una)) {
bbr_collapse_rtt(tp, bbr, TCP_REXMTVAL(tp));
@@ -8353,8 +8352,7 @@ bbr_process_data(struct mbuf *m, struct tcphdr *th, st
appended =
#endif
sbappendstream_locked(&so->so_rcv, m, 0);
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ sorwakeup_locked(so);
#ifdef NETFLIX_SB_LIMITS
if (so->so_rcv.sb_shlim && appended != mcnt)
counter_fo_release(so->so_rcv.sb_shlim,
@@ -8414,8 +8412,6 @@ bbr_process_data(struct mbuf *m, struct tcphdr *th, st
if (thflags & TH_FIN) {
if (TCPS_HAVERCVDFIN(tp->t_state) == 0) {
socantrcvmore(so);
- /* The socket upcall is handled by socantrcvmore. */
- tp->t_flags &= ~TF_WAKESOR;
/*
* If connection is half-synchronized (ie NEEDSYN
* flag on) then delay ACK, so it may be piggybacked
@@ -8606,8 +8602,7 @@ bbr_do_fastnewdata(struct mbuf *m, struct tcphdr *th,
sbappendstream_locked(&so->so_rcv, m, 0);
ctf_calc_rwin(so, tp);
}
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ sorwakeup_locked(so);
#ifdef NETFLIX_SB_LIMITS
if (so->so_rcv.sb_shlim && mcnt != appended)
counter_fo_release(so->so_rcv.sb_shlim, mcnt - appended);
@@ -8798,7 +8793,7 @@ bbr_fastack(struct mbuf *m, struct tcphdr *th, struct
&tcp_savetcp, 0);
#endif
/* Wake up the socket if we have room to write more */
- tp->t_flags |= TF_WAKESOW;
+ sowwakeup(so);
if (tp->snd_una == tp->snd_max) {
/* Nothing left outstanding */
bbr_log_progress_event(bbr, tp, ticks, PROGRESS_CLEAR, __LINE__);
@@ -11754,10 +11749,8 @@ bbr_do_segment(struct mbuf *m, struct tcphdr *th, stru
}
retval = bbr_do_segment_nounlock(m, th, so, tp,
drop_hdrlen, tlen, iptos, 0, &tv);
- if (retval == 0) {
- tcp_handle_wakeup(tp, so);
+ if (retval == 0)
INP_WUNLOCK(tp->t_inpcb);
- }
}
/*
--- sys/netinet/tcp_stacks/rack.c.orig 2021-04-09 08:36:23.622821000 -0700
+++ sys/netinet/tcp_stacks/rack.c 2021-04-09 08:41:24.096687000 -0700
@@ -8344,8 +8344,7 @@ rack_process_ack(struct mbuf *m, struct tcphdr *th, st
*/
ourfinisacked = 1;
}
- SOCKBUF_UNLOCK(&so->so_snd);
- tp->t_flags |= TF_WAKESOW;
+ sowwakeup_locked(so);
m_freem(mfree);
if (rack->r_ctl.rc_early_recovery == 0) {
if (IN_RECOVERY(tp->t_flags)) {
@@ -8665,8 +8664,7 @@ rack_process_data(struct mbuf *m, struct tcphdr *th, s
appended =
#endif
sbappendstream_locked(&so->so_rcv, m, 0);
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ sorwakeup_locked(so);
#ifdef NETFLIX_SB_LIMITS
if (so->so_rcv.sb_shlim && appended != mcnt)
counter_fo_release(so->so_rcv.sb_shlim,
@@ -8731,8 +8729,6 @@ rack_process_data(struct mbuf *m, struct tcphdr *th, s
if (thflags & TH_FIN) {
if (TCPS_HAVERCVDFIN(tp->t_state) == 0) {
socantrcvmore(so);
- /* The socket upcall is handled by socantrcvmore. */
- tp->t_flags &= ~TF_WAKESOR;
/*
* If connection is half-synchronized (ie NEEDSYN
* flag on) then delay ACK, so it may be piggybacked
@@ -8924,8 +8920,7 @@ rack_do_fastnewdata(struct mbuf *m, struct tcphdr *th,
sbappendstream_locked(&so->so_rcv, m, 0);
ctf_calc_rwin(so, tp);
}
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ sorwakeup_locked(so);
#ifdef NETFLIX_SB_LIMITS
if (so->so_rcv.sb_shlim && mcnt != appended)
counter_fo_release(so->so_rcv.sb_shlim, mcnt - appended);
@@ -9142,7 +9137,7 @@ rack_fastack(struct mbuf *m, struct tcphdr *th, struct
rack_timer_cancel(tp, rack, rack->r_ctl.rc_rcvtime, __LINE__);
}
/* Wake up the socket if we have room to write more */
- tp->t_flags |= TF_WAKESOW;
+ sowwakeup(so);
if (sbavail(&so->so_snd)) {
rack->r_wanted_output = 1;
}
@@ -11205,10 +11200,8 @@ rack_do_segment(struct mbuf *m, struct tcphdr *th, str
tcp_get_usecs(&tv);
}
if(rack_do_segment_nounlock(m, th, so, tp,
- drop_hdrlen, tlen, iptos, 0, &tv) == 0) {
- tcp_handle_wakeup(tp, so);
+ drop_hdrlen, tlen, iptos, 0, &tv) == 0)
INP_WUNLOCK(tp->t_inpcb);
- }
}
struct rack_sendmap *
--- sys/netinet/tcp_stacks/rack_bbr_common.c.orig 2021-04-09 08:45:26.721521000 -0700
+++ sys/netinet/tcp_stacks/rack_bbr_common.c 2021-04-09 08:46:58.580234000 -0700
@@ -458,7 +458,6 @@ ctf_do_queued_segments(struct socket *so, struct tcpcb
/* We lost the tcpcb (maybe a RST came in)? */
return(1);
}
- tcp_handle_wakeup(tp, so);
}
return (0);
}
--- sys/netinet/tcp_input.c.orig 2021-04-05 01:07:00.342559000 -0700
+++ sys/netinet/tcp_input.c 2021-04-09 07:58:03.262815000 -0700
@@ -1472,29 +1472,6 @@ tcp_autorcvbuf(struct mbuf *m, struct tcphdr *th, stru
}
void
-tcp_handle_wakeup(struct tcpcb *tp, struct socket *so)
-{
- /*
- * Since tp might be gone if the session entered
- * the TIME_WAIT state before coming here, we need
- * to check if the socket is still connected.
- */
- if ((so->so_state & SS_ISCONNECTED) == 0)
- return;
- INP_LOCK_ASSERT(tp->t_inpcb);
- if (tp->t_flags & TF_WAKESOR) {
- tp->t_flags &= ~TF_WAKESOR;
- SOCKBUF_UNLOCK_ASSERT(&so->so_rcv);
- sorwakeup(so);
- }
- if (tp->t_flags & TF_WAKESOW) {
- tp->t_flags &= ~TF_WAKESOW;
- SOCKBUF_UNLOCK_ASSERT(&so->so_snd);
- sowwakeup(so);
- }
-}
-
-void
tcp_do_segment(struct mbuf *m, struct tcphdr *th, struct socket *so,
struct tcpcb *tp, int drop_hdrlen, int tlen, uint8_t iptos)
{
@@ -1863,7 +1840,7 @@ tcp_do_segment(struct mbuf *m, struct tcphdr *th, stru
else if (!tcp_timer_active(tp, TT_PERSIST))
tcp_timer_activate(tp, TT_REXMT,
tp->t_rxtcur);
- tp->t_flags |= TF_WAKESOW;
+ sowwakeup(so);
if (sbavail(&so->so_snd))
(void) tp->t_fb->tfb_tcp_output(tp);
goto check_delack;
@@ -1928,8 +1905,8 @@ tcp_do_segment(struct mbuf *m, struct tcphdr *th, stru
m_adj(m, drop_hdrlen); /* delayed header drop */
sbappendstream_locked(&so->so_rcv, m, 0);
}
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ /* NB: sorwakeup_locked() does an implicit unlock. */
+ sorwakeup_locked(so);
if (DELAY_ACK(tp, tlen)) {
tp->t_flags |= TF_DELACK;
} else {
@@ -2925,8 +2902,8 @@ process_ACK:
tp->snd_wnd = 0;
ourfinisacked = 0;
}
- SOCKBUF_UNLOCK(&so->so_snd);
- tp->t_flags |= TF_WAKESOW;
+ /* NB: sowwakeup_locked() does an implicit unlock. */
+ sowwakeup_locked(so);
m_freem(mfree);
/* Detect una wraparound. */
if (!IN_RECOVERY(tp->t_flags) &&
@@ -3147,8 +3124,8 @@ dodata: /* XXX */
m_freem(m);
else
sbappendstream_locked(&so->so_rcv, m, 0);
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ /* NB: sorwakeup_locked() does an implicit unlock. */
+ sorwakeup_locked(so);
} else {
/*
* XXX: Due to the header drop above "th" is
@@ -3215,8 +3192,6 @@ dodata: /* XXX */
if (thflags & TH_FIN) {
if (TCPS_HAVERCVDFIN(tp->t_state) == 0) {
socantrcvmore(so);
- /* The socket upcall is handled by socantrcvmore. */
- tp->t_flags &= ~TF_WAKESOR;
/*
* If connection is half-synchronized
* (ie NEEDSYN flag on) then delay ACK,
@@ -3280,7 +3255,6 @@ check_delack:
tp->t_flags &= ~TF_DELACK;
tcp_timer_activate(tp, TT_DELACK, tcp_delacktime);
}
- tcp_handle_wakeup(tp, so);
INP_WUNLOCK(tp->t_inpcb);
return;
@@ -3314,7 +3288,6 @@ dropafterack:
TCP_PROBE3(debug__input, tp, th, m);
tp->t_flags |= TF_ACKNOW;
(void) tp->t_fb->tfb_tcp_output(tp);
- tcp_handle_wakeup(tp, so);
INP_WUNLOCK(tp->t_inpcb);
m_freem(m);
return;
@@ -3322,7 +3295,6 @@ dropafterack:
dropwithreset:
if (tp != NULL) {
tcp_dropwithreset(m, th, tp, tlen, rstreason);
- tcp_handle_wakeup(tp, so);
INP_WUNLOCK(tp->t_inpcb);
} else
tcp_dropwithreset(m, th, NULL, tlen, rstreason);
@@ -3338,10 +3310,8 @@ drop:
&tcp_savetcp, 0);
#endif
TCP_PROBE3(debug__input, tp, th, m);
- if (tp != NULL) {
- tcp_handle_wakeup(tp, so);
+ if (tp != NULL)
INP_WUNLOCK(tp->t_inpcb);
- }
m_freem(m);
}
--- sys/netinet/tcp_reass.c.orig 2021-04-09 08:18:10.599092000 -0700
+++ sys/netinet/tcp_reass.c 2021-04-09 08:19:54.912378000 -0700
@@ -959,8 +959,7 @@ new_entry:
} else {
sbappendstream_locked(&so->so_rcv, m, 0);
}
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ sorwakeup_locked(so);
return (flags);
}
if (tcp_new_limits) {
@@ -1108,7 +1107,6 @@ present:
#ifdef TCP_REASS_LOGGING
tcp_reass_log_dump(tp);
#endif
- SOCKBUF_UNLOCK(&so->so_rcv);
- tp->t_flags |= TF_WAKESOR;
+ sorwakeup_locked(so);
return (flags);
}
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968924C8AEAB808888D75B1DD499>