Date: Mon, 9 Dec 2024 17:56:11 +0100 From: Michael Tuexen <michael.tuexen@lurchi.franken.de> To: Pavel Vazharov <pavel@x3me.net> Cc: freebsd-net@freebsd.org Subject: Re: Interaction between the re-transmit and keep-alive logic. Message-ID: <8A8F8BFA-B407-4F76-AE93-AD679961BCC4@lurchi.franken.de> In-Reply-To: <CAJEV1ijkd1p%2Bs3PGgqraiFrBrQ9N0JVAOrC5%2B_OC8tccE-i42Q@mail.gmail.com> References: <CAJEV1ijkd1p%2Bs3PGgqraiFrBrQ9N0JVAOrC5%2B_OC8tccE-i42Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 9. Dec 2024, at 10:50, Pavel Vazharov <pavel@x3me.net> wrote:
>=20
> Hi there,
>=20
> We are using the network stack of FreeBSD 13 on top of DPDK in our =
application.
> During the last tests in the lab I stumbled upon the following =
situation:
> 1. It's a test where 5000 parallel connections are opened by Apache
> Bench and each one downloads 1MB data. It causes the client NIC to
> start dropping packets due to overflows which is intentional behavior.
> 2. The server side is our application with the FreeBSD stack. The
> client side Ubuntu 24.04 with Linux 6.8.0.
> 3. So, a connection is opened and the download starts on it. At some
> point the first drops occur and according to the TCP dump, from the
> client side, they take a few seconds before the connection heals up.
> However, these drops lead to increased values of t_srtt, t_rttvar and
> thus to increased value of t_rxtcur.
Do you observe increased values of t_rxtcur due to exponential backoff
or due to extreme values of t_srtt and t_rttvar?
>=20
> 4. The window opens again up to 100-200 KB with lots of packets
> in-flight and the drops start again. They cause the re-transmit timer
> from the FreeBSD side to be started but with an interval of something
> like 18-20 seconds (according to my printf debugging on this side).
> 5. At the same time the TCP keep-alive timer is also started for the
> same connection (it's enabled for all connections) with a timeout of
> 15 seconds.
> 6. Nothing happens on this connection for the next 15 seconds. I'm not
> sure why the Linux stack didn't send any "wake-up" ACK packets or
> something but the tcpdump from the client side shows full silence
> between 14-th and 29-th second.
> 7. Next the FreeBSD keep-alive logic kicks-in and sends an ACK packet
> which is ACK-ed by the Linux stack immediately. However, this ACK
> packet received by the FreeBSD stack leads to restart of the
> retransmit timer and with the interval which is bigger than the
> keep-alive interval.
> 8. Point 6 and 7 repeat one more time before the apache bench client
> gives up on this connection and declares that it's timed-out. My
> understanding is that the connection can "loop" in 6-7 for a very long
> time and a packet with data will never be retransmitted.
Can you provide a .pcap file?
Best regards
Michael
> 9. As far as I debugged the situation from the FreeBSD side the
> restart of the retransmit timer happens in the code after the
> `process_ACK` label, in the else branch here:
> ```
> if (th->th_ack =3D=3D tp->snd_max) {
> tcp_timer_activate(tp, TT_REXMT, 0);
> needoutput =3D 1;
> } else if (!tcp_timer_active(tp, TT_PERSIST))
> tcp_timer_activate(tp, TT_REXMT, tp->t_rxtcur);
> ```
>=20
> So, based on the above situation I've the following questions:
> 1. Would it be correct if the re-transmit timer is not restarted by
> keep-alive ACK packets?
> 2. Assuming that the above change won't break anything else, is there
> a way for detecting that an ACK packet acknowledges previously sent
> keep-alive packet?
>=20
> Regards,
> Pavel.
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8A8F8BFA-B407-4F76-AE93-AD679961BCC4>