Date: Tue, 2 May 2023 00:46:01 -0700 From: Chen Shuo <chenshuo@chenshuo.com> To: freebsd-net@freebsd.org Subject: Cwnd grows slowly during slow-start due to LRO of the receiver side. Message-ID: <CAKZ7KuJdwaUZJrb4=XdQrVo_Nq2T%2B%2BzN_B3PT9W5Kh7ORx26HQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
As per newreno_ack_received() in sys/netinet/cc/cc_newreno.c,
FreeBSD TCP sender strictly follows RFC 5681 with RFC 3465 extension
That is, during slow-start, when receiving an ACK of 'bytes_acked'
cwnd += min(bytes_acked, abc_l_var * SMSS); // abc_l_var = 2 dflt
As discussed in sec3.2 of RFC 3465, L=2*SMSS bytes exactly balances
the negative impact of the delayed ACK algorithm. RFC 5681 also
requires that a receiver SHOULD generate an ACK for at least every
second full-sized segment, so bytes_acked per ACK is at most 2 * SMSS.
If both sender and receiver follow it. cwnd should grow exponentially
during slow-slow:
cwnd *= 2 (per RTT)
However, LRO and TSO are widely used today, so receiver may generate
much less ACKs than it used to do. As I observed, Both FreeBSD and
Linux generates at most one ACK per segment assembled by LRO/GRO.
The worst case is one ACK per 45 MSS, as 45 * 1448 = 65160 < 65535.
Sending 1MB over a link of 100ms delay from FreeBSD 13.2:
0.000 IP sender > sink: Flags [S], seq 205083268, win 65535, options
[mss 1460,nop,wscale 10,sackOK,TS val 495212525 ecr 0], length 0
0.100 IP sink > sender: Flags [S.], seq 708257395, ack 205083269, win
65160, options [mss 1460,sackOK,TS val 563185696 ecr
495212525,nop,wscale 7], length 0
0.100 IP sender > sink: Flags [.], ack 1, win 65, options [nop,nop,TS
val 495212626 ecr 563185696], length 0
// TSopt omitted below for brevity.
// cwnd = 10 * MSS, sent 10 * MSS
0.101 IP sender > sink: Flags [.], seq 1:14481, ack 1, win 65, length 14480
// got one ACK for 10 * MSS, cwnd += 2 * MSS, sent 12 * MSS
0.201 IP sink > sender: Flags [.], ack 14481, win 427, length 0
0.201 IP sender > sink: Flags [.], seq 14481:31857, ack 1, win 65, length 17376
// got ACK of 12*MSS above, cwnd += 2 * MSS, sent 14 * MSS
0.301 IP sink > sender: Flags [.], ack 31857, win 411, length 0
0.301 IP sender > sink: Flags [.], seq 31857:52129, ack 1, win 65, length 20272
// got ACK of 14*MSS above, cwnd += 2 * MSS, sent 16 * MSS
0.402 IP sink > sender: Flags [.], ack 52129, win 395, length 0
0.402 IP sender > sink: Flags [P.], seq 52129:73629, ack 1, win 65,
length 21500
0.402 IP sender > sink: Flags [.], seq 73629:75077, ack 1, win 65, length 1448
As a consequence, instead of growing exponentially, cwnd grows
more-or-less quadratically during slow-start, unless abc_l_var is
set to a sufficiently large value.
NewReno took more than 20 seconds to ramp up throughput to 100Mbps
over an emulated 100ms delay link. While Linux took ~2 seconds.
I can provide the pcap file if anyone is interested.
Switching to CUBIC won't help, because it uses the logic in NewReno
ack_received() for slow start.
Is this a well-known issue and abc_l_var is the only cure for it?
https://calomel.org/freebsd_network_tuning.html
Thank you!
Best,
Shuo Chen
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKZ7KuJdwaUZJrb4=XdQrVo_Nq2T%2B%2BzN_B3PT9W5Kh7ORx26HQ>