From nobody Tue May 2 13:55:20 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9hRd3hxrz491Ww; Tue, 2 May 2023 13:55:25 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9hRd0grrz44VK; Tue, 2 May 2023 13:55:24 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Authentication-Results: mx1.freebsd.org; none Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 342DtKZD021077; Tue, 2 May 2023 06:55:20 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 342DtKWj021076; Tue, 2 May 2023 06:55:20 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <202305021355.342DtKWj021076@gndrsh.dnsmgr.net> Subject: Re: Cwnd grows slowly during slow-start due to LRO of the receiver side. In-Reply-To: <202305021346.342Dkt32021016@gndrsh.dnsmgr.net> To: "Rodney W. Grimes" Date: Tue, 2 May 2023 06:55:20 -0700 (PDT) CC: Chen Shuo , freebsd-net@FreeBSD.org, freebsd-transport@FreeBSD.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 4Q9hRd0grrz44VK X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Second attempt, first one failed due to not being a member of the list :-(. > Adding freebsd-transport@freebsd.org to get that specific groups > eyes on this issue. > > Rod > > > As per newreno_ack_received() in sys/netinet/cc/cc_newreno.c, > > FreeBSD TCP sender strictly follows RFC 5681 with RFC 3465 extension > > That is, during slow-start, when receiving an ACK of 'bytes_acked' > > > > cwnd += min(bytes_acked, abc_l_var * SMSS); // abc_l_var = 2 dflt > > > > As discussed in sec3.2 of RFC 3465, L=2*SMSS bytes exactly balances > > the negative impact of the delayed ACK algorithm. RFC 5681 also > > requires that a receiver SHOULD generate an ACK for at least every > > second full-sized segment, so bytes_acked per ACK is at most 2 * SMSS. > > If both sender and receiver follow it. cwnd should grow exponentially > > during slow-slow: > > > > cwnd *= 2 (per RTT) > > > > However, LRO and TSO are widely used today, so receiver may generate > > much less ACKs than it used to do. As I observed, Both FreeBSD and > > Linux generates at most one ACK per segment assembled by LRO/GRO. > > The worst case is one ACK per 45 MSS, as 45 * 1448 = 65160 < 65535. > > > > Sending 1MB over a link of 100ms delay from FreeBSD 13.2: > > > > 0.000 IP sender > sink: Flags [S], seq 205083268, win 65535, options > > [mss 1460,nop,wscale 10,sackOK,TS val 495212525 ecr 0], length 0 > > 0.100 IP sink > sender: Flags [S.], seq 708257395, ack 205083269, win > > 65160, options [mss 1460,sackOK,TS val 563185696 ecr > > 495212525,nop,wscale 7], length 0 > > 0.100 IP sender > sink: Flags [.], ack 1, win 65, options [nop,nop,TS > > val 495212626 ecr 563185696], length 0 > > // TSopt omitted below for brevity. > > > > // cwnd = 10 * MSS, sent 10 * MSS > > 0.101 IP sender > sink: Flags [.], seq 1:14481, ack 1, win 65, length 14480 > > > > // got one ACK for 10 * MSS, cwnd += 2 * MSS, sent 12 * MSS > > 0.201 IP sink > sender: Flags [.], ack 14481, win 427, length 0 > > 0.201 IP sender > sink: Flags [.], seq 14481:31857, ack 1, win 65, length 17376 > > > > // got ACK of 12*MSS above, cwnd += 2 * MSS, sent 14 * MSS > > 0.301 IP sink > sender: Flags [.], ack 31857, win 411, length 0 > > 0.301 IP sender > sink: Flags [.], seq 31857:52129, ack 1, win 65, length 20272 > > > > // got ACK of 14*MSS above, cwnd += 2 * MSS, sent 16 * MSS > > 0.402 IP sink > sender: Flags [.], ack 52129, win 395, length 0 > > 0.402 IP sender > sink: Flags [P.], seq 52129:73629, ack 1, win 65, > > length 21500 > > 0.402 IP sender > sink: Flags [.], seq 73629:75077, ack 1, win 65, length 1448 > > > > As a consequence, instead of growing exponentially, cwnd grows > > more-or-less quadratically during slow-start, unless abc_l_var is > > set to a sufficiently large value. > > > > NewReno took more than 20 seconds to ramp up throughput to 100Mbps > > over an emulated 100ms delay link. While Linux took ~2 seconds. > > I can provide the pcap file if anyone is interested. > > > > Switching to CUBIC won't help, because it uses the logic in NewReno > > ack_received() for slow start. > > > > Is this a well-known issue and abc_l_var is the only cure for it? > > https://calomel.org/freebsd_network_tuning.html > > > > Thank you! > > > > Best, > > Shuo Chen > > > > > > -- > Rod Grimes rgrimes@freebsd.org > > -- Rod Grimes rgrimes@freebsd.org From nobody Thu May 4 18:47:00 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QC2qD6Hcfz49PqK for ; Thu, 4 May 2023 18:47:04 +0000 (UTC) (envelope-from rrs@netflix.com) Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QC2qD0plwz4HVm for ; Thu, 4 May 2023 18:47:04 +0000 (UTC) (envelope-from rrs@netflix.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netflix.com header.s=google header.b=ni5d+QGW; spf=pass (mx1.freebsd.org: domain of rrs@netflix.com designates 2607:f8b0:4864:20::72a as permitted sender) smtp.mailfrom=rrs@netflix.com; dmarc=pass (policy=reject) header.from=netflix.com Received: by mail-qk1-x72a.google.com with SMTP id af79cd13be357-74cebbb7bc5so72728885a.3 for ; Thu, 04 May 2023 11:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; t=1683226023; x=1685818023; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=15GOBIxQUcLqUrJAG0318sTIf3HcXUuG1g8U1lrRDxs=; b=ni5d+QGWLNn9UlwK3Gv/QpifVPE3yHulSUqr1zRRWSq1jPIYIOzOeql7xjonYIE6Oo 8y3j+nPH4casCaUc65/kSE0jf5++KMtFgbO9quq4vjaCB+mldZJECFFz2y62FNC2fAWj BBdQPmQZwmuj8PDoX6p/7BHTuNoqKHwYFJ8U8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683226023; x=1685818023; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=15GOBIxQUcLqUrJAG0318sTIf3HcXUuG1g8U1lrRDxs=; b=F62yF3GhkjHv1niAmB5HYoAmr1IMl+1vqSQjh39RlUyOqOMr4n+mY6u8Z4+amC0idH VWkU1tDE5n4v1eV3butKajhhMzEe8ZAapXPg3egN6D2JbltzvSunoqujTYalJNaVuXgo Ll8vv0NYxOrV7gkEU4XKuzOWv6QQuWo2iIWwVlPVkkXbQLVd9LGyn3d0gk3XxkQLGnQq 576hgIoDzOFb9HDrwRvulGVaQ5h+0cCKpRxZ+qgqm9Zf5shZYP8n4aLVRlu30MS0n6TS RPhAjx55BbsJFzIdIfGW9YnhbNTdOOAInoaTODzINqV4r2N/MrhhdwjgmC28JStnJyyf ydYw== X-Gm-Message-State: AC+VfDwKnQQp9gR+yWk6iF5HAPxHqyHRPcqZ3w6mZSSVjSKiqpHm93ky WJ8b+c1F/PsaIgZCh+YRBDvkBNw1QWePpvlBbqfKGA== X-Google-Smtp-Source: ACHHUZ6ZZRHqFe8mwmaDImW59dBXVyFRXJhC/YB4j/oOxdG95IfIiJOA/t62Vur3PfWxwv2qba6Xtg== X-Received: by 2002:a05:6214:ac1:b0:616:7e1f:b004 with SMTP id g1-20020a0562140ac100b006167e1fb004mr17394820qvi.35.1683226022931; Thu, 04 May 2023 11:47:02 -0700 (PDT) Received: from smtpclient.apple ([2607:fb10:7061:101::62e]) by smtp.gmail.com with ESMTPSA id dl9-20020a05620a1d0900b0074d4cf8f9fcsm11739564qkb.107.2023.05.04.11.47.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 May 2023 11:47:02 -0700 (PDT) From: Randall Stewart Message-Id: <56338AD8-60B6-4B6B-AE1D-B48ED8D28909@netflix.com> Content-Type: multipart/signed; boundary="Apple-Mail=_1B855FBF-EC26-4490-9A2C-6AA3C5357C0D"; protocol="application/pkcs7-signature"; micalg=sha-256 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\)) Subject: Re: Cwnd grows slowly during slow-start due to LRO of the receiver side. Date: Thu, 4 May 2023 14:47:00 -0400 In-Reply-To: <202305021355.342DtKWj021076@gndrsh.dnsmgr.net> Cc: Chen Shuo , freebsd-net , freebsd-transport@freebsd.org To: "Rodney W. Grimes" References: <202305021355.342DtKWj021076@gndrsh.dnsmgr.net> X-Mailer: Apple Mail (2.3696.120.41.1.2) X-Spamd-Result: default: False [-15.59 / 15.00]; WHITELIST_DMARC(-7.00)[netflix.com:D:+]; WHITELIST_SPF_DKIM(-3.00)[netflix.com:d:+,netflix.com:s:+]; SIGNED_SMIME(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[netflix.com,reject]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[netflix.com:s=google]; HAS_GOOGLE_REDIR(0.01)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72a:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-transport@freebsd.org]; RCPT_COUNT_THREE(0.00)[4]; PREVIOUSLY_DELIVERED(0.00)[freebsd-transport@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; HAS_ATTACHMENT(0.00)[]; TO_DN_SOME(0.00)[]; FREEFALL_USER(0.00)[rrs]; DKIM_TRACE(0.00)[netflix.com:+]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4QC2qD0plwz4HVm X-Spamd-Bar: --------------- X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_1B855FBF-EC26-4490-9A2C-6AA3C5357C0D Content-Type: multipart/alternative; boundary="Apple-Mail=_FB2B8B55-52E3-4FD9-9253-EDCFAB8012E6" --Apple-Mail=_FB2B8B55-52E3-4FD9-9253-EDCFAB8012E6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Rodney/Chen This is a real issue in the internet=E2=80=A6 and its not just LRO/TSO = making this all happen. You have cable modem technology that will batch up and keep = the most recent ack and thus aggregate some number of acks (I have seen up = to 10 acks eaten this way.. each of those for 2 segments)..=20 You have other middle boxes as well doing similar things and then there = is the channel access technology that at least gives you all the acks only = issue is they store them up and release them all at once so forget getting a nice ack-clocking coming out of the stack. The only way to deal with it is to generally raise abc_l_var to a much = larger value. That way has you get an aggregated ack your cwnd will open.. down = side is this lets you be more bursty=E2=80=A6 pacing can help here but only = the bbr and rack pace in FreeBSD=E2=80=A6 R > On May 2, 2023, at 9:55 AM, Rodney W. Grimes = wrote: >=20 > Second attempt, first one failed due to not being a member > of the list :-(. >=20 >> Adding freebsd-transport@freebsd.org to get that specific groups >> eyes on this issue. >>=20 >> Rod >>=20 >>> As per newreno_ack_received() in sys/netinet/cc/cc_newreno.c, >>> FreeBSD TCP sender strictly follows RFC 5681 with RFC 3465 extension >>> That is, during slow-start, when receiving an ACK of 'bytes_acked' >>>=20 >>> cwnd +=3D min(bytes_acked, abc_l_var * SMSS); // abc_l_var =3D 2 = dflt >>>=20 >>> As discussed in sec3.2 of RFC 3465, L=3D2*SMSS bytes exactly = balances >>> the negative impact of the delayed ACK algorithm. RFC 5681 also >>> requires that a receiver SHOULD generate an ACK for at least every >>> second full-sized segment, so bytes_acked per ACK is at most 2 * = SMSS. >>> If both sender and receiver follow it. cwnd should grow = exponentially >>> during slow-slow: >>>=20 >>> cwnd *=3D 2 (per RTT) >>>=20 >>> However, LRO and TSO are widely used today, so receiver may generate >>> much less ACKs than it used to do. As I observed, Both FreeBSD and >>> Linux generates at most one ACK per segment assembled by LRO/GRO. >>> The worst case is one ACK per 45 MSS, as 45 * 1448 =3D 65160 < = 65535. >>>=20 >>> Sending 1MB over a link of 100ms delay from FreeBSD 13.2: >>>=20 >>> 0.000 IP sender > sink: Flags [S], seq 205083268, win 65535, options >>> [mss 1460,nop,wscale 10,sackOK,TS val 495212525 ecr 0], length 0 >>> 0.100 IP sink > sender: Flags [S.], seq 708257395, ack 205083269, = win >>> 65160, options [mss 1460,sackOK,TS val 563185696 ecr >>> 495212525,nop,wscale 7], length 0 >>> 0.100 IP sender > sink: Flags [.], ack 1, win 65, options = [nop,nop,TS >>> val 495212626 ecr 563185696], length 0 >>> // TSopt omitted below for brevity. >>>=20 >>> // cwnd =3D 10 * MSS, sent 10 * MSS >>> 0.101 IP sender > sink: Flags [.], seq 1:14481, ack 1, win 65, = length 14480 >>>=20 >>> // got one ACK for 10 * MSS, cwnd +=3D 2 * MSS, sent 12 * MSS >>> 0.201 IP sink > sender: Flags [.], ack 14481, win 427, length 0 >>> 0.201 IP sender > sink: Flags [.], seq 14481:31857, ack 1, win 65, = length 17376 >>>=20 >>> // got ACK of 12*MSS above, cwnd +=3D 2 * MSS, sent 14 * MSS >>> 0.301 IP sink > sender: Flags [.], ack 31857, win 411, length 0 >>> 0.301 IP sender > sink: Flags [.], seq 31857:52129, ack 1, win 65, = length 20272 >>>=20 >>> // got ACK of 14*MSS above, cwnd +=3D 2 * MSS, sent 16 * MSS >>> 0.402 IP sink > sender: Flags [.], ack 52129, win 395, length 0 >>> 0.402 IP sender > sink: Flags [P.], seq 52129:73629, ack 1, win 65, >>> length 21500 >>> 0.402 IP sender > sink: Flags [.], seq 73629:75077, ack 1, win 65, = length 1448 >>>=20 >>> As a consequence, instead of growing exponentially, cwnd grows >>> more-or-less quadratically during slow-start, unless abc_l_var is >>> set to a sufficiently large value. >>>=20 >>> NewReno took more than 20 seconds to ramp up throughput to 100Mbps >>> over an emulated 100ms delay link. While Linux took ~2 seconds. >>> I can provide the pcap file if anyone is interested. >>>=20 >>> Switching to CUBIC won't help, because it uses the logic in NewReno >>> ack_received() for slow start. >>>=20 >>> Is this a well-known issue and abc_l_var is the only cure for it? >>> = https://www.google.com/url?q=3Dhttps://calomel.org/freebsd_network_tuning.= html&source=3Dgmail-imap&ust=3D1683640529000000&usg=3DAOvVaw0MoyDmFAOg9MlB= 5yX3FzJP >>>=20 >>> Thank you! >>>=20 >>> Best, >>> Shuo Chen >>>=20 >>>=20 >>=20 >> --=20 >> Rod Grimes = rgrimes@freebsd.org >>=20 >>=20 >=20 > --=20 > Rod Grimes = rgrimes@freebsd.org >=20 ------ Randall Stewart rrs@netflix.com --Apple-Mail=_FB2B8B55-52E3-4FD9-9253-EDCFAB8012E6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Rodney/Chen

This is = a real issue in the internet=E2=80=A6 and its not just LRO/TSO making = this
all happen. You have cable modem technology that = will batch up and keep the
most recent ack and thus = aggregate some number of acks (I have seen up to
10 = acks eaten this way.. each of those for 2 segments).. 

You have other middle = boxes as well doing similar things and then there is the
channel access technology that at least gives you all the = acks only issue is
they store them up and release = them all at once so forget getting a nice
ack-clocking coming out of the stack.

The only way to deal with it is to = generally raise abc_l_var to a much larger
value. = That way has you get an aggregated ack your cwnd will open.. down = side
is this lets you be more bursty=E2=80=A6 = pacing can help here but only the bbr and rack
pace = in FreeBSD=E2=80=A6

R

On May 2, 2023, at 9:55 AM, Rodney W. Grimes = <freebsd-rwg@gndrsh.dnsmgr.net> wrote:

Second= attempt, first one failed due to not being a member
of = the list :-(.

Adding freebsd-transport@freebsd.org to get that specific = groups
eyes on this issue.

Rod

As per newreno_ack_received() in = sys/netinet/cc/cc_newreno.c,
FreeBSD TCP sender strictly = follows RFC 5681 with RFC 3465 extension
That is, during = slow-start, when receiving an ACK of 'bytes_acked'

   cwnd +=3D min(bytes_acked, abc_l_var * = SMSS);  // abc_l_var =3D 2 dflt

As = discussed in sec3.2 of RFC 3465, L=3D2*SMSS bytes exactly balances
the negative impact of the delayed ACK algorithm.  RFC = 5681 also
requires that a receiver SHOULD generate an ACK = for at least every
second full-sized segment, so = bytes_acked per ACK is at most 2 * SMSS.
If both sender = and receiver follow it. cwnd should grow exponentially
during slow-slow:

=    cwnd *=3D 2    (per RTT)

However, LRO and TSO are widely used today, so = receiver may generate
much less ACKs than it used to do. =  As I observed, Both FreeBSD and
Linux generates at = most one ACK per segment assembled by LRO/GRO.
The worst = case is one ACK per 45 MSS, as 45 * 1448 =3D 65160 < 65535.

Sending 1MB over a link of 100ms delay from = FreeBSD 13.2:

0.000 IP sender > sink: = Flags [S], seq 205083268, win 65535, options
[mss = 1460,nop,wscale 10,sackOK,TS val 495212525 ecr 0], length 0
= 0.100 IP sink > sender: Flags [S.], seq 708257395, ack 205083269, = win
65160, options [mss 1460,sackOK,TS val 563185696 = ecr
495212525,nop,wscale 7], length 0
0.100 = IP sender > sink: Flags [.], ack 1, win 65, options [nop,nop,TS
val 495212626 ecr 563185696], length 0
// = TSopt omitted below for brevity.

// cwnd =3D= 10 * MSS, sent 10 * MSS
0.101 IP sender > sink: Flags = [.], seq 1:14481, ack 1, win 65, length 14480

// got one ACK for 10 * MSS, cwnd +=3D 2 * MSS, sent 12 * = MSS
0.201 IP sink > sender: Flags [.], ack 14481, win = 427, length 0
0.201 IP sender > sink: Flags [.], seq = 14481:31857, ack 1, win 65, length 17376

= // got ACK of 12*MSS above, cwnd +=3D 2 * MSS, sent 14 * MSS
0.301 IP sink > sender: Flags [.], ack 31857, win 411, = length 0
0.301 IP sender > sink: Flags [.], seq = 31857:52129, ack 1, win 65, length 20272

= // got ACK of 14*MSS above, cwnd +=3D 2 * MSS, sent 16 * MSS
0.402 IP sink > sender: Flags [.], ack 52129, win 395, = length 0
0.402 IP sender > sink: Flags [P.], seq = 52129:73629, ack 1, win 65,
length 21500
= 0.402 IP sender > sink: Flags [.], seq 73629:75077, ack 1, win 65, = length 1448

As a consequence, instead of = growing exponentially, cwnd grows
more-or-less = quadratically during slow-start, unless abc_l_var is
set = to a sufficiently large value.

NewReno took = more than 20 seconds to ramp up throughput to 100Mbps
over = an emulated 100ms delay link.  While Linux took ~2 seconds.
I can provide the pcap file if anyone is interested.

Switching to CUBIC won't help, because it uses = the logic in NewReno
ack_received() for slow start.

Is this a well-known issue and abc_l_var is = the only cure for it?
https://www.google.com/url?q=3Dhttps://calomel.org/freebsd_netw= ork_tuning.html&source=3Dgmail-imap&ust=3D1683640529000000&usg= =3DAOvVaw0MoyDmFAOg9MlB5yX3FzJP

Thank = you!

Best,
Shuo Chen



-- =
Rod Grimes =             &n= bsp;           &nbs= p;            =            rgrimes@freebsd.org



-- =
Rod Grimes =             &n= bsp;           &nbs= p;            =            rgrimes@freebsd.org


------
Randall = Stewart



= --Apple-Mail=_FB2B8B55-52E3-4FD9-9253-EDCFAB8012E6-- --Apple-Mail=_1B855FBF-EC26-4490-9A2C-6AA3C5357C0D Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCAzYw ggMyMIICGqADAgECAgqxywKqrHPB2ybTMA0GCSqGSIb3DQEBCwUAMEcxGDAWBgNVBAMTD1JhbmRh bGwgU3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzAe Fw0yMTAxMjQxMjIwMTRaFw0yNjAxMjQxMjIwMTRaMEcxGDAWBgNVBAMTD1JhbmRhbGwgU3Rld2Fy dDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBAMUAht2nr/NFlK+tmmN9PdO3DBPfeYh9fLcbVihR+/dipO41 AsFy9y+2uDVaFhTEvp406P0o9PQQTuYXqrCr76eWQIj3V787e1WKjTup1mIyQeWHGf1gvb/7vmI2 zHg6QZEIC4W8xeO8SLKyHiwlFHZn8Rn1HxtB7Ge+NulygkgUgJYhXD5E29jVGXAc6Qn9Vr9AexPf KaOhHCaNB/Twcinayz6D8CO/Ym1LOs3+ceSOa4cB07fepmbqDSXDkOeA3U7KLaluHrRTlj6DO+JU nqKXX7jJ68KTYSZ0qH4fZsk8cxFkwYI/3HDJi+oF+FDkf7SRo1Q2w+e3M/5MReLIQ7sCAwEAAaMg MB4wDwYJKoZIhvcvAQEKBAIFADALBgNVHQ8EBAMCB4AwDQYJKoZIhvcNAQELBQADggEBAHJfum1j 1WIVFjOJT/hqMIN751aXkablmwesW94lNJKjslPULbbcP5nZGg2lGpHcZ+0I5F/1TTiEsT2H2rhA uAnSsUxTpxRA+aoe+xtJOa5vle3CprhFkHAvB7EIoLiNaPd0DNK6kKYsbvr5Z5Eq7TF3SIO77Qh9 /8VgUfOb0ARDgix59Q6MM0NmIabEwh0cDWQYlGgDDtN9DNk5PGM4pjs48suwEdLmFTMOmGTkCp7I Vq6iHDNinBiB6+BB4VYMAO1o9qS+0pnfdmPJybt0zVGrhm/c1Fmm3Jec7NEuiKeXmhPIwdwMkKyp AsX0sHuFCYwioBTHHZpvnir+H2rRakgxggHrMIIB5wIBATBVMEcxGDAWBgNVBAMTD1JhbmRhbGwg U3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUwIKscsC qqxzwdsm0zANBglghkgBZQMEAgEFAKBpMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI hvcNAQkFMQ8XDTIzMDUwNDE4NDcwMFowLwYJKoZIhvcNAQkEMSIEIFFnJMxE05F2HZyiCdD2i9Vl WelG7tDsvnF5RtBNkzSRMA0GCSqGSIb3DQEBCwUABIIBAB2CbIeFUgbuByvgHuns/SIxc2hQ5DC4 KY6ElUa5h6mTLjt4OtJVEeduI5NPLEB6Q+ZUJRtNmOWEHte3i+QA/lnQ+91o3VCvYLn6S6SqiCQ+ Jw0cKsM4hMjczWrXSyljADhsj+rrBh4b8vBF19lJ4y/8Wur5CV9YUJmP8bOCSdmPSojedA0Pd7RM F1NvMgNSvb8AH/CFtuEBKvLpNcthv0i2yCjbD9jac8gsWssYVtNK0EQubV4pCSTXUgL0pbTGG3tS QZ3p54Bn1WGA6s/ETTzU/3j6yhGY2BbAqecMdGDiGzdmQC4rt2wx1RJuOmd9Jm5D+rKZtEUlwM0f ZVNwnLsAAAAAAAA= --Apple-Mail=_1B855FBF-EC26-4490-9A2C-6AA3C5357C0D--