From nobody Tue May 2 07:46:01 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9XG82srPz48fCF for ; Tue, 2 May 2023 07:46:40 +0000 (UTC) (envelope-from chenshuo@chenshuo.com) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9XG66BGgz4bxj for ; Tue, 2 May 2023 07:46:38 +0000 (UTC) (envelope-from chenshuo@chenshuo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chenshuo-com.20221208.gappssmtp.com header.s=20221208 header.b=w4mzD5Af; spf=none (mx1.freebsd.org: domain of chenshuo@chenshuo.com has no SPF policy when checking 2607:f8b0:4864:20::635) smtp.mailfrom=chenshuo@chenshuo.com; dmarc=none Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1a9253d4551so25890625ad.0 for ; Tue, 02 May 2023 00:46:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chenshuo-com.20221208.gappssmtp.com; s=20221208; t=1683013597; x=1685605597; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=7wjbU3WfAf4a9AVU1UF7g117hL1L7EscYzzA5i4I+AQ=; b=w4mzD5AfPwbQ0lZlrdItRB7LUgSdpPlSqslo7WOI72sJTGeFXwGnxQbPcOaDwdTYIz ytswMVi3IMBWkPg/xBxlePozFO0Y33F7IfpMIlXwULm9nFXEjIJ+fXa7tXsE0zRdwa7D 2LU5aV0i+olvJdDz+D9y4X9vpmLiUNy+MbEi6O3XxPTftmbkF+SOJTSURRF7o18f+Qzp dtGhk6OemtwXa2cmlTb7wpNzUKWRdPik0WO5Pk5yjoxrvf3Ptzplw3t9DaCd8v3p/DWj ca2EtHmY6HpAnA4V2TGsTJtVUycwUhZlrVC78PUddHhWbUn4TmwSf28G58uHN4sVoI7P Mxtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683013597; x=1685605597; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=7wjbU3WfAf4a9AVU1UF7g117hL1L7EscYzzA5i4I+AQ=; b=FJclGaVlaIDysmFL46+aozGE0d96RvpKB//AyoEDWleB2jYTLtslKoKptL7G1hN/eR W9GsWkKc4rGewcA3tldZsGp4IwVWn+jbW06+2YHkZZljdx8QtB5UTLJlFkQ3tQtL1zAo l2cqEvOV0Ec60E1UkCukACmY2BSmjM/n8v3MrwA62XWExCkWDw3fp7kwDd97cPCdlY0Q 5Myl42PqvvMKN4l1z1IFhsp1XpmV+TbCBPY+OHhSob5D+P+nf057Oy8S4bPTu4ribgBE 0nTDV+jOC/usqkBrYNKJW+5livcEFneXqfalPyUI9obA7hdZKHJJyVqqF2ewmXd7Pn9g a3JQ== X-Gm-Message-State: AC+VfDz9bhBLbGzDLjAMXPKkX2PWamhNWHFApJsJoSruC3PHwTGMtXR8 nMgc+bx+ce6xR4QwrBzk7OZR8QY+vFYykTvZvFao8z/YD+sdI27iFg4= X-Google-Smtp-Source: ACHHUZ6u7BsJZWHM+3pR0f9bflAB+mlaEJ6Im/P7L2U6ulUIu48XMO4CnRS1461UuTAq7A0518qjzRsvcaLYJfEGhpE= X-Received: by 2002:a17:902:6ac9:b0:1a6:bb04:a020 with SMTP id i9-20020a1709026ac900b001a6bb04a020mr17103276plt.46.1683013597410; Tue, 02 May 2023 00:46:37 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 From: Chen Shuo Date: Tue, 2 May 2023 00:46:01 -0700 Message-ID: Subject: Cwnd grows slowly during slow-start due to LRO of the receiver side. To: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; R_DKIM_ALLOW(-0.20)[chenshuo-com.20221208.gappssmtp.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::635:from]; RCVD_TLS_LAST(0.00)[]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::635:server fail]; DMARC_NA(0.00)[chenshuo.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[chenshuo-com.20221208.gappssmtp.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Q9XG66BGgz4bxj X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N As per newreno_ack_received() in sys/netinet/cc/cc_newreno.c, FreeBSD TCP sender strictly follows RFC 5681 with RFC 3465 extension That is, during slow-start, when receiving an ACK of 'bytes_acked' cwnd += min(bytes_acked, abc_l_var * SMSS); // abc_l_var = 2 dflt As discussed in sec3.2 of RFC 3465, L=2*SMSS bytes exactly balances the negative impact of the delayed ACK algorithm. RFC 5681 also requires that a receiver SHOULD generate an ACK for at least every second full-sized segment, so bytes_acked per ACK is at most 2 * SMSS. If both sender and receiver follow it. cwnd should grow exponentially during slow-slow: cwnd *= 2 (per RTT) However, LRO and TSO are widely used today, so receiver may generate much less ACKs than it used to do. As I observed, Both FreeBSD and Linux generates at most one ACK per segment assembled by LRO/GRO. The worst case is one ACK per 45 MSS, as 45 * 1448 = 65160 < 65535. Sending 1MB over a link of 100ms delay from FreeBSD 13.2: 0.000 IP sender > sink: Flags [S], seq 205083268, win 65535, options [mss 1460,nop,wscale 10,sackOK,TS val 495212525 ecr 0], length 0 0.100 IP sink > sender: Flags [S.], seq 708257395, ack 205083269, win 65160, options [mss 1460,sackOK,TS val 563185696 ecr 495212525,nop,wscale 7], length 0 0.100 IP sender > sink: Flags [.], ack 1, win 65, options [nop,nop,TS val 495212626 ecr 563185696], length 0 // TSopt omitted below for brevity. // cwnd = 10 * MSS, sent 10 * MSS 0.101 IP sender > sink: Flags [.], seq 1:14481, ack 1, win 65, length 14480 // got one ACK for 10 * MSS, cwnd += 2 * MSS, sent 12 * MSS 0.201 IP sink > sender: Flags [.], ack 14481, win 427, length 0 0.201 IP sender > sink: Flags [.], seq 14481:31857, ack 1, win 65, length 17376 // got ACK of 12*MSS above, cwnd += 2 * MSS, sent 14 * MSS 0.301 IP sink > sender: Flags [.], ack 31857, win 411, length 0 0.301 IP sender > sink: Flags [.], seq 31857:52129, ack 1, win 65, length 20272 // got ACK of 14*MSS above, cwnd += 2 * MSS, sent 16 * MSS 0.402 IP sink > sender: Flags [.], ack 52129, win 395, length 0 0.402 IP sender > sink: Flags [P.], seq 52129:73629, ack 1, win 65, length 21500 0.402 IP sender > sink: Flags [.], seq 73629:75077, ack 1, win 65, length 1448 As a consequence, instead of growing exponentially, cwnd grows more-or-less quadratically during slow-start, unless abc_l_var is set to a sufficiently large value. NewReno took more than 20 seconds to ramp up throughput to 100Mbps over an emulated 100ms delay link. While Linux took ~2 seconds. I can provide the pcap file if anyone is interested. Switching to CUBIC won't help, because it uses the logic in NewReno ack_received() for slow start. Is this a well-known issue and abc_l_var is the only cure for it? https://calomel.org/freebsd_network_tuning.html Thank you! Best, Shuo Chen