From owner-freebsd-transport@freebsd.org Wed Sep 11 12:04:23 2019 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2C401D1945 for ; Wed, 11 Sep 2019 12:04:23 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-eopbgr780055.outbound.protection.outlook.com [40.107.78.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46T0ts6LYHz3MJn; Wed, 11 Sep 2019 12:04:21 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eIMvLjPm5081wbOq+i+hbt+khehrRDNROkDeZUSWLRYIpP/IzNsLTgh9mCAcFrM03t2KyYoMvUUfTTSeritJWAPXNmpmSI/019udDqWF23MUNxODqJu2IrP9STXGqr64CwS5WdqL4N0frOuZ7K8acMZ/4uzVoN1e17NZTFdJ33qOeLFpFXzdsWqKugBQDElkJBVcKKy+RJAPjKZ5pcEXSzVnY6NYCSOvxewA+YZTQGFa7J3Mg17JoK7W+JguQlB+MRuqofbnyUFZYUOsKHSWoJK9nX+joE4Ganjz+9pGlwTsXWZ67TBUyjxP4uzSv/TpSOkzX+0jMEpa+NFwT4HXxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A3acbGy2YhzzYC+FrL8077DZCrvNKyz/zUUWhDbagpg=; b=nqws2qeYJs1zgHOLts3bNN404NCQbx3uZma3A17fQUty4q22U0ll1Aw6eNGv92QzmHKEq7peMj9Xz/Lmj4OQ/9ZQzhyRg4dz0wb0C9vW6VkRIzGTZA2i/gSyod2rwsNU6wlnIRV3uNSn/0LmnEeereM4gPkXESJt1rorNnSfyf/i/1RCgZlveF4/pHMBWjQR072HixQWqj0EjNmgQKt8bE77FrMzeFWD0MKTEOKLtg7oqbtvotQcwkgEedeMj8CPttYku/LLBksWeYMkx/ptQjURxcC0s++dMvRKBg5THBoRyB5b5b+X2A33Vg7dDiwAXRzKE2M3PVEgFOlXvq6VoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.onmicrosoft.com; s=selector2-netapp-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A3acbGy2YhzzYC+FrL8077DZCrvNKyz/zUUWhDbagpg=; b=df7tICx4u/ORCnq8yKQMQlQvljBHE6XAQmQVvqyqnfDm67kngSeTUpHKEnx5aXgWl7Ip0KXZv6x5oeQpARlfF8pSj6CLhvruKp1F+VBBrtV0uAbvH4d1WzGrNN4mXO2Oiq75Tu1E64xrX+5dXEqS+TQAVbuhTRcA+oQS+PR6ldY= Received: from CY4PR0601MB3715.namprd06.prod.outlook.com (52.132.101.140) by CY4PR0601MB3604.namprd06.prod.outlook.com (52.132.101.13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.20; Wed, 11 Sep 2019 12:04:19 +0000 Received: from CY4PR0601MB3715.namprd06.prod.outlook.com ([fe80::5940:900d:c556:6f6e]) by CY4PR0601MB3715.namprd06.prod.outlook.com ([fe80::5940:900d:c556:6f6e%6]) with mapi id 15.20.2241.018; Wed, 11 Sep 2019 12:04:19 +0000 From: "Scheffenegger, Richard" To: Randall Stewart , Lawrence Stewart , Michael Tuexen , Jonathan Looney CC: "freebsd-transport@freebsd.org" , "Cui, Cheng" , Tom Jones , "bz@freebsd.org" , "Eggert, Lars" Subject: reno cwnd growth while app limited... Thread-Topic: reno cwnd growth while app limited... Thread-Index: AdVol0YZlgizeTZ3S0afXPdfEBACLg== Date: Wed, 11 Sep 2019 12:04:19 +0000 Message-ID: Accept-Language: de-AT, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-dg-ref: PG1ldGE+PGF0IG5tPSJpbWFnZTAwMi5qcGciIHA9IiIgc3o9IjAiIHQ9IjAiIGg9IiIgaWQ9IiIgYmw9IjAiIGJvPSIwIi8+PGF0IG5tPSJpbWFnZTAwNC5qcGciIHA9IiIgc3o9IjAiIHQ9IjAiIGg9IiIgaWQ9IiIgYmw9IjAiIGJvPSIwIi8+PGF0IG5tPSJpbWFnZTAwNi5qcGciIHA9IiIgc3o9IjAiIHQ9IjAiIGg9IiIgaWQ9IiIgYmw9IjAiIGJvPSIwIi8+PGF0IG5tPSJpbWFnZTAxMi5qcGciIHA9IiIgc3o9IjAiIHQ9IjAiIGg9IiIgaWQ9IiIgYmw9IjAiIGJvPSIwIi8+PGF0IG5tPSJib2R5Lmh0bWwiIHA9ImM6XHVzZXJzXHNyaWNoYXJkXGFwcGRhdGFccm9hbWluZ1wwOWQ4NDliNi0zMmQzLTRhNDAtODVlZS02Yjg0YmEyOWUzNWJcbXNnc1xtc2ctNDU0NjY5YjgtZDQ4Yy0xMWU5LWI2MDktMDAxOWQyZTRlY2Q3XGFtZS10ZXN0XDQ1NDY2OWI5LWQ0OGMtMTFlOS1iNjA5LTAwMTlkMmU0ZWNkN2JvZHkuaHRtbCIgc3o9IjEyNTg4IiB0PSIxMzIxMjY3NzA1NDU5NDk2MTEiIGg9IjRJUy9aWXljWmkyNkJKenhTQXptbnVWWmRNMD0iIGlkPSIiIGJsPSIwIiBibz0iMSIvPjwvbWV0YT4= x-originating-ip: [213.143.121.76] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7fa72ec9-7d52-4822-8f2a-08d736b02d16 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(1401327)(2017052603328)(49563074)(7193020); SRVR:CY4PR0601MB3604; x-ms-traffictypediagnostic: CY4PR0601MB3604: x-ms-exchange-purlcount: 6 x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0157DEB61B x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(366004)(376002)(396003)(136003)(39860400002)(346002)(189003)(199004)(99286004)(476003)(486006)(733005)(54906003)(478600001)(7696005)(186003)(110136005)(71190400001)(71200400001)(7736002)(790700001)(6116002)(3846002)(8936002)(14444005)(33656002)(256004)(6436002)(66446008)(606006)(64756008)(66476007)(66556008)(66576008)(81156014)(81166006)(66946007)(2906002)(8676002)(4326008)(25786009)(26005)(76116006)(52536014)(316002)(86362001)(14454004)(966005)(55016002)(9686003)(236005)(74316002)(107886003)(5660300002)(6306002)(54896002)(6506007)(53936002)(99936001)(861006)(102836004)(66066001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR0601MB3604; H:CY4PR0601MB3715.namprd06.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: netapp.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: B1rJ1kkJSzekZ4N5PgoTIBIlZiUuiViqngwKn0v5HHeKW3+9779DYca2y1OABS0aVMj/HI3cuJl5k/FEQxYXl54vwXF8lDW6e+43SuyTAW7FPul0479OzVdgp+FEbxLwa0F3iMZmGHfhIDLNyl/OIvy+d3SjHvAEwsXxb/HL1BRaBwYae4mKPqLjEMDzIFvFuVvRx2Ct43RPkbT/CCmNMsWuxXIZgZEmz1ZSJPs1m327s9V/T0ZGGhQV1E0uTsLv8+d7srDHUfnrDXDk3CmfCmpyeDqImDrSDb0j5FCRhu93sExLfD9cy+37kChqYecZddS2bkT9jsJCUt/pG49pKQTsMyUptHo+g1Glt3c0nhnsNyY7wIzHhUOSkvBHvC2sXD18BwbQX0oVHip6kNhPp5bO29bJvGifbpvZfPPP2ZI= MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7fa72ec9-7d52-4822-8f2a-08d736b02d16 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2019 12:04:19.3612 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4UnOqJlfhFrano3UVprI6Bq54MMWGHLYqE8/Co4uv6RFwM1HtUnuHqr15GXSZU8t372KHY6pg3rxzfdrQzhc4g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR0601MB3604 X-Rspamd-Queue-Id: 46T0ts6LYHz3MJn X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.onmicrosoft.com header.s=selector2-netapp-onmicrosoft-com header.b=df7tICx4; dmarc=none; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 40.107.78.55 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-2.74 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[netapp.onmicrosoft.com:s=selector2-netapp-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain]; DMARC_NA(0.00)[netapp.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[netapp.onmicrosoft.com:+]; NEURAL_SPAM_LONG(1.00)[1.000,0]; RCPT_COUNT_SEVEN(0.00)[9]; RCVD_IN_DNSWL_NONE(0.00)[55.78.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.24)[ipnet: 40.64.0.0/10(-3.65), asn: 8075(-2.50), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~,5:~,6:~,7:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Sep 2019 12:04:23 -0000 Hi, I was just looking at some graph data running two parallel dctcp flows agai= nst a cubic receiver (some internal validation) with traditional ecn feedba= ck. [cid:image002.jpg@01D568A9.CB143AC0] Now, in the beginning, a single flow can not overutilize the link capacity,= and never runs into any loss/mark... but the snd_cwnd grows unbounded (sin= ce DCTCP is using the newreno "cc_ack_received" mechanism). However, newreno_ack_received is only to grow snd_cwnd, when CCF_CWND_LIMIT= ED is set, which remains set as long as snd_cwnd < snd_wnd (the receiver si= gnaled receive-window). But is this still* the correct behavior? Say, the data flow rate is application limited (ever n milliseconds, a few = kB), and the receiver has a large window signalled - cwnd will grow until i= t matches the receivers window. If then the application chooses to no longe= r restrict itself, it would possibly burst out significantly more data than= the queuing of the path can handle... So, shouldn't there be a second condition for cwnd growth, that e.g. pipe (= flightsize) is close to cwnd (factor 0.5 during slow start, and say 0.85 du= ring congestion avoidance), to prevent sudden large bursts when a flow come= s out of being application limited? The intention here would be to restrict= the worst case burst that could be sent out (which is dealt will different= ly in other stacks), to ideally still fit into the path's queues... RFC5681 is silent on application limited flows though (but one could thing = of application limiting a flow being another form of congestion, during whi= ch cwnd shouldn't grow...) In the example above, growing cwnd up to about 500 kB and then remaining th= ere should be approximately the expected setting - based on the average of = two competing flows hovering at aroud 200-250 kB... *) I'm referring to the much higher likelihood nowadays, that the applicati= on itself pacing and transfer volume violates the design principle of TCP, = where the implicit assumption was that the sender has unlimited data to sen= d, with the timing controlled at the full disgression of TCP. Richard Scheffenegger Consulting Solution Architect NAS & Networking NetApp +43 1 3676 811 3157 Direct Phone +43 664 8866 1857 Mobile Phone Richard.Scheffenegger@netapp.com [Welcome to Data Driven] [Facebook] [Twitter] #DataDriven https://ts.la/richard49892 From owner-freebsd-transport@freebsd.org Wed Sep 11 12:18:07 2019 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A47F7D2051 for ; Wed, 11 Sep 2019 12:18:07 +0000 (UTC) (envelope-from rrs@netflix.com) Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46T1Bk5Sbxz3Mjp for ; Wed, 11 Sep 2019 12:18:06 +0000 (UTC) (envelope-from rrs@netflix.com) Received: by mail-qk1-x744.google.com with SMTP id d26so20624991qkk.2 for ; Wed, 11 Sep 2019 05:18:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=0JQSrvbA8yyRFScaz75CujHGh8XA6Lvf9UXBzQkhuYs=; b=Vn++dXuMVz9Iw9vtl57iyperxu+T0jNwU545fpEjiibkc83YChd2gy7nZvlxdpMJKA O0Lj5dPsYQqoTFxj2Tc+t19TSWIC2S0N25c6o2r8Vpbi0mfGo+6ME3grcU+OPY/AB3g8 DSe/LFWZoeuc/q9F2gakNHe0bzDYgyLreAoiUHns4SunTftUbf4kxO+TsO+dxplSS7Wx 8QDmt3zZ5BOYY3R3R0eKI+cqa7TfuNk8b/oackLAD3c8gSv028rOGqO9DND7yRYrBPke QTBPe4CLM21RmHw7cWOsI5jGgpas6yPpKG8ZzXq0QZ9m34xv2tOdObyGWGf3xTMtuU4G SCwA== X-Gm-Message-State: APjAAAUKHP/Dwj0RPiCPVcF/q05lNhqIeD+OOAwJpdsvVItkU7UjZBhS NQ2qT7jOFgB3GqfozzpayGLt5A== X-Google-Smtp-Source: APXvYqzGRSpmXisUY4n060MTxqe1aiEtdc+rTPut0nf3O4rcA74Mbaumgfqkf1SX6Z+VEL0Nq4jEgQ== X-Received: by 2002:a05:620a:1304:: with SMTP id o4mr33871427qkj.341.1568204285438; Wed, 11 Sep 2019 05:18:05 -0700 (PDT) Received: from ?IPv6:2607:fb10:7061:7fd::4c92? ([2607:fb10:7061:7fd::4c92]) by smtp.gmail.com with ESMTPSA id c8sm2001925qkk.43.2019.09.11.05.18.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Sep 2019 05:18:04 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: reno cwnd growth while app limited... From: Randall Stewart In-Reply-To: Date: Wed, 11 Sep 2019 08:18:03 -0400 Cc: Lawrence Stewart , Michael Tuexen , Jonathan Looney , "freebsd-transport@freebsd.org" , "Cui, Cheng" , Tom Jones , "bz@freebsd.org" , "Eggert, Lars" Content-Transfer-Encoding: quoted-printable Message-Id: References: To: "Scheffenegger, Richard" X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 46T1Bk5Sbxz3Mjp X-Spamd-Bar: ------------- X-Spamd-Result: default: False [-13.04 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[netflix.com:+]; DMARC_POLICY_ALLOW(-0.50)[netflix.com,reject]; RCPT_COUNT_SEVEN(0.00)[9]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.54)[ip: (2.31), ipnet: 2607:f8b0::/32(-2.72), asn: 15169(-2.25), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[netflix.com:s=google]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-transport@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; WHITELIST_DMARC(-7.00)[netflix.com:D:+]; RCVD_IN_DNSWL_NONE(0.00)[4.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; WHITELIST_SPF_DKIM(-3.00)[netflix.com:d:+,netflix.com:s:+]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Sep 2019 12:18:07 -0000 Interesting graph :) I know that years ago I had a discussion along these lines (talking = about burst-limits) with Kacheong Poon and Mark Allman. IIRR Kacheong said, at that time, sun = limited the cwnd to something like 4MSS more than the flight size (I could have that mixed = up though and it might have been Mark proposing that.. its been a while sun was still a company = then :D). On the other hand I am not sure that such a tight limit takes into = account all of the ack-artifacts that seem to be rabid in the internet now.. BBR took the approach of = limiting its cwnd to 2xBDP (or at least what it thought was the BDP).. which is more along the lines of = your .5 if I am reading you right. It might be something worth looking into but I would want to contemplate = it for a while :) R > On Sep 11, 2019, at 8:04 AM, Scheffenegger, Richard = wrote: >=20 > Hi, > =20 > I was just looking at some graph data running two parallel dctcp flows = against a cubic receiver (some internal validation) with traditional ecn = feedback. > =20 > > =20 > =20 > Now, in the beginning, a single flow can not overutilize the link = capacity, and never runs into any loss/mark=E2=80=A6 but the snd_cwnd = grows unbounded (since DCTCP is using the newreno =E2=80=9Ccc_ack_received= =E2=80=9D mechanism). > =20 > However, newreno_ack_received is only to grow snd_cwnd, when = CCF_CWND_LIMITED is set, which remains set as long as snd_cwnd < snd_wnd = (the receiver signaled receive-window). > =20 > But is this still* the correct behavior? > =20 > Say, the data flow rate is application limited (ever n milliseconds, a = few kB), and the receiver has a large window signalled =E2=80=93 cwnd = will grow until it matches the receivers window. If then the application = chooses to no longer restrict itself, it would possibly burst out = significantly more data than the queuing of the path can handle=E2=80=A6 > =20 > So, shouldn=E2=80=99t there be a second condition for cwnd growth, = that e.g. pipe (flightsize) is close to cwnd (factor 0.5 during slow = start, and say 0.85 during congestion avoidance), to prevent sudden = large bursts when a flow comes out of being application limited? The = intention here would be to restrict the worst case burst that could be = sent out (which is dealt will differently in other stacks), to ideally = still fit into the path=E2=80=99s queues=E2=80=A6 > =20 > RFC5681 is silent on application limited flows though (but one could = thing of application limiting a flow being another form of congestion, = during which cwnd shouldn=E2=80=99t grow=E2=80=A6) > =20 > In the example above, growing cwnd up to about 500 kB and then = remaining there should be approximately the expected setting =E2=80=93 = based on the average of two competing flows hovering at aroud 200-250 = kB=E2=80=A6 > =20 > *) I=E2=80=99m referring to the much higher likelihood nowadays, that = the application itself pacing and transfer volume violates the design = principle of TCP, where the implicit assumption was that the sender has = unlimited data to send, with the timing controlled at the full = disgression of TCP. > =20 > =20 > Richard Scheffenegger > Consulting Solution Architect > NAS & Networking > =20 > NetApp > +43 1 3676 811 3157 Direct Phone > +43 664 8866 1857 Mobile Phone > Richard.Scheffenegger@netapp.com > =20 > =20 > >=20 > > #DataDriven > =20 > https://ts.la/richard49892 ------ Randall Stewart rrs@netflix.com From owner-freebsd-transport@freebsd.org Wed Sep 11 12:26:02 2019 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9685DD2386 for ; Wed, 11 Sep 2019 12:26:02 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-eopbgr770058.outbound.protection.outlook.com [40.107.77.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46T1Ms4YdGz3N5x; Wed, 11 Sep 2019 12:26:01 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IW2nMTCgkQ4FXCF9+t8AX+5vy+bZChvqLrVXKtOIPPwXyf4kLTZ+LCx/yrU/p0uO+wEq/Ewq9VunzG4Y8vMhh/Z+Qrkdg5AKS5zViaGzMqR2eef5YuoKNLa7QMCFvwpgBIgmLTmZKNR3kGoTXUtAqnpvlPmat/nvUXUz4m69IBPKeJshfnBpxWjBucELqxYFGm4j+EbhCiXLq2NacE1sktgnQdOx7k0Sz2bQHMrfwDTHrCEOApx6wW0YR97lWKp0mTaChp+IKVUIZX64jyUZXbSVjAH24knEMO4ev7cwhvvwLGQhGUXMjcWFtEPOq8yHkqohZfaSTLW0qUx4Zh40vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Utk0L4LYD44UEfgrLM7LrinDf4MTbFLKvvz8wsddq2w=; b=d0hZav2bMXQdfr4WXSKTkTVrhZIIegxqHD4BKjtl86hYq9GN4uHUjCtQ7y/p8nbnsGO7/xoALfZjy1fLtHBBFqomr/9FecnL38CiNvWvRIvZbaw8DgYmL/gNFnXHPYv6ul1O+uWsLqzjjJhBKJovjGFSs2IZ8gwLMsbKgTXFMWjVIRpQxkAwrhifTTt2JFWuWAHnbdnY8Lxvp3SyqEZwWzF+0dbJlb8BhsTFFXGIvvkmodNvMH2qMRanC+Elkh5dxvQ7f9a7VkOyeLOecrkPUezubJw4WYItCDUZtJOyRrBe0V2WlaqvaWl26I9bvR6is1BVJCDEsqm6IUIzTJx6Pw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.onmicrosoft.com; s=selector2-netapp-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Utk0L4LYD44UEfgrLM7LrinDf4MTbFLKvvz8wsddq2w=; b=X5LqUhMPB8BiIZ2ySvfY/KBPoWsVfzBBZ6mBvy2f02oqaFzJyvsw6wK+ARZ5qMYTeudZAx5dF1/7Tf4ZSX0ikvRlj02cLs4Y7zgbAqyO+l3xCktgrus85cgTpGwsvYuboi9ZB0rnSo04/eXlVQZIHKxrxgEHXXga+1S+o4ml97E= Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (10.167.151.152) by SN4PR0601MB3679.namprd06.prod.outlook.com (10.167.139.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.15; Wed, 11 Sep 2019 12:25:58 +0000 Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::344b:f99a:6c23:a837]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::344b:f99a:6c23:a837%7]) with mapi id 15.20.2263.016; Wed, 11 Sep 2019 12:25:58 +0000 From: "Scheffenegger, Richard" To: Randall Stewart CC: Lawrence Stewart , Michael Tuexen , Jonathan Looney , "freebsd-transport@freebsd.org" , "Cui, Cheng" , Tom Jones , "bz@freebsd.org" , "Eggert, Lars" Subject: RE: reno cwnd growth while app limited... Thread-Topic: reno cwnd growth while app limited... Thread-Index: AdVol0YZlgizeTZ3S0afXPdfEBACLgAA69iAAAAQH5A= Date: Wed, 11 Sep 2019 12:25:58 +0000 Message-ID: References: In-Reply-To: Accept-Language: de-AT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-dg-ref: PG1ldGE+PGF0IG5tPSJib2R5LnR4dCIgcD0iYzpcdXNlcnNcc3JpY2hhcmRcYXBwZGF0YVxyb2FtaW5nXDA5ZDg0OWI2LTMyZDMtNGE0MC04NWVlLTZiODRiYTI5ZTM1Ylxtc2dzXG1zZy00ZDYwNmZkNi1kNDhmLTExZTktYjYwOS0wMDE5ZDJlNGVjZDdcYW1lLXRlc3RcNGQ2MDZmZDctZDQ4Zi0xMWU5LWI2MDktMDAxOWQyZTRlY2Q3Ym9keS50eHQiIHN6PSI1MDcwIiB0PSIxMzIxMjY3ODM1NjY1ODYyODciIGg9IlNWeVVPR29za3BIUXduRXZWNS8yNHB3UHRoST0iIGlkPSIiIGJsPSIwIiBibz0iMSIvPjwvbWV0YT4= x-dg-rorf: x-originating-ip: [217.70.211.16] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e95e1486-854b-4fba-098d-08d736b3337b x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:SN4PR0601MB3679; x-ms-traffictypediagnostic: SN4PR0601MB3679: x-ms-exchange-purlcount: 2 x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0157DEB61B x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(376002)(366004)(346002)(39860400002)(396003)(189003)(199004)(13464003)(76176011)(107886003)(6506007)(966005)(7736002)(11346002)(66446008)(8936002)(71200400001)(71190400001)(64756008)(102836004)(476003)(486006)(52536014)(6116002)(305945005)(3846002)(76116006)(4326008)(66946007)(66574012)(81156014)(66556008)(66476007)(81166006)(478600001)(8676002)(54906003)(7696005)(256004)(33656002)(5660300002)(99286004)(14444005)(5024004)(53546011)(55016002)(6916009)(86362001)(74316002)(229853002)(53936002)(6306002)(26005)(9686003)(66066001)(6436002)(6246003)(2906002)(316002)(14454004)(25786009)(446003)(186003); DIR:OUT; SFP:1101; SCL:1; SRVR:SN4PR0601MB3679; H:SN4PR0601MB3728.namprd06.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: netapp.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: GThimWnIfshPBj01+wEDHzMILwAGV8fUYCKVwp+uFvEJ5HyJAeaoelwZJTQYZxU3huynY2K+8QyXM50+5ZvGKWA6vz+AxQCRWAgtRFZJz/4Wq+Pb3iibIjUZc9k2Tr1izlHQjUzuPWpaxLT68HtNtilHcRcGgnQm0x1ZMy1LquqjplZ+vc4Vf7SN3N7Wju7FDGn9caV+eWfW4h0yk3puNqXlIYDHuFlUIe/nCkXkJqaOKRt4h8kcL8PE7O1R7X5A9uU8/RXT2KS46GnNniOhhwdSQQE7GEAlK62UKuBUkYDjFrQhFLKLUm9dGIbkFdsKbr5Sq/hX/A0e4T9CzzQL7pp88cDmV5bDMHbzJAziYs7e6s9k84ZGxxztsmT+ptvnM8bNeUi3UhnvKBKJtXo0BpveT+UFbGXcufZtcxdyK4A= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-Network-Message-Id: e95e1486-854b-4fba-098d-08d736b3337b X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2019 12:25:58.4543 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wn0LKdjAZ00ZEJ00HxcLX2MlFkBJyqKt21F8wWSpXmuC9Kd5PdJRtjLINbFmEzBqup3Tu6oFU7voII81+CCwYA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR0601MB3679 X-Rspamd-Queue-Id: 46T1Ms4YdGz3N5x X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.onmicrosoft.com header.s=selector2-netapp-onmicrosoft-com header.b=X5LqUhMP; dmarc=none; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 40.107.77.58 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-4.64 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[netapp.onmicrosoft.com:s=selector2-netapp-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[netapp.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[netapp.onmicrosoft.com:+]; MIME_BASE64_TEXT(0.10)[]; RCVD_IN_DNSWL_NONE(0.00)[58.77.107.40.list.dnswl.org : 127.0.3.0]; RCPT_COUNT_SEVEN(0.00)[9]; IP_SCORE(-1.24)[ipnet: 40.64.0.0/10(-3.65), asn: 8075(-2.50), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Sep 2019 12:26:02 -0000 VGhhbmtzIFJhbmRhbGwgZm9yIHRoZSBxdWljayByZXNwb25zZS4NCg0KDQpJIHdhcyBqdXN0IHJl LXJlYWRpbmcgUkZDNzY2MSAoTmV3Q1dWKS4uLi4NCg0KV2hpbGUgSSBzdGlsbCBsaWtlIHRoZSBp ZGVhIG9mIGRlY2F5aW5nIGN3bmQgb3ZlciB0aW1lLCB3aGVuIGEgZmxvdyBkb2Vzbid0IHV0aWxp emUgaXQncyBjd25kLCBJIHRoaW5rIGFzIGEgZmlyc3Qgc3RlcCB0byBjb25zdHJhaW4gdGhlIGdy b3d0aCBvZiBjd25kIHdoZW4gaXQgaXMgPj0gMiogZmxpZ2h0c2l6ZShpbnN0YW50YW5lb3VzKSBt YXkgYmUgYSBnb29kIHRoaW5nIGFscmVhZHkuDQoNCkJhc2ljYWxseSwgYSBzaW1wbGlzdGljIHZl cnNpb24gb2YgdGhlIHR3byBwYXJhZ3JhcGhzIG9mIHNlY3Rpb24gNC40IGh0dHBzOi8vdG9vbHMu aWV0Zi5vcmcvaHRtbC9yZmM3NjYxI3NlY3Rpb24tNC40DQoNCg0KSW4gTmV3Q1dWLCBhZGRpdGlv bmFsIHRpbWVycywgYW5kIHNtb290aGluZyAoMyByZWNlbnQsIHBlci1ydHQgc2FtcGxlcyBvZiBm bGlnaHRzaXplKSBhcmUgYmVpbmcgdXNlZC4uLiBCdXQgSSBkb24ndCB3YW50IHRvIG1lc3Mgd2l0 aCB0Y3BjYiByaWdodCBub3cgKGFnYWluKSwgYmVmb3JlIHNvbWUgc3VwZXJmbHVvdXMgdmFyaWFi bGVzLCBhbmQgZXhjZXNzaXZlbHkgbGFyZ2UgY291bnRlcnMgZ290IGZpeGVkIHRvIHJlZHVjZSBp dHMgc2l6ZS4NCg0KQmVzdCByZWdhcmRzLA0KDQpSaWNoYXJkIFNjaGVmZmVuZWdnZXINCkNvbnN1 bHRpbmcgU29sdXRpb24gQXJjaGl0ZWN0DQpOQVMgJiBOZXR3b3JraW5nDQoNCk5ldEFwcA0KKzQz IDEgMzY3NiA4MTEgMzE1NyBEaXJlY3QgUGhvbmUNCis0M8KgNjY0IDg4NjYgMTg1NyBNb2JpbGUg UGhvbmUNClJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tDQoNCmh0dHBzOi8vdHMubGEv cmljaGFyZDQ5ODkyDQoNCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IFJhbmRh bGwgU3Rld2FydCA8cnJzQG5ldGZsaXguY29tPiANClNlbnQ6IE1pdHR3b2NoLCAxMS4gU2VwdGVt YmVyIDIwMTkgMTQ6MTgNClRvOiBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkIDxSaWNoYXJkLlNjaGVm ZmVuZWdnZXJAbmV0YXBwLmNvbT4NCkNjOiBMYXdyZW5jZSBTdGV3YXJ0IDxsc3Rld2FydEBuZXRm bGl4LmNvbT47IE1pY2hhZWwgVHVleGVuIDx0dWV4ZW5ARnJlZUJTRC5vcmc+OyBKb25hdGhhbiBM b29uZXkgPGp0bEBuZXRmbGl4LmNvbT47IGZyZWVic2QtdHJhbnNwb3J0QGZyZWVic2Qub3JnOyBD dWksIENoZW5nIDxDaGVuZy5DdWlAbmV0YXBwLmNvbT47IFRvbSBKb25lcyA8dGhqQGZyZWVic2Qu b3JnPjsgYnpAZnJlZWJzZC5vcmc7IEVnZ2VydCwgTGFycyA8bGFyc0BuZXRhcHAuY29tPg0KU3Vi amVjdDogUmU6IHJlbm8gY3duZCBncm93dGggd2hpbGUgYXBwIGxpbWl0ZWQuLi4NCg0KTmV0QXBw IFNlY3VyaXR5IFdBUk5JTkc6IFRoaXMgaXMgYW4gZXh0ZXJuYWwgZW1haWwuIERvIG5vdCBjbGlj ayBsaW5rcyBvciBvcGVuIGF0dGFjaG1lbnRzIHVubGVzcyB5b3UgcmVjb2duaXplIHRoZSBzZW5k ZXIgYW5kIGtub3cgdGhlIGNvbnRlbnQgaXMgc2FmZS4NCg0KDQoNCg0KSW50ZXJlc3RpbmcgZ3Jh cGggOikNCg0KDQpJIGtub3cgdGhhdCB5ZWFycyBhZ28gSSBoYWQgYSBkaXNjdXNzaW9uIGFsb25n IHRoZXNlIGxpbmVzICh0YWxraW5nIGFib3V0IGJ1cnN0LWxpbWl0cykgd2l0aCBLYWNoZW9uZyBQ b29uIGFuZCBNYXJrIEFsbG1hbi4gSUlSUiBLYWNoZW9uZyBzYWlkLCBhdCB0aGF0IHRpbWUsIHN1 biBsaW1pdGVkIHRoZSBjd25kIHRvIHNvbWV0aGluZyBsaWtlIDRNU1MgbW9yZSB0aGFuIHRoZSBm bGlnaHQgc2l6ZSAoSSBjb3VsZCBoYXZlIHRoYXQgbWl4ZWQgdXAgdGhvdWdoIGFuZCBpdCBtaWdo dCBoYXZlIGJlZW4gTWFyayBwcm9wb3NpbmcgdGhhdC4uIGl0cyBiZWVuIGEgd2hpbGUgc3VuIHdh cyBzdGlsbCBhIGNvbXBhbnkgdGhlbiA6RCkuDQoNCk9uIHRoZSBvdGhlciBoYW5kIEkgYW0gbm90 IHN1cmUgdGhhdCBzdWNoIGEgdGlnaHQgbGltaXQgdGFrZXMgaW50byBhY2NvdW50IGFsbCBvZiB0 aGUgYWNrLWFydGlmYWN0cyB0aGF0IHNlZW0gdG8gYmUgcmFiaWQgaW4gdGhlIGludGVybmV0IG5v dy4uICBCQlIgdG9vayB0aGUgYXBwcm9hY2ggb2YgbGltaXRpbmcgaXRzIGN3bmQgdG8gMnhCRFAg KG9yIGF0IGxlYXN0IHdoYXQgaXQgdGhvdWdodCB3YXMgdGhlIEJEUCkuLiB3aGljaCBpcyBtb3Jl IGFsb25nIHRoZSBsaW5lcyBvZiB5b3VyIC41IGlmIEkgYW0gcmVhZGluZyB5b3UgcmlnaHQuDQoN Ckl0IG1pZ2h0IGJlIHNvbWV0aGluZyB3b3J0aCBsb29raW5nIGludG8gYnV0IEkgd291bGQgd2Fu dCB0byBjb250ZW1wbGF0ZSBpdCBmb3IgYSB3aGlsZSA6KQ0KDQpSDQoNCj4gT24gU2VwIDExLCAy MDE5LCBhdCA4OjA0IEFNLCBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkIDxSaWNoYXJkLlNjaGVmZmVu ZWdnZXJAbmV0YXBwLmNvbT4gd3JvdGU6DQo+DQo+IEhpLA0KPg0KPiBJIHdhcyBqdXN0IGxvb2tp bmcgYXQgc29tZSBncmFwaCBkYXRhIHJ1bm5pbmcgdHdvIHBhcmFsbGVsIGRjdGNwIGZsb3dzIGFn YWluc3QgYSBjdWJpYyByZWNlaXZlciAoc29tZSBpbnRlcm5hbCB2YWxpZGF0aW9uKSB3aXRoIHRy YWRpdGlvbmFsIGVjbiBmZWVkYmFjay4NCj4NCj4gPGltYWdlMDAyLmpwZz4NCj4NCj4NCj4gTm93 LCBpbiB0aGUgYmVnaW5uaW5nLCBhIHNpbmdsZSBmbG93IGNhbiBub3Qgb3ZlcnV0aWxpemUgdGhl IGxpbmsgY2FwYWNpdHksIGFuZCBuZXZlciBydW5zIGludG8gYW55IGxvc3MvbWFya+KApiBidXQg dGhlIHNuZF9jd25kIGdyb3dzIHVuYm91bmRlZCAoc2luY2UgRENUQ1AgaXMgdXNpbmcgdGhlIG5l d3Jlbm8g4oCcY2NfYWNrX3JlY2VpdmVk4oCdIG1lY2hhbmlzbSkuDQo+DQo+IEhvd2V2ZXIsIG5l d3Jlbm9fYWNrX3JlY2VpdmVkIGlzIG9ubHkgdG8gZ3JvdyBzbmRfY3duZCwgd2hlbiBDQ0ZfQ1dO RF9MSU1JVEVEIGlzIHNldCwgd2hpY2ggcmVtYWlucyBzZXQgYXMgbG9uZyBhcyBzbmRfY3duZCA8 IHNuZF93bmQgKHRoZSByZWNlaXZlciBzaWduYWxlZCByZWNlaXZlLXdpbmRvdykuDQo+DQo+IEJ1 dCBpcyB0aGlzIHN0aWxsKiB0aGUgY29ycmVjdCBiZWhhdmlvcj8NCj4NCj4gU2F5LCB0aGUgZGF0 YSBmbG93IHJhdGUgaXMgYXBwbGljYXRpb24gbGltaXRlZCAoZXZlciBuIG1pbGxpc2Vjb25kcywg YSANCj4gZmV3IGtCKSwgYW5kIHRoZSByZWNlaXZlciBoYXMgYSBsYXJnZSB3aW5kb3cgc2lnbmFs bGVkIOKAkyBjd25kIHdpbGwgDQo+IGdyb3cgdW50aWwgaXQgbWF0Y2hlcyB0aGUgcmVjZWl2ZXJz IHdpbmRvdy4gSWYgdGhlbiB0aGUgYXBwbGljYXRpb24gDQo+IGNob29zZXMgdG8gbm8gbG9uZ2Vy IHJlc3RyaWN0IGl0c2VsZiwgaXQgd291bGQgcG9zc2libHkgYnVyc3Qgb3V0IA0KPiBzaWduaWZp Y2FudGx5IG1vcmUgZGF0YSB0aGFuIHRoZSBxdWV1aW5nIG9mIHRoZSBwYXRoIGNhbiBoYW5kbGXi gKYNCj4NCj4gU28sIHNob3VsZG7igJl0IHRoZXJlIGJlIGEgc2Vjb25kIGNvbmRpdGlvbiBmb3Ig Y3duZCBncm93dGgsIHRoYXQgZS5nLiANCj4gcGlwZSAoZmxpZ2h0c2l6ZSkgaXMgY2xvc2UgdG8g Y3duZCAoZmFjdG9yIDAuNSBkdXJpbmcgc2xvdyBzdGFydCwgYW5kIA0KPiBzYXkgMC44NSBkdXJp bmcgY29uZ2VzdGlvbiBhdm9pZGFuY2UpLCB0byBwcmV2ZW50IHN1ZGRlbiBsYXJnZSBidXJzdHMg DQo+IHdoZW4gYSBmbG93IGNvbWVzIG91dCBvZiBiZWluZyBhcHBsaWNhdGlvbiBsaW1pdGVkPyBU aGUgaW50ZW50aW9uIGhlcmUgDQo+IHdvdWxkIGJlIHRvIHJlc3RyaWN0IHRoZSB3b3JzdCBjYXNl IGJ1cnN0IHRoYXQgY291bGQgYmUgc2VudCBvdXQgDQo+ICh3aGljaCBpcyBkZWFsdCB3aWxsIGRp ZmZlcmVudGx5IGluIG90aGVyIHN0YWNrcyksIHRvIGlkZWFsbHkgc3RpbGwgDQo+IGZpdCBpbnRv IHRoZSBwYXRo4oCZcyBxdWV1ZXPigKYNCj4NCj4gUkZDNTY4MSBpcyBzaWxlbnQgb24gYXBwbGlj YXRpb24gbGltaXRlZCBmbG93cyB0aG91Z2ggKGJ1dCBvbmUgY291bGQgDQo+IHRoaW5nIG9mIGFw cGxpY2F0aW9uIGxpbWl0aW5nIGEgZmxvdyBiZWluZyBhbm90aGVyIGZvcm0gb2YgY29uZ2VzdGlv biwgDQo+IGR1cmluZyB3aGljaCBjd25kIHNob3VsZG7igJl0IGdyb3figKYpDQo+DQo+IEluIHRo ZSBleGFtcGxlIGFib3ZlLCBncm93aW5nIGN3bmQgdXAgdG8gYWJvdXQgNTAwIGtCIGFuZCB0aGVu IA0KPiByZW1haW5pbmcgdGhlcmUgc2hvdWxkIGJlIGFwcHJveGltYXRlbHkgdGhlIGV4cGVjdGVk IHNldHRpbmcg4oCTIGJhc2VkIA0KPiBvbiB0aGUgYXZlcmFnZSBvZiB0d28gY29tcGV0aW5nIGZs b3dzIGhvdmVyaW5nIGF0IGFyb3VkIDIwMC0yNTAga0LigKYNCj4NCj4gKikgSeKAmW0gcmVmZXJy aW5nIHRvIHRoZSBtdWNoIGhpZ2hlciBsaWtlbGlob29kIG5vd2FkYXlzLCB0aGF0IHRoZSBhcHBs aWNhdGlvbiBpdHNlbGYgcGFjaW5nIGFuZCB0cmFuc2ZlciB2b2x1bWUgdmlvbGF0ZXMgdGhlIGRl c2lnbiBwcmluY2lwbGUgb2YgVENQLCB3aGVyZSB0aGUgaW1wbGljaXQgYXNzdW1wdGlvbiB3YXMg dGhhdCB0aGUgc2VuZGVyIGhhcyB1bmxpbWl0ZWQgZGF0YSB0byBzZW5kLCB3aXRoIHRoZSB0aW1p bmcgY29udHJvbGxlZCBhdCB0aGUgZnVsbCBkaXNncmVzc2lvbiBvZiBUQ1AuDQo+DQo+DQo+IFJp Y2hhcmQgU2NoZWZmZW5lZ2dlcg0KPiBDb25zdWx0aW5nIFNvbHV0aW9uIEFyY2hpdGVjdA0KPiBO QVMgJiBOZXR3b3JraW5nDQo+DQo+IE5ldEFwcA0KPiArNDMgMSAzNjc2IDgxMSAzMTU3IERpcmVj dCBQaG9uZQ0KPiArNDMgNjY0IDg4NjYgMTg1NyBNb2JpbGUgUGhvbmUNCj4gUmljaGFyZC5TY2hl ZmZlbmVnZ2VyQG5ldGFwcC5jb20NCj4NCj4NCj4gPGltYWdlMDA0LmpwZz4NCj4NCj4gPGltYWdl MDA2LmpwZz4gPGltYWdlMDEyLmpwZz4NCj4gICNEYXRhRHJpdmVuDQo+DQo+IGh0dHBzOi8vdHMu bGEvcmljaGFyZDQ5ODkyDQoNCi0tLS0tLQ0KUmFuZGFsbCBTdGV3YXJ0DQpycnNAbmV0ZmxpeC5j b20NCg0KDQoNCg== From owner-freebsd-transport@freebsd.org Thu Sep 12 19:41:24 2019 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B8793D8583 for ; Thu, 12 Sep 2019 19:41:24 +0000 (UTC) (envelope-from rrs@netflix.com) Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46Tpzm05wCz4g1h for ; Thu, 12 Sep 2019 19:41:23 +0000 (UTC) (envelope-from rrs@netflix.com) Received: by mail-qk1-x743.google.com with SMTP id q203so25712406qke.1 for ; Thu, 12 Sep 2019 12:41:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=T7pc5pNJbxsGxLBJaY6vGedIvZjtlHUzImI8o1AjF2Q=; b=RRueZe6MVLEfWVwRrME1JEeDswFoSTOabwYiEv+D0pZk4K6FpyrcAKhRVsBQXqRqJo Rzji2SKx9RHEAytbxkK3qHdoWaVnViIIm4QPSpw+vgqGIXZcdpdHM/+7EK5vLCpVsdvs AeXR/JfrQvpU9sEKYwYB5kxk7gqIQ8EjHdMiWxmEMkKsrjrk8sMD38DrAv3SGtrRSpGl ij26ahjtNPtO34x/BrPCU/8Z3YtBByJXS45VjtApd+Zsc7ZQ0yLYrZuyWi5p7Y3/isRh z32jM86UTFLBS+hicjS5+B9Fky8/6QRVB5m8spaH1tlfDgK1BZ+/5HIz+naQZi+1UrLG GcHQ== X-Gm-Message-State: APjAAAXJ//EBmFFUyLRLzxzASay4BQZxN2bsukXjYtn5LoGtQ+Xqqz9U p7MCQ8LM4qtUUDFBApdq6rD9+A== X-Google-Smtp-Source: APXvYqwCY8Zimx6XMYj1aXx0bmTYjrBZkPpRAEo4s3cmmRLb2unev+N2ap0boV3aKspHF+5cJnSd7Q== X-Received: by 2002:a37:a91:: with SMTP id 139mr41417496qkk.418.1568317282360; Thu, 12 Sep 2019 12:41:22 -0700 (PDT) Received: from ?IPv6:2607:fb10:7061:7fd::4fdb? ([2607:fb10:7061:7fd::4fdb]) by smtp.gmail.com with ESMTPSA id l3sm11492755qtc.33.2019.09.12.12.41.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Sep 2019 12:41:21 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: reno cwnd growth while app limited... From: Randall Stewart In-Reply-To: Date: Thu, 12 Sep 2019 15:41:19 -0400 Cc: Michael Tuexen , Lawrence Stewart , Jonathan Looney , "freebsd-transport@freebsd.org" , "Cui, Cheng" , Tom Jones , "bz@freebsd.org" , "Eggert, Lars" Content-Transfer-Encoding: quoted-printable Message-Id: <1E528E13-B087-4318-A9CA-F08957B3F03A@netflix.com> References: To: "Scheffenegger, Richard" X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 46Tpzm05wCz4g1h X-Spamd-Bar: ------------ X-Spamd-Result: default: False [-12.97 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[netflix.com:+]; DMARC_POLICY_ALLOW(-0.50)[netflix.com,reject]; RCPT_COUNT_SEVEN(0.00)[9]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.47)[ip: (2.68), ipnet: 2607:f8b0::/32(-2.71), asn: 15169(-2.25), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[netflix.com:s=google]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-transport@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; WHITELIST_DMARC(-7.00)[netflix.com:D:+]; RCVD_IN_DNSWL_NONE(0.00)[3.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; WHITELIST_SPF_DKIM(-3.00)[netflix.com:d:+,netflix.com:s:+]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Sep 2019 19:41:24 -0000 Richard Yes that is something one could do i.e. NewCWV=E2=80=A6 we had patches = for it at one time but it was largely untested. Pacing of course changes all of this as well.. i.e. BBR for example does = not worry about this since its pacing the packets so you never get a burst. We will soon = be committing and updated rack stack that also has this ability R > On Sep 12, 2019, at 2:49 PM, Scheffenegger, Richard = wrote: >=20 > Michael, > =20 > Thanks a lot for pointing out the uperf utility - this could be = configured exactly in the way I wanted to demonstrate this... > =20 > In the below graphs, I traced the evolution of cwnd for a flow across = the loopback in a VM. > =20 > The application is doing 3060 times 10kB writes, 10ms pausing (well = below the 230ms minimum TCP idle period), and once that phase is over, = floods the session with 8x writes of 10MB each. > =20 > Currently, the stack will initially grow cwnd up to the limit set by = the receiver's window (set to 1.2 MB) - during the low bandwidth rate = phase, where no loss occurs... > =20 > Thus the application can send out a massive burst of data in a single = RTT (or at linerate) when it chooses to do so... > =20 > =20 > Using the guidance given by NewCWV (RFC7661), and growing cwnd only, = when flightsize is larger than half of cwnd, the congestion window = remains in more reasonable ranges during the application limited phase, = thus limiting the maximum burst size. > =20 > Growth of cwnd in SS or CA otherwise is normal, but the inverse case = (application transisioning from high throughput to low) is not = addressed; but I wonder if a reduction could be achieved without the = timer infrastructure described in 7661 (e.g. reducing cwnd by 1 mss, = when flightsize is < =C2=BD cwnd, while not doing recovery=E2=80=A6 > =20 > > Unlimited ssthresh: > > =20 > =20 > > =20 > =20 > Richard Scheffenegger > Consulting Solution Architect > NAS & Networking > =20 > NetApp > +43 1 3676 811 3157 Direct Phone > +43 664 8866 1857 Mobile Phone > Richard.Scheffenegger@netapp.com > =20 > https://ts.la/richard49892 > =20 > =20 > -----Original Message----- > From: Randall Stewart =20 > Sent: Mittwoch, 11. September 2019 14:18 > To: Scheffenegger, Richard > Cc: Lawrence Stewart ; Michael Tuexen = ; Jonathan Looney ; = freebsd-transport@freebsd.org; Cui, Cheng ; Tom = Jones ; bz@freebsd.org; Eggert, Lars > Subject: Re: reno cwnd growth while app limited... > =20 > NetApp Security WARNING: This is an external email. Do not click links = or open attachments unless you recognize the sender and know the content = is safe. > =20 > =20 > =20 > =20 > Interesting graph :) > =20 > =20 > I know that years ago I had a discussion along these lines (talking = about burst-limits) with > Kacheong Poon and Mark Allman. IIRR Kacheong said, at that time, sun = limited the cwnd to > something like 4MSS more than the flight size (I could have that mixed = up though and it might > have been Mark proposing that.. its been a while sun was still a = company then :D). > =20 > On the other hand I am not sure that such a tight limit takes into = account all of the ack-artifacts that > seem to be rabid in the internet now.. BBR took the approach of = limiting its cwnd to 2xBDP (or at > least what it thought was the BDP).. which is more along the lines of = your .5 if I am reading you right. > =20 > It might be something worth looking into but I would want to = contemplate it for a while :) > =20 > R > =20 > > On Sep 11, 2019, at 8:04 AM, Scheffenegger, Richard = wrote: > >=20 > > Hi, > >=20 > > I was just looking at some graph data running two parallel dctcp = flows against a cubic receiver (some internal validation) with = traditional ecn feedback. > >=20 > > > >=20 > >=20 > > Now, in the beginning, a single flow can not overutilize the link = capacity, and never runs into any loss/mark=E2=80=A6 but the snd_cwnd = grows unbounded (since DCTCP is using the newreno =E2=80=9Ccc_ack_received= =E2=80=9D mechanism). > >=20 > > However, newreno_ack_received is only to grow snd_cwnd, when = CCF_CWND_LIMITED is set, which remains set as long as snd_cwnd < snd_wnd = (the receiver signaled receive-window). > >=20 > > But is this still* the correct behavior? > >=20 > > Say, the data flow rate is application limited (ever n milliseconds, = a few kB), and the receiver has a large window signalled =E2=80=93 cwnd = will grow until it matches the receivers window. If then the application = chooses to no longer restrict itself, it would possibly burst out = significantly more data than the queuing of the path can handle=E2=80=A6 > >=20 > > So, shouldn=E2=80=99t there be a second condition for cwnd growth, = that e.g. pipe (flightsize) is close to cwnd (factor 0.5 during slow = start, and say 0.85 during congestion avoidance), to prevent sudden = large bursts when a flow comes out of being application limited? The = intention here would be to restrict the worst case burst that could be = sent out (which is dealt will differently in other stacks), to ideally = still fit into the path=E2=80=99s queues=E2=80=A6 > >=20 > > RFC5681 is silent on application limited flows though (but one could = thing of application limiting a flow being another form of congestion, = during which cwnd shouldn=E2=80=99t grow=E2=80=A6) > >=20 > > In the example above, growing cwnd up to about 500 kB and then = remaining there should be approximately the expected setting =E2=80=93 = based on the average of two competing flows hovering at aroud 200-250 = kB=E2=80=A6 > >=20 > > *) I=E2=80=99m referring to the much higher likelihood nowadays, = that the application itself pacing and transfer volume violates the = design principle of TCP, where the implicit assumption was that the = sender has unlimited data to send, with the timing controlled at the = full disgression of TCP. > >=20 > >=20 > > Richard Scheffenegger > > Consulting Solution Architect > > NAS & Networking > >=20 > > NetApp > > +43 1 3676 811 3157 Direct Phone > > +43 664 8866 1857 Mobile Phone > > Richard.Scheffenegger@netapp.com > >=20 > >=20 > > > >=20 > > > > #DataDriven > >=20 > > https://ts.la/richard49892 > =20 > ------ > Randall Stewart > rrs@netflix.com ------ Randall Stewart rrs@netflix.com