From nobody Mon Jul 3 15:43:36 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qvqvw43w7z4lWMX for ; Mon, 3 Jul 2023 15:43:40 +0000 (UTC) (envelope-from muralik1@vmware.com) Received: from CO1PR02CU001.outbound.protection.outlook.com (mail-westus2azon11011004.outbound.protection.outlook.com [52.101.47.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qvqvv6rvXz4FtY; Mon, 3 Jul 2023 15:43:39 +0000 (UTC) (envelope-from muralik1@vmware.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CZaTUjC597A5wIjaJCSI3xqdqwVmlx+aoklz5qcKcjApcNebdcG9asnK2lX7gdcnjg3H1XpTUI2ftZbkSaYVBSq4Wh/bUZ/6MEKUFDckE5wxZ9RSLD3hYeGzEJ8u5Kka8PZ8lZH1eJse/DklVjVWUysYDUtI/oU4lkZe4vQmhs5bMhFpAwoMo7ZvJK1djnZHkzrsmR1f6v2jO3ApQnYIrf2HZ8r6i6Eweq1QD4IfgnMLGCrcW4Ony5SFVV65PisnqPE8aXTx13AW0Jco3nogFTwp0HboCbhRX/Bhmbjj7fEWV1o8ojuOjgeeJnLIryAKF7s9zQaNeOCdoDofAyAHlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DdlXdZzG2sKD46p3FriJC20dcFj0DTniHQXGy1tbKx8=; b=j2oVyVPXBffdihE510izv6Bjzg5Th4206KVcx9M8QVZym6x9Zykk/Xfo9MFyAjNncwQRq+Ip8IkrsrBHGu0EJepuUMmwkblFkhDU9xaJ4kCcKBleiNPGn5rUSosEXFNyrRPueRTBEu3vULp81KwFzcYwQdEeD1jNYzFB9B5jyGe6RCRR1/yfjasJBms/xw27nQYvzkBRFCO6GLHZljB4Q0LMs4pPkknmUbtvnRWbcfFc4tDqy37zNWOAdvapb/RBLgL2vDGGs6Bkh9raYrDwuiXvR6gVlWenaN3v1LRMnbHnn6Ru/rCj7J7ouKdugYYfzknZsTm3vP/BDnlHjlgO7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DdlXdZzG2sKD46p3FriJC20dcFj0DTniHQXGy1tbKx8=; b=2AY7dgVh3Ap6PXSN//+pHKfY4/bAQZo/bRNg/kZf5zTJjc/J2cyAZ1Pkv5+v8TMr23tK4ETKuueyOrZTTP7OdX+hAhrCl5k0zKDB6Nc4V4btHNKXNHxCBTgLQ/gWPH+P1wjkCiTuICfG7t70EdmWAPt65RZfUAFaoS09H5mIFwg= Received: from PH0PR05MB10064.namprd05.prod.outlook.com (2603:10b6:510:29d::8) by DS7PR05MB9969.namprd05.prod.outlook.com (2603:10b6:8:db::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Mon, 3 Jul 2023 15:43:36 +0000 Received: from PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593]) by PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593%4]) with mapi id 15.20.6544.019; Mon, 3 Jul 2023 15:43:36 +0000 From: Murali Krishnamurthy To: "Scheffenegger, Richard" , FreeBSD Transport Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Thread-Topic: FreeBSD TCP (with iperf3) comparison with Linux Thread-Index: AQHZrYmyiK08kPjzWEy1GrMDiqKRKq+oI//y Date: Mon, 3 Jul 2023 15:43:36 +0000 Message-ID: References: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org> In-Reply-To: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org> Accept-Language: en-IN, en-US Content-Language: en-IN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR05MB10064:EE_|DS7PR05MB9969:EE_ x-ms-office365-filtering-correlation-id: 8ffb8688-3605-4a09-d813-08db7bdc43cd x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: UZdczEgPZIU0yrHYyur2GvVWWeSUpZiy3tBvc0NIVpj3GBsuUANTDdWIUAztGcnCGYSrJe6qU7syxk70mSOh6aMJN8oPL4l8BFsRKNmdq5mZYET+IUguNvqoeTgQtpz/8iP9bVdH1kwJt+O+zLFYRjWVuHSgk6HbYH4pckOheB+/A25q1KDn3EVDUt2zr/2rXEge7fEBk7afdphyEQZ5txjRWHcdkbMbrXp0FTu4kpSSUq3Z76u8YFVN0zcPCU5CC69BaaejY2B1r+/S78BaphwN/xpR/s1o8KrpeOekRRFxVNfKX0WfEqztCQwLVhbVKUzfeiKh+tul01gt4E2c++0tzSf9Zd9mlteDwRtDhxxL+4CbGBr8tNzbqDqE8uVn2xxGZJcnrJozgeuZ1v6qaVP8Z6prLzpgDpR7xc7PBqelihEYHc+zEiLMbRv1LoEk44yXeTTNLGWiErtDCqoC8m1d9HllyF52ORi9RO0/Acmt8ttGFETs6z7xxb6t49bRK6pym71/wp3EcdYs8Ie/xfKOhtW4Cfi1wE9YeeZufb8IGBOkZHU48BXfTwytABlo+q/G/Z1kpvyRqWWNnCe8GyqTlP9oSJzjL6Fdz8KmNl5uMpMe/sqiIGTLpeJVfgrtJul+YXEGFQo/IbYSIynD23RHxRiydyUzGiSPA7POScnk6Y1GPEk016RO4IbfV3aYtjpQptb2IpsuGhOFJfgVkA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR05MB10064.namprd05.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(376002)(39860400002)(346002)(396003)(136003)(366004)(451199021)(41300700001)(86362001)(38100700002)(7696005)(71200400001)(166002)(83380400001)(122000001)(26005)(186003)(6506007)(53546011)(9686003)(966005)(55016003)(38070700005)(110136005)(478600001)(2906002)(316002)(450100002)(66476007)(91956017)(64756008)(66446008)(76116006)(66556008)(66946007)(8936002)(33656002)(8676002)(52536014)(5660300002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?lTa/hugUAxofk+5k3U7tPTZaoVp+jqE6LWxF9Z7nVCoyhY3lC6qhv4jp?= =?Windows-1252?Q?VxahvNyO0O9g9uHcLAKjA0ICwlD16Sk1kvnSf7GSquyvuCfZIbBrgbNS?= =?Windows-1252?Q?MIqeun3dCs0hxs9vWqRlDzZq20TL2RCq39A6hR54+H2MIfkPCuu3f0Yu?= =?Windows-1252?Q?9HNFV8RwdTwCEHS7OCFWXmJUHLdosDlMb3wws8PgvbEw3zrHo/wd8JY3?= =?Windows-1252?Q?zBfXA7Kh/jx3xSWyJDEUQyMXbNGIfpZjBIGh5RiNBh+TqH8+Wn8vIFxZ?= =?Windows-1252?Q?NRWYfi6SYPBj4w5CnEku0v0sZnb68P2S5WZTO/mc+fSHJ+Rxep2m8VB+?= =?Windows-1252?Q?3PdqmZfdD90e6QIrFW7jGVsKO1B9yC21X+7YhMBw1iLi6a+P9S7WZPiW?= =?Windows-1252?Q?F9h+/H16MMMWeWTmV6bTNRGqI/A1H3UNPyzpMseMDBZnHJiPeGDh8IpK?= =?Windows-1252?Q?LaZ6LlTtfUWg//WekgMj0Tz2/gVMuHUh0Rm6PrsdlpYVtVTyiTf1ZXy+?= =?Windows-1252?Q?xgSBI26eIBxD6r1BuamTV3VbHmPNAyQ2jRSisIJlTWou5MQ8nD/EoeAr?= =?Windows-1252?Q?mrcwP+PTc35rbYhVRB55WfT6JpdzHF27GmKsppYPLx6NpM3+050ZRFnk?= =?Windows-1252?Q?1TMh6xmBvnd+jI9aBKabjYJJ7RhhqL3MnU+zKyOVRDMwrBCy9SVCAFk4?= =?Windows-1252?Q?F2ovlmEevDMYuadkpXl674EOrjM7vEvb+eXTC4LV3jRzdmI8SSbdWZOQ?= =?Windows-1252?Q?stn41ht1Br5BCO8cs0T63UhxWyqzrTellyjtUAUv8jsnH31aEQ7+ZftM?= =?Windows-1252?Q?oNr4R0WMxpA238eYorYuvE5Pc/u+Mg3yUV5GtFmvKQ1XaJsllNa2V6At?= =?Windows-1252?Q?ZsiovdO7ZQr89NLlxOMD3KSK7Lsb7AWf9hqFbJhmEa4zc8sivVDURWDz?= =?Windows-1252?Q?Q7SBKlm4kJGrAtqQQkOlsydtjvnZGfZ+VxIDVOzEkGu3c6PIpdFix+Fd?= =?Windows-1252?Q?i3KUBJWTCwR+9X6QT852zQKbe6HaXklwiUQK/1SJO3EPyYO0axRL1g0G?= =?Windows-1252?Q?vS5ykLfm2dEk3lhzdQ3bCTCeFkGk5CuEme5YVqSggHS/of/UnoEIOo5V?= =?Windows-1252?Q?sa/WRHE97iIopEKMeVyafPH0XRu6KTsMKjbAa7wumzeKIFFUrOhM3FYn?= =?Windows-1252?Q?7OBbs3r6orUK2s6fR7uI1yT7dcEP/UxybZgBQ0e2XQaZZ/QpOCLpGdBN?= =?Windows-1252?Q?3m1+1j/Xx78zJqdCKJlc4s3LZYxA9LARil7mlqqU01GGBIvG0j0IH1/j?= =?Windows-1252?Q?g3N6wVfMDPGMADgTeqFs1BeS5tAyRMrgTjWiLNKvaNMsx/nWnn+IQvEe?= =?Windows-1252?Q?dTclk7JTBxl/TlS8I1NE7nUGwhwNYuZobbVH5vVuZQxC+7CIUlP4smvv?= =?Windows-1252?Q?9uetypbf8dxWaVNZXL0ZLH7IeF7p+XGnjyquTnGBZNwPIleZWRFHjpi/?= =?Windows-1252?Q?nTGFrXHbpUlEtL2Ho95CNi1LzZaLGUolf794fTD8HEGZqnHOWtvtxImj?= =?Windows-1252?Q?LrG8Z3ve/a8ajTh930oxxgjljtJYK8itDRGEqN1qxDnMFZ2dRRau0FU7?= =?Windows-1252?Q?OZ9BJuUV6QZTk6KT30au6yTE9nxEd/v5quRGn47zHHElRUmg2I3ArV7K?= =?Windows-1252?Q?E6+G2FYs55jK7AWDFV2h7zJ6swWI8Q2R5V0dxAEnUMpllyYEyJ7JUQ?= =?Windows-1252?Q?=3D=3D?= Content-Type: multipart/alternative; boundary="_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_" List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR05MB10064.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8ffb8688-3605-4a09-d813-08db7bdc43cd X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jul 2023 15:43:36.3306 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7AUFED+1PnssT+k/QWV9HUge8W4xldplD9Kp5Wtvi6ULQDtamRtiSakVRdQdlMIbXYOLK7LyBUG7P9AQr5wTuA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR05MB9969 X-Rspamd-Queue-Id: 4Qvqvv6rvXz4FtY X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:52.96.0.0/12, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Richard, =93How having high timer granularity pacing (as what RACK does) can prevent= microbursts?=94 =93 That TCP can, at high transmission rates, overrun the NIC driver memory= buffers?=94 Yes. Any relevant doc/write-up on these would help for better understanding= . I know high transmission of packets could lead to packet drops at NIC buffe= rs. Linux somehow seem to be avoiding that problem. =93What did your experiments with the suggestions by @cc show with the base= stack?=94 Just posted a follow up query to Cheng on this. On enabling RACK: It seems we need to rebuild freebsd with compile time opt= ion. Is there a build (iso) available in repository with RACK enabled alrea= dy? Regards Murali On 03/07/23, 2:08 PM, "Scheffenegger, Richard" wrote: Hi Murali, > Appreciate the useful inputs you have shared so far. Will try to figure o= ut regarding packet drops. > > Regarding HyStart, I see even BSD code base has support for this. May I k= now by when can we see that in an release, if not already available ? You "simply" switch to the RACK stack. Just as I mentioned earlier. Note th= at this is a completely refactored stack by itself - so please validate tha= t all your userspace applications (or kernel patches) still work when you a= ctivate it. https://man.freebsd.org/cgi/man.cgi?query=3Dtcp_rack&sektion=3D4&manpath=3D= FreeBSD+13.2-RELEASE+and+Ports > > Regarding this point : =93Switching to other cc modules may give some mor= e insights. But again, I suspect that momentary (microsecond) burstiness of= BSD may be causing this significantly higher loss rate.=94 > Is there some info somewhere where I can understand more on this in detai= l? What detail? How to toggle to a different congestion control? How having high timer granularity pacing (as what RACK does) can prevent mi= crobursts? That TCP can, at high transmission rates, overrun the NIC driver memory buf= fers? What did your experiments with the suggestions by @cc show with the base st= ack? And again, the RACK stack does HyStart and high-granularity pacing, address= ing many issues which are important to smooth streaming workflows and preve= nting momentary burst loads. However, the RACK stack certainly needs to be = validated with particular applications such as the kernel NFS server/client= , which are deeply embedded and may rely on some things in the base stack t= o work properly. I suspect that until now, noone has done a thorough invest= igation of the implications of RACK with NFS really. Best regards, Richard > > Regards > Murali --_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Richard,<= o:p>

&nbs= p;

&nbs= p;

=93How having high timer granularity pacing (as what= RACK does) can prevent microbursts?=94

=93 That TCP can, at high transmission rates, overrun the NIC driver memory b= uffers?=94

 

Yes. Any relevant doc/write-up on these would help f= or better understanding.

 

I know high transmission of packets could lead to pa= cket drops at NIC buffers. Linux somehow seem to be avoiding that problem.

 

&nbs= p;

=93What did your experiments with the suggestions by @cc show with the base s= tack?=94

Just post= ed a follow up query to Cheng on this.

&nbs= p;

On enabli= ng RACK: It seems we need to rebuild freebsd with compile time option. Is t= here a build (iso) available in repository with RACK enabled already?<= /o:p>

&nbs= p;

&nbs= p;

Regards

Murali

&nbs= p;

&nbs= p;

On 03/07/23, 2:08 PM,= "Scheffenegger, Richard" <rscheff@freebsd.org> wrote:=

Hi Murali,

 

> Appreciate the useful inputs you have shared so= far. Will try to figure out regarding packet drops.

>

> Regarding HyStart, I see even BSD code base has= support for this. May I know by when can we see that in an release, if not= already available ?

 

You "simply" switch to the RACK stack. Jus= t as I mentioned earlier. Note that this is a completely refactored stack b= y itself - so please validate that all your userspace applications (or kern= el patches) still work when you activate it.

 

 

>

> Regarding this point : =93Switching to other cc= modules may give some more insights. But again, I suspect that momentary (= microsecond) burstiness of BSD may be causing this significantly higher los= s rate.=94

> Is there some info somewhere where I can unders= tand more on this in detail?

 

What detail? How to toggle to a different congestion= control?

 

How having high timer granularity pacing (as what RA= CK does) can prevent microbursts?

 

That TCP can, at high transmission rates, overrun th= e NIC driver memory buffers?

 

What did your experiments with the suggestions by @c= c show with the base stack?

 

And again, the RACK stack does HyStart and high-gran= ularity pacing, addressing many issues which are important to smooth stream= ing workflows and preventing momentary burst loads. However, the RACK stack= certainly needs to be validated with particular applications such as the kernel NFS server/client, which are de= eply embedded and may rely on some things in the base stack to work properl= y. I suspect that until now, noone has done a thorough investigation of the= implications of RACK with NFS really.

 

 

Best regards,

   Richard

 

>

> Regards

> Murali

 

--_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_--