From nobody Fri Jun 30 16:32:08 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qt17W2Lfwz4kL5x for ; Fri, 30 Jun 2023 16:32:23 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qt17V64w0z3N5K; Fri, 30 Jun 2023 16:32:22 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-5636425bf98so1222207eaf.1; Fri, 30 Jun 2023 09:32:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688142741; x=1690734741; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=equMdtkC/tM+aozNsnL8YTIziCHoRqd7CL+9IXiwaN8=; b=Ob3E1xlGsIwfrQzm34odb26laCm2tdsNt9vei0e+qxsJdniAekmj/DaAXW4H65Z8jZ birhb4Axh+hMQkRxSdME+q6enwwtfuiigZU8gK2PesC88sNdKF6WVVsaEbfdz0SVkpIR C1sxtBzigkK7XJ4YNmp32ZVmFGovlMWp2f7+4V1Jl1nWl0INqTrLxtHC1Z/alGjVyT4f oL7gg6TZK/1aNT2vPTfcnjYz0wTy3bKqsh8rYNcIxHxRBqRvHii3if9vqrABvJ8IA8bm RnzhnnqzPauuDkW8NNhjtQbXEZKQxHxkWqiK/31Iz+w2Y+6j0V8hD6Z4NHV8AlB6NJdc vVQA== X-Gm-Message-State: AC+VfDyxsAIw5tRIpq+9KWcPmBMSfOESmK7Ozzextx3KuXv0iKgdE2Jk a01DPabIW4yBiANoXIk3sBtR6gqR+G4= X-Google-Smtp-Source: ACHHUZ5tqgiSY0dePtEA0OhT6mFFBLpzKCj2/BFUJEc7BAX5fHq3qFPuFrW3eqelVeON5vS1Tvm06g== X-Received: by 2002:a4a:2c04:0:b0:563:516e:ae3d with SMTP id o4-20020a4a2c04000000b00563516eae3dmr2551536ooo.6.1688142740841; Fri, 30 Jun 2023 09:32:20 -0700 (PDT) Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com. [209.85.160.50]) by smtp.gmail.com with ESMTPSA id bd20-20020a4aee14000000b0051134f333d3sm2111873oob.16.2023.06.30.09.32.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 30 Jun 2023 09:32:20 -0700 (PDT) Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-1b055510c9dso1575454fac.0; Fri, 30 Jun 2023 09:32:20 -0700 (PDT) X-Received: by 2002:a05:6870:b506:b0:1b0:60ff:b745 with SMTP id v6-20020a056870b50600b001b060ffb745mr3336068oap.48.1688142739897; Fri, 30 Jun 2023 09:32:19 -0700 (PDT) List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 References: <53aff274-b1a8-0730-6971-2755c7e7b688@freebsd.org> In-Reply-To: From: Cheng Cui Date: Fri, 30 Jun 2023 12:32:08 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux To: Murali Krishnamurthy Cc: "Scheffenegger, Richard" , FreeBSD Transport Content-Type: multipart/alternative; boundary="00000000000057840405ff5b5cdc" X-Rspamd-Queue-Id: 4Qt17V64w0z3N5K X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000057840405ff5b5cdc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I used an emulation testbed from Emulab.net with Dummynet traffic shaper adding 100ms RTT between two nodes, the link capacity is 1Gbps and both nodes are using freebsd13.2. cc@s1:~ % ping -c 3 r1 PING r1-link1 (10.1.1.3): 56 data bytes 64 bytes from 10.1.1.3: icmp_seq=3D0 ttl=3D64 time=3D100.091 ms 64 bytes from 10.1.1.3: icmp_seq=3D1 ttl=3D64 time=3D99.995 ms 64 bytes from 10.1.1.3: icmp_seq=3D2 ttl=3D64 time=3D99.979 ms --- r1-link1 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 99.979/100.022/100.091/0.049 ms cc@s1:~ % iperf3 -c r1 -t 10 -i 1 -C cubic Connecting to host r1, port 5201 [ 5] local 10.1.1.2 port 56089 connected to 10.1.1.3 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.19 MBytes 35.2 Mbits/sec 0 1.24 MBytes [ 5] 1.00-2.00 sec 56.5 MBytes 474 Mbits/sec 6 2.41 MBytes [ 5] 2.00-3.00 sec 58.6 MBytes 492 Mbits/sec 18 7.17 MBytes [ 5] 3.00-4.00 sec 65.6 MBytes 550 Mbits/sec 14 606 KBytes [ 5] 4.00-5.00 sec 60.8 MBytes 510 Mbits/sec 18 7.22 MBytes [ 5] 5.00-6.00 sec 62.1 MBytes 521 Mbits/sec 12 7.86 MBytes [ 5] 6.00-7.00 sec 60.9 MBytes 512 Mbits/sec 14 3.43 MBytes [ 5] 7.00-8.00 sec 62.8 MBytes 527 Mbits/sec 16 372 KBytes [ 5] 8.00-9.00 sec 59.3 MBytes 497 Mbits/sec 14 1.77 MBytes [ 5] 9.00-10.00 sec 57.0 MBytes 477 Mbits/sec 18 7.13 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 548 MBytes 459 Mbits/sec 130 sende= r [ 5] 0.00-10.10 sec 540 MBytes 449 Mbits/sec receiver iperf Done. cc@s1:~ % ifconfig bce4 bce4: flags=3D8843 metric 0 mtu 150= 0 options=3Dc01bb ether 00:10:18:56:94:d4 inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 media: Ethernet 1000baseT status: active nd6 options=3D29 I believe the default values for bce tx/rx pages are 2. And I happened to find this problem before that when the tx queue was full, it would not enqueue packets and started return errors. And this error was misunderstood by the TCP layer as retransmission. After adding hw.bce.tx_pages=3D4 and hw.bce.rx_pages=3D4 in /boot/loader.co= nf and reboot: cc@s1:~ % iperf3 -c r1 -t 10 -i 1 -C cubic Connecting to host r1, port 5201 [ 5] local 10.1.1.2 port 20478 connected to 10.1.1.3 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.15 MBytes 34.8 Mbits/sec 0 1.17 MBytes [ 5] 1.00-2.00 sec 83.1 MBytes 697 Mbits/sec 0 12.2 MBytes [ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 12.2 MBytes [ 5] 3.00-4.00 sec 113 MBytes 944 Mbits/sec 0 12.2 MBytes [ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 12.2 MBytes [ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 12.2 MBytes [ 5] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 0 12.2 MBytes [ 5] 7.00-8.00 sec 113 MBytes 944 Mbits/sec 0 12.2 MBytes [ 5] 8.00-9.00 sec 112 MBytes 938 Mbits/sec 0 12.2 MBytes [ 5] 9.00-10.00 sec 113 MBytes 947 Mbits/sec 0 12.2 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 985 MBytes 826 Mbits/sec 0 sende= r [ 5] 0.00-10.11 sec 982 MBytes 815 Mbits/sec receiver iperf Done. Best Regards, Cheng Cui On Fri, Jun 30, 2023 at 12:26=E2=80=AFPM Murali Krishnamurthy wrote: > Richard, > > > > Appreciate the useful inputs you have shared so far. Will try to figure > out regarding packet drops. > > > > Regarding HyStart, I see even BSD code base has support for this. May I > know by when can we see that in an release, if not already available ? > > > > Regarding this point : *=E2=80=9C**Switching to other cc modules may give= some > more insights. But again, I suspect that momentary (microsecond) burstine= ss > of BSD may be causing this significantly higher loss rate.**=E2=80=9D* > > Is there some info somewhere where I can understand more on this in detai= l? > > > > Regards > > Murali > > > > > > On 30/06/23, 9:35 PM, "owner-freebsd-transport@freebsd.org" < > owner-freebsd-transport@freebsd.org> wrote: > > > > Hi Murali, > > > > > Q. Since you mention two hypervisors - what is the phyiscal network > topology in between these two servers? What theoretical link rates would = be > attainable? > > > > > > Here is the topology > > > > > > Iperf end points are on 2 different hypervisors. > > > > > > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94 =E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94 > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 =E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94-=E2=80=94 > > > | Linux VM1 | | BSD 13 VM > 1 | > | Linux VM2 | | BSD 13 VM 2 | > > > |___________| |_ ____ ____ ___ > | = |___________ > | |_ ____ ____ ___ | > > > | | > | > | | > > > > | | > | | > > > > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 = =E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > > > | ESX Hypervisor 1 | 10G link connected vi= a > L2 Switch | ESX Hypervisor 2 | > > > | > |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > | | > > > |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > | > |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94| > > > > > > > > > Nic is of 10G capacity on both ESX server and it has below config. > > > > > > So, when both VMs run on the same Hypervisor, maybe with another VM to > simulate the 100ms delay, can you attain a lossless baseline scenario? > > > > > > > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ > 1024 =3D 1.25 Gbps > > > > > > So theoretically we should see close to 1.25Gbps of Bitrate and we see > Linux reaching close to this number. > > > > Under no loss, yes. > > > > > > > But BSD is not able to do that. > > > > > > > > > Q. Did you run iperf3? Did the transmitting endpoint report any > retransmissions between Linux or FBSD hosts? > > > > > > Yes, we used iper3. I see Linux doing less number retransmissions > compared to BSD. > > > On BSD, the best performance was around 600 Mbps bitrate and the number > of retransmissions for this number seen is around 32K > > > On Linux, the best performance was around 1.15 Gbps bitrate and the > number of retransmissions for this number seen is only 2K. > > > So as you pointed the number of retransmissions in BSD could be the rea= l > issue here. > > > > There are other cc modules available; but I believe one major deviation i= s > that Linux can perform mechanisms like hystart; ACKing every packet when > the client detects slow start; perform pacing to achieve more uniform > packet transmissions. > > > > I think the next step would be to find out, at which queue those packet > discards are coming from (external switch? delay generator? Vswitch? Eth > stack inside the VM?) > > > > Or alternatively, provide your ESX hypervisors with vastly more link > speed, to rule out any L2 induced packet drops - provided your delay > generator is not the source when momentarily overloaded. > > > > > Is there a way to reduce this packet loss by fine tuning some parameter= s > w.r.t ring buffer or any other areas? > > > > Finding where these arise (looking at queue and port counters) would be > the next step. But this is not really my specific area of expertise beyon= d > the high level, vendor independent observations. > > > > Switching to other cc modules may give some more insights. But again, I > suspect that momentary (microsecond) burstiness of BSD may be causing thi= s > significantly higher loss rate. > > > > TCP RACK would be another option. That stack has pacing, more fine-graine= d > timing, the RACK loss recovery mechanisms etc. Maybe that helps reduce th= e > observed packet drops by iperf, and consequently, yield a higher overall > throuhgput. > > > > > > > > > --00000000000057840405ff5b5cdc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+SSB1c2VkIGFuIGVtdWxhdGlvbiB0ZXN0YmVkIGZyb20gRW11bGFiLm5l dCB3aXRoIER1bW15bmV0IHRyYWZmaWMgc2hhcGVyIGFkZGluZyAxMDBtcyBSVFQ8YnI+YmV0d2Vl biB0d28gbm9kZXMsIHRoZSBsaW5rIGNhcGFjaXR5IGlzIDFHYnBzIGFuZCBib3RoIG5vZGVzIGFy ZSB1c2luZyBmcmVlYnNkMTMuMi48YnI+PGJyPmNjQHMxOn4gJSBwaW5nIC1jIDMgcjE8YnI+UElO RyByMS1saW5rMSAoMTAuMS4xLjMpOiA1NiBkYXRhIGJ5dGVzPGJyPjY0IGJ5dGVzIGZyb20gPGEg aHJlZj0iaHR0cDovLzEwLjEuMS4zIj4xMC4xLjEuMzwvYT46IGljbXBfc2VxPTAgdHRsPTY0IHRp bWU9MTAwLjA5MSBtczxicj42NCBieXRlcyBmcm9tIDxhIGhyZWY9Imh0dHA6Ly8xMC4xLjEuMyI+ MTAuMS4xLjM8L2E+OiBpY21wX3NlcT0xIHR0bD02NCB0aW1lPTk5Ljk5NSBtczxicj42NCBieXRl cyBmcm9tIDxhIGhyZWY9Imh0dHA6Ly8xMC4xLjEuMyI+MTAuMS4xLjM8L2E+OiBpY21wX3NlcT0y IHR0bD02NCB0aW1lPTk5Ljk3OSBtczxicj48YnI+LS0tIHIxLWxpbmsxIHBpbmcgc3RhdGlzdGlj cyAtLS08YnI+MyBwYWNrZXRzIHRyYW5zbWl0dGVkLCAzIHBhY2tldHMgcmVjZWl2ZWQsIDAuMCUg cGFja2V0IGxvc3M8YnI+cm91bmQtdHJpcCBtaW4vYXZnL21heC9zdGRkZXYgPSA5OS45NzkvMTAw LjAyMi8xMDAuMDkxLzAuMDQ5IG1zPGJyPjxicj48YnI+Y2NAczE6fiAlIGlwZXJmMyAtYyByMSAt dCAxMCAtaSAxIC1DIGN1YmljPGJyPkNvbm5lY3RpbmcgdG8gaG9zdCByMSwgcG9ydCA1MjAxPGJy PlsgwqA1XSBsb2NhbCAxMC4xLjEuMiBwb3J0IDU2MDg5IGNvbm5lY3RlZCB0byAxMC4xLjEuMyBw b3J0IDUyMDE8YnI+WyBJRF0gSW50ZXJ2YWwgwqAgwqAgwqAgwqAgwqAgVHJhbnNmZXIgwqAgwqAg Qml0cmF0ZSDCoCDCoCDCoCDCoCBSZXRyIMKgQ3duZDxicj5bIMKgNV0gwqAgMC4wMC0xLjAwIMKg IHNlYyDCoDQuMTkgTUJ5dGVzIMKgMzUuMiBNYml0cy9zZWMgwqAgwqAwIMKgIDEuMjQgTUJ5dGVz IMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgMS4wMC0yLjAwIMKgIHNlYyDCoDU2LjUgTUJ5dGVzIMKg IDQ3NCBNYml0cy9zZWMgwqAgwqA2IMKgIDIuNDEgTUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0g wqAgMi4wMC0zLjAwIMKgIHNlYyDCoDU4LjYgTUJ5dGVzIMKgIDQ5MiBNYml0cy9zZWMgwqAgMTgg wqAgNy4xNyBNQnl0ZXMgwqAgwqAgwqAgPGJyPlsgwqA1XSDCoCAzLjAwLTQuMDAgwqAgc2VjIMKg NjUuNiBNQnl0ZXMgwqAgNTUwIE1iaXRzL3NlYyDCoCAxNCDCoCDCoDYwNiBLQnl0ZXMgwqAgwqAg wqAgPGJyPlsgwqA1XSDCoCA0LjAwLTUuMDAgwqAgc2VjIMKgNjAuOCBNQnl0ZXMgwqAgNTEwIE1i aXRzL3NlYyDCoCAxOCDCoCA3LjIyIE1CeXRlcyDCoCDCoCDCoCA8YnI+WyDCoDVdIMKgIDUuMDAt Ni4wMCDCoCBzZWMgwqA2Mi4xIE1CeXRlcyDCoCA1MjEgTWJpdHMvc2VjIMKgIDEyIMKgIDcuODYg TUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgNi4wMC03LjAwIMKgIHNlYyDCoDYwLjkgTUJ5 dGVzIMKgIDUxMiBNYml0cy9zZWMgwqAgMTQgwqAgMy40MyBNQnl0ZXMgwqAgwqAgwqAgPGJyPlsg wqA1XSDCoCA3LjAwLTguMDAgwqAgc2VjIMKgNjIuOCBNQnl0ZXMgwqAgNTI3IE1iaXRzL3NlYyDC oCAxNiDCoCDCoDM3MiBLQnl0ZXMgwqAgwqAgwqAgPGJyPlsgwqA1XSDCoCA4LjAwLTkuMDAgwqAg c2VjIMKgNTkuMyBNQnl0ZXMgwqAgNDk3IE1iaXRzL3NlYyDCoCAxNCDCoCAxLjc3IE1CeXRlcyDC oCDCoCDCoCA8YnI+WyDCoDVdIMKgIDkuMDAtMTAuMDAgwqBzZWMgwqA1Ny4wIE1CeXRlcyDCoCA0 NzcgTWJpdHMvc2VjIMKgIDE4IMKgIDcuMTMgTUJ5dGVzIMKgIMKgIMKgIDxicj4tIC0gLSAtIC0g LSAtIC0gLSAtIC0gLSAtIC0gLSAtIC0gLSAtIC0gLSAtIC0gLSAtPGJyPlsgSURdIEludGVydmFs IMKgIMKgIMKgIMKgIMKgIFRyYW5zZmVyIMKgIMKgIEJpdHJhdGUgwqAgwqAgwqAgwqAgUmV0cjxi cj5bIMKgNV0gwqAgMC4wMC0xMC4wMCDCoHNlYyDCoCA1NDggTUJ5dGVzIMKgIDQ1OSBNYml0cy9z ZWMgwqAxMzAgwqAgwqAgwqAgwqAgwqAgwqAgc2VuZGVyPGJyPlsgwqA1XSDCoCAwLjAwLTEwLjEw IMKgc2VjIMKgIDU0MCBNQnl0ZXMgwqAgNDQ5IE1iaXRzL3NlYyDCoCDCoCDCoCDCoCDCoCDCoCDC oCDCoCDCoHJlY2VpdmVyPGJyPjxicj5pcGVyZiBEb25lLjxicj48YnI+Y2NAczE6fiAlIGlmY29u ZmlnIGJjZTQ8YnI+YmNlNDogZmxhZ3M9ODg0MyZsdDtVUCxCUk9BRENBU1QsUlVOTklORyxTSU1Q TEVYLE1VTFRJQ0FTVCZndDsgbWV0cmljIDAgbXR1IDE1MDA8YnI+CW9wdGlvbnM9YzAxYmImbHQ7 UlhDU1VNLFRYQ1NVTSxWTEFOX01UVSxWTEFOX0hXVEFHR0lORyxKVU1CT19NVFUsVkxBTl9IV0NT VU0sVFNPNCxWTEFOX0hXVFNPLExJTktTVEFURSZndDs8YnI+CWV0aGVyIDAwOjEwOjE4OjU2Ojk0 OmQ0PGJyPglpbmV0IDEwLjEuMS4yIG5ldG1hc2sgMHhmZmZmZmYwMCBicm9hZGNhc3QgMTAuMS4x LjI1NTxicj4JbWVkaWE6IEV0aGVybmV0IDEwMDBiYXNlVCAmbHQ7ZnVsbC1kdXBsZXgmZ3Q7PGJy PglzdGF0dXM6IGFjdGl2ZTxicj4JbmQ2IG9wdGlvbnM9MjkmbHQ7UEVSRk9STU5VRCxJRkRJU0FC TEVELEFVVE9fTElOS0xPQ0FMJmd0Ozxicj48YnI+SSBiZWxpZXZlIHRoZSBkZWZhdWx0IHZhbHVl cyBmb3IgYmNlIHR4L3J4IHBhZ2VzIGFyZSAyLiBBbmQgSSBoYXBwZW5lZCB0byBmaW5kPGJyPnRo aXMgcHJvYmxlbSBiZWZvcmUgdGhhdCB3aGVuIHRoZSB0eCBxdWV1ZSB3YXMgZnVsbCwgaXQgd291 bGQgbm90IGVucXVldWUgcGFja2V0czxicj5hbmQgc3RhcnRlZCByZXR1cm4gZXJyb3JzLjxicj5B bmQgdGhpcyBlcnJvciB3YXMgbWlzdW5kZXJzdG9vZCBieSB0aGUgVENQIGxheWVyIGFzIHJldHJh bnNtaXNzaW9uLjxicj48YnI+QWZ0ZXIgYWRkaW5nIGh3LmJjZS50eF9wYWdlcz00IGFuZCBody5i Y2UucnhfcGFnZXM9NCBpbiAvYm9vdC9sb2FkZXIuY29uZiBhbmQgcmVib290Ojxicj48YnI+Y2NA czE6fiAlIGlwZXJmMyAtYyByMSAtdCAxMCAtaSAxIC1DIGN1YmljPGJyPkNvbm5lY3RpbmcgdG8g aG9zdCByMSwgcG9ydCA1MjAxPGJyPlsgwqA1XSBsb2NhbCAxMC4xLjEuMiBwb3J0IDIwNDc4IGNv bm5lY3RlZCB0byAxMC4xLjEuMyBwb3J0IDUyMDE8YnI+WyBJRF0gSW50ZXJ2YWwgwqAgwqAgwqAg wqAgwqAgVHJhbnNmZXIgwqAgwqAgQml0cmF0ZSDCoCDCoCDCoCDCoCBSZXRyIMKgQ3duZDxicj5b IMKgNV0gwqAgMC4wMC0xLjAwIMKgIHNlYyDCoDQuMTUgTUJ5dGVzIMKgMzQuOCBNYml0cy9zZWMg wqAgwqAwIMKgIDEuMTcgTUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgMS4wMC0yLjAwIMKg IHNlYyDCoDgzLjEgTUJ5dGVzIMKgIDY5NyBNYml0cy9zZWMgwqAgwqAwIMKgIDEyLjIgTUJ5dGVz IMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgMi4wMC0zLjAwIMKgIHNlYyDCoCAxMTIgTUJ5dGVzIMKg IDkzOSBNYml0cy9zZWMgwqAgwqAwIMKgIDEyLjIgTUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0g wqAgMy4wMC00LjAwIMKgIHNlYyDCoCAxMTMgTUJ5dGVzIMKgIDk0NCBNYml0cy9zZWMgwqAgwqAw IMKgIDEyLjIgTUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgNC4wMC01LjAwIMKgIHNlYyDC oCAxMTIgTUJ5dGVzIMKgIDk0MCBNYml0cy9zZWMgwqAgwqAwIMKgIDEyLjIgTUJ5dGVzIMKgIMKg IMKgIDxicj5bIMKgNV0gwqAgNS4wMC02LjAwIMKgIHNlYyDCoCAxMTIgTUJ5dGVzIMKgIDk0MiBN Yml0cy9zZWMgwqAgwqAwIMKgIDEyLjIgTUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgNi4w MC03LjAwIMKgIHNlYyDCoCAxMTIgTUJ5dGVzIMKgIDkzOCBNYml0cy9zZWMgwqAgwqAwIMKgIDEy LjIgTUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgNy4wMC04LjAwIMKgIHNlYyDCoCAxMTMg TUJ5dGVzIMKgIDk0NCBNYml0cy9zZWMgwqAgwqAwIMKgIDEyLjIgTUJ5dGVzIMKgIMKgIMKgIDxi cj5bIMKgNV0gwqAgOC4wMC05LjAwIMKgIHNlYyDCoCAxMTIgTUJ5dGVzIMKgIDkzOCBNYml0cy9z ZWMgwqAgwqAwIMKgIDEyLjIgTUJ5dGVzIMKgIMKgIMKgIDxicj5bIMKgNV0gwqAgOS4wMC0xMC4w MCDCoHNlYyDCoCAxMTMgTUJ5dGVzIMKgIDk0NyBNYml0cy9zZWMgwqAgwqAwIMKgIDEyLjIgTUJ5 dGVzIMKgIMKgIMKgIDxicj4tIC0gLSAtIC0gLSAtIC0gLSAtIC0gLSAtIC0gLSAtIC0gLSAtIC0g LSAtIC0gLSAtPGJyPlsgSURdIEludGVydmFsIMKgIMKgIMKgIMKgIMKgIFRyYW5zZmVyIMKgIMKg IEJpdHJhdGUgwqAgwqAgwqAgwqAgUmV0cjxicj5bIMKgNV0gwqAgMC4wMC0xMC4wMCDCoHNlYyDC oCA5ODUgTUJ5dGVzIMKgIDgyNiBNYml0cy9zZWMgwqAgwqAwIMKgIMKgIMKgIMKgIMKgIMKgIHNl bmRlcjxicj5bIMKgNV0gwqAgMC4wMC0xMC4xMSDCoHNlYyDCoCA5ODIgTUJ5dGVzIMKgIDgxNSBN Yml0cy9zZWMgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqByZWNlaXZlcjxicj48YnI+aXBlcmYg RG9uZS48ZGl2PjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEtc21h cnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPjxkaXYgZGlyPSJsdHIiPjxkaXY+PGJyPjwvZGl2PkJl c3QgUmVnYXJkcyw8ZGl2PkNoZW5nIEN1aTwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pjxicj48L2Rp dj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFp bF9hdHRyIj5PbiBGcmksIEp1biAzMCwgMjAyMyBhdCAxMjoyNuKAr1BNIE11cmFsaSBLcmlzaG5h bXVydGh5ICZsdDs8YSBocmVmPSJtYWlsdG86bXVyYWxpazFAdm13YXJlLmNvbSI+bXVyYWxpazFA dm13YXJlLmNvbTwvYT4mZ3Q7IHdyb3RlOjxicj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21h aWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4 IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBjbGFzcz0ibXNn NzI3Njc4OTg1Nzg1NDI4NDgzMiI+DQoNCg0KDQoNCg0KPGRpdiBzdHlsZT0ib3ZlcmZsb3ctd3Jh cDogYnJlYWstd29yZDsiIGxhbmc9IkVOLUlOIj4NCjxkaXYgY2xhc3M9Im1fLTg5MzUxNjA5MzAz MjkzMzM4NTdXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4+UmljaGFy ZCw8dT48L3U+PHU+PC91Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bhbj48 dT48L3U+wqA8dT48L3U+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuPkFw cHJlY2lhdGUgdGhlIHVzZWZ1bCBpbnB1dHMgeW91IGhhdmUgc2hhcmVkIHNvIGZhci4gV2lsbCB0 cnkgdG8gZmlndXJlIG91dCByZWdhcmRpbmcgcGFja2V0IGRyb3BzLjx1PjwvdT48dT48L3U+PC9z cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuPjx1PjwvdT7CoDx1PjwvdT48L3Nw YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4+UmVnYXJkaW5nIEh5U3RhcnQsIEkg c2VlIGV2ZW4gQlNEIGNvZGUgYmFzZSBoYXMgc3VwcG9ydCBmb3IgdGhpcy4gTWF5IEkga25vdyBi eSB3aGVuIGNhbiB3ZSBzZWUgdGhhdCBpbiBhbiByZWxlYXNlLCBpZiBub3QgYWxyZWFkeSBhdmFp bGFibGUgPzx1PjwvdT48dT48L3U+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuPjx1PjwvdT7CoDx1PjwvdT48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4+UmVnYXJkaW5nIHRoaXMgcG9pbnQgOg0KPGk+4oCcPC9pPjwvc3Bhbj48aT5Td2l0Y2hpbmcg dG8gb3RoZXIgY2MgbW9kdWxlcyBtYXkgZ2l2ZSBzb21lIG1vcmUgaW5zaWdodHMuIEJ1dCBhZ2Fp biwgSSBzdXNwZWN0IHRoYXQgbW9tZW50YXJ5IChtaWNyb3NlY29uZCkgYnVyc3RpbmVzcyBvZiBC U0QgbWF5IGJlIGNhdXNpbmcgdGhpcyBzaWduaWZpY2FudGx5IGhpZ2hlciBsb3NzIHJhdGUuPC9p PjxpPjxzcGFuPuKAnTwvc3Bhbj48L2k+PHNwYW4+PHU+PC91Pjx1PjwvdT48L3NwYW4+PC9wPg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4+SXMgdGhlcmUgc29tZSBpbmZvIHNvbWV3aGVyZSB3 aGVyZSBJIGNhbiB1bmRlcnN0YW5kIG1vcmUgb24gdGhpcyBpbiBkZXRhaWw/PC9zcGFuPjx1Pjwv dT48dT48L3U+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4+PHU+PC91PsKgPHU+PC91 Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bhbj5SZWdhcmRzPHU+PC91Pjx1 PjwvdT48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4+TXVyYWxpPHU+PC91 Pjx1PjwvdT48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4+PHU+PC91PsKg PHU+PC91Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bhbj48dT48L3U+wqA8 dT48L3U+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90 dG9tOjEycHQiPk9uIDMwLzA2LzIzLCA5OjM1IFBNLCAmcXVvdDs8YSBocmVmPSJtYWlsdG86b3du ZXItZnJlZWJzZC10cmFuc3BvcnRAZnJlZWJzZC5vcmciIHRhcmdldD0iX2JsYW5rIj5vd25lci1m cmVlYnNkLXRyYW5zcG9ydEBmcmVlYnNkLm9yZzwvYT4mcXVvdDsgJmx0OzxhIGhyZWY9Im1haWx0 bzpvd25lci1mcmVlYnNkLXRyYW5zcG9ydEBmcmVlYnNkLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPm93 bmVyLWZyZWVic2QtdHJhbnNwb3J0QGZyZWVic2Qub3JnPC9hPiZndDsgd3JvdGU6PHU+PC91Pjx1 PjwvdT48L3A+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PsKgPHU+PC91Pjwv cD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPkhpIE11cmFsaSw8dT48L3U+ PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7C oDx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IFEu IFNpbmNlIHlvdSBtZW50aW9uIHR3byBoeXBlcnZpc29ycyAtIHdoYXQgaXMgdGhlIHBoeWlzY2Fs IG5ldHdvcmsgdG9wb2xvZ3kgaW4gYmV0d2VlbiB0aGVzZSB0d28gc2VydmVycz8gV2hhdCB0aGVv cmV0aWNhbCBsaW5rIHJhdGVzIHdvdWxkIGJlIGF0dGFpbmFibGU/PHU+PC91Pjx1PjwvdT48L3A+ DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7wqDCoDx1PjwvdT48dT48 L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyBIZXJlIGlz IHRoZSB0b3BvbG9neTx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCI+Jmd0OyA8dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiPiZndDsgSXBlcmYgZW5kIHBvaW50cyBhcmUgb24gMiBkaWZmZXJlbnQg aHlwZXJ2aXNvcnMuIDx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCI+Jmd0OyA8dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiPiZndDvCoMKg4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU wqDCoMKgwqDCoMKgwqDCoOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKA lOKAlOKAlMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAg4oCU 4oCU4oCU4oCU4oCU4oCUwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDigJTigJTigJTi gJTigJTigJQt4oCUwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgPHU+PC91Pjx1PjwvdT48L3A+DQo8 L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IHwgTGludXggVk0xIHzCoMKg wqDCoMKgwqB8wqDCoEJTRCAxMyBWTSAxwqDCoHzCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgIHzCoMKgTGludXggVk0ywqDCoHzCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoHzCoMKgQlNEIDEzIFZNIDLCoMKgfDx1PjwvdT48dT48L3U+PC9wPg0KPC9k aXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyB8X19fX19fX19fX198wqDCoMKg wqDCoMKgfF8gX19fXyBfX19fIF9fXyB8wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqB8X19fX19fX19fX18gfMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgfF8gX19fXyBfX19fIF9fXyB8PHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IHzCoMKgwqDCoMKgwqDCoMKgwqDCoHzCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgfMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHzCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCB8PHU+PC91Pjx1PjwvdT48 L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7wqDCoMKgwqDCoMKg wqDCoMKgwqAgfMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqB8wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgfMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgIHw8dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiPiZndDsg4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCUwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDigJTigJTigJTi gJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJQ8dT48L3U+PHU+PC91PjwvcD4NCjwv ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZndDsgfMKgwqDCoMKgwqDCoMKgwqDC oMKgIEVTWCBIeXBlcnZpc29yIDHCoMKgwqDCoMKgwqDCoMKgwqDCoHzCoMKgwqDCoMKgwqDCoMKg wqDCoCAxMEcgbGluayBjb25uZWN0ZWQgdmlhIEwyIFN3aXRjaMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgfMKgwqDCoMKgwqDCoMKgwqDCoMKgIEVTWCBIeXBlcnZp c29ywqDCoDLCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqB8PHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rp dj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IHzCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoCB84oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCUwqDCoCB8wqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgfDx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyB84oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCUIHzCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCB84oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCUfDx1Pjwv dT48dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyA8 dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZn dDsgPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs Ij4mZ3Q7IE5pYyBpcyBvZiAxMEcgY2FwYWNpdHkgb24gYm90aCBFU1ggc2VydmVyIGFuZCBpdCBo YXMgYmVsb3cgY29uZmlnLjx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xh c3M9Ik1zb05vcm1hbCI+PHU+PC91PsKgPHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj5Tbywgd2hlbiBib3RoIFZNcyBydW4gb24gdGhlIHNhbWUgSHlwZXJ2 aXNvciwgbWF5YmUgd2l0aCBhbm90aGVyIFZNIHRvIHNpbXVsYXRlIHRoZSAxMDBtcyBkZWxheSwg Y2FuIHlvdSBhdHRhaW4gYSBsb3NzbGVzcyBiYXNlbGluZSBzY2VuYXJpbz88dT48L3U+PHU+PC91 PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1Pjwv dT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48dT48L3U+wqA8dT48 L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyBCRFAgZm9y IDE2TUIgU29ja2V0IGJ1ZmZlcjogMTYgTUIgKiAoMTAwMCBtcyAqIDEwMG1zIGxhdGVuY3kpICog OCBiaXRzLyAxMDI0ID0gMS4yNSBHYnBzPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IDx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyBTbyB0aGVvcmV0aWNhbGx5IHdlIHNob3Vs ZCBzZWUgY2xvc2UgdG8gMS4yNUdicHMgb2YgQml0cmF0ZSBhbmQgd2Ugc2VlIExpbnV4IHJlYWNo aW5nIGNsb3NlIHRvIHRoaXMgbnVtYmVyLjx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2 Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PsKgPHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRp dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPlVuZGVyIG5vIGxvc3MsIHllcy48dT48L3U+PHU+PC91 PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1Pjwv dT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48dT48L3U+wqA8dT48 L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyBCdXQgQlNE IGlzIG5vdCBhYmxlIHRvIGRvIHRoYXQuPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IDx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyA8dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2 Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZndDsgUS4gRGlkIHlvdSBydW4gaXBlcmYz PyBEaWQgdGhlIHRyYW5zbWl0dGluZyBlbmRwb2ludCByZXBvcnQgYW55IHJldHJhbnNtaXNzaW9u cyBiZXR3ZWVuIExpbnV4IG9yIEZCU0QgaG9zdHM/PHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4N CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IDx1PjwvdT48dT48L3U+PC9wPg0KPC9k aXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jmd0OyBZZXMsIHdlIHVzZWQgaXBlcjMu IEkgc2VlIExpbnV4IGRvaW5nIGxlc3MgbnVtYmVyIHJldHJhbnNtaXNzaW9ucyBjb21wYXJlZCB0 byBCU0QuDQo8dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29O b3JtYWwiPiZndDsgT24gQlNELCB0aGUgYmVzdCBwZXJmb3JtYW5jZSB3YXMgYXJvdW5kIDYwMCBN YnBzIGJpdHJhdGUgYW5kIHRoZSBudW1iZXIgb2YgcmV0cmFuc21pc3Npb25zIGZvciB0aGlzIG51 bWJlciBzZWVuIGlzIGFyb3VuZCAzMks8dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZndDsgT24gTGludXgsIHRoZSBiZXN0IHBlcmZvcm1hbmNl IHdhcyBhcm91bmQgMS4xNSBHYnBzIGJpdHJhdGUgYW5kIHRoZSBudW1iZXIgb2YgcmV0cmFuc21p c3Npb25zIGZvciB0aGlzIG51bWJlciBzZWVuIGlzIG9ubHkgMksuDQo8dT48L3U+PHU+PC91Pjwv cD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZndDsgU28gYXMgeW91IHBv aW50ZWQgdGhlIG51bWJlciBvZiByZXRyYW5zbWlzc2lvbnMgaW4gQlNEIGNvdWxkIGJlIHRoZSBy ZWFsIGlzc3VlIGhlcmUuPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIj48dT48L3U+wqA8dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xh c3M9Ik1zb05vcm1hbCI+VGhlcmUgYXJlIG90aGVyIGNjIG1vZHVsZXMgYXZhaWxhYmxlOyBidXQg SSBiZWxpZXZlIG9uZSBtYWpvciBkZXZpYXRpb24gaXMgdGhhdCBMaW51eCBjYW4gcGVyZm9ybSBt ZWNoYW5pc21zIGxpa2UgaHlzdGFydDsgQUNLaW5nIGV2ZXJ5IHBhY2tldCB3aGVuIHRoZSBjbGll bnQgZGV0ZWN0cyBzbG93IHN0YXJ0OyBwZXJmb3JtIHBhY2luZyB0byBhY2hpZXZlIG1vcmUgdW5p Zm9ybSBwYWNrZXQgdHJhbnNtaXNzaW9ucy48dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRp dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1PjwvdT48L3A+DQo8L2Rpdj4NCjxk aXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JIHRoaW5rIHRoZSBuZXh0IHN0ZXAgd291bGQgYmUg dG8gZmluZCBvdXQsIGF0IHdoaWNoIHF1ZXVlIHRob3NlIHBhY2tldCBkaXNjYXJkcyBhcmUgY29t aW5nIGZyb20gKGV4dGVybmFsIHN3aXRjaD8gZGVsYXkgZ2VuZXJhdG9yPyBWc3dpdGNoPyBFdGgg c3RhY2sgaW5zaWRlIHRoZSBWTT8pPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48dT48L3U+wqA8dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+T3IgYWx0ZXJuYXRpdmVseSwgcHJvdmlkZSB5b3VyIEVTWCBo eXBlcnZpc29ycyB3aXRoIHZhc3RseSBtb3JlIGxpbmsgc3BlZWQsIHRvIHJ1bGUgb3V0IGFueSBM MiBpbmR1Y2VkIHBhY2tldCBkcm9wcyAtIHByb3ZpZGVkIHlvdXIgZGVsYXkgZ2VuZXJhdG9yIGlz IG5vdCB0aGUgc291cmNlIHdoZW4gbW9tZW50YXJpbHkgb3ZlcmxvYWRlZC48dT48L3U+PHU+PC91 PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1Pjwv dT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4mZ3Q7IElzIHRoZXJl IGEgd2F5IHRvIHJlZHVjZSB0aGlzIHBhY2tldCBsb3NzIGJ5IGZpbmUgdHVuaW5nIHNvbWUgcGFy YW1ldGVycyB3LnIudCByaW5nIGJ1ZmZlciBvciBhbnkgb3RoZXIgYXJlYXM/DQo8dT48L3U+PHU+ PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1 PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5GaW5kaW5nIHdo ZXJlIHRoZXNlIGFyaXNlIChsb29raW5nIGF0IHF1ZXVlIGFuZCBwb3J0IGNvdW50ZXJzKSB3b3Vs ZCBiZSB0aGUgbmV4dCBzdGVwLiBCdXQgdGhpcyBpcyBub3QgcmVhbGx5IG15IHNwZWNpZmljIGFy ZWEgb2YgZXhwZXJ0aXNlIGJleW9uZCB0aGUgaGlnaCBsZXZlbCwgdmVuZG9yIGluZGVwZW5kZW50 IG9ic2VydmF0aW9ucy48dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIj5Td2l0Y2hpbmcgdG8gb3RoZXIgY2MgbW9kdWxlcyBtYXkgZ2l2ZSBzb21l IG1vcmUgaW5zaWdodHMuIEJ1dCBhZ2FpbiwgSSBzdXNwZWN0IHRoYXQgbW9tZW50YXJ5IChtaWNy b3NlY29uZCkgYnVyc3RpbmVzcyBvZiBCU0QgbWF5IGJlIGNhdXNpbmcgdGhpcyBzaWduaWZpY2Fu dGx5IGhpZ2hlciBsb3NzIHJhdGUuPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48dT48L3U+wqA8dT48L3U+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+VENQIFJBQ0sgd291bGQgYmUgYW5vdGhlciBvcHRpb24uIFRo YXQgc3RhY2sgaGFzIHBhY2luZywgbW9yZSBmaW5lLWdyYWluZWQgdGltaW5nLCB0aGUgUkFDSyBs b3NzIHJlY292ZXJ5IG1lY2hhbmlzbXMgZXRjLiBNYXliZSB0aGF0IGhlbHBzIHJlZHVjZSB0aGUg b2JzZXJ2ZWQgcGFja2V0IGRyb3BzIGJ5IGlwZXJmLCBhbmQgY29uc2VxdWVudGx5LCB5aWVsZCBh IGhpZ2hlciBvdmVyYWxsIHRocm91aGdwdXQuPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj4NCjxk aXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48dT48L3U+wqA8dT48L3U+PC9wPg0KPC9kaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PsKgPHU+PC91PjwvcD4NCjwvZGl2Pg0K PGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT7CoDx1PjwvdT48L3A+DQo8L2Rpdj4N CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48dT48L3U+wqA8dT48L3U+PC9wPg0KPC9kaXY+ DQo8L2Rpdj4NCjwvZGl2Pg0KDQo8L2Rpdj48L2Jsb2NrcXVvdGU+PC9kaXY+DQo= --00000000000057840405ff5b5cdc-- From nobody Mon Jul 3 08:38:10 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QvfT31wpZz4m6kl for ; Mon, 3 Jul 2023 08:38:15 +0000 (UTC) (envelope-from rscheff@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QvfT310Hrz49ss; Mon, 3 Jul 2023 08:38:15 +0000 (UTC) (envelope-from rscheff@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688373495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=/5efIOLNKK7rKe+hDMp1XRQuetAknw84km8UVLem3F0=; b=BVvdZnGYc4zI75+kL4gtlRCckvT6Ndi/f/ZpNN6jx48sFlfnUr9+KLIc8oIsZxM8ZkFgsg MnMDKxJOgRRwagYRfdvogKDWPtj/pehbnF5V0PDM5SJasIOzPPipNNSUSxOSzAKruk328N xb9znH5Lpoq8HOZGWZ5ySp+054HD6A1PkfLG8ZSA7S9gORQJuSeDnpJfGnpCpHS2df/0Er y/UBTl5l+tmqac6M2wR4VXXp6SoKeYeyOK3Ac6/SjX4OVKBSV2lh1B29ZtH+IWhLaWU43h AXTvYCgbekOvOv2yh6y56ClaOFxEAJvpCuazesK/kq7YcUZ682Vq4bwaGY3l8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688373495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=/5efIOLNKK7rKe+hDMp1XRQuetAknw84km8UVLem3F0=; b=NHmhd/ksLvqfvPlwx050EdnwPjJGf6sHhk5CORS+S1O9CMec1eG7z65sX+95FAM6ez/G1H FIqvvc3lromvLvkbNPEVswImWsAHFbWmKRVT27oRrZc1ICjLyfdX2TPLQpta3iYkHvmP9V B4Lnp5DN5MYNyt1QgxFkiqhiKrIoSbstMS5nDXJx7Wtc3dLVV78myKtx+En/Vn4FQb13vB CDF3SVZyiDEGRaP3AYHL1UTrSfAseMNaxjtB3lhixiGYGg/8wroKQyi1hhsHtKnazt9Kck +mbnwHruUO4IzN+SINJEZe/8t60NqqFsuLtGADUZrnlG/2grgyf48k1OJwkGSQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688373495; a=rsa-sha256; cv=none; b=R7q6hSzEvbQWuGbO5kifBiqbwTlK4+tEfn+q6PhaOmtzVXctvpfXfYMJul76wvebN23DXb W/sQAKnAE96BZYNJBMot+21BmFFhNyIw76EqFbqyNYJUhGBnZxJCj/6A53/kOej1BYSKpu SezFSzZkST/Puz7tAbU0zc1hS7f7ZKCxY3cvKnFy/S8EP7/8lZWb/Dm3W/4CI0/S3KLPOH LYGK+YyupDuY3rwLR+5LQXZgkAuolvDaCDIByxpI1rpbgLcpjPjk4ANdEb7i5USqctHu41 dnCY6Jq7il9CwYCfes/4s7OWK/KZW099Wi1Q3o5LtQJQaBPGjsrKb86sIpozkQ== Received: from [192.168.233.109] (unknown [185.236.167.136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: rscheff/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QvfT24pCFzxsh; Mon, 3 Jul 2023 08:38:14 +0000 (UTC) (envelope-from rscheff@freebsd.org) Message-ID: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org> Date: Mon, 3 Jul 2023 10:38:10 +0200 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: Murali Krishnamurthy , FreeBSD Transport From: "Scheffenegger, Richard" Autocrypt: addr=rscheff@freebsd.org; keydata= xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iziPF0FzrN K1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9yZz7CmgQTFgoAQhYh BDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGABQsJCAcCAyICAQYVCgkICwIE FgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/nvht8kExJ31M+3qpjOqdVypMp+/Ojvh5Z lsk96QEA5HCBkteJcrohwRA7llZvLH3m25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdV AQUBAQdA1Dim8ZWpXRS5i9hb3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2 S7eZrINEWrPNtHEXvliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bA EPqOH+JCIND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8= Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------1IGOA5Q7xo00VAW9CFOGpMCx" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------1IGOA5Q7xo00VAW9CFOGpMCx Content-Type: multipart/mixed; boundary="------------RpDGyvGRkOgbW5QSEGuyImgb"; protected-headers="v1" From: "Scheffenegger, Richard" To: Murali Krishnamurthy , FreeBSD Transport Message-ID: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org> Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux --------------RpDGyvGRkOgbW5QSEGuyImgb Content-Type: multipart/mixed; boundary="------------8GVHGHs1u66fLi3ndKzy4zoZ" --------------8GVHGHs1u66fLi3ndKzy4zoZ Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 SGkgTXVyYWxpLA0KDQo+IEFwcHJlY2lhdGUgdGhlIHVzZWZ1bCBpbnB1dHMgeW91IGhhdmUg c2hhcmVkIHNvIGZhci4gV2lsbCB0cnkgdG8gZmlndXJlIG91dCByZWdhcmRpbmcgcGFja2V0 IGRyb3BzLg0KPiANCj4gUmVnYXJkaW5nIEh5U3RhcnQsIEkgc2VlIGV2ZW4gQlNEIGNvZGUg YmFzZSBoYXMgc3VwcG9ydCBmb3IgdGhpcy4gTWF5IEkga25vdyBieSB3aGVuIGNhbiB3ZSBz ZWUgdGhhdCBpbiBhbiByZWxlYXNlLCBpZiBub3QgYWxyZWFkeSBhdmFpbGFibGUgPw0KDQpZ b3UgInNpbXBseSIgc3dpdGNoIHRvIHRoZSBSQUNLIHN0YWNrLiBKdXN0IGFzIEkgbWVudGlv bmVkIGVhcmxpZXIuIE5vdGUgdGhhdCB0aGlzIGlzIGEgY29tcGxldGVseSByZWZhY3RvcmVk IHN0YWNrIGJ5IGl0c2VsZiAtIHNvIHBsZWFzZSB2YWxpZGF0ZSB0aGF0IGFsbCB5b3VyIHVz ZXJzcGFjZSBhcHBsaWNhdGlvbnMgKG9yIGtlcm5lbCBwYXRjaGVzKSBzdGlsbCB3b3JrIHdo ZW4geW91IGFjdGl2YXRlIGl0Lg0KDQpodHRwczovL21hbi5mcmVlYnNkLm9yZy9jZ2kvbWFu LmNnaT9xdWVyeT10Y3BfcmFjayZzZWt0aW9uPTQmbWFucGF0aD1GcmVlQlNEKzEzLjItUkVM RUFTRSthbmQrUG9ydHMNCg0KPiANCj4gUmVnYXJkaW5nIHRoaXMgcG9pbnQgOiDigJxTd2l0 Y2hpbmcgdG8gb3RoZXIgY2MgbW9kdWxlcyBtYXkgZ2l2ZSBzb21lIG1vcmUgaW5zaWdodHMu IEJ1dCBhZ2FpbiwgSSBzdXNwZWN0IHRoYXQgbW9tZW50YXJ5IChtaWNyb3NlY29uZCkgYnVy c3RpbmVzcyBvZiBCU0QgbWF5IGJlIGNhdXNpbmcgdGhpcyBzaWduaWZpY2FudGx5IGhpZ2hl ciBsb3NzIHJhdGUu4oCdDQo+IElzIHRoZXJlIHNvbWUgaW5mbyBzb21ld2hlcmUgd2hlcmUg SSBjYW4gdW5kZXJzdGFuZCBtb3JlIG9uIHRoaXMgaW4gZGV0YWlsPw0KDQpXaGF0IGRldGFp bD8gSG93IHRvIHRvZ2dsZSB0byBhIGRpZmZlcmVudCBjb25nZXN0aW9uIGNvbnRyb2w/DQoN CkhvdyBoYXZpbmcgaGlnaCB0aW1lciBncmFudWxhcml0eSBwYWNpbmcgKGFzIHdoYXQgUkFD SyBkb2VzKSBjYW4gcHJldmVudCBtaWNyb2J1cnN0cz8NCg0KVGhhdCBUQ1AgY2FuLCBhdCBo aWdoIHRyYW5zbWlzc2lvbiByYXRlcywgb3ZlcnJ1biB0aGUgTklDIGRyaXZlciBtZW1vcnkg YnVmZmVycz8NCg0KV2hhdCBkaWQgeW91ciBleHBlcmltZW50cyB3aXRoIHRoZSBzdWdnZXN0 aW9ucyBieSBAY2Mgc2hvdyB3aXRoIHRoZSBiYXNlIHN0YWNrPw0KDQpBbmQgYWdhaW4sIHRo ZSBSQUNLIHN0YWNrIGRvZXMgSHlTdGFydCBhbmQgaGlnaC1ncmFudWxhcml0eSBwYWNpbmcs IGFkZHJlc3NpbmcgbWFueSBpc3N1ZXMgd2hpY2ggYXJlIGltcG9ydGFudCB0byBzbW9vdGgg c3RyZWFtaW5nIHdvcmtmbG93cyBhbmQgcHJldmVudGluZyBtb21lbnRhcnkgYnVyc3QgbG9h ZHMuIEhvd2V2ZXIsIHRoZSBSQUNLIHN0YWNrIGNlcnRhaW5seSBuZWVkcyB0byBiZSB2YWxp ZGF0ZWQgd2l0aCBwYXJ0aWN1bGFyIGFwcGxpY2F0aW9ucyBzdWNoIGFzIHRoZSBrZXJuZWwg TkZTIHNlcnZlci9jbGllbnQsIHdoaWNoIGFyZSBkZWVwbHkgZW1iZWRkZWQgYW5kIG1heSBy ZWx5IG9uIHNvbWUgdGhpbmdzIGluIHRoZSBiYXNlIHN0YWNrIHRvIHdvcmsgcHJvcGVybHku IEkgc3VzcGVjdCB0aGF0IHVudGlsIG5vdywgbm9vbmUgaGFzIGRvbmUgYSB0aG9yb3VnaCBp bnZlc3RpZ2F0aW9uIG9mIHRoZSBpbXBsaWNhdGlvbnMgb2YgUkFDSyB3aXRoIE5GUyByZWFs bHkuDQoNCg0KQmVzdCByZWdhcmRzLA0KICAgUmljaGFyZA0KDQo+IA0KPiBSZWdhcmRzDQo+ IE11cmFsaQ0K --------------8GVHGHs1u66fLi3ndKzy4zoZ Content-Type: application/pgp-keys; name="OpenPGP_0x17BE5899E0B1439B.asc" Content-Disposition: attachment; filename="OpenPGP_0x17BE5899E0B1439B.asc" Content-Description: OpenPGP public key Content-Transfer-Encoding: quoted-printable -----BEGIN PGP PUBLIC KEY BLOCK----- xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iz iPF0FzrNK1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9y Zz7CmgQTFgoAQhYhBDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGA BQsJCAcCAyICAQYVCgkICwIEFgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/n vht8kExJ31M+3qpjOqdVypMp+/Ojvh5Zlsk96QEA5HCBkteJcrohwRA7llZvLH3m 25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdVAQUBAQdA1Dim8ZWpXRS5i9hb 3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2S7eZrINEWrPNtHEX vliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bAEPqOH+JC IND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8=3D =3DBwxS -----END PGP PUBLIC KEY BLOCK----- --------------8GVHGHs1u66fLi3ndKzy4zoZ-- --------------RpDGyvGRkOgbW5QSEGuyImgb-- --------------1IGOA5Q7xo00VAW9CFOGpMCx Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQQ2S7eZrINEWrPNtHEXvliZ4LFDmwUCZKKI8gUDAAAAAAAKCRAXvliZ4LFDm2FC AP9czg1ydkzxaStoAtUx3JXAxKP8Q+NkIbHraDxPI4mIVwD9HKJvZNuXZ9odCLobfFxJjogHThhu bWftOYr2WBlQ4AI= =J8yX -----END PGP SIGNATURE----- --------------1IGOA5Q7xo00VAW9CFOGpMCx-- From nobody Mon Jul 3 15:43:36 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qvqvw43w7z4lWMX for ; Mon, 3 Jul 2023 15:43:40 +0000 (UTC) (envelope-from muralik1@vmware.com) Received: from CO1PR02CU001.outbound.protection.outlook.com (mail-westus2azon11011004.outbound.protection.outlook.com [52.101.47.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qvqvv6rvXz4FtY; Mon, 3 Jul 2023 15:43:39 +0000 (UTC) (envelope-from muralik1@vmware.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CZaTUjC597A5wIjaJCSI3xqdqwVmlx+aoklz5qcKcjApcNebdcG9asnK2lX7gdcnjg3H1XpTUI2ftZbkSaYVBSq4Wh/bUZ/6MEKUFDckE5wxZ9RSLD3hYeGzEJ8u5Kka8PZ8lZH1eJse/DklVjVWUysYDUtI/oU4lkZe4vQmhs5bMhFpAwoMo7ZvJK1djnZHkzrsmR1f6v2jO3ApQnYIrf2HZ8r6i6Eweq1QD4IfgnMLGCrcW4Ony5SFVV65PisnqPE8aXTx13AW0Jco3nogFTwp0HboCbhRX/Bhmbjj7fEWV1o8ojuOjgeeJnLIryAKF7s9zQaNeOCdoDofAyAHlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DdlXdZzG2sKD46p3FriJC20dcFj0DTniHQXGy1tbKx8=; b=j2oVyVPXBffdihE510izv6Bjzg5Th4206KVcx9M8QVZym6x9Zykk/Xfo9MFyAjNncwQRq+Ip8IkrsrBHGu0EJepuUMmwkblFkhDU9xaJ4kCcKBleiNPGn5rUSosEXFNyrRPueRTBEu3vULp81KwFzcYwQdEeD1jNYzFB9B5jyGe6RCRR1/yfjasJBms/xw27nQYvzkBRFCO6GLHZljB4Q0LMs4pPkknmUbtvnRWbcfFc4tDqy37zNWOAdvapb/RBLgL2vDGGs6Bkh9raYrDwuiXvR6gVlWenaN3v1LRMnbHnn6Ru/rCj7J7ouKdugYYfzknZsTm3vP/BDnlHjlgO7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DdlXdZzG2sKD46p3FriJC20dcFj0DTniHQXGy1tbKx8=; b=2AY7dgVh3Ap6PXSN//+pHKfY4/bAQZo/bRNg/kZf5zTJjc/J2cyAZ1Pkv5+v8TMr23tK4ETKuueyOrZTTP7OdX+hAhrCl5k0zKDB6Nc4V4btHNKXNHxCBTgLQ/gWPH+P1wjkCiTuICfG7t70EdmWAPt65RZfUAFaoS09H5mIFwg= Received: from PH0PR05MB10064.namprd05.prod.outlook.com (2603:10b6:510:29d::8) by DS7PR05MB9969.namprd05.prod.outlook.com (2603:10b6:8:db::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Mon, 3 Jul 2023 15:43:36 +0000 Received: from PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593]) by PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593%4]) with mapi id 15.20.6544.019; Mon, 3 Jul 2023 15:43:36 +0000 From: Murali Krishnamurthy To: "Scheffenegger, Richard" , FreeBSD Transport Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Thread-Topic: FreeBSD TCP (with iperf3) comparison with Linux Thread-Index: AQHZrYmyiK08kPjzWEy1GrMDiqKRKq+oI//y Date: Mon, 3 Jul 2023 15:43:36 +0000 Message-ID: References: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org> In-Reply-To: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org> Accept-Language: en-IN, en-US Content-Language: en-IN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR05MB10064:EE_|DS7PR05MB9969:EE_ x-ms-office365-filtering-correlation-id: 8ffb8688-3605-4a09-d813-08db7bdc43cd x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: UZdczEgPZIU0yrHYyur2GvVWWeSUpZiy3tBvc0NIVpj3GBsuUANTDdWIUAztGcnCGYSrJe6qU7syxk70mSOh6aMJN8oPL4l8BFsRKNmdq5mZYET+IUguNvqoeTgQtpz/8iP9bVdH1kwJt+O+zLFYRjWVuHSgk6HbYH4pckOheB+/A25q1KDn3EVDUt2zr/2rXEge7fEBk7afdphyEQZ5txjRWHcdkbMbrXp0FTu4kpSSUq3Z76u8YFVN0zcPCU5CC69BaaejY2B1r+/S78BaphwN/xpR/s1o8KrpeOekRRFxVNfKX0WfEqztCQwLVhbVKUzfeiKh+tul01gt4E2c++0tzSf9Zd9mlteDwRtDhxxL+4CbGBr8tNzbqDqE8uVn2xxGZJcnrJozgeuZ1v6qaVP8Z6prLzpgDpR7xc7PBqelihEYHc+zEiLMbRv1LoEk44yXeTTNLGWiErtDCqoC8m1d9HllyF52ORi9RO0/Acmt8ttGFETs6z7xxb6t49bRK6pym71/wp3EcdYs8Ie/xfKOhtW4Cfi1wE9YeeZufb8IGBOkZHU48BXfTwytABlo+q/G/Z1kpvyRqWWNnCe8GyqTlP9oSJzjL6Fdz8KmNl5uMpMe/sqiIGTLpeJVfgrtJul+YXEGFQo/IbYSIynD23RHxRiydyUzGiSPA7POScnk6Y1GPEk016RO4IbfV3aYtjpQptb2IpsuGhOFJfgVkA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR05MB10064.namprd05.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(376002)(39860400002)(346002)(396003)(136003)(366004)(451199021)(41300700001)(86362001)(38100700002)(7696005)(71200400001)(166002)(83380400001)(122000001)(26005)(186003)(6506007)(53546011)(9686003)(966005)(55016003)(38070700005)(110136005)(478600001)(2906002)(316002)(450100002)(66476007)(91956017)(64756008)(66446008)(76116006)(66556008)(66946007)(8936002)(33656002)(8676002)(52536014)(5660300002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?lTa/hugUAxofk+5k3U7tPTZaoVp+jqE6LWxF9Z7nVCoyhY3lC6qhv4jp?= =?Windows-1252?Q?VxahvNyO0O9g9uHcLAKjA0ICwlD16Sk1kvnSf7GSquyvuCfZIbBrgbNS?= =?Windows-1252?Q?MIqeun3dCs0hxs9vWqRlDzZq20TL2RCq39A6hR54+H2MIfkPCuu3f0Yu?= =?Windows-1252?Q?9HNFV8RwdTwCEHS7OCFWXmJUHLdosDlMb3wws8PgvbEw3zrHo/wd8JY3?= =?Windows-1252?Q?zBfXA7Kh/jx3xSWyJDEUQyMXbNGIfpZjBIGh5RiNBh+TqH8+Wn8vIFxZ?= =?Windows-1252?Q?NRWYfi6SYPBj4w5CnEku0v0sZnb68P2S5WZTO/mc+fSHJ+Rxep2m8VB+?= =?Windows-1252?Q?3PdqmZfdD90e6QIrFW7jGVsKO1B9yC21X+7YhMBw1iLi6a+P9S7WZPiW?= =?Windows-1252?Q?F9h+/H16MMMWeWTmV6bTNRGqI/A1H3UNPyzpMseMDBZnHJiPeGDh8IpK?= =?Windows-1252?Q?LaZ6LlTtfUWg//WekgMj0Tz2/gVMuHUh0Rm6PrsdlpYVtVTyiTf1ZXy+?= =?Windows-1252?Q?xgSBI26eIBxD6r1BuamTV3VbHmPNAyQ2jRSisIJlTWou5MQ8nD/EoeAr?= =?Windows-1252?Q?mrcwP+PTc35rbYhVRB55WfT6JpdzHF27GmKsppYPLx6NpM3+050ZRFnk?= =?Windows-1252?Q?1TMh6xmBvnd+jI9aBKabjYJJ7RhhqL3MnU+zKyOVRDMwrBCy9SVCAFk4?= =?Windows-1252?Q?F2ovlmEevDMYuadkpXl674EOrjM7vEvb+eXTC4LV3jRzdmI8SSbdWZOQ?= =?Windows-1252?Q?stn41ht1Br5BCO8cs0T63UhxWyqzrTellyjtUAUv8jsnH31aEQ7+ZftM?= =?Windows-1252?Q?oNr4R0WMxpA238eYorYuvE5Pc/u+Mg3yUV5GtFmvKQ1XaJsllNa2V6At?= =?Windows-1252?Q?ZsiovdO7ZQr89NLlxOMD3KSK7Lsb7AWf9hqFbJhmEa4zc8sivVDURWDz?= =?Windows-1252?Q?Q7SBKlm4kJGrAtqQQkOlsydtjvnZGfZ+VxIDVOzEkGu3c6PIpdFix+Fd?= =?Windows-1252?Q?i3KUBJWTCwR+9X6QT852zQKbe6HaXklwiUQK/1SJO3EPyYO0axRL1g0G?= =?Windows-1252?Q?vS5ykLfm2dEk3lhzdQ3bCTCeFkGk5CuEme5YVqSggHS/of/UnoEIOo5V?= =?Windows-1252?Q?sa/WRHE97iIopEKMeVyafPH0XRu6KTsMKjbAa7wumzeKIFFUrOhM3FYn?= =?Windows-1252?Q?7OBbs3r6orUK2s6fR7uI1yT7dcEP/UxybZgBQ0e2XQaZZ/QpOCLpGdBN?= =?Windows-1252?Q?3m1+1j/Xx78zJqdCKJlc4s3LZYxA9LARil7mlqqU01GGBIvG0j0IH1/j?= =?Windows-1252?Q?g3N6wVfMDPGMADgTeqFs1BeS5tAyRMrgTjWiLNKvaNMsx/nWnn+IQvEe?= =?Windows-1252?Q?dTclk7JTBxl/TlS8I1NE7nUGwhwNYuZobbVH5vVuZQxC+7CIUlP4smvv?= =?Windows-1252?Q?9uetypbf8dxWaVNZXL0ZLH7IeF7p+XGnjyquTnGBZNwPIleZWRFHjpi/?= =?Windows-1252?Q?nTGFrXHbpUlEtL2Ho95CNi1LzZaLGUolf794fTD8HEGZqnHOWtvtxImj?= =?Windows-1252?Q?LrG8Z3ve/a8ajTh930oxxgjljtJYK8itDRGEqN1qxDnMFZ2dRRau0FU7?= =?Windows-1252?Q?OZ9BJuUV6QZTk6KT30au6yTE9nxEd/v5quRGn47zHHElRUmg2I3ArV7K?= =?Windows-1252?Q?E6+G2FYs55jK7AWDFV2h7zJ6swWI8Q2R5V0dxAEnUMpllyYEyJ7JUQ?= =?Windows-1252?Q?=3D=3D?= Content-Type: multipart/alternative; boundary="_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_" List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR05MB10064.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8ffb8688-3605-4a09-d813-08db7bdc43cd X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jul 2023 15:43:36.3306 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7AUFED+1PnssT+k/QWV9HUge8W4xldplD9Kp5Wtvi6ULQDtamRtiSakVRdQdlMIbXYOLK7LyBUG7P9AQr5wTuA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR05MB9969 X-Rspamd-Queue-Id: 4Qvqvv6rvXz4FtY X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:52.96.0.0/12, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Richard, =93How having high timer granularity pacing (as what RACK does) can prevent= microbursts?=94 =93 That TCP can, at high transmission rates, overrun the NIC driver memory= buffers?=94 Yes. Any relevant doc/write-up on these would help for better understanding= . I know high transmission of packets could lead to packet drops at NIC buffe= rs. Linux somehow seem to be avoiding that problem. =93What did your experiments with the suggestions by @cc show with the base= stack?=94 Just posted a follow up query to Cheng on this. On enabling RACK: It seems we need to rebuild freebsd with compile time opt= ion. Is there a build (iso) available in repository with RACK enabled alrea= dy? Regards Murali On 03/07/23, 2:08 PM, "Scheffenegger, Richard" wrote: Hi Murali, > Appreciate the useful inputs you have shared so far. Will try to figure o= ut regarding packet drops. > > Regarding HyStart, I see even BSD code base has support for this. May I k= now by when can we see that in an release, if not already available ? You "simply" switch to the RACK stack. Just as I mentioned earlier. Note th= at this is a completely refactored stack by itself - so please validate tha= t all your userspace applications (or kernel patches) still work when you a= ctivate it. https://man.freebsd.org/cgi/man.cgi?query=3Dtcp_rack&sektion=3D4&manpath=3D= FreeBSD+13.2-RELEASE+and+Ports > > Regarding this point : =93Switching to other cc modules may give some mor= e insights. But again, I suspect that momentary (microsecond) burstiness of= BSD may be causing this significantly higher loss rate.=94 > Is there some info somewhere where I can understand more on this in detai= l? What detail? How to toggle to a different congestion control? How having high timer granularity pacing (as what RACK does) can prevent mi= crobursts? That TCP can, at high transmission rates, overrun the NIC driver memory buf= fers? What did your experiments with the suggestions by @cc show with the base st= ack? And again, the RACK stack does HyStart and high-granularity pacing, address= ing many issues which are important to smooth streaming workflows and preve= nting momentary burst loads. However, the RACK stack certainly needs to be = validated with particular applications such as the kernel NFS server/client= , which are deeply embedded and may rely on some things in the base stack t= o work properly. I suspect that until now, noone has done a thorough invest= igation of the implications of RACK with NFS really. Best regards, Richard > > Regards > Murali --_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Richard,<= o:p>

&nbs= p;

&nbs= p;

=93How having high timer granularity pacing (as what= RACK does) can prevent microbursts?=94

=93 That TCP can, at high transmission rates, overrun the NIC driver memory b= uffers?=94

 

Yes. Any relevant doc/write-up on these would help f= or better understanding.

 

I know high transmission of packets could lead to pa= cket drops at NIC buffers. Linux somehow seem to be avoiding that problem.

 

&nbs= p;

=93What did your experiments with the suggestions by @cc show with the base s= tack?=94

Just post= ed a follow up query to Cheng on this.

&nbs= p;

On enabli= ng RACK: It seems we need to rebuild freebsd with compile time option. Is t= here a build (iso) available in repository with RACK enabled already?<= /o:p>

&nbs= p;

&nbs= p;

Regards

Murali

&nbs= p;

&nbs= p;

On 03/07/23, 2:08 PM,= "Scheffenegger, Richard" <rscheff@freebsd.org> wrote:=

Hi Murali,

 

> Appreciate the useful inputs you have shared so= far. Will try to figure out regarding packet drops.

>

> Regarding HyStart, I see even BSD code base has= support for this. May I know by when can we see that in an release, if not= already available ?

 

You "simply" switch to the RACK stack. Jus= t as I mentioned earlier. Note that this is a completely refactored stack b= y itself - so please validate that all your userspace applications (or kern= el patches) still work when you activate it.

 

 

>

> Regarding this point : =93Switching to other cc= modules may give some more insights. But again, I suspect that momentary (= microsecond) burstiness of BSD may be causing this significantly higher los= s rate.=94

> Is there some info somewhere where I can unders= tand more on this in detail?

 

What detail? How to toggle to a different congestion= control?

 

How having high timer granularity pacing (as what RA= CK does) can prevent microbursts?

 

That TCP can, at high transmission rates, overrun th= e NIC driver memory buffers?

 

What did your experiments with the suggestions by @c= c show with the base stack?

 

And again, the RACK stack does HyStart and high-gran= ularity pacing, addressing many issues which are important to smooth stream= ing workflows and preventing momentary burst loads. However, the RACK stack= certainly needs to be validated with particular applications such as the kernel NFS server/client, which are de= eply embedded and may rely on some things in the base stack to work properl= y. I suspect that until now, noone has done a thorough investigation of the= implications of RACK with NFS really.

 

 

Best regards,

   Richard

 

>

> Regards

> Murali

 

--_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_-- From nobody Mon Jul 3 20:24:00 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qvy7g3Wqmz4ln27 for ; Mon, 3 Jul 2023 20:24:15 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qvy7f2VYnz4NvX; Mon, 3 Jul 2023 20:24:14 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of ccfreebsd@gmail.com designates 209.85.210.45 as permitted sender) smtp.mailfrom=ccfreebsd@gmail.com; dmarc=none Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-6b886456f66so2892834a34.0; Mon, 03 Jul 2023 13:24:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688415853; x=1691007853; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kL2SvjOXZLq1YS2WATLstRGlnwz+SOIQSfWPaz1q6HM=; b=azbanqN4T+aUa6T5DB4Pl+HBmjKvs7f8V7j+Ep2mPPtaHFsHeJwTXuLz3Wm/iImlOy ySaYnMCG05aDWr6QpyOlXrgWmCaFL5J6ePbEguJms0o/fEYJdMMjyibXwNqE9mJc3EAi sxTEZqGX+6vFhztHGdukKW1N/DxGeWKnBnU5NO7CfqNBVzy01Fg7+QQTnVtka3Q1H9cQ 5vYOBz0rw4HKmUE15aYekuG1I3zI359yeDiSVDfNC4IbMm7AY56vwsAkM4hHeFfy2Bsm pduVwQZGaAWxhD15t4cFgjvsgPzyEsVpT8jxvxuoG201VvYSYYlTNuh5yzcghuuqjqOx beaw== X-Gm-Message-State: AC+VfDxvnArD1PSE83uQ2U6qRjdnfZZ0OVeADDhgsuplDRbtEXf+jmjK whWXVpM1l3amQ5bkFbtKCuKKLT1qr74= X-Google-Smtp-Source: ACHHUZ4GaEmea7EiI2GeuU72+8PG7/lp7yQHp1OeoYEi4kAxzDy4E4H4Lm9Jpjuji6cue3KYZOP1xQ== X-Received: by 2002:a05:6830:149a:b0:6b7:1fd6:50b3 with SMTP id s26-20020a056830149a00b006b71fd650b3mr9123807otq.31.1688415852611; Mon, 03 Jul 2023 13:24:12 -0700 (PDT) Received: from mail-oa1-f52.google.com (mail-oa1-f52.google.com. [209.85.160.52]) by smtp.gmail.com with ESMTPSA id k9-20020a0568301be900b006b87c41a57esm5498400otb.11.2023.07.03.13.24.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Jul 2023 13:24:12 -0700 (PDT) Received: by mail-oa1-f52.google.com with SMTP id 586e51a60fabf-19a427d7b57so3257872fac.2; Mon, 03 Jul 2023 13:24:11 -0700 (PDT) X-Received: by 2002:a05:6870:5d10:b0:1b0:e98:163b with SMTP id fv16-20020a0568705d1000b001b00e98163bmr7460373oab.21.1688415851747; Mon, 03 Jul 2023 13:24:11 -0700 (PDT) List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 References: <53aff274-b1a8-0730-6971-2755c7e7b688@freebsd.org> In-Reply-To: From: Cheng Cui Date: Mon, 3 Jul 2023 16:24:00 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux To: Murali Krishnamurthy Cc: "Scheffenegger, Richard" , FreeBSD Transport Content-Type: multipart/alternative; boundary="000000000000139feb05ff9af368" X-Spamd-Result: default: False [-0.11 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_LONG(-0.99)[-0.990]; NEURAL_HAM_SHORT(-0.99)[-0.985]; NEURAL_SPAM_MEDIUM(0.96)[0.962]; FORGED_SENDER(0.30)[cc@freebsd.org,ccfreebsd@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RWL_MAILSPIKE_GOOD(-0.10)[209.85.210.45:from]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-transport@freebsd.org]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[209.85.210.45:from,209.85.160.52:received]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[freebsd.org]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[cc@freebsd.org,ccfreebsd@gmail.com] X-Rspamd-Queue-Id: 4Qvy7f2VYnz4NvX X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N --000000000000139feb05ff9af368 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I see. Sorry about a straight description in my previous email. If you found the iperf3 report shows bad throughput and increasing numbers in the "Retr" field, also the "netstat -sp tcp" shows retransmitted packets without SACK recovery episodes (SACK is enabled by default). Then, you are likely hitting the problem I described, and the root cause is the TX queue drops. The tcpdump trace file won't show any packet retransmissions and the peer won't be aware of packet loss, as this is a local problem. cc@s1:~ % netstat -sp tcp | egrep "tcp:|retrans|SACK" tcp: 139 data packets (300416 bytes) retransmitted << 0 data packets unnecessarily retransmitted 3 retransmit timeouts 0 retransmitted 0 SACK recovery episodes << 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK retransmissions lost 0 SACK scoreboard overflow Local packet drops due to TX full can be found from this command, for example cc@s1:~ % netstat -i -I bce4 -nd Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop bce4 1500 00:10:18:56:94:d4 286184 0 0 148079 0 0 54 << bce4 - 10.1.1.0/24 10.1.1.2 286183 - - 582111 - - - cc@s1:= ~ % Hope the above stats can help you better root cause analysis. Also, increasing the TX queue size is a workaround and is specific to a particular NIC. But you get the idea. Best Regards, Cheng Cui On Mon, Jul 3, 2023 at 11:34=E2=80=AFAM Murali Krishnamurthy wrote: > Cheng, > > > > Thanks for your inputs. > > > > Sorry, I am not familiar with this area. > > > > Few queries, > > > > =E2=80=9CI believe the default values for bce tx/rx pages are 2. And I ha= ppened to > find > this problem before that when the tx queue was full, it would not enqueue > packets > and started return errors. > And this error was misunderstood by the TCP layer as retransmission.=E2= =80=9D > > > > Could you please elaborate what is misunderstood by TCP here? Loss of > packets should anyway lead to retransmissions. > > > > Could you point to some stats where I can see such drops due to queue > getting full? > > > > I have a vmx interface in my VM and I have attached the screenshot of > ifconfig command for that. > > Anything we can understand from that? > > Will your suggestion of increasing tx_pages=3D4 and rx_pages=3D4 work for= this > ? If so, I assume names would be hw.vmx.tx_pages=3D4 and hw.vmx.rx_pages = ? > > > > Regards > > Murali > > > > > > *From: *Cheng Cui > *Date: *Friday, 30 June 2023 at 10:02 PM > *To: *Murali Krishnamurthy > *Cc: *Scheffenegger, Richard , FreeBSD Transport < > freebsd-transport@freebsd.org> > *Subject: *Re: FreeBSD TCP (with iperf3) comparison with Linux > > *!! External Email* > > I used an emulation testbed from Emulab.net with Dummynet traffic shaper > adding 100ms RTT > between two nodes, the link capacity is 1Gbps and both nodes are using > freebsd13.2. > > cc@s1:~ % ping -c 3 r1 > PING r1-link1 (10.1.1.3): 56 data bytes > 64 bytes from 10.1.1.3: icmp_seq=3D0 ttl=3D64 time=3D100.091 ms > 64 bytes from 10.1.1.3: icmp_seq=3D1 ttl=3D64 time=3D99.995 ms > 64 bytes from 10.1.1.3: icmp_seq=3D2 ttl=3D64 time=3D99.979 ms > > --- r1-link1 ping statistics --- > 3 packets transmitted, 3 packets received, 0.0% packet loss > round-trip min/avg/max/stddev =3D 99.979/100.022/100.091/0.049 ms > > > cc@s1:~ % iperf3 -c r1 -t 10 -i 1 -C cubic > Connecting to host r1, port 5201 > [ 5] local 10.1.1.2 port 56089 connected to 10.1.1.3 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 4.19 MBytes 35.2 Mbits/sec 0 1.24 MBytes > > [ 5] 1.00-2.00 sec 56.5 MBytes 474 Mbits/sec 6 2.41 MBytes > > [ 5] 2.00-3.00 sec 58.6 MBytes 492 Mbits/sec 18 7.17 MBytes > > [ 5] 3.00-4.00 sec 65.6 MBytes 550 Mbits/sec 14 606 KBytes > > [ 5] 4.00-5.00 sec 60.8 MBytes 510 Mbits/sec 18 7.22 MBytes > > [ 5] 5.00-6.00 sec 62.1 MBytes 521 Mbits/sec 12 7.86 MBytes > > [ 5] 6.00-7.00 sec 60.9 MBytes 512 Mbits/sec 14 3.43 MBytes > > [ 5] 7.00-8.00 sec 62.8 MBytes 527 Mbits/sec 16 372 KBytes > > [ 5] 8.00-9.00 sec 59.3 MBytes 497 Mbits/sec 14 1.77 MBytes > > [ 5] 9.00-10.00 sec 57.0 MBytes 477 Mbits/sec 18 7.13 MBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 548 MBytes 459 Mbits/sec 130 > sender > [ 5] 0.00-10.10 sec 540 MBytes 449 Mbits/sec > receiver > > iperf Done. > > cc@s1:~ % ifconfig bce4 > bce4: flags=3D8843 metric 0 mtu 1= 500 > > options=3Dc01bb > ether 00:10:18:56:94:d4 > inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 > media: Ethernet 1000baseT > status: active > nd6 options=3D29 > > I believe the default values for bce tx/rx pages are 2. And I happened to > find > this problem before that when the tx queue was full, it would not enqueue > packets > and started return errors. > And this error was misunderstood by the TCP layer as retransmission. > > After adding hw.bce.tx_pages=3D4 and hw.bce.rx_pages=3D4 in /boot/loader.= conf > and reboot: > > cc@s1:~ % iperf3 -c r1 -t 10 -i 1 -C cubic > Connecting to host r1, port 5201 > [ 5] local 10.1.1.2 port 20478 connected to 10.1.1.3 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 4.15 MBytes 34.8 Mbits/sec 0 1.17 MBytes > > [ 5] 1.00-2.00 sec 83.1 MBytes 697 Mbits/sec 0 12.2 MBytes > > [ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 12.2 MBytes > > [ 5] 3.00-4.00 sec 113 MBytes 944 Mbits/sec 0 12.2 MBytes > > [ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 12.2 MBytes > > [ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 12.2 MBytes > > [ 5] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 0 12.2 MBytes > > [ 5] 7.00-8.00 sec 113 MBytes 944 Mbits/sec 0 12.2 MBytes > > [ 5] 8.00-9.00 sec 112 MBytes 938 Mbits/sec 0 12.2 MBytes > > [ 5] 9.00-10.00 sec 113 MBytes 947 Mbits/sec 0 12.2 MBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 985 MBytes 826 Mbits/sec 0 > sender > [ 5] 0.00-10.11 sec 982 MBytes 815 Mbits/sec > receiver > > iperf Done. > > > > Best Regards, > > Cheng Cui > > > > > > On Fri, Jun 30, 2023 at 12:26=E2=80=AFPM Murali Krishnamurthy > wrote: > > Richard, > > > > Appreciate the useful inputs you have shared so far. Will try to figure > out regarding packet drops. > > > > Regarding HyStart, I see even BSD code base has support for this. May I > know by when can we see that in an release, if not already available ? > > > > Regarding this point : *=E2=80=9CSwitching to other cc modules may give s= ome more > insights. But again, I suspect that momentary (microsecond) burstiness of > BSD may be causing this significantly higher loss rate.=E2=80=9D* > > Is there some info somewhere where I can understand more on this in detai= l? > > > > Regards > > Murali > > > > > > On 30/06/23, 9:35 PM, "owner-freebsd-transport@freebsd.org" < > owner-freebsd-transport@freebsd.org> wrote: > > > > Hi Murali, > > > > > Q. Since you mention two hypervisors - what is the phyiscal network > topology in between these two servers? What theoretical link rates would = be > attainable? > > > > > > Here is the topology > > > > > > Iperf end points are on 2 different hypervisors. > > > > > > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94 =E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94 > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 =E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94-=E2=80=94 > > > | Linux VM1 | | BSD 13 VM > 1 | > | Linux VM2 | | BSD 13 VM 2 | > > > |___________| |_ ____ ____ ___ > | = |___________ > | |_ ____ ____ ___ | > > > | | > | > | | > > > > | | > | | > > > > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 = =E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > > > | ESX Hypervisor 1 | 10G link connected vi= a > L2 Switch | ESX Hypervisor 2 | > > > | > |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > | | > > > |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > | > |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94| > > > > > > > > > Nic is of 10G capacity on both ESX server and it has below config. > > > > > > So, when both VMs run on the same Hypervisor, maybe with another VM to > simulate the 100ms delay, can you attain a lossless baseline scenario? > > > > > > > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ > 1024 =3D 1.25 Gbps > > > > > > So theoretically we should see close to 1.25Gbps of Bitrate and we see > Linux reaching close to this number. > > > > Under no loss, yes. > > > > > > > But BSD is not able to do that. > > > > > > > > > Q. Did you run iperf3? Did the transmitting endpoint report any > retransmissions between Linux or FBSD hosts? > > > > > > Yes, we used iper3. I see Linux doing less number retransmissions > compared to BSD. > > > On BSD, the best performance was around 600 Mbps bitrate and the number > of retransmissions for this number seen is around 32K > > > On Linux, the best performance was around 1.15 Gbps bitrate and the > number of retransmissions for this number seen is only 2K. > > > So as you pointed the number of retransmissions in BSD could be the rea= l > issue here. > > > > There are other cc modules available; but I believe one major deviation i= s > that Linux can perform mechanisms like hystart; ACKing every packet when > the client detects slow start; perform pacing to achieve more uniform > packet transmissions. > > > > I think the next step would be to find out, at which queue those packet > discards are coming from (external switch? delay generator? Vswitch? Eth > stack inside the VM?) > > > > Or alternatively, provide your ESX hypervisors with vastly more link > speed, to rule out any L2 induced packet drops - provided your delay > generator is not the source when momentarily overloaded. > > > > > Is there a way to reduce this packet loss by fine tuning some parameter= s > w.r.t ring buffer or any other areas? > > > > Finding where these arise (looking at queue and port counters) would be > the next step. But this is not really my specific area of expertise beyon= d > the high level, vendor independent observations. > > > > Switching to other cc modules may give some more insights. But again, I > suspect that momentary (microsecond) burstiness of BSD may be causing thi= s > significantly higher loss rate. > > > > TCP RACK would be another option. That stack has pacing, more fine-graine= d > timing, the RACK loss recovery mechanisms etc. Maybe that helps reduce th= e > observed packet drops by iperf, and consequently, yield a higher overall > throuhgput. > > > > > > > > > > > > *!! External Email:* This email originated from outside of the > organization. Do not click links or open attachments unless you recognize > the sender. > > > --000000000000139feb05ff9af368 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I see. Sorry about a straight descri= ption in my previous email.

If you found the = iperf3 report shows bad throughput and increasing numbers in the "Retr= " field, also the "netstat -sp tcp" shows retransmitted pack= ets without SACK recovery episodes (SACK is enabled by default). Then, you = are likely hitting the problem I described, and the root cause is the TX qu= eue drops. The tcpdump trace file won't show any packet retransmissions= and the peer won't be aware of packet loss, as this is a local problem= .

cc@s1:~ % netstat -sp tcp | egrep "tcp:= |retrans|SACK"
tcp:
139 data packets (300416 bytes) retransmitted =C2=A0 =C2=A0 =C2=A0 <= <
0 data packets unnecessarily retransmitted
3 retransmit t= imeouts
0 retransmitted
0 SACK recovery episodes =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0<<
0 segment rexmits in SACK recovery episodes=
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks= ) received
0 SACK options (SACK blocks) sent
0 SACK retransmissions l= ost
0 SACK scoreboard overflow

Local packet dro= ps due to TX full can be found from this command, for example
cc@s1:~ % netstat -i= -I bce4 -nd Name Mtu Network = Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop bce4 1500 <Link#5> 00:10:18:56:94:d4 286= 184 0 0 148079 0 0 54 << bce4 - 10.1.1.0/24 10.1.1.2 286183 - - 5= 82111 - - -=20 cc@s1:~ %

Hope the above stats can help you better root cause analys= is. Also, increasing the TX queue size is a workaround and is specific to a= particular NIC. But you get the idea.

Best Regards= ,
Cheng Cui


On Mon, Jul 3, 2023 at 11:= 34=E2=80=AFAM Murali Krishnamurthy <muralik1@vmware.com> wrote:

Cheng,

=C2=A0

Thanks for your inputs.

=C2=A0

Sorry, I am not familiar with this area.

=C2=A0

Few queries,

=C2=A0

=E2=80=9CI believe the default values for bce tx/rx = pages are 2. And I happened to find
this problem before that when the tx queue was full, it would not enqueue p= ackets
and started return errors.
And this error was misunderstood by the TCP layer as retransmission.=E2=80= =9D

=C2=A0

Could you please elaborate what is misunderstood by = TCP here? Loss of packets should anyway lead to retransmissions.<= /u>

=C2=A0

Could you point to some stats where I can see such d= rops due to queue getting full?

=C2=A0

I have a vmx interface in my VM and I have attached = the screenshot of ifconfig command for that.

Anything we can understand from that?<= /p>

Will your suggestion of increasing tx_pages=3D4 and = rx_pages=3D4 work for this ? If so, I assume names would be hw.vmx.tx_pages= =3D4 and hw.vmx.rx_pages ?

=C2=A0

Regards

Murali

=C2=A0

=C2=A0

From: Cheng Cui <cc@freebsd.org>
Date: Friday, 30 June 2023 at 10:02 PM
To: Murali Krishnamurthy <muralik1@vmware.com>
Cc: Scheffenegger, Richard <rscheff@freebsd.org>, FreeBSD Transport <freebsd-tran= sport@freebsd.org>
Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux<= u>

!! External Email

I used an emulation testbed from Emulab.net with Dum= mynet traffic shaper adding 100ms RTT
between two nodes, the link capacity is 1Gbps and both nodes are using free= bsd13.2.

cc@s1:~ % ping -c 3 r1
PING r1-link1 (10.1.1.3): 56 data bytes
64 bytes from 10.1.1.3: = icmp_seq=3D0 ttl=3D64 time=3D100.091 ms
64 bytes from 10.1.1.3: = icmp_seq=3D1 ttl=3D64 time=3D99.995 ms
64 bytes from 10.1.1.3: = icmp_seq=3D2 ttl=3D64 time=3D99.979 ms

--- r1-link1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev =3D 99.979/100.022/100.091/0.049 ms


cc@s1:~ % iperf3 -c r1 -t 10 -i 1 -C cubic
Connecting to host r1, port 5201
[ =C2=A05] local 10.1.1.2 port 56089 connected to 10.1.1.3 port 5201
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer =C2=A0 =C2=A0 Bi= trate =C2=A0 =C2=A0 =C2=A0 =C2=A0 Retr =C2=A0Cwnd
[ =C2=A05] =C2=A0 0.00-1.00 =C2=A0 sec =C2=A04.19 MBytes =C2=A035.2 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 1.24 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 1.00-2.00 =C2=A0 sec =C2=A056.5 MBytes =C2=A0 474 Mbits/s= ec =C2=A0 =C2=A06 =C2=A0 2.41 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 2.00-3.00 =C2=A0 sec =C2=A058.6 MBytes =C2=A0 492 Mbits/s= ec =C2=A0 18 =C2=A0 7.17 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 3.00-4.00 =C2=A0 sec =C2=A065.6 MBytes =C2=A0 550 Mbits/s= ec =C2=A0 14 =C2=A0 =C2=A0606 KBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 4.00-5.00 =C2=A0 sec =C2=A060.8 MBytes =C2=A0 510 Mbits/s= ec =C2=A0 18 =C2=A0 7.22 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 5.00-6.00 =C2=A0 sec =C2=A062.1 MBytes =C2=A0 521 Mbits/s= ec =C2=A0 12 =C2=A0 7.86 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 6.00-7.00 =C2=A0 sec =C2=A060.9 MBytes =C2=A0 512 Mbits/s= ec =C2=A0 14 =C2=A0 3.43 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 7.00-8.00 =C2=A0 sec =C2=A062.8 MBytes =C2=A0 527 Mbits/s= ec =C2=A0 16 =C2=A0 =C2=A0372 KBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 8.00-9.00 =C2=A0 sec =C2=A059.3 MBytes =C2=A0 497 Mbits/s= ec =C2=A0 14 =C2=A0 1.77 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 9.00-10.00 =C2=A0sec =C2=A057.0 MBytes =C2=A0 477 Mbits/s= ec =C2=A0 18 =C2=A0 7.13 MBytes =C2=A0 =C2=A0 =C2=A0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer =C2=A0 =C2=A0 Bi= trate =C2=A0 =C2=A0 =C2=A0 =C2=A0 Retr
[ =C2=A05] =C2=A0 0.00-10.00 =C2=A0sec =C2=A0 548 MBytes =C2=A0 459 Mbits/s= ec =C2=A0130 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sender
[ =C2=A05] =C2=A0 0.00-10.10 =C2=A0sec =C2=A0 540 MBytes =C2=A0 449 Mbits/s= ec =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0receiver
iperf Done.

cc@s1:~ % ifconfig bce4
bce4: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 m= tu 1500
options=3Dc01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWC= SUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 00:10:18:56:94:d4
inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I believe the default values for bce tx/rx pages are 2. And I happened to f= ind
this problem before that when the tx queue was full, it would not enqueue p= ackets
and started return errors.
And this error was misunderstood by the TCP layer as retransmission.

After adding hw.bce.tx_pages=3D4 and hw.bce.rx_pages=3D4 in /boot/loader.co= nf and reboot:

cc@s1:~ % iperf3 -c r1 -t 10 -i 1 -C cubic
Connecting to host r1, port 5201
[ =C2=A05] local 10.1.1.2 port 20478 connected to 10.1.1.3 port 5201
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer =C2=A0 =C2=A0 Bi= trate =C2=A0 =C2=A0 =C2=A0 =C2=A0 Retr =C2=A0Cwnd
[ =C2=A05] =C2=A0 0.00-1.00 =C2=A0 sec =C2=A04.15 MBytes =C2=A034.8 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 1.17 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 1.00-2.00 =C2=A0 sec =C2=A083.1 MBytes =C2=A0 697 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 2.00-3.00 =C2=A0 sec =C2=A0 112 MBytes =C2=A0 939 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 3.00-4.00 =C2=A0 sec =C2=A0 113 MBytes =C2=A0 944 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 4.00-5.00 =C2=A0 sec =C2=A0 112 MBytes =C2=A0 940 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 5.00-6.00 =C2=A0 sec =C2=A0 112 MBytes =C2=A0 942 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 6.00-7.00 =C2=A0 sec =C2=A0 112 MBytes =C2=A0 938 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 7.00-8.00 =C2=A0 sec =C2=A0 113 MBytes =C2=A0 944 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 8.00-9.00 =C2=A0 sec =C2=A0 112 MBytes =C2=A0 938 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
[ =C2=A05] =C2=A0 9.00-10.00 =C2=A0sec =C2=A0 113 MBytes =C2=A0 947 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 12.2 MBytes =C2=A0 =C2=A0 =C2=A0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer =C2=A0 =C2=A0 Bi= trate =C2=A0 =C2=A0 =C2=A0 =C2=A0 Retr
[ =C2=A05] =C2=A0 0.00-10.00 =C2=A0sec =C2=A0 985 MBytes =C2=A0 826 Mbits/s= ec =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sender
[ =C2=A05] =C2=A0 0.00-10.11 =C2=A0sec =C2=A0 982 MBytes =C2=A0 815 Mbits/s= ec =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0receiver
iperf Done.

=C2=A0

Best Regards,

Cheng Cui

=C2=A0

=C2=A0

On Fri, Jun 30, 2023 at 12:26=E2=80=AFPM Murali Kris= hnamurthy <mura= lik1@vmware.com> wrote:

Richard,

=C2=A0

Appreciate the useful inputs you have shared so far.= Will try to figure out regarding packet drops.

=C2=A0

Regarding HyStart, I see even BSD code base has supp= ort for this. May I know by when can we see that in an release, if not alre= ady available ?

=C2=A0

Regarding this point : =E2=80=9CSwitching to other cc modules may give some more insights. But = again, I suspect that momentary (microsecond) burstiness of BSD may be caus= ing this significantly higher loss rate.=E2=80=9D

Is there some info somewhere where I can understand = more on this in detail?

=C2=A0

Regards

Murali

=C2=A0

=C2=A0

On 30/06/23, 9:35 PM, &= quot;owner-freebsd-transport@freebsd.org" <owner-freebsd-transport@= freebsd.org> wrote:

=C2=A0

Hi Murali,

=C2=A0

> Q. Since you mention two hypervisors - what is = the phyiscal network topology in between these two servers? What theoretica= l link rates would be attainable?

>=C2=A0=C2=A0

> Here is the topology

>

> Iperf end points are on 2 different hypervisors= .

>

>=C2=A0=C2=A0=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94-=E2=80=94=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<= /p>

> | Linux VM1 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0|=C2=A0=C2=A0BSD 13 VM 1=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |= =C2=A0=C2=A0Linux VM2=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0|=C2=A0=C2=A0BSD 1= 3 VM 2=C2=A0=C2=A0|

> |___________|=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0|_ ____ ____ ___ |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0|___________= |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0|_ ____ ____ ___ |

> |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |

>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |

> =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94

> |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 ESX Hypervisor 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 10G link connected via L2 Switch=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0|=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 ESX Hypervisor=C2=A0=C2=A02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0|

> |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0|

> |=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94|

>

>

> Nic is of 10G capacity on both ESX server and i= t has below config.

=C2=A0

=C2=A0

So, when both VMs run on the same Hypervisor, maybe = with another VM to simulate the 100ms delay, can you attain a lossless base= line scenario?

=C2=A0

=C2=A0

> BDP for 16MB Socket buffer: 16 MB * (1000 ms * = 100ms latency) * 8 bits/ 1024 =3D 1.25 Gbps

>

> So theoretically we should see close to 1.25Gbp= s of Bitrate and we see Linux reaching close to this number.<= /p>

=C2=A0

Under no loss, yes.

=C2=A0

=C2=A0

> But BSD is not able to do that.

>

>

> Q. Did you run iperf3? Did the transmitting end= point report any retransmissions between Linux or FBSD hosts?=

>

> Yes, we used iper3. I see Linux doing less numb= er retransmissions compared to BSD.

> On BSD, the best performance was around 600 Mbp= s bitrate and the number of retransmissions for this number seen is around = 32K

> On Linux, the best performance was around 1.15 = Gbps bitrate and the number of retransmissions for this number seen is only= 2K.

> So as you pointed the number of retransmissions= in BSD could be the real issue here.

=C2=A0

There are other cc modules available; but I believe = one major deviation is that Linux can perform mechanisms like hystart; ACKi= ng every packet when the client detects slow start; perform pacing to achieve more uniform packet transmissions.=

=C2=A0

I think the next step would be to find out, at which= queue those packet discards are coming from (external switch? delay genera= tor? Vswitch? Eth stack inside the VM?)

=C2=A0

Or alternatively, provide your ESX hypervisors with = vastly more link speed, to rule out any L2 induced packet drops - provided = your delay generator is not the source when momentarily overloaded.

=C2=A0

> Is there a way to reduce this packet loss by fi= ne tuning some parameters w.r.t ring buffer or any other areas?

=C2=A0

Finding where these arise (looking at queue and port= counters) would be the next step. But this is not really my specific area = of expertise beyond the high level, vendor independent observations.

=C2=A0

Switching to other cc modules may give some more ins= ights. But again, I suspect that momentary (microsecond) burstiness of BSD = may be causing this significantly higher loss rate.

=C2=A0

TCP RACK would be another option. That stack has pac= ing, more fine-grained timing, the RACK loss recovery mechanisms etc. Maybe= that helps reduce the observed packet drops by iperf, and consequently, yield a higher overall throuhgput.

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

!! External Email: This email originated from outside of the organi= zation. Do not click links or open attachments unless you recognize the sender.

=C2=A0

--000000000000139feb05ff9af368--