Date: Fri, 30 Jun 2023 15:03:44 +0000 From: Murali Krishnamurthy <muralik1@vmware.com> To: "Scheffenegger, Richard" <rscheff@freebsd.org>, FreeBSD Transport <freebsd-transport@freebsd.org> Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Message-ID: <PH0PR05MB10064ECAB726B2636AF410338FB2AA@PH0PR05MB10064.namprd05.prod.outlook.com> In-Reply-To: <0826320c-936a-e3a4-78e8-809b8bab5f71@freebsd.org> References: <0826320c-936a-e3a4-78e8-809b8bab5f71@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--_000_PH0PR05MB10064ECAB726B2636AF410338FB2AAPH0PR05MB10064na_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Richard,
Please see my answers for your queries,
Q. Since you mention two hypervisors - what is the phyiscal network topolog=
y in between these two servers? What theoretical link rates would be attain=
able?
Here is the topology
Iperf end points are on 2 different hypervisors.
=97=97=97=97=97=97 =97=97=97=97=97=97-=97 =
=97=97=97=97=97=97 =
=97=97=97=97=97=97-=97
| Linux VM1 | | BSD 13 VM 1 | =
| Linux VM2 | |=
BSD 13 VM 2 |
|___________| |_ ____ ____ ___ | =
|___________ | |=
_ ____ ____ ___ |
| | | =
| =
|
| | =
| =
|
=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97 =
=97=97=97=97=97=97=97=
=97=97=97=97=97=97=97=97
| ESX Hypervisor 1 | 10G link connected via L2=
Switch | ESX Hypervisor 2 |
| |=97=97=97=97=97=97=97=97=
=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97 | =
|
|=97=97=97=97=97=97=97=97=97=97=97=97=97=97 | =
|=97=97=97=97=97=97=97=
=97=97=97=97=97=97=97|
Nic is of 10G capacity on both ESX server and it has below config.
Name PCI Driver Link Speed =
MAC Address MTU Descrip=
tion
vmnic4 0000:81:00.0 ixgben Up 10000Mbps Full Du=
plex a0:36:9f:61:ca:d4 1500 Intel(R) Ethernet 10G 2P X520=
Adapter
iperf3 was run between,
1. BSD 13 VM 1 <-> BSD 13 VM 2
2. Linux VM 1 <-> Linux VM 2
BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 102=
4 =3D 1.25 Gbps
So theoretically we should see close to 1.25Gbps of Bitrate and we see Linu=
x reaching close to this number.
But BSD is not able to do that.
Q. Did you run iperf3? Did the transmitting endpoint report any retransmiss=
ions between Linux or FBSD hosts?
Yes, we used iper3. I see Linux doing less number retransmissions compared =
to BSD.
On BSD, the best performance was around 600 Mbps bitrate and the number of =
retransmissions for this number seen is around 32K
On Linux, the best performance was around 1.15 Gbps bitrate and the number =
of retransmissions for this number seen is only 2K.
So as you pointed the number of retransmissions in BSD could be the real is=
sue here.
Is there a way to reduce this packet loss by fine tuning some parameters w.=
r.t ring buffer or any other areas?
Q. Did you contrast your cc_cubic findings with cc_newreno?
New reno does not perform well with long RTT networks. I could only see 200=
Mbps bitrate even with no packet loss.
So we did not want to pursue with New Reno with long RTT connections.
Hope the answers are clear. Please let me know if you have any other sugges=
tions to reduce the packet loss.
Regards
Murali
On 29/06/23, 9:44 PM, "Scheffenegger, Richard" <rscheff@freebsd.org> wrote:
Hi Murali,
> Hello FreeBSD Transport experts,
>
> We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in long =
RTT setup and happened to compare the performance with Linux VM with same h=
ypervisor.
> We see a substantially better performance with Linux getting close to the=
BDP limit, whereas BSD 13 not filling up the pipe enough.
> We are trying to figure out what could lead to such a huge difference and=
feel we could be missing something here.
>
> Could you please help us to know if there is a way to make it perform bet=
ter?
>
> Setup details:
>
> We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu 23.0=
4/Linux kernel 6.2) were launched on each hypervisor.
> Then we ran iperf between,
> a) BSD 13 <-> BSD 13
> b) Ubuntu <-> Ubuntu
Since you mention two hypervisors - what is the phyiscal network topology i=
n between these two servers? What theoretical link rates would be attainabl=
e?
Or did you want to say that each test was run within one and the same hyper=
visor, presumably with the same hardware and virtual switch/hypervisor prov=
ided on both test scenarios?
So that the theoretical bandwidth would be whatever packets can be shuffled=
in memory between VM1, Vswitch (with delay simulator), VM2?
Did you run iperf3? Did the transmitting endpoint report any retransmission=
s between Linux or FBSD hosts?
In theory, in a true back-to-back, lossless environment, it should be fairl=
y irrelevant which CC you are running; if there are packet drops (retransmi=
ssions) in the iperf3 output, please compare their frequency between the Li=
nux and FBSD side - a slightly higher incidence rate of packet drops (e.g. =
ring-buffer overflows used to be a prominent issue at high speed links) cou=
ld possibly explain what appears to be such a huge differential (the square=
of the loss probability is relevant for classic TCPs - so a minor change c=
an have a huge impact).
Did you contrast your cc_cubic findings with cc_newreno?
> Even though the network environment were same in both cases, we see Ubunt=
u performing much better.
>
> Below are connection parameters:
> Socket buffer: 16MB
> TCP CC Algo: Cubic. We used this as this is suitable for Long Fat Network=
s.
> Ping RTT: 100 ms between the two end points.
> We kept all other parameters to default on both Linux and BSD.
>
> BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 1=
024 =3D 1.25 Gbps
>
> Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP li=
mit.
> FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So it=
seems to be doing half as good as Linux.
> For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same an=
d able to meet BDP of 300 Mbps consistently.
> Larger socket buffer seems to have an issue.
>
> Please let us know if there are ways to fine tune the system parameters t=
o make BSD perform better.
> Or any other suggestions/queries welcome.
>
> Regards
> Murali
--_000_PH0PR05MB10064ECAB726B2636AF410338FB2AAPH0PR05MB10064na_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">;
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1=
252">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Helvetica Neue";
panose-1:2 0 5 3 0 0 0 2 0 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0cm;
font-size:10.0pt;
font-family:"Helvetica Neue";}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0cm;
font-size:10.0pt;
font-family:"Helvetica Neue";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:541751583;
mso-list-type:hybrid;
mso-list-template-ids:52449262 567314720 134807577 134807579 134807567 134=
807577 134807579 134807567 134807577 134807579;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1529487027;
mso-list-type:hybrid;
mso-list-template-ids:-495026070 134807567 134807577 134807579 134807567 1=
34807577 134807579 134807567 134807577 134807579;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2
{mso-list-id:1607497547;
mso-list-type:hybrid;
mso-list-template-ids:152591530 233208848 134807555 134807557 134807553 13=
4807555 134807557 134807553 134807555 134807557;}
@list l2:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0D8 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang=3D"EN-IN" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:=
break-word">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Richard,<=
o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Please se=
e my answers for your queries,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Q. </span=
>Since you mention two hypervisors - what is the phyiscal network topology =
in between these two servers? What theoretical link rates would be attainab=
le?<o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Here is t=
he topology<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Iperf end=
points are on 2 different hypervisors.
<o:p></o:p></span></p>
<p class=3D"p1"><o:p> </o:p></p>
<p class=3D"p2">=97=97=97=97=97=97 <span class=3D"apple-converted-space">&n=
bsp; </span>=97=97=97=97=97=97-=97<span class=3D"apple-ta=
b-span"> &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p;
</span>=97=97=97=97=97=97 <span class=3D"apple-converted-space"> &nbs=
p; </span>=97=
=97=97=97=97=97-=97<span class=3D"apple-tab-span"> &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p;
</span><o:p></o:p></p>
<p class=3D"p2">|<span class=3D"apple-converted-space"> </span>Linux =
VM1 |<span class=3D"apple-converted-space"> </span=
>|<span class=3D"apple-converted-space">
</span>BSD 13 VM 1<span class=3D"apple-converted-space"> </span>|<spa=
n class=3D"apple-tab-span"> =
&nb=
sp; =
&nb=
sp; =
&nb=
sp;
</span>|<span class=3D"apple-converted-space"> </span>Linux VM2<span =
class=3D"apple-converted-space">
</span>|<span class=3D"apple-converted-space"> &nb=
sp; </span>|<span class=3D"=
apple-converted-space">
</span>BSD 13 VM 2<span class=3D"apple-converted-space"> </span>|<o:p=
></o:p></p>
<p class=3D"p2">|___________|<span class=3D"apple-converted-space"> &=
nbsp; </span>|_ ____ ____ ___ |<span class=3D"apple-tab-span">&=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp;
</span>|___________ |<span class=3D"apple-converted-space"> &n=
bsp; </span>|_ =
____ ____ ___ |<o:p></o:p></p>
<p class=3D"p2"><span class=3D"apple-converted-space">| =
</span>|<span class=3D"apple-tab-span"> &nb=
sp;  =
;
</span>|<span class=3D"apple-tab-span"> =
&nb=
sp; =
&nb=
sp; =
&nb=
sp; =
&=
nbsp;
</span>|<span class=3D"apple-tab-span"> =
&nb=
sp;  =
; </span>|<o:p></o:p></p>
<p class=3D"p2"> |<sp=
an class=3D"apple-tab-span">  =
; &n=
bsp; </span>
|<span class=3D"apple-tab-span"> &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p;  =
; &n=
bsp;
</span>|<span class=3D"apple-tab-span"> =
&nb=
sp;  =
; </span>|<o:p></o:p></p>
<p class=3D"p2">=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97<span class=3D=
"apple-tab-span"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp;
</span>=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97<o:p></o:p></p>
<p class=3D"p2">|<span class=3D"apple-tab-span"> &nb=
sp; </span>ESX Hypervisor 1<span class=3D"app=
le-tab-span">
</span>|<span class=3D"apple-tab-span"> =
</span>10G link connected via L2 Switch<span class=
=3D"apple-tab-span"> =
</span>|<span class=3D"apple-tab-span"> =
</span>ESX Hypervisor<span class=3D"apple-converte=
d-space">
</span>2<span class=3D"apple-tab-span"> =
</span>|<o:p></o:p></p>
<p class=3D"p2">|<span class=3D"apple-tab-span"> &nb=
sp; =
&nb=
sp; =
</span>|=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=
=97=97 |<span class=3D"apple-tab-span"> =
&nb=
sp;  =
; &=
nbsp;
</span>|<o:p></o:p></p>
<p class=3D"p2">|=97=97=97=97=97=97=97=97=97=97=97=97=97=97 |<span class=3D=
"apple-tab-span"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp;
</span>|=97=97=97=97=97=97=97=97=97=97=97=97=97=97|<o:p></o:p></p>
<p class=3D"p1"><o:p> </o:p></p>
<p class=3D"p1"><o:p> </o:p></p>
<p class=3D"p2">Nic is of 10G capacity on both ESX server and it has below =
config.<o:p></o:p></p>
<p class=3D"p1"><o:p> </o:p></p>
<p class=3D"p2">Name<span class=3D"apple-converted-space"> </s=
pan>PCI<span class=3D"apple-converted-space"> &n=
bsp;
</span><span class=3D"apple-tab-span"> &=
nbsp; &nbs=
p; </span>Driver<span class=3D"apple-converted-space"> &=
nbsp;
</span>Link <span class=3D"apple-tab-span"> </span>Speed<span c=
lass=3D"apple-converted-space">
</span><span class=3D"apple-tab-span"> &=
nbsp; &nb=
sp; </sp=
an>MAC Address
<span class=3D"apple-converted-space"> </span><span cla=
ss=3D"apple-tab-span">  =
;
</span>MTU<span class=3D"apple-converted-space"> </span>Descri=
ption<span class=3D"apple-converted-space"> &nbs=
p; </span><o:p></o:p></p>
<p class=3D"p2">vmnic4<span class=3D"apple-converted-space"> </span>0=
000:81:00.0 <span class=3D"apple-tab-span">
&n=
bsp; </span>ixgben<span class=3D"apple-converted-space">  =
; </span>Up <span class=3D"apple-converted-space">
</span><span class=3D"apple-tab-span"></span>10000Mbps<span class=3D=
"apple-converted-space">
</span><span class=3D"apple-tab-span"> </span>Full Duplex=
<span class=3D"apple-converted-space">
</span><span class=3D"apple-tab-span"> </span=
>a0:36:9f:61:ca:d4 1500 <s=
pan class=3D"apple-converted-space">
</span>Intel(R) Ethernet 10G 2P X520 Adapter<o:p></o:p></p>
<p class=3D"p1"><o:p> </o:p></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">iperf3 wa=
s run between,<o:p></o:p></span></p>
<ol style=3D"margin-top:0cm" start=3D"1" type=3D"1">
<li class=3D"MsoListParagraph" style=3D"margin-left:0cm;mso-list:l0 level1 =
lfo3">BSD 13 VM 1 <-> BSD 13 VM 2<span style=3D"mso-fareast-language:=
EN-US"><o:p></o:p></span></li><li class=3D"MsoListParagraph" style=3D"margi=
n-left:0cm;mso-list:l0 level1 lfo3">Linux VM 1 <-> Linux VM 2<span st=
yle=3D"mso-fareast-language:EN-US"><o:p></o:p></span></li></ol>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal">BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms=
latency) * 8 bits/ 1024 =3D 1.25 Gbps<o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">So theore=
tically we should see close to 1.25Gbps of Bitrate and we see Linux reachin=
g close to this number.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">But BSD i=
s not able to do that.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal">Q. Did you run iperf3? Did the transmitting endpoint=
report any retransmissions between Linux or FBSD hosts?<o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Yes, we u=
sed iper3. I see Linux doing less number retransmissions compared to BSD.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">On BSD, t=
he best performance was around 600 Mbps bitrate and the number of retransmi=
ssions for this number seen is around 32K<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">On Linux,=
the best performance was around 1.15 Gbps bitrate and the number of retran=
smissions for this number seen is only 2K.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">So as you=
pointed the number of retransmissions in BSD could be the real issue here.=
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Is there =
a way to reduce this packet loss by fine tuning some parameters w.r.t ring =
buffer or any other areas?
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Q. </span=
>Did you contrast your cc_cubic findings with cc_newreno?<o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">New reno =
does not perform well with long RTT networks. I could only see 200 Mbps bit=
rate even with no packet loss.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">So we did=
not want to pursue with New Reno with long RTT connections.<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Hope the =
answers are clear. Please let me know if you have any other suggestions to =
reduce the packet loss.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Regards<o=
:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Murali<o:=
p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs=
p;</o:p></span></p>
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt">On 29/06/23, 9:44 PM,=
"Scheffenegger, Richard" <rscheff@freebsd.org> wrote:<o:p>=
</o:p></p>
<div>
<p class=3D"MsoNormal">Hi Murali,<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Hello FreeBSD Transport experts,<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> We are evaluating performance of FreeBSD 13 VM =
on ESX hypervisor in long RTT setup and happened to compare the performance=
with Linux VM with same hypervisor.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> We see a substantially better performance with =
Linux getting close to the BDP limit, whereas BSD 13 not filling up the pip=
e enough.
<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> We are trying to figure out what could lead to =
such a huge difference and feel we could be missing something here.<o:p></o=
:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Could you please help us to know if there is a =
way to make it perform better?<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Setup details:<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> We have 2 ESX hypervisors where 2 VMs (one Free=
BSD 13 and one Ubuntu 23.04/Linux kernel 6.2) were launched on each hypervi=
sor.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Then we ran iperf between,<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> a)<span class=3D"apple-tab-span"> &n=
bsp; </span>BSD 13 <-> BSD 13<o:p></o:p=
></p>
</div>
<div>
<p class=3D"MsoNormal">> b)<span class=3D"apple-tab-span"> &n=
bsp; </span>Ubuntu <-> Ubuntu<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Since you mention two hypervisors - what is the phyi=
scal network topology in between these two servers? What theoretical link r=
ates would be attainable?<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Or did you want to say that each test was run within=
one and the same hypervisor, presumably with the same hardware and virtual=
switch/hypervisor provided on both test scenarios?<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">So that the theoretical bandwidth would be whatever =
packets can be shuffled in memory between VM1, Vswitch (with delay simulato=
r), VM2?<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Did you run iperf3? Did the transmitting endpoint re=
port any retransmissions between Linux or FBSD hosts?<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">In theory, in a true back-to-back, lossless environm=
ent, it should be fairly irrelevant which CC you are running; if there are =
packet drops (retransmissions) in the iperf3 output, please compare their f=
requency between the Linux and FBSD
side - a slightly higher incidence rate of packet drops (e.g. ring-buffer =
overflows used to be a prominent issue at high speed links) could possibly =
explain what appears to be such a huge differential (the square of the loss=
probability is relevant for classic
TCPs - so a minor change can have a huge impact).<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Did you contrast your cc_cubic findings with cc_newr=
eno?<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Even though the network environment were same i=
n both cases, we see Ubuntu performing much better.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Below are connection parameters:<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Socket buffer: 16MB<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> TCP CC Algo: Cubic. We used this as this is sui=
table for Long Fat Networks.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Ping RTT: 100 ms between the two end=
points.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> We kept all other parameters to default on both=
Linux and BSD.
<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> BDP for 16MB Socket buffer: 16 MB * (1000 ms * =
100ms latency) * 8 bits/ 1024 =3D 1.25 Gbps<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Ubuntu consistently hits around 1 Gbps Bitrate =
almost reaching the BDP limit.
<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> FreeBSD 13 shows a Bit rate between the range o=
f 300-600 Mbps only. So it seems to be doing half as good as Linux.<o:p></o=
:p></p>
</div>
<div>
<p class=3D"MsoNormal">> For lower socket buffer of 4MB, both FreeBSD an=
d Linux seem to do same and able to meet BDP of 300 Mbps consistently.<o:p>=
</o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Larger socket buffer seems to have an issue.<o:=
p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Please let us know if there are ways to fine tu=
ne the system parameters to make BSD perform better.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Or any other suggestions/queries welcome.<o:p><=
/o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> <o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Regards<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal">> Murali<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>
--_000_PH0PR05MB10064ECAB726B2636AF410338FB2AAPH0PR05MB10064na_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?PH0PR05MB10064ECAB726B2636AF410338FB2AA>