Date: Thu, 29 Jun 2023 09:51:11 +0000 From: Murali Krishnamurthy <muralik1@vmware.com> To: "freebsd-transport@FreeBSD.org" <freebsd-transport@FreeBSD.org> Subject: FreeBSD TCP (with iperf3) comparison with Linux Message-ID: <PH0PR05MB10064288C78F8F3A0116D023DFB469@PH0PR05MB10064.namprd05.prod.outlook.com>
next in thread | raw e-mail | index | archive | help
--_000_PH0PR05MB10064288C78F8F3A0116D023DFB469PH0PR05MB10064na_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello FreeBSD Transport experts, We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in long RT= T setup and happened to compare the performance with Linux VM with same hyp= ervisor. We see a substantially better performance with Linux getting close to the B= DP limit, whereas BSD 13 not filling up the pipe enough. We are trying to figure out what could lead to such a huge difference and f= eel we could be missing something here. Could you please help us to know if there is a way to make it perform bette= r? Setup details: We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu 23.04/= Linux kernel 6.2) were launched on each hypervisor. Then we ran iperf between, 1. BSD 13 <-> BSD 13 2. Ubuntu <-> Ubuntu Even though the network environment were same in both cases, we see Ubuntu = performing much better. Below are connection parameters: Socket buffer: 16MB TCP CC Algo: Cubic. We used this as this is suitable for Long Fat Networks. Ping RTT: 100 ms between the two end points. We kept all other parameters to default on both Linux and BSD. BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 102= 4 =3D 1.25 Gbps Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP limi= t. FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So it s= eems to be doing half as good as Linux. For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same and = able to meet BDP of 300 Mbps consistently. Larger socket buffer seems to have an issue. Please let us know if there are ways to fine tune the system parameters to = make BSD perform better. Or any other suggestions/queries welcome. Regards Murali --_000_PH0PR05MB10064288C78F8F3A0116D023DFB469PH0PR05MB10064na_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">; <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:"Segoe UI"; panose-1:2 11 5 2 4 2 4 2 2 3;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ligatures:standardcontextual; mso-fareast-language:EN-US;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0cm; margin-right:0cm; margin-bottom:0cm; margin-left:36.0pt; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ligatures:standardcontextual; mso-fareast-language:EN-US;} span.apple-converted-space {mso-style-name:apple-converted-space;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; font-family:"Calibri",sans-serif; mso-ligatures:none; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:124734190; mso-list-type:hybrid; mso-list-template-ids:-1953459250 134807575 134807577 134807579 134807567 = 134807577 134807579 134807567 134807577 134807579;} @list l0:level1 {mso-level-number-format:alpha-lower; mso-level-text:"%1\)"; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt;} @list l0:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt;} @list l0:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l0:level4 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt;} @list l0:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt;} @list l0:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l0:level7 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt;} @list l0:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt;} @list l0:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l1 {mso-list-id:532770188; mso-list-template-ids:-411673588;} @list l1:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:36.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:72.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level3 {mso-level-number-format:alpha-lower; mso-level-tab-stop:108.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level4 {mso-level-number-format:alpha-lower; mso-level-tab-stop:144.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:180.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level6 {mso-level-number-format:alpha-lower; mso-level-tab-stop:216.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level7 {mso-level-number-format:alpha-lower; mso-level-tab-stop:252.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:288.0pt; mso-level-number-position:left; text-indent:-18.0pt;} @list l1:level9 {mso-level-number-format:alpha-lower; mso-level-tab-stop:324.0pt; mso-level-number-position:left; text-indent:-18.0pt;} ol {margin-bottom:0cm;} ul {margin-bottom:0cm;} --></style> </head> <body lang=3D"EN-IN" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:= break-word"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span lang=3D"EN-US">Hello FreeBSD Transport experts= ,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">We are evaluating performance o= f FreeBSD 13 VM on ESX hypervisor in long RTT setup and happened to compare= the performance with Linux VM with same hypervisor.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">We see a substantially better p= erformance with Linux getting close to the BDP limit, whereas BSD 13 not fi= lling up the pipe enough. <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">We are trying to figure out wha= t could lead to such a huge difference and feel we could be missing somethi= ng here.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Could you please help us to kno= w if there is a way to make it perform better?<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Setup details:<o:p></o:p></span= ></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">We have 2 ESX hypervisors where= 2 VMs (one FreeBSD 13 and one Ubuntu 23.04/Linux kernel 6.2) were launched= on each hypervisor.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Then we ran iperf between,<o:p>= </o:p></span></p> <ol style=3D"margin-top:0cm" start=3D"1" type=3D"a"> <li class=3D"MsoListParagraph" style=3D"margin-left:0cm;mso-list:l0 level1 = lfo3"><span lang=3D"EN-US">BSD 13 <-> BSD 13<o:p></o:p></span></li><l= i class=3D"MsoListParagraph" style=3D"margin-left:0cm;mso-list:l0 level1 lf= o3"><span lang=3D"EN-US">Ubuntu <-> Ubuntu<o:p></o:p></span></li></ol= > <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Even though the network environ= ment were same in both cases, we see Ubuntu performing much better.<o:p></o= :p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Below are connection parameters= :<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Socket buffer: 16MB<o:p></o:p><= /span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">TCP CC Algo: Cubic. We used thi= s as this is suitable for Long Fat Networks.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Ping RTT: 100 ms between = the two end points.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">We kept all other parameters to= default on both Linux and BSD. <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"> <o:p></o:p></span></p> <p class=3D"MsoNormal"><strong><i><span style=3D"font-size:10.5pt;font-fami= ly:"Segoe UI",sans-serif;color:#172B4D">BDP for 16MB</span></i></= strong><span class=3D"apple-converted-space"><i><span style=3D"font-size:10= .5pt;font-family:"Segoe UI",sans-serif;color:#172B4D"> </spa= n></i></span><em><b><span style=3D"font-size:10.5pt;font-family:"Segoe= UI",sans-serif;color:#172B4D">Socket buffer:</span></b></em><span class=3D"apple-converted-space"><i><span styl= e=3D"font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#17= 2B4D"> </span></i></span><em><span style=3D"font-size:10.5pt;font-fami= ly:"Segoe UI",sans-serif;color:#172B4D">16 MB * (1000 ms * 100ms latency) * 8 bits/ 1024</span></em><span class=3D"apple-convert= ed-space"><span style=3D"font-size:10.5pt;font-family:"Segoe UI",= sans-serif;color:#172B4D"> </span></span><span style=3D"font-size:10.5= pt;font-family:"Segoe UI",sans-serif;color:#172B4D">=3D<span clas= s=3D"apple-converted-space"><i> </i></span><strong><i><span style=3D"f= ont-family:"Segoe UI",sans-serif">1.25 Gbps</span></i></strong></span><span lang=3D"EN-US"><o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Ubuntu consistently hits around= 1 Gbps Bitrate almost reaching the BDP limit. <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">FreeBSD 13 shows a Bit rate bet= ween the range of </span><strong><span style=3D"font-size:10.5pt;font-family:"Segoe UI&q= uot;,sans-serif;color:#172B4D">300-600 Mbps</span></strong><span class=3D"a= pple-converted-space"><span style=3D"font-size:10.5pt;font-family:"Seg= oe UI",sans-serif;color:#172B4D;background:white"> </span></span>= <span style=3D"font-size:10.5pt;font-family:"Segoe UI",sans-serif= ;color:#172B4D;background:white">only. So it seems to be doing half as good as Linux.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">For lower socket buffer of 4MB,= both FreeBSD and Linux seem to do same and able to meet BDP of 300 Mbps co= nsistently.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Larger socket buffer seems to h= ave an issue.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Please let us know if there are= ways to fine tune the system parameters to make BSD perform better.<o:p></= o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Or any other suggestions/querie= s welcome.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Regards<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Murali<o:p></o:p></span></p> </div> </body> </html> --_000_PH0PR05MB10064288C78F8F3A0116D023DFB469PH0PR05MB10064na_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?PH0PR05MB10064288C78F8F3A0116D023DFB469>