Date: Thu, 29 Jun 2023 07:46:00 -0400 From: Randall Stewart <rrs@netflix.com> To: Murali Krishnamurthy <muralik1@vmware.com> Cc: "freebsd-transport@FreeBSD.org" <freebsd-transport@FreeBSD.org> Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Message-ID: <089C6BBB-C978-46DE-9479-1285B112E90D@netflix.com> In-Reply-To: <PH0PR05MB10064288C78F8F3A0116D023DFB469@PH0PR05MB10064.namprd05.prod.outlook.com> References: <PH0PR05MB10064288C78F8F3A0116D023DFB469@PH0PR05MB10064.namprd05.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036 Content-Type: multipart/alternative; boundary="Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707" --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Greetings Murali: So I am unclear from your =E2=80=9CSocket buffer=E2=80=9D as to what you = did.. Did you set both the send and receive windows to 16Mbps with the SO_SNDBUF and = SO_RCVBUF options? Or were you just using auto-scaling to have the socket buffers advance? Thanks R > On Jun 29, 2023, at 5:51 AM, Murali Krishnamurthy = <muralik1@vmware.com> wrote: >=20 > Hello FreeBSD Transport experts, > =20 > We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in = long RTT setup and happened to compare the performance with Linux VM = with same hypervisor. > We see a substantially better performance with Linux getting close to = the BDP limit, whereas BSD 13 not filling up the pipe enough. > We are trying to figure out what could lead to such a huge difference = and feel we could be missing something here. > =20 > Could you please help us to know if there is a way to make it perform = better? > =20 > Setup details: > =20 > We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu = 23.04/Linux kernel 6.2) were launched on each hypervisor. > Then we ran iperf between, > BSD 13 <-> BSD 13 > Ubuntu <-> Ubuntu > =20 > Even though the network environment were same in both cases, we see = Ubuntu performing much better. > =20 > Below are connection parameters: > Socket buffer: 16MB > TCP CC Algo: Cubic. We used this as this is suitable for Long Fat = Networks. > Ping RTT: 100 ms between the two end points. > We kept all other parameters to default on both Linux and BSD. > =20 > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 = bits/ 1024 =3D 1.25 Gbps > =20 > Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP = limit. > FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So = it seems to be doing half as good as Linux. > For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same = and able to meet BDP of 300 Mbps consistently. > Larger socket buffer seems to have an issue. > =20 > Please let us know if there are ways to fine tune the system = parameters to make BSD perform better. > Or any other suggestions/queries welcome. > =20 > Regards > Murali ------ Randall Stewart rrs@netflix.com --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" = class=3D"">Greetings Murali:<div class=3D""><br class=3D""></div><div = class=3D"">So I am unclear from your =E2=80=9CSocket buffer=E2=80=9D as = to what you did.. Did you</div><div class=3D"">set both the send and = receive windows to 16Mbps with the SO_SNDBUF and SO_RCVBUF = options?</div><div class=3D"">Or were you just using auto-scaling to = have the socket buffers advance?</div><div class=3D""><br = class=3D""></div><div class=3D"">Thanks</div><div class=3D""><br = class=3D""></div><div class=3D"">R</div><div class=3D""><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On Jun = 29, 2023, at 5:51 AM, Murali Krishnamurthy <<a = href=3D"mailto:muralik1@vmware.com" class=3D"">muralik1@vmware.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><meta= charset=3D"UTF-8" class=3D""><div class=3D"WordSection1" style=3D"page: = WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 14px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><div style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">Hello FreeBSD = Transport experts,<o:p class=3D""></o:p></span></div><div style=3D"margin:= 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span= lang=3D"EN-US" class=3D""><o:p class=3D""> </o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">We are = evaluating performance of FreeBSD 13 VM on ESX hypervisor in long RTT = setup and happened to compare the performance with Linux VM with same = hypervisor.<o:p class=3D""></o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">We see a substantially better performance with = Linux getting close to the BDP limit, whereas BSD 13 not filling up the = pipe enough.<o:p class=3D""></o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">We are trying to figure out what could lead to = such a huge difference and feel we could be missing something here.<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""> </o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Could you please help us to know if there is a = way to make it perform better?<o:p class=3D""></o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D""><o:p = class=3D""> </o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Setup details:<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""> </o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">We have 2 ESX hypervisors where 2 VMs (one = FreeBSD 13 and one Ubuntu 23.04/Linux kernel 6.2) were launched on each = hypervisor.<o:p class=3D""></o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Then we ran iperf between,<o:p = class=3D""></o:p></span></div><ol start=3D"1" type=3D"a" = style=3D"margin-bottom: 0cm; margin-top: 0cm;" class=3D""><li = class=3D"MsoListParagraph" style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;"><span lang=3D"EN-US" class=3D"">BSD = 13 <-> BSD 13<o:p class=3D""></o:p></span></li><li = class=3D"MsoListParagraph" style=3D"margin: 0cm; font-size: 11pt; = font-family: Calibri, sans-serif;"><span lang=3D"EN-US" class=3D"">Ubuntu = <-> Ubuntu<o:p class=3D""></o:p></span></li></ol><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D""><o:p = class=3D""> </o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Even though the network environment were same = in both cases, we see Ubuntu performing much better.<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""> </o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Below are connection parameters:<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D"">Socket buffer: 16MB<o:p class=3D""></o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">TCP CC Algo: = Cubic. We used this as this is suitable for Long Fat Networks.<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D"">Ping RTT: 100 ms between the two end points.<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D"">We kept all other parameters to default on both Linux and = BSD.<o:p class=3D""></o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D""> <o:p class=3D""></o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><strong class=3D""><i class=3D""><span = style=3D"font-size: 10.5pt; font-family: "Segoe UI", = sans-serif; color: rgb(23, 43, 77);" class=3D"">BDP for = 16MB</span></i></strong><span class=3D"apple-converted-space"><i = class=3D""><span style=3D"font-size: 10.5pt; font-family: "Segoe = UI", sans-serif; color: rgb(23, 43, 77);" = class=3D""> </span></i></span><em class=3D""><b class=3D""><span = style=3D"font-size: 10.5pt; font-family: "Segoe UI", = sans-serif; color: rgb(23, 43, 77);" class=3D"">Socket = buffer:</span></b></em><span class=3D"apple-converted-space"><i = class=3D""><span style=3D"font-size: 10.5pt; font-family: "Segoe = UI", sans-serif; color: rgb(23, 43, 77);" = class=3D""> </span></i></span><em class=3D""><span = style=3D"font-size: 10.5pt; font-family: "Segoe UI", = sans-serif; color: rgb(23, 43, 77);" class=3D"">16 MB * (1000 ms * 100ms = latency) * 8 bits/ 1024</span></em><span = class=3D"apple-converted-space"><span style=3D"font-size: 10.5pt; = font-family: "Segoe UI", sans-serif; color: rgb(23, 43, 77);" = class=3D""> </span></span><span style=3D"font-size: 10.5pt; = font-family: "Segoe UI", sans-serif; color: rgb(23, 43, 77);" = class=3D"">=3D<span class=3D"apple-converted-space"><i = class=3D""> </i></span><strong class=3D""><i class=3D""><span = style=3D"font-family: "Segoe UI", sans-serif;" class=3D"">1.25 = Gbps</span></i></strong></span><span lang=3D"EN-US" class=3D""><o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""> </o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Ubuntu consistently hits around 1 Gbps Bitrate = almost reaching the BDP limit.<o:p class=3D""></o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">FreeBSD 13 shows = a Bit rate between the range of<span = class=3D"Apple-converted-space"> </span></span><strong = class=3D""><span style=3D"font-size: 10.5pt; font-family: "Segoe = UI", sans-serif; color: rgb(23, 43, 77);" class=3D"">300-600 = Mbps</span></strong><span class=3D"apple-converted-space"><span = style=3D"font-size: 10.5pt; font-family: "Segoe UI", = sans-serif; color: rgb(23, 43, 77); background-color: white;" = class=3D""> </span></span><span style=3D"font-size: 10.5pt; = font-family: "Segoe UI", sans-serif; color: rgb(23, 43, 77); = background-color: white;" class=3D"">only. So it seems to be doing half = as good as Linux.<o:p class=3D""></o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">For lower socket buffer of 4MB, both FreeBSD = and Linux seem to do same and able to meet BDP of 300 Mbps = consistently.<o:p class=3D""></o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Larger socket buffer seems to have an = issue.<o:p class=3D""></o:p></span></div><div style=3D"margin: 0cm; = font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D""><o:p class=3D""> </o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" class=3D"">Please let us = know if there are ways to fine tune the system parameters to make BSD = perform better.<o:p class=3D""></o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Or any other suggestions/queries welcome.<o:p = class=3D""></o:p></span></div><div style=3D"margin: 0cm; font-size: = 11pt; font-family: Calibri, sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D""><o:p class=3D""> </o:p></span></div><div style=3D"margin: = 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=3D""><span = lang=3D"EN-US" class=3D"">Regards<o:p class=3D""></o:p></span></div><div = style=3D"margin: 0cm; font-size: 11pt; font-family: Calibri, = sans-serif;" class=3D""><span lang=3D"EN-US" = class=3D"">Murali</span></div></div></div></blockquote></div><br = class=3D""><div class=3D""> <div style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none;">------</div><div style=3D"caret-color: rgb(0, 0, = 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;">Randall = Stewart</div><div style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none;"><a href=3D"mailto:rrs@netflix.com" = class=3D"">rrs@netflix.com</a></div><div style=3D"caret-color: rgb(0, 0, = 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><br = class=3D""></div><br class=3D"Apple-interchange-newline"> </div> <br class=3D""></div></body></html>= --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707-- --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCAzYw ggMyMIICGqADAgECAgqxywKqrHPB2ybTMA0GCSqGSIb3DQEBCwUAMEcxGDAWBgNVBAMTD1JhbmRh bGwgU3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzAe Fw0yMTAxMjQxMjIwMTRaFw0yNjAxMjQxMjIwMTRaMEcxGDAWBgNVBAMTD1JhbmRhbGwgU3Rld2Fy dDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBAMUAht2nr/NFlK+tmmN9PdO3DBPfeYh9fLcbVihR+/dipO41 AsFy9y+2uDVaFhTEvp406P0o9PQQTuYXqrCr76eWQIj3V787e1WKjTup1mIyQeWHGf1gvb/7vmI2 zHg6QZEIC4W8xeO8SLKyHiwlFHZn8Rn1HxtB7Ge+NulygkgUgJYhXD5E29jVGXAc6Qn9Vr9AexPf KaOhHCaNB/Twcinayz6D8CO/Ym1LOs3+ceSOa4cB07fepmbqDSXDkOeA3U7KLaluHrRTlj6DO+JU nqKXX7jJ68KTYSZ0qH4fZsk8cxFkwYI/3HDJi+oF+FDkf7SRo1Q2w+e3M/5MReLIQ7sCAwEAAaMg MB4wDwYJKoZIhvcvAQEKBAIFADALBgNVHQ8EBAMCB4AwDQYJKoZIhvcNAQELBQADggEBAHJfum1j 1WIVFjOJT/hqMIN751aXkablmwesW94lNJKjslPULbbcP5nZGg2lGpHcZ+0I5F/1TTiEsT2H2rhA uAnSsUxTpxRA+aoe+xtJOa5vle3CprhFkHAvB7EIoLiNaPd0DNK6kKYsbvr5Z5Eq7TF3SIO77Qh9 /8VgUfOb0ARDgix59Q6MM0NmIabEwh0cDWQYlGgDDtN9DNk5PGM4pjs48suwEdLmFTMOmGTkCp7I Vq6iHDNinBiB6+BB4VYMAO1o9qS+0pnfdmPJybt0zVGrhm/c1Fmm3Jec7NEuiKeXmhPIwdwMkKyp AsX0sHuFCYwioBTHHZpvnir+H2rRakgxggHrMIIB5wIBATBVMEcxGDAWBgNVBAMTD1JhbmRhbGwg U3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUwIKscsC qqxzwdsm0zANBglghkgBZQMEAgEFAKBpMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI hvcNAQkFMQ8XDTIzMDYyOTExNDYwMFowLwYJKoZIhvcNAQkEMSIEILaabTtURq4atG0MExNLW/wO 8Ouv2lkNCnBERFs6wJwSMA0GCSqGSIb3DQEBCwUABIIBAIdtF+DXNOHZ2pJB1YkkLSaNc3uTedsy w45aDrz//YcaClhr/KWJnZ2vStcGv1O6tqXQEmfXPYACQFB/iOwsIahe8NjY71ysxl1f+37JEmnY 1GlOEIkjSoHr7JPmPDvRGtbaVQWsjeLDXJ5fZ1cUOmKB2h1g+sBdD4fTITRfN9z2JTDjWkVlSdw0 BvI1H+jke17uLoGLqVtPPNyW3NpWoKaTWz+1Ug7Bk/Qkz14Yhcq8elF0w+IZ/ySZQc32j9Hhjwl/ xOFH/xpyxgNx8yFEOJFNpwurBpGvfO14uEAzAxJ8DZva9sn6C5VfoQKa+ApkYlvo2xCib/tlT2on tra52DwAAAAAAAA= --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?089C6BBB-C978-46DE-9479-1285B112E90D>