Date: Mon, 3 Jul 2023 15:43:36 +0000 From: Murali Krishnamurthy <muralik1@vmware.com> To: "Scheffenegger, Richard" <rscheff@freebsd.org>, FreeBSD Transport <freebsd-transport@freebsd.org> Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Message-ID: <PH0PR05MB1006405C3E6C421E99CF6E853FB29A@PH0PR05MB10064.namprd05.prod.outlook.com> In-Reply-To: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org> References: <22e39458-2f07-6634-1c30-c9df9d6bfaea@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Richard, =93How having high timer granularity pacing (as what RACK does) can prevent= microbursts?=94 =93 That TCP can, at high transmission rates, overrun the NIC driver memory= buffers?=94 Yes. Any relevant doc/write-up on these would help for better understanding= . I know high transmission of packets could lead to packet drops at NIC buffe= rs. Linux somehow seem to be avoiding that problem. =93What did your experiments with the suggestions by @cc show with the base= stack?=94 Just posted a follow up query to Cheng on this. On enabling RACK: It seems we need to rebuild freebsd with compile time opt= ion. Is there a build (iso) available in repository with RACK enabled alrea= dy? Regards Murali On 03/07/23, 2:08 PM, "Scheffenegger, Richard" <rscheff@freebsd.org> wrote: Hi Murali, > Appreciate the useful inputs you have shared so far. Will try to figure o= ut regarding packet drops. > > Regarding HyStart, I see even BSD code base has support for this. May I k= now by when can we see that in an release, if not already available ? You "simply" switch to the RACK stack. Just as I mentioned earlier. Note th= at this is a completely refactored stack by itself - so please validate tha= t all your userspace applications (or kernel patches) still work when you a= ctivate it. https://man.freebsd.org/cgi/man.cgi?query=3Dtcp_rack&sektion=3D4&manpath=3D= FreeBSD+13.2-RELEASE+and+Ports > > Regarding this point : =93Switching to other cc modules may give some mor= e insights. But again, I suspect that momentary (microsecond) burstiness of= BSD may be causing this significantly higher loss rate.=94 > Is there some info somewhere where I can understand more on this in detai= l? What detail? How to toggle to a different congestion control? How having high timer granularity pacing (as what RACK does) can prevent mi= crobursts? That TCP can, at high transmission rates, overrun the NIC driver memory buf= fers? What did your experiments with the suggestions by @cc show with the base st= ack? And again, the RACK stack does HyStart and high-granularity pacing, address= ing many issues which are important to smooth streaming workflows and preve= nting momentary burst loads. However, the RACK stack certainly needs to be = validated with particular applications such as the kernel NFS server/client= , which are deeply embedded and may rely on some things in the base stack t= o work properly. I suspect that until now, noone has done a thorough invest= igation of the implications of RACK with NFS really. Best regards, Richard > > Regards > Murali --_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">; <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} span.EmailStyle18 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; mso-ligatures:none;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style> </head> <body lang=3D"EN-IN" link=3D"blue" vlink=3D"purple" style=3D"word-wrap:brea= k-word"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Richard,<= o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal">=93How having high timer granularity pacing (as what= RACK does) can prevent microbursts?=94<o:p></o:p></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">=93 </spa= n>That TCP can, at high transmission rates, overrun the NIC driver memory b= uffers?=94<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Yes. Any relevant doc/write-up on these would help f= or better understanding. <o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">I know high transmission of packets could lead to pa= cket drops at NIC buffers. Linux somehow seem to be avoiding that problem. <o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">=93</span= >What did your experiments with the suggestions by @cc show with the base s= tack?=94<o:p></o:p></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Just post= ed a follow up query to Cheng on this.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">On enabli= ng RACK: It seems we need to rebuild freebsd with compile time option. Is t= here a build (iso) available in repository with RACK enabled already?<o:p><= /o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Regards<o= :p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US">Murali<o:= p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"><o:p>&nbs= p;</o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt">On 03/07/23, 2:08 PM,= "Scheffenegger, Richard" <rscheff@freebsd.org> wrote:<o:p>= </o:p></p> <div> <p class=3D"MsoNormal">Hi Murali,<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">> Appreciate the useful inputs you have shared so= far. Will try to figure out regarding packet drops.<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal">> <o:p></o:p></p> </div> <div> <p class=3D"MsoNormal">> Regarding HyStart, I see even BSD code base has= support for this. May I know by when can we see that in an release, if not= already available ?<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">You "simply" switch to the RACK stack. Jus= t as I mentioned earlier. Note that this is a completely refactored stack b= y itself - so please validate that all your userspace applications (or kern= el patches) still work when you activate it.<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal"><a href=3D"https://man.freebsd.org/cgi/man.cgi?query= =3Dtcp_rack&sektion=3D4&manpath=3DFreeBSD+13.2-RELEASE+and+Ports">h= ttps://man.freebsd.org/cgi/man.cgi?query=3Dtcp_rack&sektion=3D4&man= path=3DFreeBSD+13.2-RELEASE+and+Ports</a><o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">> <o:p></o:p></p> </div> <div> <p class=3D"MsoNormal">> Regarding this point : =93Switching to other cc= modules may give some more insights. But again, I suspect that momentary (= microsecond) burstiness of BSD may be causing this significantly higher los= s rate.=94<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal">> Is there some info somewhere where I can unders= tand more on this in detail?<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">What detail? How to toggle to a different congestion= control?<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">How having high timer granularity pacing (as what RA= CK does) can prevent microbursts?<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">That TCP can, at high transmission rates, overrun th= e NIC driver memory buffers?<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">What did your experiments with the suggestions by @c= c show with the base stack?<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">And again, the RACK stack does HyStart and high-gran= ularity pacing, addressing many issues which are important to smooth stream= ing workflows and preventing momentary burst loads. However, the RACK stack= certainly needs to be validated with particular applications such as the kernel NFS server/client, which are de= eply embedded and may rely on some things in the base stack to work properl= y. I suspect that until now, noone has done a thorough investigation of the= implications of RACK with NFS really.<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">Best regards,<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"> Richard<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <p class=3D"MsoNormal">> <o:p></o:p></p> </div> <div> <p class=3D"MsoNormal">> Regards<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal">> Murali<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </div> </body> </html> --_000_PH0PR05MB1006405C3E6C421E99CF6E853FB29APH0PR05MB10064na_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?PH0PR05MB1006405C3E6C421E99CF6E853FB29A>