Date: Wed, 1 Feb 2023 16:07:15 -0500 From: Paul Mather <paul@gromit.dlib.vt.edu> To: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> Cc: stable@freebsd.org Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? Message-ID: <BE435158-34CD-4A54-870E-481A6646D4DE@gromit.dlib.vt.edu> In-Reply-To: <d890e3c5-9570-80ed-15b4-207e47ecc614@plan-b.pwste.edu.pl> References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <4ed8b724-041f-f561-ae60-ab966aefbb68@plan-b.pwste.edu.pl> <282AF730-E5E0-4A50-9F47-E7301B36E5C8@gromit.dlib.vt.edu> <2ed582b9-b544-74bb-2047-99d04924b46b@plan-b.pwste.edu.pl> <8AE3B49C-6C7F-4A20-B2DC-0D4B1343FB59@gromit.dlib.vt.edu> <83E43236-60F8-4949-8840-54E66D327EE9@gromit.dlib.vt.edu> <d890e3c5-9570-80ed-15b4-207e47ecc614@plan-b.pwste.edu.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_233E2C73-6AEB-4008-BE76-E608477332C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Feb 1, 2023, at 3:14 PM, Marek Zarychta = <zarychtam@plan-b.pwste.edu.pl> wrote: > W dniu 1.02.2023 o 20:33, Paul Mather pisze: >> It looks like we may have a winner, folks. I built and enabled the = extra TCP stacks and for the first time was able to max out my = connection to the remote FreeBSD system. I get consistently higher = throughput over the 15-hop WAN path to the remote FreeBSD system when = using the RACK TCP stack than when using the default "freebsd" stack. >>=20 >> Although the speeds are consistently higher when using the setting = "net.inet.tcp.functions_default=3Drack", they are still variable. = However, rather than the 3--4 MB/s I saw that kicked off this thread, I = now average over 10 MB/s. >>=20 >> I actually get the best results with = "net.inet.tcp.functions_default=3Dbbr" (having loaded tcp_bbr). That = behaves very much like the Linux hosts in that speeds climb very quickly = until it saturates the WAN connection. I get the same high speeds from = the remote FreeBSD system using tcp_bbr as I do to the Linux hosts. I = will stick with tcp_bbr for now as the default on my remote FreeBSD = servers. It appears to put them on a par with Linux for this WAN link. > Thanks for the feedback Paul. Please bear in mind that BBR 1 which is = implemented in FreeBSD is not a fair[1] congestion control algorithm. = Maybe in the future, we will have BBR v2 in the stack, but for now, I = don't recommend using BBR, unless you want to act slightly as a hm.... = network leecher. Maybe Linux hosts behave this way, maybe they have = implemented BBR v2, I am not familiar with Linux TCP stack enhancements. > On the other hand, tcp_rack(4) is performant, well-tested in the = FreeBSD stack, considered fair and more acceptable for a fileserver, = though not ideal, ie. probably more computationally expensive and still = missing some features like TCP-MD5. >=20 >=20 > [1] https://www.mdpi.com/1424-8220/21/12/4128 >=20 That is a fair and astute observation, Marek. I am also not familiar = with Linux TCP stack implementations but it had occurred to me that = maybe Linux was not being an entirely good netizen whereas FreeBSD was = behaving with impeccable net manners when it came to congestion control = and being fair to others, and that is why Linux was getting faster = speeds for me. Then again, perhaps not. :-) In the case of the remote FreeBSD hosts I use at $JOB, they have low = numbers of users and so are more akin to endpoints than servers, so I'm = not worried about "leeching" from them. Also, my ISP download bandwidth = is 1/5th of each FreeBSD system, so hopefully there is still plenty to = go around after I max out my bulk downloads. (Plus, I believe $JOB = prefers my downloads to take half [or less] the time.) :-) Hopefully we will get BBR v2 (or something even fairer) at some point. = IIRC, the FreeBSD Foundation has been highlighting some of this network = stack work. It would be a pity for it not to be enabled by default so = more people could use it on -RELEASE without building a custom kernel. = I'm just glad right now I'm not stuck with 3--4 MB/s downloads any more. Cheers, Paul. --Apple-Mail=_233E2C73-6AEB-4008-BE76-E608477332C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;">On Feb 1, = 2023, at 3:14 PM, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> = wrote:<br><div><br><blockquote type=3D"cite">W dniu 1.02.2023 = o 20:33, Paul Mather pisze:<br><div><div> <blockquote type=3D"cite" = cite=3D"mid:83E43236-60F8-4949-8840-54E66D327EE9@gromit.dlib.vt.edu"> <pre class=3D"moz-quote-pre" wrap=3D"">It looks like we may have a = winner, folks. I built and enabled the extra TCP stacks and for the = first time was able to max out my connection to the remote FreeBSD = system. I get consistently higher throughput over the 15-hop WAN path = to the remote FreeBSD system when using the RACK TCP stack than when = using the default "freebsd" stack. Although the speeds are consistently higher when using the setting = "net.inet.tcp.functions_default=3Drack", they are still variable. = However, rather than the 3--4 MB/s I saw that kicked off this thread, I = now average over 10 MB/s. I actually get the best results with = "net.inet.tcp.functions_default=3Dbbr" (having loaded tcp_bbr). That = behaves very much like the Linux hosts in that speeds climb very quickly = until it saturates the WAN connection. I get the same high speeds from = the remote FreeBSD system using tcp_bbr as I do to the Linux hosts. I = will stick with tcp_bbr for now as the default on my remote FreeBSD = servers. It appears to put them on a par with Linux for this WAN link. </pre> </blockquote><p><span style=3D"white-space: pre-wrap; display: = block; width: 98vw;">Thanks for the feedback Paul. Please bear in mind = that BBR 1 which is implemented in FreeBSD is not a fair[1] congestion = control algorithm. Maybe in the future, we will have BBR v2 in the = stack, but for now, I don't recommend using BBR, unless you want to act = slightly as a hm.... network leecher. Maybe Linux hosts behave this way, = maybe they have implemented BBR v2, I am not familiar with Linux TCP = stack enhancements. On the other hand, tcp_rack(4) is performant, well-tested in the FreeBSD = stack, considered fair and more acceptable for a fileserver, though not = ideal, ie. probably more computationally expensive and still missing = some features like TCP-MD5.</span><br> <br> [1] <a class=3D"moz-txt-link-freetext" = href=3D"https://www.mdpi.com/1424-8220/21/12/4128">https://www.mdpi.com/14= 24-8220/21/12/4128</a></p></div></div></blockquote><br></div><div>That = is a fair and astute observation, Marek. I am also not familiar = with Linux TCP stack implementations but it had occurred to me that = maybe Linux was not being an entirely good netizen whereas FreeBSD was = behaving with impeccable net manners when it came to congestion control = and being fair to others, and that is why Linux was getting faster = speeds for me. Then again, perhaps not. = :-)</div><div><br></div><div>In the case of the remote FreeBSD hosts I = use at $JOB, they have low numbers of users and so are more akin to = endpoints than servers, so I'm not worried about "leeching" from them. = Also, my ISP download bandwidth is 1/5th of each FreeBSD system, = so hopefully there is still plenty to go around after I max out my bulk = downloads. (Plus, I believe $JOB prefers my downloads to take half = [or less] the time.) :-)</div><div><br></div><div>Hopefully we will get = BBR v2 (or something even fairer) at some point. IIRC, the FreeBSD = Foundation has been highlighting some of this network stack work. = It would be a pity for it not to be enabled by default so more = people could use it on -RELEASE without building a custom kernel. = I'm just glad right now I'm not stuck with 3--4 MB/s downloads any = more.</div><div><br></div><div>Cheers,</div><div><br></div><div>Paul.</div= ><br></body></html>= --Apple-Mail=_233E2C73-6AEB-4008-BE76-E608477332C2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BE435158-34CD-4A54-870E-481A6646D4DE>