Date: Wed, 1 Feb 2023 15:54:09 -0700 From: David <2yt@gmx.com> To: freebsd-stable@freebsd.org Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? Message-ID: <cebab4bd-ff1e-fb8d-cdc2-8720da9fd31a@gmx.com> In-Reply-To: <BE435158-34CD-4A54-870E-481A6646D4DE@gromit.dlib.vt.edu> References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <4ed8b724-041f-f561-ae60-ab966aefbb68@plan-b.pwste.edu.pl> <282AF730-E5E0-4A50-9F47-E7301B36E5C8@gromit.dlib.vt.edu> <2ed582b9-b544-74bb-2047-99d04924b46b@plan-b.pwste.edu.pl> <8AE3B49C-6C7F-4A20-B2DC-0D4B1343FB59@gromit.dlib.vt.edu> <83E43236-60F8-4949-8840-54E66D327EE9@gromit.dlib.vt.edu> <d890e3c5-9570-80ed-15b4-207e47ecc614@plan-b.pwste.edu.pl> <BE435158-34CD-4A54-870E-481A6646D4DE@gromit.dlib.vt.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2/1/23 14:07, Paul Mather wrote: > On Feb 1, 2023, at 3:14 PM, Marek Zarychta > <zarychtam@plan-b.pwste.edu.pl> wrote: > >> W dniu 1.02.2023 o 20:33, Paul Mather pisze: >>> It looks like we may have a winner, folks. I built and enabled the extra TCP stacks and for the first time was able to max out my connection to the remote FreeBSD system. I get consistently higher throughput over the 15-hop WAN path to the remote FreeBSD system when using the RACK TCP stack than when using the default "freebsd" stack. >>> >>> Although the speeds are consistently higher when using the setting "net.inet.tcp.functions_default=rack", they are still variable. However, rather than the 3--4 MB/s I saw that kicked off this thread, I now average over 10 MB/s. >>> >>> I actually get the best results with "net.inet.tcp.functions_default=bbr" (having loaded tcp_bbr). That behaves very much like the Linux hosts in that speeds climb very quickly until it saturates the WAN connection. I get the same high speeds from the remote FreeBSD system using tcp_bbr as I do to the Linux hosts. I will stick with tcp_bbr for now as the default on my remote FreeBSD servers. It appears to put them on a par with Linux for this WAN link. >> >> Thanks for the feedback Paul. Please bear in mind that BBR 1 which is >> implemented in FreeBSD is not a fair[1] congestion control algorithm. >> Maybe in the future, we will have BBR v2 in the stack, but for now, I >> don't recommend using BBR, unless you want to act slightly as a hm.... >> network leecher. Maybe Linux hosts behave this way, maybe they have >> implemented BBR v2, I am not familiar with Linux TCP stack >> enhancements. On the other hand, tcp_rack(4) is performant, >> well-tested in the FreeBSD stack, considered fair and more acceptable >> for a fileserver, though not ideal, ie. probably more computationally >> expensive and still missing some features like TCP-MD5. >> >> [1] https://www.mdpi.com/1424-8220/21/12/4128 >> > > That is a fair and astute observation, Marek. I am also not familiar > with Linux TCP stack implementations but it had occurred to me that > maybe Linux was not being an entirely good netizen whereas FreeBSD was > behaving with impeccable net manners when it came to congestion control > and being fair to others, and that is why Linux was getting faster > speeds for me. Then again, perhaps not. :-) > > In the case of the remote FreeBSD hosts I use at $JOB, they have low > numbers of users and so are more akin to endpoints than servers, so I'm > not worried about "leeching" from them. Also, my ISP download bandwidth > is 1/5th of each FreeBSD system, so hopefully there is still plenty to > go around after I max out my bulk downloads. (Plus, I believe $JOB > prefers my downloads to take half [or less] the time.) :-) > > Hopefully we will get BBR v2 (or something even fairer) at some point. > IIRC, the FreeBSD Foundation has been highlighting some of this > network stack work. It would be a pity for it not to be enabled by > default so more people could use it on -RELEASE without building a > custom kernel. I'm just glad right now I'm not stuck with 3--4 MB/s > downloads any more. > > Cheers, > > Paul. > Word of caution: It would appear not all FreeBSD applications like BBR or RACK. I run a Magento (e-commerce) VM and was getting weird pauses (hang for a bit then resume) on the website. Running Magneto requires several other TCP services and something wasn't happy. Not going to debug the problem, just wanted to give a heads up. -- David
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?cebab4bd-ff1e-fb8d-cdc2-8720da9fd31a>