Date: Tue, 31 Jan 2023 19:27:42 -0500 From: Mark Saad <nonesuch@longcount.org> To: Paul Mather <paul@gromit.dlib.vt.edu> Cc: David <2yt@gmx.com>, freebsd-stable@freebsd.org Subject: Re: Slow WAN traffic to FreeBSD hosts but not to Linux hosts---how to debug/fix? Message-ID: <CAMXt9NY_u5Qvf%2BevhbE%2BD4Z_pueLdFME_0sHO1VU_XxCihSn_A@mail.gmail.com> In-Reply-To: <A3E14937-0CC7-4C4F-ABAE-7DA37CDD7BE7@gromit.dlib.vt.edu> References: <95EDCFCA-7E3F-458F-85A6-856D606B9D98@gromit.dlib.vt.edu> <tr9g44$d3l$1@ciao.gmane.io> <CANwXMPNBkWQHzT4f%2Bjy0RV1gK7R6nJOWYsG5NZTf9RadRDNfXg@mail.gmail.com> <ee8df505-cd76-41fb-25a6-cb5c67dc5d8c@gmx.com> <A3E14937-0CC7-4C4F-ABAE-7DA37CDD7BE7@gromit.dlib.vt.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000e1ae7205f39884bb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jan 31, 2023 at 5:03 PM Paul Mather <paul@gromit.dlib.vt.edu> wrote= : > On Jan 31, 2023, at 10:39 AM, David <2yt@gmx.com> wrote: > > > On 1/30/23 16:30, Matt Garber wrote: > >> > Any help/insight is gratefully appreciated. > >> > > >> > Cheers, > >> > > >> > Paul. > >> > > >> sysctl net.inet.tcp.cc.algorithm=3Dhtcp > >> I would set "htcp" on the server and home computer to improve > >> through in > >> your type of situation. > >> There may be other FreeBSD sysctls that have bad defaults in this > scenario and could be better tuned, but I doubt changing the CC algorithm > at this point is the problem =E2=80=94 at least not so much a problem tha= t=E2=80=99s > causing throughput to be reduced so drastically. Happy to be wrong if tha= t > does help things quickly and easily, though. > >> (Since OP mentioned that FreeBSD CC was set to CUBIC, that would match > what the Linux boxes are using by default, too, unless they=E2=80=99ve be= en changed > to something newer like BBR=E2=80=A6 so seems like CUBIC *should* be perf= orming > fine on this WAN link, and the difference is something else.) > >> =E2=80=94Matt > > > > I love FreeBSD and very much appreciate the efforts of those people muc= h > smarter. But... I don't think the defaults get enough testing in real wor= ld > conditions. > > > > I came across Paul's issue several years ago and spent a few a hours > testing and found the defaults performed very well on a LAN but could > perform terribly on a many hop WAN. HTCP performs marginally worse on a L= AN > or close WAN connection, but much much better on a many hop WAN connectio= n. > > > [[...]] > > > In my opinion HTCP is a better default for the current state of the > internet. > > > It looks like they already changed the default from NewReno to CUBIC in > FreeBSD-CURRENT. > > I agree with your observation about the defaults vs. real world > conditions. As you observed, I also get great performance in a high-spee= d > LAN, but not so much in a many-hop WAN to an asymmetric ISP connection. > > I actually started down this rabbit hole when I noticed I couldn't manage > more than about 3--4 MB in a single stream and thought my ISP was > throttling me. But then I noticed I would actually get fast/maximum > speeds, e.g., when doing "brew upgrade -v" and Homebrew would be > downloading packages, and so then wondered whether they were throttling > non-HTTP traffic. That led me to discover that even HTTP downloads were > slow to the FreeBSD servers I use remotely at $JOB and, furthermore, that > all traffic to Linux systems I use at $JOB didn't seem to be throttled or > incapable of getting maximum single stream bandwidth matching my ISP's > quoted speeds. :-\ > > I accept that this may just be a peculiarity of my local and remote setup= , > and so appreciate the help and suggestions people have offered in trying = to > debug the issue. > > Cheers, > > Paul. > Paul I was reading this and I wanted to just add two bits, one congestion control only really affects tcp sockets, so if you are using a freebsd router, setting the cc algo here does not do all that much. Second thing, the "ergonomics" of the tcp tunings on FreeBSD vs Redhat are very different. . IMHO try aligning the two boxes tcp tunings and also make sure the network cards used in each server are also aligned. IE do you have LRO enabled on one server and not the other. Is one server using an Intel 10G nic vs a Chelsio 10G nic ? --=20 mark saad | nonesuch@longcount.org --000000000000e1ae7205f39884bb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Tue, Jan 31, 2023 at 5:03 PM Paul = Mather <<a href=3D"mailto:paul@gromit.dlib.vt.edu">paul@gromit.dlib.vt.e= du</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi= n:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex= ">On Jan 31, 2023, at 10:39 AM, David <<a href=3D"mailto:2yt@gmx.com" ta= rget=3D"_blank">2yt@gmx.com</a>> wrote:<br> <br> > On 1/30/23 16:30, Matt Garber wrote:<br> >>=C2=A0 =C2=A0 =C2=A0> Any help/insight is gratefully appreciated= .<br> >>=C2=A0 =C2=A0 =C2=A0><br> >>=C2=A0 =C2=A0 =C2=A0> Cheers,<br> >>=C2=A0 =C2=A0 =C2=A0><br> >>=C2=A0 =C2=A0 =C2=A0> Paul.<br> >>=C2=A0 =C2=A0 =C2=A0><br> >>=C2=A0 =C2=A0 sysctl net.inet.tcp.cc.algorithm=3Dhtcp<br> >>=C2=A0 =C2=A0 I would set "htcp" on the server and home c= omputer to improve<br> >>=C2=A0 =C2=A0 through in<br> >>=C2=A0 =C2=A0 your type of situation.<br> >> There may be other FreeBSD sysctls that have bad defaults in this = scenario and could be better tuned, but I doubt changing the CC algorithm a= t this point is the problem =E2=80=94 at least not so much a problem that= =E2=80=99s causing throughput to be reduced so drastically. Happy to be wro= ng if that does help things quickly and easily, though.<br> >> (Since OP mentioned that FreeBSD CC was set to CUBIC, that would m= atch what the Linux boxes are using by default, too, unless they=E2=80=99ve= been changed to something newer like BBR=E2=80=A6 so seems like CUBIC *sho= uld* be performing fine on this WAN link, and the difference is something e= lse.)<br> >> =E2=80=94Matt<br> > <br> > I love FreeBSD and very much appreciate the efforts of those people mu= ch smarter. But... I don't think the defaults get enough testing in rea= l world conditions.<br> > <br> > I came across Paul's issue several years ago and spent a few a hou= rs testing and found the defaults performed very well on a LAN but could pe= rform terribly on a many hop WAN. HTCP performs marginally worse on a LAN o= r close WAN connection, but much much better on a many hop WAN connection.<= br> > <br> [[...]]<br> <br> > In my opinion HTCP is a better default for the current state of the in= ternet.<br> <br> <br> It looks like they already changed the default from NewReno to CUBIC in Fre= eBSD-CURRENT.<br> <br> I agree with your observation about the defaults vs. real world conditions.= =C2=A0 As you observed, I also get great performance in a high-speed LAN, b= ut not so much in a many-hop WAN to an asymmetric ISP connection.<br> <br> I actually started down this rabbit hole when I noticed I couldn't mana= ge more than about 3--4 MB in a single stream and thought my ISP was thrott= ling me.=C2=A0 But then I noticed I would actually get fast/maximum speeds,= e.g., when doing "brew upgrade -v" and Homebrew would be downloa= ding packages, and so then wondered whether they were throttling non-HTTP t= raffic.=C2=A0 That led me to discover that even HTTP downloads were slow to= the FreeBSD servers I use remotely at $JOB and, furthermore, that all traf= fic to Linux systems I use at $JOB didn't seem to be throttled or incap= able of getting maximum single stream bandwidth matching my ISP's quote= d speeds. :-\<br> <br> I accept that this may just be a peculiarity of my local and remote setup, = and so appreciate the help and suggestions people have offered in trying to= debug the issue.<br> <br> Cheers,<br> <br> Paul.<br> </blockquote></div><div>Paul</div><div>=C2=A0 I was reading this and I want= ed to just add two bits, one congestion control only really affects tcp soc= kets, so if you are using a freebsd router, setting the cc algo here does n= ot do all that much. Second thing, the "ergonomics" of the tcp tu= nings on FreeBSD vs Redhat are very different.=C2=A0 .=C2=A0 IMHO try align= ing the two boxes tcp tunings and also make sure the network cards used in = each server are also aligned. IE do you have LRO enabled on one server and = not the other. Is one=C2=A0 server using an Intel 10G nic vs a Chelsio 10G = nic ? <br></div><div><br>-- <br><div dir=3D"ltr" class=3D"gmail_signature">= mark saad | <a href=3D"mailto:nonesuch@longcount.org" target=3D"_blank">non= esuch@longcount.org</a><br></div></div></div> --000000000000e1ae7205f39884bb--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMXt9NY_u5Qvf%2BevhbE%2BD4Z_pueLdFME_0sHO1VU_XxCihSn_A>