Date: Mon, 14 Aug 2023 14:32:08 +0300 From: Odhiambo Washington <odhiambo@gmail.com> To: freebsd-virtualization@freebsd.org Subject: Re: bhyve VM not getting as much upload bandwidth as the host Message-ID: <CAAdA2WO0mNK0mSz--=XBNLK4q=d1Ubh%2B0bh5HbUVMbbSA1gbOQ@mail.gmail.com> In-Reply-To: <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de> References: <CAAdA2WN6FTT=Bn-p3EEGDk_-8=g_PoFicUb%2Be-ybcpKxjY5bwQ@mail.gmail.com> <096b2d73-b28c-bd6b-a6d3-2a3c3dbaea7e@h4ck.io> <CAAdA2WM8Qrro-md-Rg%2BXfj5BxNrP_A8chD6TGrxQD2G14oC86g@mail.gmail.com> <38587cf9-29a7-1246-3b64-a3cf35933717@quip.cz> <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000c7b1450602e06b7e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Aug 14, 2023 at 12:38=E2=80=AFPM Patrick M. Hausen <hausen@punkt.de= > wrote: > Hi all, > > > Am 14.08.2023 um 11:30 schrieb Miroslav Lachman <000.fbsd@quip.cz>: > > > > On 14/08/2023 10:49, Odhiambo Washington wrote: > >> I had the following in sysctl.conf: > >> net.link.tap.up_on_open=3D1 > >> net.link.bridge.pfil_member=3D0 > >> net.link.bridge.pfil_bridge=3D0 > >> net.link.bridge.pfil_local_phys=3D1 > >> So I only did: > >> sysctl -w net.link.bridge.pfil_onlyip=3D0 > >> sysctl -w net.link.bridge.pfil_local_phys=3D0 > >> Nothing has changed for the linux VM. > >> Windows11 VM is getting download/upload speed of 40Mbps/37Mbps while a > Debian12 VM is getting download/upload of 37Mbps/45Kbps. > >> Maybe there is an issue with the Linux itself? > > > > I never had this solved. Even my FreeBSD guest on FreeBSD host with > VirtualBox is slow as few hunderds kB/s > > It was like 10Mbps with Bhyve. I only use VMs for testing but installin= g > packages is always so slow. So you are not alone. I would really like to > know how to improve the network speed with virtualized environment. > > This looks weird to me. I run lots of VMs in production on TrueNAS CORE > - essentially FreeBSD 13-STABLE with bhyve and all of them get near gigab= it > speed with bridged networking. > > Guests: > > Windows > Ubuntu > FreeBSD (OPNsense) > > Specifically the OPNsense VM can route 700-800 Mbit/s across gigabit > interfaces. > > All my VMs use VirtIO network drivers - do yours? > > Odhiambo, another minor thing: > > > ifconfig_em1=3D"inet w.x.y.z netmask 255.255.255.0 mtu 1492 -tso -lro > -txcsum -rxcsum" > > A bridge member interface must not have a layer 3 address. You must put > the IP address > on the bridge interface itself and only configure > > ifconfig_em1=3D"mtu 1492 -tso -lro -txcsum -rxcsum up" > em1 is NOT a bridge member. It's the host's interface that is connected to the Internet. So the suggestion by @Wanpeng was "you have to turn off TSO LRO TXCSUM etc for the host interface which is bridge to VM", which I understood as my main interface. My bridge configuration is as follows: cloned_interfaces=3D"bridge0 tap0 tap1 tap2 tap3" ifconfig_bridge0_name=3D"em1bridge" ifconfig_em1bridge=3D"addm em1 addm tap0 addm tap1 addm tap2 addm tap3 up" ifconfig_tap0=3D"inet 172.16.1.1/24" ifconfig_tap1=3D"inet 172.16.2.1/24" ifconfig_tap2=3D"inet 172.16.3.1/24" ifconfig_tap3=3D"inet 172.16.4.1/24" When I create a VM and tie it on tap0, I give the VM an IP like 172.16.1.10/24, with a gateway of 172.16.1.1. But this is most probably not connected to your performance problem. It > just breaks > other things if you have an IP address on a bridge member ;-) > So is the setup above problematic or I am on the right track? > Why are you lowering the MTU of em1? I don't quite remember why I did that as it's a relic from the past :) > Does that carry over to the bridge interface? > Of course, because em1 is a bridge member, no? > What's the MTU of the emulated interface in your Linux VM? > Removing the MTU change on em1 has resolved the problem. The Debian12 VM now gets download/upload of 45Mbps/39Mbps! One last question for today (although I should just go and RTFM): Do I really need several tap devices? Can't I just have all my VMs on tap0? Each with it's own IP in that range? --=20 Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' =C2=AF\_(=E3=83=84)_/=C2=AF :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html] --000000000000c7b1450602e06b7e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Mon, Aug 14, 2023 at 12:38=E2=80= =AFPM Patrick M. Hausen <<a href=3D"mailto:hausen@punkt.de" target=3D"_b= lank">hausen@punkt.de</a>> wrote:<br></div><blockquote class=3D"gmail_qu= ote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20= 4);padding-left:1ex">Hi all,<br> <br> > Am 14.08.2023 um 11:30 schrieb Miroslav Lachman <<a href=3D"mailto:= 000.fbsd@quip.cz" target=3D"_blank">000.fbsd@quip.cz</a>>:<br> > <br> > On 14/08/2023 10:49, Odhiambo Washington wrote:<br> >> I had the following in sysctl.conf:<br> >> net.link.tap.up_on_open=3D1<br> >> net.link.bridge.pfil_member=3D0<br> >> net.link.bridge.pfil_bridge=3D0<br> >> net.link.bridge.pfil_local_phys=3D1<br> >> So I only did:<br> >> sysctl -w net.link.bridge.pfil_onlyip=3D0<br> >> sysctl -w net.link.bridge.pfil_local_phys=3D0<br> >> Nothing has changed for the linux VM.<br> >> Windows11 VM is getting download/upload speed of 40Mbps/37Mbps whi= le a Debian12 VM is getting download/upload of 37Mbps/45Kbps.<br> >> Maybe there is an issue with the Linux itself?<br> > <br> > I never had this solved. Even my FreeBSD guest on FreeBSD host with Vi= rtualBox is slow as few hunderds kB/s<br> > It was like 10Mbps with Bhyve. I only use VMs for testing but installi= ng packages is always so slow. So you are not alone. I would really like to= know how to improve the network speed with virtualized environment.<br> <br> This looks weird to me. I run lots of VMs in production on TrueNAS CORE<br> - essentially FreeBSD 13-STABLE with bhyve and all of them get near gigabit= <br> speed with bridged networking.<br> <br> Guests:<br> <br> Windows<br> Ubuntu<br> FreeBSD (OPNsense)<br> <br> Specifically the OPNsense VM can route 700-800 Mbit/s across gigabit interf= aces.<br> <br> All my VMs use VirtIO network drivers - do yours?<br> <br> Odhiambo, another minor thing:<br> <br> > ifconfig_em1=3D"inet w.x.y.z netmask 255.255.255.0 mtu 1492 -tso = -lro -txcsum -rxcsum"<br> <br> A bridge member interface must not have a layer 3 address. You must put the= IP address<br> on the bridge interface itself and only configure<br> <br> ifconfig_em1=3D"mtu 1492 -tso -lro -txcsum -rxcsum up"<br></block= quote><div><br></div><div>em1 is NOT a bridge member. It's the host'= ;s interface that is connected to the Internet.</div><div>So the suggestion= by=C2=A0=C2=A0@Wanpeng was "you have to turn off TSO LRO TXCSUM etc f= or the host interface which is bridge to VM", which I understood</div>= <div>as my main interface.</div><div>My bridge configuration is as follows:= </div><div>cloned_interfaces=3D"bridge0 tap0 tap1 tap2 tap3"<br>i= fconfig_bridge0_name=3D"em1bridge"<br>ifconfig_em1bridge=3D"= addm em1 addm tap0 addm tap1 addm tap2 addm tap3 up"<br>ifconfig_tap0= =3D"inet <a href=3D"http://172.16.1.1/24" target=3D"_blank">172.16.1.1= /24</a>"<br>ifconfig_tap1=3D"inet <a href=3D"http://172.16.2.1/24= " target=3D"_blank">172.16.2.1/24</a>"<br>ifconfig_tap2=3D"inet <= a href=3D"http://172.16.3.1/24" target=3D"_blank">172.16.3.1/24</a>"<b= r>ifconfig_tap3=3D"inet <a href=3D"http://172.16.4.1/24" target=3D"_bl= ank">172.16.4.1/24</a>"<br></div><div><br></div><div>When I create a V= M and tie it on tap0, I give the VM an IP like <a href=3D"http://172.16.1.1= 0/24" target=3D"_blank">172.16.1.10/24</a>, with a gateway of 172.16.1.1.</= div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> But this is most probably not connected to your performance problem. It jus= t breaks<br> other things if you have an IP address on a bridge member ;-)<br></blockquo= te><div><br></div><div>So is the setup above problematic or I am on the rig= ht track?</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"= margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef= t:1ex"> Why are you lowering the MTU of em1? </blockquote><div>I don't quite re= member why I did that as it's a relic from the past :)</div><div>=C2=A0= </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b= order-left:1px solid rgb(204,204,204);padding-left:1ex">Does that carry ove= r to the bridge interface?<br></blockquote><div><br></div><div>Of course, b= ecause em1 is a bridge member, no?</div><div>=C2=A0</div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex"> What's the MTU of the emulated interface in your Linux VM?<br></blockqu= ote><div><br></div><div>Removing the MTU change on em1 has resolved the pro= blem. The Debian12 VM now gets download/upload of 45Mbps/39Mbps!</div><div>= <br></div><div>=C2=A0One last question for today (although I should just go= and RTFM): Do I really need several tap devices? Can't I just have all= my VMs on tap0? Each with it's own IP in that range?</div></div><div><= br></div><span class=3D"gmail_signature_prefix">-- </span><br><div dir=3D"l= tr" class=3D"gmail_signature"><div dir=3D"ltr"><div dir=3D"ltr"><div>Best r= egards,<br>Odhiambo WASHINGTON,<br>Nairobi,KE<br>+254 7 3200 0004/+254 7 22= 74 3223<br>"<span style=3D"font-size:12.8px">Oh, the cruft.</span><spa= n style=3D"font-size:12.8px">",=C2=A0</span><span style=3D"font-size:1= 2.8px">egrep -v '^$|^.*#'=C2=A0</span><span style=3D"background-col= or:rgb(34,34,34);color:rgb(238,238,238);font-family:"Lucida Console&qu= ot;,Consolas,"Courier New",monospace;font-size:13.6px">=C2=AF\_(= =E3=83=84)_/=C2=AF</span><span style=3D"font-size:12.8px">=C2=A0:-)</span><= /div><div><span style=3D"font-size:12.8px">[How to ask smart questions:=C2= =A0</span><span style=3D"font-size:12.8px"><a href=3D"http://www.catb.org/~= esr/faqs/smart-questions.html" target=3D"_blank">http://www.catb.org/~esr/f= aqs/smart-questions.html</a>]</span></div></div></div></div></div> --000000000000c7b1450602e06b7e--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAdA2WO0mNK0mSz--=XBNLK4q=d1Ubh%2B0bh5HbUVMbbSA1gbOQ>