Date: Mon, 14 Aug 2023 14:32:08 +0300 From: Odhiambo Washington <odhiambo@gmail.com> To: freebsd-virtualization@freebsd.org Subject: Re: bhyve VM not getting as much upload bandwidth as the host Message-ID: <CAAdA2WO0mNK0mSz--=XBNLK4q=d1Ubh%2B0bh5HbUVMbbSA1gbOQ@mail.gmail.com> In-Reply-To: <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de> References: <CAAdA2WN6FTT=Bn-p3EEGDk_-8=g_PoFicUb%2Be-ybcpKxjY5bwQ@mail.gmail.com> <096b2d73-b28c-bd6b-a6d3-2a3c3dbaea7e@h4ck.io> <CAAdA2WM8Qrro-md-Rg%2BXfj5BxNrP_A8chD6TGrxQD2G14oC86g@mail.gmail.com> <38587cf9-29a7-1246-3b64-a3cf35933717@quip.cz> <96160FBF-4E48-48E3-B26C-026CEB5B03C2@punkt.de>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Mon, Aug 14, 2023 at 12:38 PM Patrick M. Hausen <hausen@punkt.de> wrote: > Hi all, > > > Am 14.08.2023 um 11:30 schrieb Miroslav Lachman <000.fbsd@quip.cz>: > > > > On 14/08/2023 10:49, Odhiambo Washington wrote: > >> I had the following in sysctl.conf: > >> net.link.tap.up_on_open=1 > >> net.link.bridge.pfil_member=0 > >> net.link.bridge.pfil_bridge=0 > >> net.link.bridge.pfil_local_phys=1 > >> So I only did: > >> sysctl -w net.link.bridge.pfil_onlyip=0 > >> sysctl -w net.link.bridge.pfil_local_phys=0 > >> Nothing has changed for the linux VM. > >> Windows11 VM is getting download/upload speed of 40Mbps/37Mbps while a > Debian12 VM is getting download/upload of 37Mbps/45Kbps. > >> Maybe there is an issue with the Linux itself? > > > > I never had this solved. Even my FreeBSD guest on FreeBSD host with > VirtualBox is slow as few hunderds kB/s > > It was like 10Mbps with Bhyve. I only use VMs for testing but installing > packages is always so slow. So you are not alone. I would really like to > know how to improve the network speed with virtualized environment. > > This looks weird to me. I run lots of VMs in production on TrueNAS CORE > - essentially FreeBSD 13-STABLE with bhyve and all of them get near gigabit > speed with bridged networking. > > Guests: > > Windows > Ubuntu > FreeBSD (OPNsense) > > Specifically the OPNsense VM can route 700-800 Mbit/s across gigabit > interfaces. > > All my VMs use VirtIO network drivers - do yours? > > Odhiambo, another minor thing: > > > ifconfig_em1="inet w.x.y.z netmask 255.255.255.0 mtu 1492 -tso -lro > -txcsum -rxcsum" > > A bridge member interface must not have a layer 3 address. You must put > the IP address > on the bridge interface itself and only configure > > ifconfig_em1="mtu 1492 -tso -lro -txcsum -rxcsum up" > em1 is NOT a bridge member. It's the host's interface that is connected to the Internet. So the suggestion by @Wanpeng was "you have to turn off TSO LRO TXCSUM etc for the host interface which is bridge to VM", which I understood as my main interface. My bridge configuration is as follows: cloned_interfaces="bridge0 tap0 tap1 tap2 tap3" ifconfig_bridge0_name="em1bridge" ifconfig_em1bridge="addm em1 addm tap0 addm tap1 addm tap2 addm tap3 up" ifconfig_tap0="inet 172.16.1.1/24" ifconfig_tap1="inet 172.16.2.1/24" ifconfig_tap2="inet 172.16.3.1/24" ifconfig_tap3="inet 172.16.4.1/24" When I create a VM and tie it on tap0, I give the VM an IP like 172.16.1.10/24, with a gateway of 172.16.1.1. But this is most probably not connected to your performance problem. It > just breaks > other things if you have an IP address on a bridge member ;-) > So is the setup above problematic or I am on the right track? > Why are you lowering the MTU of em1? I don't quite remember why I did that as it's a relic from the past :) > Does that carry over to the bridge interface? > Of course, because em1 is a bridge member, no? > What's the MTU of the emulated interface in your Linux VM? > Removing the MTU change on em1 has resolved the problem. The Debian12 VM now gets download/upload of 45Mbps/39Mbps! One last question for today (although I should just go and RTFM): Do I really need several tap devices? Can't I just have all my VMs on tap0? Each with it's own IP in that range? -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html] [-- Attachment #2 --] <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 14, 2023 at 12:38 PM Patrick M. Hausen <<a href="mailto:hausen@punkt.de" target="_blank">hausen@punkt.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br> <br> > Am 14.08.2023 um 11:30 schrieb Miroslav Lachman <<a href="mailto:000.fbsd@quip.cz" target="_blank">000.fbsd@quip.cz</a>>:<br> > <br> > On 14/08/2023 10:49, Odhiambo Washington wrote:<br> >> I had the following in sysctl.conf:<br> >> net.link.tap.up_on_open=1<br> >> net.link.bridge.pfil_member=0<br> >> net.link.bridge.pfil_bridge=0<br> >> net.link.bridge.pfil_local_phys=1<br> >> So I only did:<br> >> sysctl -w net.link.bridge.pfil_onlyip=0<br> >> sysctl -w net.link.bridge.pfil_local_phys=0<br> >> Nothing has changed for the linux VM.<br> >> Windows11 VM is getting download/upload speed of 40Mbps/37Mbps while a Debian12 VM is getting download/upload of 37Mbps/45Kbps.<br> >> Maybe there is an issue with the Linux itself?<br> > <br> > I never had this solved. Even my FreeBSD guest on FreeBSD host with VirtualBox is slow as few hunderds kB/s<br> > It was like 10Mbps with Bhyve. I only use VMs for testing but installing packages is always so slow. So you are not alone. I would really like to know how to improve the network speed with virtualized environment.<br> <br> This looks weird to me. I run lots of VMs in production on TrueNAS CORE<br> - essentially FreeBSD 13-STABLE with bhyve and all of them get near gigabit<br> speed with bridged networking.<br> <br> Guests:<br> <br> Windows<br> Ubuntu<br> FreeBSD (OPNsense)<br> <br> Specifically the OPNsense VM can route 700-800 Mbit/s across gigabit interfaces.<br> <br> All my VMs use VirtIO network drivers - do yours?<br> <br> Odhiambo, another minor thing:<br> <br> > ifconfig_em1="inet w.x.y.z netmask 255.255.255.0 mtu 1492 -tso -lro -txcsum -rxcsum"<br> <br> A bridge member interface must not have a layer 3 address. You must put the IP address<br> on the bridge interface itself and only configure<br> <br> ifconfig_em1="mtu 1492 -tso -lro -txcsum -rxcsum up"<br></blockquote><div><br></div><div>em1 is NOT a bridge member. It's the host's interface that is connected to the Internet.</div><div>So the suggestion by @Wanpeng was "you have to turn off TSO LRO TXCSUM etc for the host interface which is bridge to VM", which I understood</div><div>as my main interface.</div><div>My bridge configuration is as follows:</div><div>cloned_interfaces="bridge0 tap0 tap1 tap2 tap3"<br>ifconfig_bridge0_name="em1bridge"<br>ifconfig_em1bridge="addm em1 addm tap0 addm tap1 addm tap2 addm tap3 up"<br>ifconfig_tap0="inet <a href="http://172.16.1.1/24" target="_blank">172.16.1.1/24</a>"<br>ifconfig_tap1="inet <a href="http://172.16.2.1/24" target="_blank">172.16.2.1/24</a>"<br>ifconfig_tap2="inet <a href="http://172.16.3.1/24" target="_blank">172.16.3.1/24</a>"<br>ifconfig_tap3="inet <a href="http://172.16.4.1/24" target="_blank">172.16.4.1/24</a>"<br></div><div><br></div><div>When I create a VM and tie it on tap0, I give the VM an IP like <a href="http://172.16.1.10/24" target="_blank">172.16.1.10/24</a>, with a gateway of 172.16.1.1.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> But this is most probably not connected to your performance problem. It just breaks<br> other things if you have an IP address on a bridge member ;-)<br></blockquote><div><br></div><div>So is the setup above problematic or I am on the right track?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Why are you lowering the MTU of em1? </blockquote><div>I don't quite remember why I did that as it's a relic from the past :)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Does that carry over to the bridge interface?<br></blockquote><div><br></div><div>Of course, because em1 is a bridge member, no?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> What's the MTU of the emulated interface in your Linux VM?<br></blockquote><div><br></div><div>Removing the MTU change on em1 has resolved the problem. The Debian12 VM now gets download/upload of 45Mbps/39Mbps!</div><div><br></div><div> One last question for today (although I should just go and RTFM): Do I really need several tap devices? Can't I just have all my VMs on tap0? Each with it's own IP in that range?</div></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div>Best regards,<br>Odhiambo WASHINGTON,<br>Nairobi,KE<br>+254 7 3200 0004/+254 7 2274 3223<br>"<span style="font-size:12.8px">Oh, the cruft.</span><span style="font-size:12.8px">", </span><span style="font-size:12.8px">egrep -v '^$|^.*#' </span><span style="background-color:rgb(34,34,34);color:rgb(238,238,238);font-family:"Lucida Console",Consolas,"Courier New",monospace;font-size:13.6px">¯\_(ツ)_/¯</span><span style="font-size:12.8px"> :-)</span></div><div><span style="font-size:12.8px">[How to ask smart questions: </span><span style="font-size:12.8px"><a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a>]</span></div></div></div></div></div>;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAdA2WO0mNK0mSz--=XBNLK4q=d1Ubh%2B0bh5HbUVMbbSA1gbOQ>