Date: Thu, 30 Jun 2022 19:47:08 -0700 From: John-Mark Gurney <jmg@funkthat.com> To: Nils Beyer <nbe@vkf-renzel.de> Cc: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: slow network throughput between guest VM and host (and vice versa)? Message-ID: <20220701024708.GC88842@funkthat.com> In-Reply-To: <c68c6a34-61f7-48a8-19b8-43a253d3826d@vkf-renzel.de> References: <c68c6a34-61f7-48a8-19b8-43a253d3826d@vkf-renzel.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Nils Beyer wrote this message on Thu, Jun 30, 2022 at 12:45 +0200:
> Hi,
>
> I've setup a FreeBSD VM and am using a VirtIO network interface within. Then I've setup
> an ip address 192.168.0.2/30 to that VirtIO NIC of the guest using:
>
> ifconfig vtnet0 192.168.0.2/30 up
>
> and on the TAP interface of the host an ip address 192.168.0.1/30 using
>
> ifconfig tap0 192.168.0.1/30 up
>
> Trying an iperf3-transfer (iperf3-server on the host, iperf3-client on the guest) with a TCP window size of
> 128k I only get around 2.45Gbit/s:
>
> > # env LD_LIBRARY_PATH=. ./iperf3 -c 192.168.0.1 -w 128k
> > Connecting to host 192.168.0.1, port 5201
> > [ 5] local 192.168.0.2 port 25651 connected to 192.168.0.1 port 5201
> > [ ID] Interval Transfer Bitrate Retr Cwnd
> > [ 5] 0.00-1.00 sec 314 MBytes 2.63 Gbits/sec 0 1.43 MBytes
> > [ 5] 1.00-2.00 sec 301 MBytes 2.52 Gbits/sec 0 1.43 MBytes
> > [ 5] 2.00-3.00 sec 264 MBytes 2.21 Gbits/sec 0 1.43 MBytes
> > [ 5] 3.00-4.00 sec 284 MBytes 2.38 Gbits/sec 0 1.43 MBytes
> > [ 5] 4.00-5.00 sec 296 MBytes 2.48 Gbits/sec 0 1.43 MBytes
> > [ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec 0 1.43 MBytes
> > [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 1.43 MBytes
> > [ 5] 7.00-8.00 sec 310 MBytes 2.60 Gbits/sec 0 1.43 MBytes
> > [ 5] 8.00-9.00 sec 302 MBytes 2.53 Gbits/sec 0 1.43 MBytes
> > [ 5] 9.00-10.00 sec 333 MBytes 2.79 Gbits/sec 0 1.43 MBytes
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval Transfer Bitrate Retr
> > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec 0 sender
> > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec receiver
>
>
> Switching the roles (iperf3-server on the guest, iperf3-client on the host) with a TCP windows size of 128k,
> I get 4.04Gbit/s:
>
> > #iperf3 -c 192.168.0.2 -w 128k
> > Connecting to host 192.168.0.2, port 5201
> > [ 5] local 192.168.0.1 port 56892 connected to 192.168.0.2 port 5201
> > [ ID] Interval Transfer Bitrate Retr Cwnd
> > [ 5] 0.00-1.00 sec 411 MBytes 3.44 Gbits/sec 40 973 KBytes
> > [ 5] 1.00-2.00 sec 483 MBytes 4.05 Gbits/sec 5 1.03 MBytes
> > [ 5] 2.00-3.00 sec 507 MBytes 4.26 Gbits/sec 1 1.21 MBytes
> > [ 5] 3.00-4.00 sec 514 MBytes 4.31 Gbits/sec 15 561 KBytes
> > [ 5] 4.00-5.00 sec 498 MBytes 4.18 Gbits/sec 10 966 KBytes
> > [ 5] 5.00-6.00 sec 491 MBytes 4.12 Gbits/sec 19 841 KBytes
> > [ 5] 6.00-7.00 sec 513 MBytes 4.31 Gbits/sec 0 1.43 MBytes
> > [ 5] 7.00-8.00 sec 504 MBytes 4.23 Gbits/sec 0 1.43 MBytes
> > [ 5] 8.00-9.00 sec 459 MBytes 3.85 Gbits/sec 0 1.43 MBytes
> > [ 5] 9.00-10.00 sec 435 MBytes 3.65 Gbits/sec 0 1.43 MBytes
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval Transfer Bitrate Retr
> > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec 90 sender
> > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec receiver
>
>
> Increasing MTU on the Virtio interface and on the TAP interface to 9000 helps a little bit:
> getting 8.38Gbit/s guest->host and 10.3Gbit/s host->guest.
>
> Increasing TCP windows size to 1024k only produces more retries and does nothing on the
> throughput.
>
> Is that expected that I'm not able to get more throughput within the bhyve network
> stack (guest <-> host)? I was expecting way more then 10Gbit/s...
What does CPU usage look like on the host and in the VM? This sounds
like hitting a CPU bandwidth limit in that you're limited by pps...
--
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20220701024708.GC88842>