Date: Thu, 30 Jun 2022 12:45:54 +0200 From: Nils Beyer <nbe@vkf-renzel.de> To: freebsd-virtualization@FreeBSD.org Subject: bhyve: slow network throughput between guest VM and host (and vice versa)? Message-ID: <c68c6a34-61f7-48a8-19b8-43a253d3826d@vkf-renzel.de>
next in thread | raw e-mail | index | archive | help
Hi, I've setup a FreeBSD VM and am using a VirtIO network interface within. Then I've setup an ip address 192.168.0.2/30 to that VirtIO NIC of the guest using: ifconfig vtnet0 192.168.0.2/30 up and on the TAP interface of the host an ip address 192.168.0.1/30 using ifconfig tap0 192.168.0.1/30 up Trying an iperf3-transfer (iperf3-server on the host, iperf3-client on the guest) with a TCP window size of 128k I only get around 2.45Gbit/s: > # env LD_LIBRARY_PATH=. ./iperf3 -c 192.168.0.1 -w 128k > Connecting to host 192.168.0.1, port 5201 > [ 5] local 192.168.0.2 port 25651 connected to 192.168.0.1 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 314 MBytes 2.63 Gbits/sec 0 1.43 MBytes > [ 5] 1.00-2.00 sec 301 MBytes 2.52 Gbits/sec 0 1.43 MBytes > [ 5] 2.00-3.00 sec 264 MBytes 2.21 Gbits/sec 0 1.43 MBytes > [ 5] 3.00-4.00 sec 284 MBytes 2.38 Gbits/sec 0 1.43 MBytes > [ 5] 4.00-5.00 sec 296 MBytes 2.48 Gbits/sec 0 1.43 MBytes > [ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec 0 1.43 MBytes > [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 1.43 MBytes > [ 5] 7.00-8.00 sec 310 MBytes 2.60 Gbits/sec 0 1.43 MBytes > [ 5] 8.00-9.00 sec 302 MBytes 2.53 Gbits/sec 0 1.43 MBytes > [ 5] 9.00-10.00 sec 333 MBytes 2.79 Gbits/sec 0 1.43 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec 0 sender > [ 5] 0.00-10.00 sec 2.89 GBytes 2.49 Gbits/sec receiver Switching the roles (iperf3-server on the guest, iperf3-client on the host) with a TCP windows size of 128k, I get 4.04Gbit/s: > #iperf3 -c 192.168.0.2 -w 128k > Connecting to host 192.168.0.2, port 5201 > [ 5] local 192.168.0.1 port 56892 connected to 192.168.0.2 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 411 MBytes 3.44 Gbits/sec 40 973 KBytes > [ 5] 1.00-2.00 sec 483 MBytes 4.05 Gbits/sec 5 1.03 MBytes > [ 5] 2.00-3.00 sec 507 MBytes 4.26 Gbits/sec 1 1.21 MBytes > [ 5] 3.00-4.00 sec 514 MBytes 4.31 Gbits/sec 15 561 KBytes > [ 5] 4.00-5.00 sec 498 MBytes 4.18 Gbits/sec 10 966 KBytes > [ 5] 5.00-6.00 sec 491 MBytes 4.12 Gbits/sec 19 841 KBytes > [ 5] 6.00-7.00 sec 513 MBytes 4.31 Gbits/sec 0 1.43 MBytes > [ 5] 7.00-8.00 sec 504 MBytes 4.23 Gbits/sec 0 1.43 MBytes > [ 5] 8.00-9.00 sec 459 MBytes 3.85 Gbits/sec 0 1.43 MBytes > [ 5] 9.00-10.00 sec 435 MBytes 3.65 Gbits/sec 0 1.43 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec 90 sender > [ 5] 0.00-10.00 sec 4.70 GBytes 4.04 Gbits/sec receiver Increasing MTU on the Virtio interface and on the TAP interface to 9000 helps a little bit: getting 8.38Gbit/s guest->host and 10.3Gbit/s host->guest. Increasing TCP windows size to 1024k only produces more retries and does nothing on the throughput. Is that expected that I'm not able to get more throughput within the bhyve network stack (guest <-> host)? I was expecting way more then 10Gbit/s... TIA and KR, Nils
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c68c6a34-61f7-48a8-19b8-43a253d3826d>