Date: Mon, 24 Jun 2019 17:58:15 +0200 From: Bjarne <bjb@darco.dk> To: freebsd-xen@freebsd.org Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <03b839d1-39fb-f85f-2781-5219f69ba9ac@darco.dk> In-Reply-To: <mailman.7.1560945600.15387.freebsd-xen@freebsd.org> References: <mailman.7.1560945600.15387.freebsd-xen@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Well, to add some datapoints, I just did a quick test om my private homeserver, testing with iperf3 -s / iperf3 -c <server-ip> I have not much knowledge of iperf3 network test and it's validity, but there is definitely some differences to be seen. Summary: FBSD <-> FBSD 3,5Gbit/sec FBSD -> Centos: 3GBit/Sec Centos -> FBSD: 12 Gbit/sec Centos <-> Centos: 12 Gbit/sec OpenSuse <-> OpenSuse 13 GBit/sec So it ssems there is some fishy when Freebsd is used as client xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm has been optimized in any way, everything is out of the box. =========================== Test 1, Freebsd 12 -> freebsd 12 =========================== Server Freebsd 12: ccepted connection from 192.168.1.100, port 39518 [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec [ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec [ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec [ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec [ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver Client, freebsd 12: Connecting to host 10.192.168.99, port 5201 [ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes [ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes [ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes [ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285 sender [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver ========================== test 2, centos 7.6 -> Freebsd 12 ========================== Server Freebsd 12: Accepted connection from 192.168.1.162, port 49180 [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec [ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec [ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec [ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec [ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec [ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec [ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver Client, centos 7.6: Connecting to host 192.168.1.99, port 5201 [ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes [ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes [ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes [ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes [ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes [ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes [ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes [ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes [ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes [ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec receiver ========================== test 3, Freebsd 12 -> Centos 7.6 ========================== Server, Centos 7.6: Accepted connection from 192.168.1.100, port 20309 [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec [ 5] 1.00-2.00 sec 379 MBytes 3.18 Gbits/sec [ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec [ 5] 3.00-4.00 sec 371 MBytes 3.11 Gbits/sec [ 5] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec [ 5] 5.00-6.00 sec 362 MBytes 3.04 Gbits/sec [ 5] 6.00-7.00 sec 381 MBytes 3.19 Gbits/sec [ 5] 7.00-8.00 sec 384 MBytes 3.22 Gbits/sec [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec [ 5] 9.00-10.00 sec 366 MBytes 3.07 Gbits/sec [ 5] 10.00-10.13 sec 56.4 MBytes 3.52 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.13 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.13 sec 3.66 GBytes 3.10 Gbits/sec receiver Client, Freebsd 12: 5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 387 MBytes 3.25 Gbits/sec 83 309 KBytes [ 5] 1.00-2.00 sec 380 MBytes 3.19 Gbits/sec 88 11.4 KBytes [ 5] 2.00-3.00 sec 365 MBytes 3.06 Gbits/sec 83 54.2 KBytes [ 5] 3.00-4.00 sec 372 MBytes 3.12 Gbits/sec 86 322 KBytes [ 5] 4.00-5.00 sec 370 MBytes 3.11 Gbits/sec 83 2.85 KBytes [ 5] 5.00-6.00 sec 366 MBytes 3.07 Gbits/sec 84 298 KBytes [ 5] 6.00-7.00 sec 382 MBytes 3.20 Gbits/sec 87 356 KBytes [ 5] 7.00-8.00 sec 380 MBytes 3.19 Gbits/sec 87 369 KBytes [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec 87 29.9 KBytes [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 86 419 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec 854 sender [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec receiver ========================== test 4, Centos 7.6 -> Centos 7.6 ========================== Server: Accepted connection from 192.168.1.162, port 47606 [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.60 GBytes 13.7 Gbits/sec [ 5] 1.00-2.00 sec 1.59 GBytes 13.7 Gbits/sec [ 5] 2.00-3.00 sec 1.60 GBytes 13.8 Gbits/sec [ 5] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec [ 5] 4.00-5.00 sec 1.49 GBytes 12.8 Gbits/sec [ 5] 5.00-6.00 sec 1.38 GBytes 11.9 Gbits/sec [ 5] 6.00-7.00 sec 1.52 GBytes 13.1 Gbits/sec [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec [ 5] 8.00-9.00 sec 1.51 GBytes 13.0 Gbits/sec [ 5] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec [ 5] 10.00-10.03 sec 48.1 MBytes 11.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.03 sec 15.5 GBytes 13.2 Gbits/sec receiver Client, Centos 7.6 Connecting to host 192.168.1.155, port 5201 [ 4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.41 MBytes [ 4] 1.00-2.00 sec 1.58 GBytes 13.5 Gbits/sec 0 1.46 MBytes [ 4] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 1.91 MBytes [ 4] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0 2.13 MBytes [ 4] 4.00-5.00 sec 1.47 GBytes 12.6 Gbits/sec 0 2.49 MBytes [ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 2.56 MBytes [ 4] 6.00-7.00 sec 1.53 GBytes 13.1 Gbits/sec 0 2.71 MBytes [ 4] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.73 MBytes [ 4] 8.00-9.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.81 MBytes [ 4] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec 0 2.84 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec receiver =============================== test4, Opensuse 15.1 -> OpenSuse 15.1 ================================ Server, Opensuse 15.1: [ 5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.51 GBytes 13.0 Gbits/sec [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec [ 5] 2.00-3.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec [ 5] 6.00-7.00 sec 1.58 GBytes 13.5 Gbits/sec [ 5] 7.00-8.00 sec 1.60 GBytes 13.8 Gbits/sec [ 5] 8.00-9.00 sec 1.60 GBytes 13.7 Gbits/sec [ 5] 9.00-10.00 sec 1.61 GBytes 13.8 Gbits/sec [ 5] 10.00-10.04 sec 68.0 MBytes 14.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver Client, OpenSuse 15.1: [ 5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.58 GBytes 13.6 Gbits/sec 0 723 KBytes [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec 0 805 KBytes [ 5] 2.00-3.00 sec 1.57 GBytes 13.5 Gbits/sec 0 926 KBytes [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec 0 926 KBytes [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec 0 970 KBytes [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.05 MBytes [ 5] 6.00-7.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.05 MBytes [ 5] 7.00-8.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.05 MBytes [ 5] 8.00-9.00 sec 1.59 GBytes 13.7 Gbits/sec 0 1.05 MBytes [ 5] 9.00-10.00 sec 1.62 GBytes 13.9 Gbits/sec 0 1.05 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 15.9 GBytes 13.7 Gbits/sec 0 sender [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec receiver > I've noticed very slow networking speed between VM's with FreeBSD on the > same host (XCP-ng 7.6.0) for more recent FreeBSD-versions. > > > I've made some tests that show me that something happened from 10.4-RELEASE > to 11.0-RELEASE that had a huge impact on network performance, and > 12.0-RELEASE is even slower. > > > My test setup: > > Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source. > > Network: Internal Private Network on the host (not connected to a PIF). > Each VM has only one VIF connected to this network. > > VM's: > > 2 x 12.0-RELEASE > > 2 x 11.0-RELEASE > > 2 x 10.4-RELEASE > > > All clean identical installs from XenOrchestra, only installed iperf on > each VM for testing. (xe-guest-utilities makes no difference in my tests, > I've tried with and without). > > > iperf -s on first server listed below, and iperf -c <ip> -r on the second > to test speed back and forth: > > > 12.0 <-> 12.0: 50Mbit as client and server > > 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server) > > 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server). > > 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server > > 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server > > > As I side note, not sure if related, but I've noticed that I can't run > iperf with -r flag on 10.4-RELEASE. I get this error message: > > > iperf -c 172.31.16.122 -r > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > write failed: Broken pipe > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > > > > I can run iperf -s fine, and iperf -c <ip> from the other 10.4 VM though: > > > > iperf -c 172.31.16.122 > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec > > > > What have I tried to solve this? > > I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via > XCP-ng. Disabled basically everything without any difference in iperf > results: other-config (MRW): ethtool-sg: off; ethtool-tso: off; > ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off > > > Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum > -rxcsum -tso -lro and no difference here either. > > > Any ideas of how to proceed now to find a solution for this? > > > Regards, > > Christian > -- mvh, Bjarne
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?03b839d1-39fb-f85f-2781-5219f69ba9ac>