Date: Thu, 4 Jul 2019 09:58:10 +0200 From: Christian M <christian.marcos@gmail.com> To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= <roger.pau@citrix.com> Cc: Eugene Grosbein <eugen@grosbein.net>, freebsd-xen@freebsd.org, freebsd-net@freebsd.org Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <CAKwR994KPhQ0gJQD34VOhrKPooVLy6=yUKYRo2JL01o645sRdA@mail.gmail.com> In-Reply-To: <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> References: <d89b2b9a-5005-c9ee-5364-9affff5fd1f5@grosbein.net> <CAKwR995FNv-QjkUQKaHw7Wb2jhA3DoiWzQ7prnA-k56qE8vifw@mail.gmail.com> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <CAKwR996i6ZJLt4iBFTedDNzPaO2mFt5jz3P6iFM-Y1SJ0pu_wA@mail.gmail.com> <ce1f2ebb-c0bf-73c5-f857-be6b24e71da1@grosbein.net> <CAKwR997Zjxkpr48DJ08csLobLK5u5bBY%2BKrQ8r22hdZYJUNREA@mail.gmail.com> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <CAKwR9968gf1RkgP7hD8XOh4Z-Ti917P6W5Lt3-Lakx6r9pei9g@mail.gmail.com> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> <CAKwR994OYS2Hjhu=QdR2DSoZVyDWEky2Nkmu_LWDQRJiggqxmA@mail.gmail.com> <20190704072354.6ycduisiidz5iy2x@Air-de-Roger>
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks Roger. I disabled pv nic entirely on my two 12.0-RELEASE test VM's. I got 1000baseT full duplex auto selected, so I expected throughput close to that, but to my surprise this are the results: $ ifconfig em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3D812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_= HWFILTER> ether 56:65:6f:f3:02:fb inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> $ iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 11247 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.50 MBytes 54.5 Mbits/sec 0 368 KBytes [ 5] 1.00-2.00 sec 7.22 MBytes 60.6 Mbits/sec 0 368 KBytes [ 5] 2.00-3.00 sec 7.26 MBytes 60.9 Mbits/sec 0 368 KBytes [ 5] 3.00-4.00 sec 7.09 MBytes 59.4 Mbits/sec 0 368 KBytes [ 5] 4.00-5.00 sec 7.32 MBytes 61.4 Mbits/sec 0 368 KBytes [ 5] 5.00-6.00 sec 7.22 MBytes 60.6 Mbits/sec 0 385 KBytes [ 5] 6.00-7.00 sec 7.24 MBytes 60.7 Mbits/sec 0 385 KBytes [ 5] 7.00-8.00 sec 7.28 MBytes 61.1 Mbits/sec 0 385 KBytes [ 5] 8.00-9.00 sec 7.55 MBytes 63.3 Mbits/sec 0 385 KBytes [ 5] 9.00-10.00 sec 7.12 MBytes 59.7 Mbits/sec 0 385 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec 0 sende= r [ 5] 0.00-10.06 sec 71.8 MBytes 59.9 Mbits/sec receiver iperf Done. $ iperf3 -c 172.31.16.126 -R Connecting to host 172.31.16.126, port 5201 Reverse mode, remote host 172.31.16.126 is sending [ 5] local 172.31.16.125 port 22443 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 5.06 MBytes 42.5 Mbits/sec [ 5] 1.00-2.00 sec 5.60 MBytes 47.0 Mbits/sec [ 5] 2.00-3.00 sec 6.22 MBytes 52.2 Mbits/sec [ 5] 3.00-4.00 sec 6.92 MBytes 58.0 Mbits/sec [ 5] 4.00-5.00 sec 7.39 MBytes 62.0 Mbits/sec [ 5] 5.00-6.00 sec 7.17 MBytes 60.1 Mbits/sec [ 5] 6.00-7.00 sec 7.18 MBytes 60.3 Mbits/sec [ 5] 7.00-8.00 sec 7.32 MBytes 61.4 Mbits/sec [ 5] 8.00-9.00 sec 6.94 MBytes 58.2 Mbits/sec [ 5] 9.00-10.00 sec 7.04 MBytes 59.0 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 67.3 MBytes 56.5 Mbits/sec 0 sende= r [ 5] 0.00-10.00 sec 66.8 MBytes 56.1 Mbits/sec receiver iperf Done. Good luck with your talk. Regards, Christian Den tors 4 juli 2019 kl 09:24 skrev Roger Pau Monn=C3=A9 <roger.pau@citrix.= com>: > On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monn=C3=A9 < > roger.pau@citrix.com>: > > > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and > .128. > > > VIF > > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's= . > > > > > > > > I feel the throughput is more consistent now, not all over the plac= e > as > > > > before, even between runs. But the Retr column (tcp retries) in > iperf3 > > > has > > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where t= he > > > issue > > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it > to > > > be, > > > > and 0 retries. > > > > > > > > 12.0-STABLE: > > > > > > > > Connecting to host 172.31.16.128, port 5201 > > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 por= t > 5201 > > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 > KBytes > > > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 > KBytes > > > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 > KBytes > > > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 > KBytes > > > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 > KBytes > > > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 > KBytes > > > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 > KBytes > > > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 > KBytes > > > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 > KBytes > > > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 > KBytes > > > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 > KBytes > > > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 > KBytes > > > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 > KBytes > > > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 > KBytes > > > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 > KBytes > > > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 > KBytes > > > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 > KBytes > > > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 > KBytes > > > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 > KBytes > > > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 > KBytes > > > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > > [ ID] Interval Transfer Bitrate Retr > > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > > > Can you paste the output of ifconfig for both the interfaces used in > > > the test? > > > > > > Are you sure all hardware offloading capabilities are turned off on > > > both interfaces? > > > > > > Can you check what's causing those retries? > > > > > > Either using tcpdump, whireshark or some other tool to analyze the > > > network traffic and detect the errors that cause such retries? > > > > > > Thanks, Roger. > > > > > > > 172.31.16.127 (12.0-STABLE): > > > > lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > > options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> > > xn0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu = 1500 > > options=3D501<RXCSUM,TSO4,LRO> > > I would try to disable rxcsum, tso4 and lro also. > > > ether 6e:83:99:ed:ce:f7 > > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > > > ethtool -k vif68.0 > > Features for vif68.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > 172.31.16.128 (12.0-STABLE): > > > > lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > > options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> > > xn0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu = 1500 > > options=3D501<RXCSUM,TSO4,LRO> > > ether 32:a7:9f:cc:94:8a > > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > > > ethtool -k vif67.0 > > Features for vif67.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? > > On the Xen host would be better, since that tcpdump will read the > packets as FreeBSD puts them on the wire. > > > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > > around just to see how 12.0-RELEASE behaved there, and It was the same > > thing, super slow between VM's. > > Yes, I don't think this is related to hardware at all. > > I will try to find some time to look into this, but it's not going to > be soon since I'm currently preparing a talk for a conference I'm > attending next week, so I won't likely be able to get to it until the > end of July, sorry. Also, I'm no networking expert, so I cannot assure > my success in fixing this. > > As a workaround you can switch to the emulated network card by > setting 'hw.xen.disable_pv_nics=3D1' in /boot/loader.conf. That will > give you worse performance than a fully working PV network card, but > at least should be consistent. There are others that have switched to > virtio-net, but I have no idea how to do that with XCP. > > Roger. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKwR994KPhQ0gJQD34VOhrKPooVLy6=yUKYRo2JL01o645sRdA>