Date: Thu, 4 Jul 2019 09:23:54 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= <roger.pau@citrix.com> To: Christian M <christian.marcos@gmail.com> Cc: Eugene Grosbein <eugen@grosbein.net>, <freebsd-xen@freebsd.org>, <freebsd-net@freebsd.org> Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> In-Reply-To: <CAKwR994OYS2Hjhu=QdR2DSoZVyDWEky2Nkmu_LWDQRJiggqxmA@mail.gmail.com> References: <d89b2b9a-5005-c9ee-5364-9affff5fd1f5@grosbein.net> <CAKwR995FNv-QjkUQKaHw7Wb2jhA3DoiWzQ7prnA-k56qE8vifw@mail.gmail.com> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <CAKwR996i6ZJLt4iBFTedDNzPaO2mFt5jz3P6iFM-Y1SJ0pu_wA@mail.gmail.com> <ce1f2ebb-c0bf-73c5-f857-be6b24e71da1@grosbein.net> <CAKwR997Zjxkpr48DJ08csLobLK5u5bBY%2BKrQ8r22hdZYJUNREA@mail.gmail.com> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <CAKwR9968gf1RkgP7hD8XOh4Z-Ti917P6W5Lt3-Lakx6r9pei9g@mail.gmail.com> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> <CAKwR994OYS2Hjhu=QdR2DSoZVyDWEky2Nkmu_LWDQRJiggqxmA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné <roger.pau@citrix.com>: > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > > VIF > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > > > I feel the throughput is more consistent now, not all over the place as > > > before, even between runs. But the Retr column (tcp retries) in iperf3 > > has > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > > issue > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > > be, > > > and 0 retries. > > > > > > 12.0-STABLE: > > > > > > Connecting to host 172.31.16.128, port 5201 > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > [ ID] Interval Transfer Bitrate Retr > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > Can you paste the output of ifconfig for both the interfaces used in > > the test? > > > > Are you sure all hardware offloading capabilities are turned off on > > both interfaces? > > > > Can you check what's causing those retries? > > > > Either using tcpdump, whireshark or some other tool to analyze the > > network traffic and detect the errors that cause such retries? > > > > Thanks, Roger. > > > > 172.31.16.127 (12.0-STABLE): > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=501<RXCSUM,TSO4,LRO> I would try to disable rxcsum, tso4 and lro also. > ether 6e:83:99:ed:ce:f7 > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > ethtool -k vif68.0 > Features for vif68.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > 172.31.16.128 (12.0-STABLE): > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=501<RXCSUM,TSO4,LRO> > ether 32:a7:9f:cc:94:8a > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > ethtool -k vif67.0 > Features for vif67.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? On the Xen host would be better, since that tcpdump will read the packets as FreeBSD puts them on the wire. > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > around just to see how 12.0-RELEASE behaved there, and It was the same > thing, super slow between VM's. Yes, I don't think this is related to hardware at all. I will try to find some time to look into this, but it's not going to be soon since I'm currently preparing a talk for a conference I'm attending next week, so I won't likely be able to get to it until the end of July, sorry. Also, I'm no networking expert, so I cannot assure my success in fixing this. As a workaround you can switch to the emulated network card by setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will give you worse performance than a fully working PV network card, but at least should be consistent. There are others that have switched to virtio-net, but I have no idea how to do that with XCP. Roger.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190704072354.6ycduisiidz5iy2x>