Date: Thu, 27 Jun 2019 12:31:35 +0200 From: Christian M <christian.marcos@gmail.com> To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= <roger.pau@citrix.com> Cc: Eugene Grosbein <eugen@grosbein.net>, freebsd-xen@freebsd.org, freebsd-net@freebsd.org Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <CAKwR994OYS2Hjhu=QdR2DSoZVyDWEky2Nkmu_LWDQRJiggqxmA@mail.gmail.com> In-Reply-To: <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> References: <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <CAKwR994qWVpY0D8s0n5SOZcY3GzOSV87y7sG0LiMWps3-_ACcg@mail.gmail.com> <d89b2b9a-5005-c9ee-5364-9affff5fd1f5@grosbein.net> <CAKwR995FNv-QjkUQKaHw7Wb2jhA3DoiWzQ7prnA-k56qE8vifw@mail.gmail.com> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <CAKwR996i6ZJLt4iBFTedDNzPaO2mFt5jz3P6iFM-Y1SJ0pu_wA@mail.gmail.com> <ce1f2ebb-c0bf-73c5-f857-be6b24e71da1@grosbein.net> <CAKwR997Zjxkpr48DJ08csLobLK5u5bBY%2BKrQ8r22hdZYJUNREA@mail.gmail.com> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <CAKwR9968gf1RkgP7hD8XOh4Z-Ti917P6W5Lt3-Lakx6r9pei9g@mail.gmail.com> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local>
next in thread | previous in thread | raw e-mail | index | archive | help
Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monn=C3=A9 <roger.pau@citrix= .com>: > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > VIF > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > I feel the throughput is more consistent now, not all over the place as > > before, even between runs. But the Retr column (tcp retries) in iperf3 > has > > jumped up considerably from hundreds/s to thousands/s. > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > issue > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > be, > > and 0 retries. > > > > 12.0-STABLE: > > > > Connecting to host 172.31.16.128, port 5201 > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 52= 01 > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KByte= s > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KByte= s > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KByte= s > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KByte= s > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KByte= s > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KByte= s > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KByte= s > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KByte= s > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KByte= s > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KByte= s > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KByte= s > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KByte= s > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KByte= s > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KByte= s > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KByte= s > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KByte= s > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KByte= s > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KByte= s > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KByte= s > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KByte= s > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > Can you paste the output of ifconfig for both the interfaces used in > the test? > > Are you sure all hardware offloading capabilities are turned off on > both interfaces? > > Can you check what's causing those retries? > > Either using tcpdump, whireshark or some other tool to analyze the > network traffic and detect the errors that cause such retries? > > Thanks, Roger. > 172.31.16.127 (12.0-STABLE): lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> xn0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3D501<RXCSUM,TSO4,LRO> ether 6e:83:99:ed:ce:f7 inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ethtool -k vif68.0 Features for vif68.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] 172.31.16.128 (12.0-STABLE): lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> xn0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3D501<RXCSUM,TSO4,LRO> ether 32:a7:9f:cc:94:8a inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ethtool -k vif67.0 Features for vif67.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Should I run tcpdump on xen host, or on FreeBSD while running iperf3? As I side note, I installed XCP-ng 7.6.0 on a older server I had laying around just to see how 12.0-RELEASE behaved there, and It was the same thing, super slow between VM's.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKwR994OYS2Hjhu=QdR2DSoZVyDWEky2Nkmu_LWDQRJiggqxmA>