From owner-freebsd-net@freebsd.org Thu Jul 4 07:25:17 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CCCF15CAF35; Thu, 4 Jul 2019 07:25:17 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa4.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E65084982; Thu, 4 Jul 2019 07:25:14 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa4.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: GndjarNKt2286LNYUFsbmBCGHDQ60bE6p39594iQQ8ZCUtdt+F0doHOYdZz7Q1YdG6PRtuX3Mf cBBGQ8eyCOluBPnX7YYLVyUqFrNkElap9B2wCa74GwLepVJgIgcDG50juMPhH4WS6eeIu/zrKT w6GPNKYv1LxVtiiOOwpF3pev+9u4Zww68utunGtr3wLbLmeS/XTbQ0+lF+Z2HIF5MlzNZU9wlY /HXy2OBaMhBTUMKDFdYYLqcIFMckWEpOQ5NfbZxZuYRPnnagNfyFt6o3gTnoa51l+MqDE6/HHs oIc= X-SBRS: 2.7 X-MesageID: 2637404 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,449,1557201600"; d="scan'208";a="2637404" Date: Thu, 4 Jul 2019 09:23:54 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Christian M CC: Eugene Grosbein , , Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> References: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: 1E65084982 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.144 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-3.67 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:216.71.155.144]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com]; NEURAL_HAM_SHORT(-0.91)[-0.907,0]; IP_SCORE(-0.85)[ip: (-1.24), ipnet: 216.71.154.0/23(-1.81), asn: 16417(-1.15), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; RCVD_IN_DNSWL_LOW(-0.10)[144.155.71.216.list.dnswl.org : 127.0.3.1]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 07:25:17 -0000 On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monné : > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > > VIF > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > > > I feel the throughput is more consistent now, not all over the place as > > > before, even between runs. But the Retr column (tcp retries) in iperf3 > > has > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > > issue > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > > be, > > > and 0 retries. > > > > > > 12.0-STABLE: > > > > > > Connecting to host 172.31.16.128, port 5201 > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > [ ID] Interval Transfer Bitrate Retr > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > Can you paste the output of ifconfig for both the interfaces used in > > the test? > > > > Are you sure all hardware offloading capabilities are turned off on > > both interfaces? > > > > Can you check what's causing those retries? > > > > Either using tcpdump, whireshark or some other tool to analyze the > > network traffic and detect the errors that cause such retries? > > > > Thanks, Roger. > > > > 172.31.16.127 (12.0-STABLE): > > lo0: flags=8049 metric 0 mtu 16384 > options=680003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21 > xn0: flags=8843 metric 0 mtu 1500 > options=501 I would try to disable rxcsum, tso4 and lro also. > ether 6e:83:99:ed:ce:f7 > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29 > > ethtool -k vif68.0 > Features for vif68.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > 172.31.16.128 (12.0-STABLE): > > lo0: flags=8049 metric 0 mtu 16384 > options=680003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21 > xn0: flags=8843 metric 0 mtu 1500 > options=501 > ether 32:a7:9f:cc:94:8a > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > media: Ethernet manual > status: active > nd6 options=29 > > ethtool -k vif67.0 > Features for vif67.0: > rx-checksumming: on [fixed] > tx-checksumming: off > tx-checksum-ipv4: off > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off > tx-scatter-gather-fraglist: off > tcp-segmentation-offload: off > tx-tcp-segmentation: off > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: off [fixed] > rx-vlan-filter: off [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-ipip-segmentation: off [fixed] > tx-sit-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > busy-poll: off [fixed] > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? On the Xen host would be better, since that tcpdump will read the packets as FreeBSD puts them on the wire. > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > around just to see how 12.0-RELEASE behaved there, and It was the same > thing, super slow between VM's. Yes, I don't think this is related to hardware at all. I will try to find some time to look into this, but it's not going to be soon since I'm currently preparing a talk for a conference I'm attending next week, so I won't likely be able to get to it until the end of July, sorry. Also, I'm no networking expert, so I cannot assure my success in fixing this. As a workaround you can switch to the emulated network card by setting 'hw.xen.disable_pv_nics=1' in /boot/loader.conf. That will give you worse performance than a fully working PV network card, but at least should be consistent. There are others that have switched to virtio-net, but I have no idea how to do that with XCP. Roger.