From owner-freebsd-xen@freebsd.org Thu Jun 27 10:31:48 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F41715B41CA; Thu, 27 Jun 2019 10:31:48 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 533D277B7F; Thu, 27 Jun 2019 10:31:47 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2b.google.com with SMTP id h6so3630316ioh.3; Thu, 27 Jun 2019 03:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KflgItLl2Jr30IONB3DvSNvbwHNnqkoi9nPf8ylLLtA=; b=Bm8KGipJibVkXcid/+U9nLyLgFySVktUVAHrN0kWF0JvZ8NuOLqzqI69GU2rBkOWez 6GMrWVeIvobAGpbMj9EPqkQfXDD27v7l5/3bLW0IfJSYb4toEvoeAOPMYDTg+3Hv0a9N fYCu1aHTbngS82FvR2sLmW5uYZ877SaI446KXg+LKYfA5r1ksCsOEiqBmKycgX/ubO6K IOFlzEEF6MZmYrB9GCPGSmJleWbWuRYpVASa0P90qhjTxyNjVqVtf4vnW5TjvTqIjqKI HCx1AVhFcRgPO7bO+K3sfNKDDToT88L3t8fXEoSYjU1mhMYfoE8a1qhJoqCdUPYk1oO1 sPYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KflgItLl2Jr30IONB3DvSNvbwHNnqkoi9nPf8ylLLtA=; b=cJX4lPaghwrBXtaNB09vft3+Mtm2FRxPqE8g009S3Y0tnZaj42i69gIsKo3BzSNb3n iVr3RMGUxA14KlF1KwBgtvfuWOaWWpSzYfUEZwBkC1D80sMZj7sWeKNR+BMbOKIyyzG1 mX1zLxj0V57FtivwwervULWbpFspnASy4nKywtpf5WbGc0j1mOYGpEWCGigDSh/Mz9kX a0yvkOv4Urrs+gB6qyTgozEmYOIjFVMsEad+BJLuZFjzB61BAt74joMnMxKIh9OXxct7 GDARjIPOEt0TUcaMEV2crtZyI8max60skDnPcVHNB7j+K8tdjJzKS5EeaKJuyB2492Es aEQg== X-Gm-Message-State: APjAAAX88ar6c04s4WX53RKaLrZ57Rs0y6wtl7W78CJr33iAlEVVCTrD DIOfRaOR/21L51F4erHuazZZTRtOywXnZbO0JMjfkcDfu10= X-Google-Smtp-Source: APXvYqwDL7LXjJoUiz+8sNJyuatbnsMtnENby80aFUMM5H9kabXUWFPlZQ4azV8Mt2HIsW+uANQI0I6RZp/PvLjEKms= X-Received: by 2002:a02:cb4b:: with SMTP id k11mr3617601jap.109.1561631506366; Thu, 27 Jun 2019 03:31:46 -0700 (PDT) MIME-Version: 1.0 References: <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> In-Reply-To: <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> From: Christian M Date: Thu, 27 Jun 2019 12:31:35 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: Eugene Grosbein , freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 533D277B7F X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Bm8KGipJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2b as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.52 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[b.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.80)[ip: (-8.45), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.34), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.71)[-0.707,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2019 10:31:48 -0000 Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monn=C3=A9 : > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > VIF > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > I feel the throughput is more consistent now, not all over the place as > > before, even between runs. But the Retr column (tcp retries) in iperf3 > has > > jumped up considerably from hundreds/s to thousands/s. > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > issue > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > be, > > and 0 retries. > > > > 12.0-STABLE: > > > > Connecting to host 172.31.16.128, port 5201 > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 52= 01 > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KByte= s > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KByte= s > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KByte= s > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KByte= s > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KByte= s > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KByte= s > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KByte= s > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KByte= s > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KByte= s > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KByte= s > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KByte= s > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KByte= s > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KByte= s > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KByte= s > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KByte= s > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KByte= s > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KByte= s > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KByte= s > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KByte= s > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KByte= s > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > Can you paste the output of ifconfig for both the interfaces used in > the test? > > Are you sure all hardware offloading capabilities are turned off on > both interfaces? > > Can you check what's causing those retries? > > Either using tcpdump, whireshark or some other tool to analyze the > network traffic and detect the errors that cause such retries? > > Thanks, Roger. > 172.31.16.127 (12.0-STABLE): lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 xn0: flags=3D8843 metric 0 mtu 1500 options=3D501 ether 6e:83:99:ed:ce:f7 inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=3D29 ethtool -k vif68.0 Features for vif68.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] 172.31.16.128 (12.0-STABLE): lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 xn0: flags=3D8843 metric 0 mtu 1500 options=3D501 ether 32:a7:9f:cc:94:8a inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=3D29 ethtool -k vif67.0 Features for vif67.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Should I run tcpdump on xen host, or on FreeBSD while running iperf3? As I side note, I installed XCP-ng 7.6.0 on a older server I had laying around just to see how 12.0-RELEASE behaved there, and It was the same thing, super slow between VM's.