From owner-freebsd-net@freebsd.org Thu Jul 4 07:58:23 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9858215CBE01; Thu, 4 Jul 2019 07:58:23 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B0D2085AA1; Thu, 4 Jul 2019 07:58:22 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2c.google.com with SMTP id w25so10985483ioc.8; Thu, 04 Jul 2019 00:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZlRMaCe0KI/YXXRnuRRHklytOAp0JrqXFp/kEl/fZNY=; b=pAjOlkxgQggihzW+VBa6OiADZPLHbAGldc22JWwEc1VFuQb2N/PStjfctKHeBjuBHL x+a3JetSbhIEc6/0PLvm7PcHNpdlrS3Oc47KIe2e1FjpfiwT3KofUliqWB8zhMvFz8lN th6eRBdLMYr7VMNOQsGzn8k4UwirOPKywsbHjSHWwNQ7d+Fo2+FG5uOAP9brLHw9WCRm QmumBhA47BCDowkWEqoskkI4nhsX+xdt1l6IK7vPBLWlQMRY82PUU70fybWZwKFdSRaV 8pO2eCBTbEPk+i7TZGWTPIcYEfGpzwTi3VTM8riRj6r43c2n7q5EpVtUuWseVs66b7Ql jEVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZlRMaCe0KI/YXXRnuRRHklytOAp0JrqXFp/kEl/fZNY=; b=jTcZ+/kodAWZCytdGvTXYTl+T/YuwfjlyK6Q95KV2UJdvF08zqi5ITZRLJzFOXI8H3 lOhNDJ6SbaTf9FxslL7P5gYNXVuk/yNwF24TkMsyMmZW5ewnYrBMU6gjw3fcG8ziaBVu W0JhtCSAsotCqe8QZvusmCBVrAlBwshCh6nKoL7QEMefuaHL8Ljna5AU7tTdWW+XLARP DFCSodkl5NcfijuVfIIpaHD8g64gXxyN4XlQE1ZDSVbelObzDsOUZHE9Q8ayGybsVbml u/6PTZQr3x6nKIUrRiPmikog7JZTBqgeZDpESB9pEQrlG9YGhMp2Ix4g0VHdXJhElsgd ovWw== X-Gm-Message-State: APjAAAUGJadUbd6bkYAILZNNt9qU+pp2oW67pZbi1ro2Mzvej1OUTpt+ M2Npah5gSic2nNJu5dT/dyyhwhlOxALtyHpVPEk= X-Google-Smtp-Source: APXvYqxbAGtrkwxE0kq+GS4RZD6Czdqu9KCfY99f1F5y7V5wctI9pOpovJSv+17AFe7uKIbQk4Q1pk/c69dWEaX5Hxg= X-Received: by 2002:a02:710f:: with SMTP id n15mr46377043jac.119.1562227101773; Thu, 04 Jul 2019 00:58:21 -0700 (PDT) MIME-Version: 1.0 References: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> In-Reply-To: <20190704072354.6ycduisiidz5iy2x@Air-de-Roger> From: Christian M Date: Thu, 4 Jul 2019 09:58:10 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: Eugene Grosbein , freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: B0D2085AA1 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=pAjOlkxg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2c as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[c.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.92)[ip: (-8.99), ipnet: 2607:f8b0::/32(-3.16), asn: 15169(-2.39), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.97)[-0.969,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jul 2019 07:58:23 -0000 Thanks Roger. I disabled pv nic entirely on my two 12.0-RELEASE test VM's. I got 1000baseT full duplex auto selected, so I expected throughput close to that, but to my surprise this are the results: $ ifconfig em0: flags=3D8843 metric 0 mtu 1500 options=3D812099 ether 56:65:6f:f3:02:fb inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D29 lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 $ iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 11247 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.50 MBytes 54.5 Mbits/sec 0 368 KBytes [ 5] 1.00-2.00 sec 7.22 MBytes 60.6 Mbits/sec 0 368 KBytes [ 5] 2.00-3.00 sec 7.26 MBytes 60.9 Mbits/sec 0 368 KBytes [ 5] 3.00-4.00 sec 7.09 MBytes 59.4 Mbits/sec 0 368 KBytes [ 5] 4.00-5.00 sec 7.32 MBytes 61.4 Mbits/sec 0 368 KBytes [ 5] 5.00-6.00 sec 7.22 MBytes 60.6 Mbits/sec 0 385 KBytes [ 5] 6.00-7.00 sec 7.24 MBytes 60.7 Mbits/sec 0 385 KBytes [ 5] 7.00-8.00 sec 7.28 MBytes 61.1 Mbits/sec 0 385 KBytes [ 5] 8.00-9.00 sec 7.55 MBytes 63.3 Mbits/sec 0 385 KBytes [ 5] 9.00-10.00 sec 7.12 MBytes 59.7 Mbits/sec 0 385 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec 0 sende= r [ 5] 0.00-10.06 sec 71.8 MBytes 59.9 Mbits/sec receiver iperf Done. $ iperf3 -c 172.31.16.126 -R Connecting to host 172.31.16.126, port 5201 Reverse mode, remote host 172.31.16.126 is sending [ 5] local 172.31.16.125 port 22443 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 5.06 MBytes 42.5 Mbits/sec [ 5] 1.00-2.00 sec 5.60 MBytes 47.0 Mbits/sec [ 5] 2.00-3.00 sec 6.22 MBytes 52.2 Mbits/sec [ 5] 3.00-4.00 sec 6.92 MBytes 58.0 Mbits/sec [ 5] 4.00-5.00 sec 7.39 MBytes 62.0 Mbits/sec [ 5] 5.00-6.00 sec 7.17 MBytes 60.1 Mbits/sec [ 5] 6.00-7.00 sec 7.18 MBytes 60.3 Mbits/sec [ 5] 7.00-8.00 sec 7.32 MBytes 61.4 Mbits/sec [ 5] 8.00-9.00 sec 6.94 MBytes 58.2 Mbits/sec [ 5] 9.00-10.00 sec 7.04 MBytes 59.0 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 67.3 MBytes 56.5 Mbits/sec 0 sende= r [ 5] 0.00-10.00 sec 66.8 MBytes 56.1 Mbits/sec receiver iperf Done. Good luck with your talk. Regards, Christian Den tors 4 juli 2019 kl 09:24 skrev Roger Pau Monn=C3=A9 : > On Thu, Jun 27, 2019 at 12:31:35PM +0200, Christian M wrote: > > Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monn=C3=A9 < > roger.pau@citrix.com>: > > > > > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and > .128. > > > VIF > > > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's= . > > > > > > > > I feel the throughput is more consistent now, not all over the plac= e > as > > > > before, even between runs. But the Retr column (tcp retries) in > iperf3 > > > has > > > > jumped up considerably from hundreds/s to thousands/s. > > > > > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where t= he > > > issue > > > > appeared first for me. 10.4-RELEASE is as fast as I could expect it > to > > > be, > > > > and 0 retries. > > > > > > > > 12.0-STABLE: > > > > > > > > Connecting to host 172.31.16.128, port 5201 > > > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 por= t > 5201 > > > > [ ID] Interval Transfer Bitrate Retr Cwnd > > > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 > KBytes > > > > > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 > KBytes > > > > > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 > KBytes > > > > > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 > KBytes > > > > > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 > KBytes > > > > > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 > KBytes > > > > > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 > KBytes > > > > > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 > KBytes > > > > > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 > KBytes > > > > > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 > KBytes > > > > > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 > KBytes > > > > > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 > KBytes > > > > > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 > KBytes > > > > > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 > KBytes > > > > > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 > KBytes > > > > > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 > KBytes > > > > > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 > KBytes > > > > > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 > KBytes > > > > > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 > KBytes > > > > > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 > KBytes > > > > > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > > [ ID] Interval Transfer Bitrate Retr > > > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > > > > > Can you paste the output of ifconfig for both the interfaces used in > > > the test? > > > > > > Are you sure all hardware offloading capabilities are turned off on > > > both interfaces? > > > > > > Can you check what's causing those retries? > > > > > > Either using tcpdump, whireshark or some other tool to analyze the > > > network traffic and detect the errors that cause such retries? > > > > > > Thanks, Roger. > > > > > > > 172.31.16.127 (12.0-STABLE): > > > > lo0: flags=3D8049 metric 0 mtu 16384 > > options=3D680003 > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=3D21 > > xn0: flags=3D8843 metric 0 mtu = 1500 > > options=3D501 > > I would try to disable rxcsum, tso4 and lro also. > > > ether 6e:83:99:ed:ce:f7 > > inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=3D29 > > > > ethtool -k vif68.0 > > Features for vif68.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > 172.31.16.128 (12.0-STABLE): > > > > lo0: flags=3D8049 metric 0 mtu 16384 > > options=3D680003 > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > inet 127.0.0.1 netmask 0xff000000 > > groups: lo > > nd6 options=3D21 > > xn0: flags=3D8843 metric 0 mtu = 1500 > > options=3D501 > > ether 32:a7:9f:cc:94:8a > > inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 > > media: Ethernet manual > > status: active > > nd6 options=3D29 > > > > ethtool -k vif67.0 > > Features for vif67.0: > > rx-checksumming: on [fixed] > > tx-checksumming: off > > tx-checksum-ipv4: off > > tx-checksum-ip-generic: off [fixed] > > tx-checksum-ipv6: off > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: off > > tx-scatter-gather: off > > tx-scatter-gather-fraglist: off > > tcp-segmentation-offload: off > > tx-tcp-segmentation: off > > tx-tcp-ecn-segmentation: off [fixed] > > tx-tcp6-segmentation: off > > udp-fragmentation-offload: off [fixed] > > generic-segmentation-offload: off > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: off [fixed] > > rx-vlan-filter: off [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: off [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-ipip-segmentation: off [fixed] > > tx-sit-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > busy-poll: off [fixed] > > > > Should I run tcpdump on xen host, or on FreeBSD while running iperf3? > > On the Xen host would be better, since that tcpdump will read the > packets as FreeBSD puts them on the wire. > > > As I side note, I installed XCP-ng 7.6.0 on a older server I had laying > > around just to see how 12.0-RELEASE behaved there, and It was the same > > thing, super slow between VM's. > > Yes, I don't think this is related to hardware at all. > > I will try to find some time to look into this, but it's not going to > be soon since I'm currently preparing a talk for a conference I'm > attending next week, so I won't likely be able to get to it until the > end of July, sorry. Also, I'm no networking expert, so I cannot assure > my success in fixing this. > > As a workaround you can switch to the emulated network card by > setting 'hw.xen.disable_pv_nics=3D1' in /boot/loader.conf. That will > give you worse performance than a fully working PV network card, but > at least should be consistent. There are others that have switched to > virtio-net, but I have no idea how to do that with XCP. > > Roger. >