From owner-freebsd-net@freebsd.org Tue Jun 25 11:55:54 2019 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DF7015C9B0E; Tue, 25 Jun 2019 11:55:54 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5984486F5E; Tue, 25 Jun 2019 11:55:53 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2e.google.com with SMTP id r185so2325797iod.6; Tue, 25 Jun 2019 04:55:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=0Q9cJyn5HIuRziOIKWB0r2lYgE6qyQIIlqM/s2ofC7U=; b=EMW1/ap4LKB1vtqJoXyizScKcZb3sNWRA0Pt818HrLweaJbq1OEAhNDGqazZOcl4iX 1/giIK10AnoDqJo8wDKzjYNLbW51MTQYIgwihibNgXKIP16EAJmQ/cpwIS9NAroUmgbj KmZQQVBrUjrmMXF6Q3vqArZlQbvWzl5w2q3XzHT/dBvrM24utu8QIlFHjginOS2+NnEv UDzwzkhNn5XOdc02nD9owU9FYKI3b/bMc4e8c3XFOgC/kxA5bBh4m30QcI6lP3TZ7Yie aSu+XI+KbXXHWs07l44qhVY69vcTN9Nh47WoK3YDFsTbPNsjgyTv+TvXUXv0e+VSie6v Nl+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=0Q9cJyn5HIuRziOIKWB0r2lYgE6qyQIIlqM/s2ofC7U=; b=DmJMLL8lrbABbSIa2ubRhPm6GntivQnyrkuGrunecL7n+3zHnZjz8DzSgr6Wf7vIuH 9WCIJre4s40lyZkhVaxsvfp6w0uF90nsMEuBI78l2BZz4cYhmBtmJ6xkJqoUwqPiogP/ ljCx5KXpw7szBKi83kBvIE90qOAXC3YYG05E1+wJkZbhwcc8ppXWtYy7xGjaFi9VbK2D cpGW/5YRWPGZXH4ZFYM1KJ6Fta2Se+F3oCGNi0R9TL3V3/YUMmkoD9S8l+GNa4Ox//AQ ly6c5Xce62+FSWAm0dCypyp05nsNVn8dn34igSJ+PdMQs+LojYN8+HQwO3u0s9Fm4M0p OxIg== X-Gm-Message-State: APjAAAX9cGl7RCGmMvNFI7ablseZEFaMlXkP9qZRiBOQGOPayvWt80ot odi7GWi7xr5YB8usprUar9uYLWCwl6lfT/CJAvck18Of7do= X-Google-Smtp-Source: APXvYqz5DtJrpGCwNyWRIU710boh8nn0Vl/3v0L3XmjIcu7M47d0nXdzWkbX3mvBGBzoxws+TaRppE17xSHaEXiHf2Q= X-Received: by 2002:a6b:4107:: with SMTP id n7mr1218546ioa.12.1561463751894; Tue, 25 Jun 2019 04:55:51 -0700 (PDT) MIME-Version: 1.0 References: <mailman.7.1560945600.15387.freebsd-xen@freebsd.org> <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <CAKwR994kQTRfB0R0_WiTYx4pgOHtR-ge7h0vP8NiM_+-MBPGGw@mail.gmail.com> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> In-Reply-To: <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> From: Christian M <christian.marcos@gmail.com> Date: Tue, 25 Jun 2019 13:55:40 +0200 Message-ID: <CAKwR996s9Y_Bea95GOhPkdEJabmJM0r6Qi43gW7_8SXE2ryddA@mail.gmail.com> Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 5984486F5E X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=EMW1/ap4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2e as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.66 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[e.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.84)[-0.841,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; IP_SCORE(-2.81)[ip: (-8.55), ipnet: 2607:f8b0::/32(-3.13), asn: 15169(-2.33), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 25 Jun 2019 11:55:54 -0000 I've made two tests while running tcpdump on the xcp-ng host. I'm not at all qualified to interpret the .pcap files from tcpdump, but I've put them on Google Drive and linked them below the two tests. Perhaps someone more qualified could have a look for anything useful in there. Please note the extremely uneven throughput for test 2 below. It's almost like the throughput increased when running tcpdump simultaneously. Host: XCP-ng 7.6.0 Network: Private Network on host, not connected to any PIF. VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) On the host I listen with tcpdump on the VIF for VM1 in both tests. VM1 as client: On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap xcp-ng-vm1-client.pcap (80M): https://drive.google.com/open?id=3D1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU On VM1: iperf3 -c 172.31.16.126 On VM2: iperf3 -s VM1 iperf3 output: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 18182 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 8.00 MBytes 67.1 Mbits/sec 156 15.6 KBytes [ 5] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec 318 1.43 KBytes [ 5] 2.00-3.00 sec 7.51 MBytes 63.1 Mbits/sec 218 1.43 KBytes [ 5] 3.00-4.00 sec 8.29 MBytes 69.3 Mbits/sec 193 1.43 KBytes [ 5] 4.00-5.01 sec 10.7 MBytes 89.4 Mbits/sec 252 1.43 KBytes [ 5] 5.01-6.00 sec 13.6 MBytes 115 Mbits/sec 313 31.3 KBytes [ 5] 6.00-7.00 sec 8.41 MBytes 70.2 Mbits/sec 309 1.43 KBytes [ 5] 7.00-8.01 sec 12.6 MBytes 106 Mbits/sec 223 1.43 KBytes [ 5] 8.01-9.00 sec 12.5 MBytes 106 Mbits/sec 227 1.43 KBytes [ 5] 9.00-10.00 sec 13.5 MBytes 113 Mbits/sec 263 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 109 MBytes 91.6 Mbits/sec 2472 sender [ 5] 0.00-10.01 sec 109 MBytes 91.3 Mbits/sec receiver VM1 as server: On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-server.pcap xcp-ng-vm1-server.pcap: (337M): https://drive.google.com/open?id=3D1N3-GccXaBc6hlzFrgshi2vhcCt8vekxF On VM1: iperf3 -s On VM2: iperf3 -c 172.31.16.125 VM1 iperf3 output: ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 172.31.16.126, port 31523 [ 5] local 172.31.16.125 port 5201 connected to 172.31.16.126 port 34605 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 10.9 MBytes 91.6 Mbits/sec [ 5] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec [ 5] 2.00-3.01 sec 4.92 MBytes 41.0 Mbits/sec [ 5] 3.01-4.00 sec 10.4 MBytes 87.9 Mbits/sec [ 5] 4.00-5.00 sec 8.52 MBytes 71.2 Mbits/sec [ 5] 5.00-6.00 sec 22.0 MBytes 185 Mbits/sec [ 5] 6.00-7.00 sec 9.25 MBytes 77.8 Mbits/sec [ 5] 7.00-8.00 sec 70.8 MBytes 594 Mbits/sec [ 5] 8.00-9.00 sec 89.5 MBytes 752 Mbits/sec [ 5] 9.00-10.00 sec 105 MBytes 884 Mbits/sec [ 5] 10.00-10.00 sec 180 KBytes 1.19 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 346 MBytes 290 Mbits/sec receiver Den tis 25 juni 2019 kl 10:21 skrev Roger Pau Monn=C3=A9 <roger.pau@citrix.= com>: > On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote: > > Thanks for your input Bjarne. > > > > Previously I tested with iperf2, but I have made som tests with iperf3 > now > > also and noticed there are a lot of "Retr" (TCP retries) in some cases. > > Went back here to your post and saw that your results also showed a lot > of > > retries in some cases. My new tests showed similar results to yours > > (although not nearly as hi throughput as you have). When Linux is clien= t > > the retries are 0, with FreeBSD as client the number is > 0. > > Can you figure out what caused those retries? Wrong MTU, bad > checksums? > > > I'm not sure > > if our numbers are considered high though, and something that actually > is a > > problem? Although, on a internal network with no external factors like > > interference, I feel this should always be 0 no matter what? > > If properly configured yes, there should be no retries when doing > intra-VM connections. > > Thanks, Roger. >