From owner-freebsd-xen@freebsd.org Thu Jun 20 14:40:41 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 693BF15BDCB5; Thu, 20 Jun 2019 14:40:41 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa5.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D0CDC8451F; Thu, 20 Jun 2019 14:40:39 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: fKXiAShAcOR0rMRt1LPcSiG3St2p6qwdURQf+k+WhowR9mRddnYpMNdpZQQ1+/qk/gWlL9+n3u jbmHYmAs/trgpkdDid9GjWO6s54TnfOVauxTS39T+kAmDZH4yKfEj1wGPSkniq62zwo0j/Di6w jY6f5gh+Y+tVgxaB24zV6N3Axj7HhL8v2OSw+k4vUUZPtRfO17upaezaAuxt4/696Imn+3Ivz+ WZb1VE9xvE3g0+JBT2Sn28phWJs5O0YDvEhjfMugLqaXT5csWmoh9KegMnjfOA3ZDeCu/rCJyd U1M= X-SBRS: 2.7 X-MesageID: 1996272 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,397,1557201600"; d="scan'208";a="1996272" Date: Thu, 20 Jun 2019 16:39:17 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Christian M CC: , Subject: Re: Very slow and inconsistent internal network speed (between VM's on the same host) for FreeBSD 11.0+ as guest on XCP-ng/XenServer Message-ID: <20190620143917.h4tq2xtgz4nbgkth@MacBook-Air-de-Roger.local> References: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: D0CDC8451F X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.168 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-4.08 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:216.71.155.168]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com]; NEURAL_HAM_SHORT(-0.90)[-0.897,0]; RCVD_IN_DNSWL_NONE(0.00)[168.155.71.216.list.dnswl.org : 127.0.3.0]; IP_SCORE(-0.87)[ip: (-1.54), ipnet: 216.71.154.0/23(-1.68), asn: 16417(-1.07), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jun 2019 14:40:41 -0000 Adding freebsd-net in case they can provide some feedback or tips about how to debug this. On Tue, Jun 18, 2019 at 04:03:00PM +0200, Christian M wrote: > I've noticed very slow networking speed between VM's with FreeBSD on the > same host (XCP-ng 7.6.0) for more recent FreeBSD-versions. Sadly inter-VM throughput has always been a problem for FreeBSD/Xen VMs. I'm not a network expert, so take my comments below with a pinch of salt. > > I've made some tests that show me that something happened from 10.4-RELEASE > to 11.0-RELEASE that had a huge impact on network performance, and > 12.0-RELEASE is even slower. I don't think there have been any major changes, the main one would be the re-write to add multiqueue support to netfront, but that change should have left the code more or less as it was, apart from adding multiqueue. > > My test setup: > > Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source. > > Network: Internal Private Network on the host (not connected to a PIF). > Each VM has only one VIF connected to this network. > > VM's: > > 2 x 12.0-RELEASE > > 2 x 11.0-RELEASE > > 2 x 10.4-RELEASE > > > All clean identical installs from XenOrchestra, only installed iperf on > each VM for testing. (xe-guest-utilities makes no difference in my tests, > I've tried with and without). I think xe-guest-utilities is just needed in order to report suspend/resume capability to XCP, but there isn't anything specially helpful in there. > > iperf -s on first server listed below, and iperf -c -r on the second > to test speed back and forth: > > > 12.0 <-> 12.0: 50Mbit as client and server > > 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server) > > 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server). > > 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server > > 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server Do you see the same issues with external connections? Have you tested throughput between two FreeBSD 12.0 VM running on different hosts? > > > As I side note, not sure if related, but I've noticed that I can't run > iperf with -r flag on 10.4-RELEASE. I get this error message: > > > iperf -c 172.31.16.122 -r > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > write failed: Broken pipe > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec Hm, OK that's weird, I don't think however it's related to Xen. Have you tried if the same happens on a bare-metal install of FreeBSD? Or when running on a different hypervisor? > > > > I can run iperf -s fine, and iperf -c from the other 10.4 VM though: > > > > iperf -c 172.31.16.122 > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec > > > > What have I tried to solve this? > > I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via > XCP-ng. Disabled basically everything without any difference in iperf > results: other-config (MRW): ethtool-sg: off; ethtool-tso: off; > ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off > > > Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum > -rxcsum -tso -lro and no difference here either. Hm, disabling offloading would be my first suggestion, but you seem to have already done that. > > Any ideas of how to proceed now to find a solution for this? Maybe you can try to run wireshark/tcpdump or some other similar software in order to try to detect if there are errors on the transmitted packets? You could run the sniffer on the host and attach it to the backend interfaces (vifX.X) or the bridge if you are using bridged networking. The 12.0 <-> 12.0 case seems quite bad, so I would start with that one. Roger.