From owner-freebsd-net@freebsd.org Thu Jun 20 09:54:27 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5287515B4111 for ; Thu, 20 Jun 2019 09:54:27 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A9D773132 for ; Thu, 20 Jun 2019 09:54:26 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2d.google.com with SMTP id e5so528491iok.4 for ; Thu, 20 Jun 2019 02:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=e2a5ecMc5WDZgzASgIcd+J0miHHpTNzwufdsTiRLds0=; b=F5XNinhpquGAZxz5f2eG0P8zd3Y2Jt7DJM3QfGAwFj2g7yzg8XM7tTk6t8Dt48fFgj ES3xk+ok74XFe9J8tHSBK/PPbpDGNw1BhS1ddlWplWPA70WyaQMELxG0q0EsIb1Qo+cQ vfiHDvkqQEavrgAh8Akzv5xrffGRIX+uTjuWUZMXycTC5hNKGh4brR0/7TTPshf+bRwd 8CAeiM0/EKcHvJB/wDt543ARyJUC+OUxFrUuwNZMZ4xfSllQ/URgR1w6RzRhzCaucB7r RTlSXMwOm3NRuYUCkAi4zRjut/LIc1lKKuHL5e9ojR4qmPMYqryJLfnZAwGUm2tB/j8F MEMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=e2a5ecMc5WDZgzASgIcd+J0miHHpTNzwufdsTiRLds0=; b=PguJujsWI+Qb6F86M4f3gZ5Cziax+MhDQ+ND/41IFn9Rvzwn/juusdOk+N4Sy9hsiZ SOtPc3m68wV6bDlXRg49L5OdoxyNj3GcgJT5WdeDGn0A1rRKfz00QM3UY3p96BJe0l/k bUmLaCkS9caSi0hYgemKgivl/zIOcH9s2/Ic8jisOQrow/3DniDM5/eeYUkZuPhCWGXe CkN/cIuvXNotuydhBZ5WK7qOZMAFO/K9lIbo5wuoreG2ckKtwhlfilcmG1a94B5lCLHK xDmy6qzdOPnFYDOA6eksMZNuMzhRc9Q7asC3ahZ7kQ4nWciBBdLoFcMbAi9IN13Piw7e YA5Q== X-Gm-Message-State: APjAAAU56S7Wf8OpAm9fuYfCBwTDbdCaxifUc+hhgsySbcAN94LrLod2 RRfAmOKhcFK6rbLTgC+zpCQowukn6xhGIXS+Z6AHzPPK X-Google-Smtp-Source: APXvYqwDioRn5xTJCk4lh13thUbOnR7Ih7eD0HtlyjE7bdnBgjG2adXoQ73e9n4uQNUPlDZ8vIpGIFpDEYELchJjHTI= X-Received: by 2002:a6b:2cc7:: with SMTP id s190mr14664500ios.29.1561024465282; Thu, 20 Jun 2019 02:54:25 -0700 (PDT) MIME-Version: 1.0 From: Christian M Date: Thu, 20 Jun 2019 11:54:14 +0200 Message-ID: Subject: Very slow and inconsistent internal network speed (between VM's on the same host) for FreeBSD 11.0+ as guest on XCP-ng/XenServer To: freebsd-net@freebsd.org X-Rspamd-Queue-Id: 6A9D773132 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=F5XNinhp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2d as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.93 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.96)[-0.964,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.95)[ip: (-9.26), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.31), country: US(-0.06)]; RCVD_IN_DNSWL_NONE(0.00)[d.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jun 2019 09:54:27 -0000 Hi. I posted this on freebsd-xen also, but I begin to believe that list is more for Xen on FreeBSD, not FreeBSD on Xen. Anyway, I've noticed that networking speed between FreeBSD VM's on the same host (XCP-ng 7.6.0) for more recent FreeBSD-versions is very slow and inconsistent between FreeBSD versions. I've made some tests that show me that something happened from 10.4-RELEASE to 11.0-RELEASE that had a huge impact on network performance, and 12.0-RELEASE is even slower. My test setup: Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source. Network: Internal Private Network on the host (not connected to a PIF). Each VM has only one VIF connected to this network. VM's: 2 x 12.0-RELEASE 2 x 11.0-RELEASE 2 x 10.4-RELEASE All clean identical installs from XenOrchestra, only installed iperf on each VM for testing. (xe-guest-utilities makes no difference in my tests, I've tried with and without). iperf -s on first server listed below, and iperf -c -r on the second to test speed back and forth: 12.0 <-> 12.0: 50Mbit as client and server 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server) 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server). 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server As I side note, not sure if related, but I've noticed that I can't run iperf with -r flag on 10.4-RELEASE. I get this error message: iperf -c 172.31.16.122 -r ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ write failed: Broken pipe ------------------------------------------------------------ Client connecting to 172.31.16.122, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001 [ ID] Interval Transfer Bandwidth [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec I can run iperf -s fine, and iperf -c from the other 10.4 VM though: iperf -c 172.31.16.122 ------------------------------------------------------------ Client connecting to 172.31.16.122, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec What have I tried to solve this? I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via XCP-ng. Disabled basically everything without any difference in iperf results: other-config (MRW): ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum -rxcsum -tso -lro and no difference here either. Any ideas of how to proceed now to find a solution for this? Regards, Christian