From owner-freebsd-xen@freebsd.org Thu Jun 27 10:14:46 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B701F15B3823; Thu, 27 Jun 2019 10:14:46 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B2DC277245; Thu, 27 Jun 2019 10:14:45 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2d.google.com with SMTP id u13so3566425iop.0; Thu, 27 Jun 2019 03:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KQbBJuYPKvFcxPLYETvyKZRoVLnzi/O5m8MlPrkCS1o=; b=UABh6ZB3si/LOB0B4UJ5S8Plb9nZAy4xauheMnAwLRvg0AyxUDoIMWDsmerQ+8PyDA +pw3/AmSMbtYmhF804oAmZ2JJfhvgja+uZUoCiajd+rt9uLvhfSZ+fFxXgS5B/wp+Zr9 HVJ+UvyqaGsXpt7hYuIb3JL4QUV+pbrEIgmTcx9sLC/sBwv6UbrUBZOGWyp6mBWNqtKR IVPM/eal5lN1zDgOciVMcXr3XiT4bCYG6ZXxUMJIM7sQvnK6cytFubbfmiBjLNws/JbZ G/jiUTTDDGBaSJbkj4XXC3I/20iqNN0Z1Nqb+uBim2rLVqDbX4vdb0gfCp7Fw+G+CvEz +NqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KQbBJuYPKvFcxPLYETvyKZRoVLnzi/O5m8MlPrkCS1o=; b=lHnmH4GXtU6Y/X2sk4N3MUzkQT21G8SJ9BuFaZYFHwjR6hWWhIBzvqdB+EoOpe/OGc 2Iuh/N373Io753XCh9Bk0zs5vK4TwocV8W/7b07qCSQMtVuI5ysOdOBgiUGqwVxJoLPE Q/Qi6gEtnikzzK7+ARMEsvqBRXOn4zjv8rRfiIA3IG+BnTRiygcEp9N8+UZUlthH0YfZ 0VpdUD/dAb5gSKA0fMv0F0ucVMFsTMa13gHA6OiM+e8ofwrrGQG9loKlqtUJVKN5Xamn wbLQezIGUzBR8oNJRRd4nwqzYF1MGZwUQN4DcaOf/Ae3QlTe30OHb7u3XyGus1FnYZuc c63w== X-Gm-Message-State: APjAAAVz3xzYXUJpg07TLPvSiLxCbk/e/ZYPgWrrg6IwtPCPeAul+lvb ZPj4TfThV386Q6lLYV4rzah2JW8MC+yReFBGWQLWxiG8 X-Google-Smtp-Source: APXvYqyfAz3p6XhkrgNjM1VyEHLHzgbTWX2e/JaI/HG+CSKuybXxiKbvw9JG+tD9poE7s7miv5pnoqXSGzVBZVN8pa0= X-Received: by 2002:a6b:b7d5:: with SMTP id h204mr3547070iof.188.1561630484693; Thu, 27 Jun 2019 03:14:44 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> In-Reply-To: <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> From: Christian M Date: Thu, 27 Jun 2019 12:14:33 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Eugene Grosbein Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: B2DC277245 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UABh6ZB3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2d as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.56 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.71)[-0.706,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[d.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.95)[ip: (-9.19), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.34), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2019 10:14:47 -0000 I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF cheksum offloading is turned off, and -txcsum for xn0 for both VM's. I feel the throughput is more consistent now, not all over the place as before, even between runs. But the Retr column (tcp retries) in iperf3 has jumped up considerably from hundreds/s to thousands/s. Just a reminder, I have tested this with 11.0-RELEASE also, where the issue appeared first for me. 10.4-RELEASE is as fast as I could expect it to be, and 0 retries. 12.0-STABLE: Connecting to host 172.31.16.128, port 5201 [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 sender [ 5] 0.00-20.00 sec 2.05 GBytes 881 Mbits/sec receiver 10.3-RELEASE (offloading enabled everywhere for these VM's VIF, and for xn0): Connecting to host 172.31.16.122, port 5201 [ 5] local 172.31.16.121 port 20712 connected to 172.31.16.122 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 832 MBytes 6.98 Gbits/sec 0 580 KBytes [ 5] 1.00-2.00 sec 1.24 GBytes 10.6 Gbits/sec 0 1.04 MBytes [ 5] 2.00-3.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.57 MBytes [ 5] 3.00-4.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.76 MBytes [ 5] 4.00-5.00 sec 1.37 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 5.00-6.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.77 MBytes [ 5] 6.00-7.00 sec 1.37 GBytes 11.8 Gbits/sec 0 1.77 MBytes [ 5] 7.00-8.00 sec 1.39 GBytes 12.0 Gbits/sec 0 1.77 MBytes [ 5] 8.00-9.00 sec 1.35 GBytes 11.6 Gbits/sec 0 1.77 MBytes [ 5] 9.00-10.00 sec 1.42 GBytes 12.2 Gbits/sec 0 1.77 MBytes [ 5] 10.00-11.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.77 MBytes [ 5] 11.00-12.00 sec 1.32 GBytes 11.4 Gbits/sec 0 1.77 MBytes [ 5] 12.00-13.00 sec 1.34 GBytes 11.6 Gbits/sec 0 1.77 MBytes [ 5] 13.00-14.00 sec 1.49 GBytes 12.8 Gbits/sec 0 1.77 MBytes [ 5] 14.00-15.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 15.00-16.00 sec 1.37 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 16.00-17.00 sec 1.48 GBytes 12.7 Gbits/sec 0 1.77 MBytes [ 5] 17.00-18.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 18.00-19.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.77 MBytes [ 5] 19.00-20.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 26.9 GBytes 11.6 Gbits/sec 0 sender [ 5] 0.00-20.01 sec 26.9 GBytes 11.6 Gbits/sec receiver