From owner-freebsd-xen@freebsd.org Mon Jun 24 14:51:54 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9347E15CFE97; Mon, 24 Jun 2019 14:51:54 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B80F7174B; Mon, 24 Jun 2019 14:51:53 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd35.google.com with SMTP id s7so2804074iob.11; Mon, 24 Jun 2019 07:51:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=N78XGcxDZPBC4lC31H9eptysrteZcTOPbm/1JgSfN0w=; b=pCU4VoA/nngBhzGBLJ0JinhUbUt2NNQ+XWzYuRecMdLnzk20Ga3twgHF1fFJic7OK8 aHpvqGK+E3UoeHhwRbyLdt6mPvpZZabuKZMMuvYTMu4Z6Dq+tcgx+i+cLq2IQJNPAwLh R0A9WtIRrwD2IdmRj2Ek4v0dJG5ChvliCrlQhxRWX3wbl06aKTSPfWjJEw2UdvCoK92a f5RBRlocx/vVapXk5jh78d5IoK5S41sWL8kiczd0MNm7xIOYHfhZ6C+nxc6PoPgUsGjv BeHClxfkvI/Fk1anIrePXJB1KYWniVBdY71Rn3cAWt4bWWwT/UMZycUiRdMcaWwqYdwA TQ1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=N78XGcxDZPBC4lC31H9eptysrteZcTOPbm/1JgSfN0w=; b=ckKfsHeZzgyS8Pg74UEC6WzXyNnLh/XbGKKSC4bY1hyrY1Z3lMlE27IJhJUQnH2R9d FVY1IFJCBoATOH5fokxHiblxey0Au6qQbHv1Zpqi9ycgi99ie23rB3+vR87R24WLTRjI XflGJ8kHu1UZxKAJKvz3E68uVGn+Xo65Ap8zalDhAwFFIvmPwCc1V5IYnkug2BpV4YgP 9BvFCl73gZlgoeQaq5jtXn/NtRW2UHexahc7mEe+oPRBJIe7gUjY90s412CfkunZv4Pk OrFk1hUg4r/fzxnktpTi90FZ6auIdOezHBEyMk1JVRBkF3tEiOKP4EKL7Yz94x37pbdR Wt3A== X-Gm-Message-State: APjAAAX53uE/3JKwm6q/+1Y4Jaxm6C2cdFrhZMqfMsAn7UB5kz/Rg3Ry EdT4gAK10Ba1up+Zx8Nja/FyHBD5J9EZfBxsS+s= X-Google-Smtp-Source: APXvYqxsR134wMRWk7crWMeki02iqd+n6Gp1WvX7wrU4S/MjPIqWZtUDaV+9WYo6Ro2XkuN7Tk8uFfiHbwh9cVZc0YE= X-Received: by 2002:a6b:b7d5:: with SMTP id h204mr5621579iof.188.1561387912474; Mon, 24 Jun 2019 07:51:52 -0700 (PDT) MIME-Version: 1.0 References: <20190620143917.h4tq2xtgz4nbgkth@MacBook-Air-de-Roger.local> In-Reply-To: <20190620143917.h4tq2xtgz4nbgkth@MacBook-Air-de-Roger.local> From: Christian M Date: Mon, 24 Jun 2019 16:51:41 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between VM's on the same host) for FreeBSD 11.0+ as guest on XCP-ng/XenServer To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 7B80F7174B X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=pCU4VoA/; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d35 as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.80 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[5.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.88)[ip: (-8.85), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.33), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.92)[-0.916,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2019 14:51:54 -0000 Den tors 20 juni 2019 kl 16:39 skrev Roger Pau Monn=C3=A9 : > > Do you see the same issues with external connections? Have you tested > throughput between two FreeBSD 12.0 VM running on different hosts? > > I've tested 12.0-RELEASE between two hosts (XCP-ng 7.6.0 and Citrix CXenserver 7.2.0) over Gbit link, and iperf results is about 600Mbit/s back and forth. I also tested 12.0-RELEASE on same XCP-ng host, but changed the network from Private Network to the Internal Network (that's connected to a PIF). That increased speed slightly, but It's still below 100mbit back and forth. > > > Hm, OK that's weird, I don't think however it's related to Xen. Have > you tried if the same happens on a bare-metal install of FreeBSD? > > Or when running on a different hypervisor? > > I have not. I only have XCP-ng 7.6.0 and Citrix XenServer 7.x to play with currently. I have no server at this moment to test a bare metal install on, but if you believe this is an important test I could perhaps find a machine to try it on. Please let me know if I should proceed with this. > > > > > Any ideas of how to proceed now to find a solution for this? > > Maybe you can try to run wireshark/tcpdump or some other similar > software in order to try to detect if there are errors on the > transmitted packets? > > You could run the sniffer on the host and attach it to the backend > interfaces (vifX.X) or the bridge if you are using bridged > networking. > > The 12.0 <-> 12.0 case seems quite bad, so I would start with that > one. > > Roger. > I have now run tshark (wireshark) to capture traffic between 12.0 <-> 12.0 on the private network (running tshark on freebsd). I'm really not qualified to interpret the output, so I'll link to the capture-output here: capture-output.pcap: https://drive.google.com/open?id=3D1zU6qhZO7OOKfLRugXJovYIQuFCLrouH6 (218MB= ) output.txt (tshark -r capture-output.pcap > output.txt): https://drive.google.com/open?id=3D1ORJy_jhDyAIgmw1WnXBsPQSlIxqEzVrF (34MB) I'm not really sure how to run it on XCP-ng, or is there perhaps another tool I can use for that other than tshark? Thanks, Christian From owner-freebsd-xen@freebsd.org Mon Jun 24 15:58:19 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F36E115D1AF4 for ; Mon, 24 Jun 2019 15:58:18 +0000 (UTC) (envelope-from bjb@darco.dk) Received: from post1.darco.dk (post1.blichsoft.dk [104.248.143.233]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4E60D74421 for ; Mon, 24 Jun 2019 15:58:18 +0000 (UTC) (envelope-from bjb@darco.dk) Received: from mail.blichsoft.dk (per [192.168.1.135]) by c3po.blichsoft.dk (Postfix) with ESMTPS id B4440224BC8 for ; Mon, 24 Jun 2019 17:58:16 +0200 (CEST) Received: by mail.blichsoft.dk (Postfix, from userid 110) id 66B38620A4; Mon, 24 Jun 2019 17:58:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darco.dk; s=mail; t=1561391896; bh=Y+un5Img1/G2nvlkIKC3RcfB2Ay/L0GtdvhSgrl6/d4=; h=From:Subject:To:References:Date:In-Reply-To; b=rFm9Oo/ZsOpu4sjOgrnCwZUHGvQZ889c8MbU/QwHot9lpxhSPWAs/oRTYiowsGYw5 JZ58iA6jcCGAR77OxWEnAE/N3hn4O56iCGKtfZ7aObGqmoCWjTCNU6L1afqXwtTd5T O1LDnW7J0x1KB7fsSA5iypStMCLCoRAHChfNB/lA= X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on per.blichsoft.dk X-Spam-Level: X-Spam-Status: No, score=-102.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, NO_DNS_FOR_FROM,USER_IN_WHITELIST autolearn=no autolearn_force=no version=3.4.2 Received: from [192.168.1.200] (sussi.darco.dk [192.168.1.200]) by mail.blichsoft.dk (Postfix) with ESMTP id A191962099 for ; Mon, 24 Jun 2019 17:58:15 +0200 (CEST) From: Bjarne Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: freebsd-xen@freebsd.org References: Message-ID: <03b839d1-39fb-f85f-2781-5219f69ba9ac@darco.dk> Date: Mon, 24 Jun 2019 17:58:15 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Status: No X-Virus-Checker-Version: clamassassin 1.2.4 with clamdscan / ClamAV 0.101.2/25489/Sun Jun 23 10:01:03 2019 X-Rspamd-Queue-Id: 4E60D74421 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=darco.dk header.s=mail header.b=rFm9Oo/Z; spf=pass (mx1.freebsd.org: domain of bjb@darco.dk designates 104.248.143.233 as permitted sender) smtp.mailfrom=bjb@darco.dk X-Spamd-Result: default: False [-2.10 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.83)[-0.829,0]; R_DKIM_ALLOW(-0.20)[darco.dk:s=mail]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:104.248.143.233]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.99)[-0.989,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[darco.dk]; NEURAL_SPAM_SHORT(0.02)[0.020,0]; DKIM_TRACE(0.00)[darco.dk:+]; MX_GOOD(-0.01)[cached: mail.blichsoft.dk]; IP_SCORE(0.21)[asn: 14061(1.12), country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.248.128.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2019 15:58:19 -0000 Well, to add some datapoints,  I just did a quick test om my private homeserver, testing with iperf3 -s  / iperf3 -c I have not much knowledge of iperf3 network test and it's validity, but there is definitely some differences to be seen. Summary: FBSD <-> FBSD   3,5Gbit/sec FBSD -> Centos:  3GBit/Sec Centos -> FBSD: 12 Gbit/sec Centos <-> Centos: 12 Gbit/sec OpenSuse <-> OpenSuse 13 GBit/sec So it ssems there is some fishy when Freebsd is used as client xen running on opensuse,  11 vm's,  freebsd , openSuse and Centos. No vm has been optimized in any way,  everything is out of the box. =========================== Test 1,  Freebsd 12 -> freebsd 12 =========================== Server Freebsd 12: ccepted connection from 192.168.1.100, port 39518 [  5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518 [ ID] Interval           Transfer     Bitrate [  5]   0.00-1.00   sec   374 MBytes  3.14 Gbits/sec [  5]   1.00-2.00   sec   444 MBytes  3.72 Gbits/sec [  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec [  5]   3.00-4.00   sec   438 MBytes  3.67 Gbits/sec [  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec [  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec [  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec [  5]   7.00-8.00   sec   432 MBytes  3.63 Gbits/sec [  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec [  5]  10.00-10.00  sec   332 KBytes  2.17 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec                  receiver Client, freebsd 12: Connecting to host 10.192.168.99, port 5201 [  5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd [  5]   0.00-1.00   sec   375 MBytes  3.14 Gbits/sec  239    178 KBytes [  5]   1.00-2.00   sec   445 MBytes  3.73 Gbits/sec  834   74.1 KBytes [  5]   2.00-3.00   sec   411 MBytes  3.45 Gbits/sec  794   38.5 KBytes [  5]   3.00-4.00   sec   438 MBytes  3.68 Gbits/sec  808    198 KBytes [  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec  777   67.0 KBytes [  5]   5.00-6.00   sec   417 MBytes  3.50 Gbits/sec  795    120 KBytes [  5]   6.00-7.00   sec   412 MBytes  3.45 Gbits/sec  763   72.7 KBytes [  5]   7.00-8.00   sec   432 MBytes  3.62 Gbits/sec  810    329 KBytes [  5]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec  777    219 KBytes [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec  688    304 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate         Retr [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec 7285             sender [  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec                  receiver ========================== test 2, centos 7.6 -> Freebsd 12 ========================== Server Freebsd 12: Accepted connection from 192.168.1.162, port 49180 [  5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182 [ ID] Interval           Transfer     Bitrate [  5]   0.00-1.00   sec  1.17 GBytes  10.0 Gbits/sec [  5]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec [  5]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec [  5]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec [  5]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec [  5]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec [  5]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec [  5]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec [  5]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec [  5]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate [  5]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec                  receiver Client, centos 7.6: Connecting to host 192.168.1.99, port 5201 [  4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201 [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd [  4]   0.00-1.00   sec  1.17 GBytes  10.1 Gbits/sec    0    822 KBytes [  4]   1.00-2.00   sec  1.47 GBytes  12.6 Gbits/sec    0   1.53 MBytes [  4]   2.00-3.00   sec  1.30 GBytes  11.2 Gbits/sec    0   1.57 MBytes [  4]   3.00-4.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.64 MBytes [  4]   4.00-5.00   sec  1.44 GBytes  12.4 Gbits/sec    0   1.70 MBytes [  4]   5.00-6.00   sec  1.29 GBytes  11.1 Gbits/sec    0   1.70 MBytes [  4]   6.00-7.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes [  4]   7.00-8.00   sec  1.36 GBytes  11.7 Gbits/sec    0   1.70 MBytes [  4]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec    0   1.70 MBytes [  4]   9.00-10.00  sec  1.41 GBytes  12.1 Gbits/sec    0   1.70 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth       Retr [  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec 0 sender [  4]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec                  receiver ========================== test 3, Freebsd 12 -> Centos 7.6 ========================== Server, Centos 7.6: Accepted connection from 192.168.1.100, port 20309 [  5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917 [ ID] Interval           Transfer     Bandwidth [  5]   0.00-1.00   sec   335 MBytes  2.81 Gbits/sec [  5]   1.00-2.00   sec   379 MBytes  3.18 Gbits/sec [  5]   2.00-3.00   sec   368 MBytes  3.08 Gbits/sec [  5]   3.00-4.00   sec   371 MBytes  3.11 Gbits/sec [  5]   4.00-5.00   sec   374 MBytes  3.14 Gbits/sec [  5]   5.00-6.00   sec   362 MBytes  3.04 Gbits/sec [  5]   6.00-7.00   sec   381 MBytes  3.19 Gbits/sec [  5]   7.00-8.00   sec   384 MBytes  3.22 Gbits/sec [  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec [  5]   9.00-10.00  sec   366 MBytes  3.07 Gbits/sec [  5]  10.00-10.13  sec  56.4 MBytes  3.52 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth [  5]   0.00-10.13  sec  0.00 Bytes  0.00 bits/sec sender [  5]   0.00-10.13  sec  3.66 GBytes  3.10 Gbits/sec                  receiver Client, Freebsd 12:   5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd [  5]   0.00-1.00   sec   387 MBytes  3.25 Gbits/sec   83    309 KBytes [  5]   1.00-2.00   sec   380 MBytes  3.19 Gbits/sec   88   11.4 KBytes [  5]   2.00-3.00   sec   365 MBytes  3.06 Gbits/sec   83   54.2 KBytes [  5]   3.00-4.00   sec   372 MBytes  3.12 Gbits/sec   86    322 KBytes [  5]   4.00-5.00   sec   370 MBytes  3.11 Gbits/sec   83   2.85 KBytes [  5]   5.00-6.00   sec   366 MBytes  3.07 Gbits/sec   84    298 KBytes [  5]   6.00-7.00   sec   382 MBytes  3.20 Gbits/sec   87    356 KBytes [  5]   7.00-8.00   sec   380 MBytes  3.19 Gbits/sec   87    369 KBytes [  5]   8.00-9.00   sec   374 MBytes  3.14 Gbits/sec   87   29.9 KBytes [  5]   9.00-10.00  sec   375 MBytes  3.15 Gbits/sec   86    419 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate         Retr [  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec 854 sender [  5]   0.00-10.00  sec  3.66 GBytes  3.15 Gbits/sec                  receiver ========================== test 4, Centos 7.6 -> Centos 7.6 ========================== Server: Accepted connection from 192.168.1.162, port 47606 [  5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608 [ ID] Interval           Transfer     Bandwidth [  5]   0.00-1.00   sec  1.60 GBytes  13.7 Gbits/sec [  5]   1.00-2.00   sec  1.59 GBytes  13.7 Gbits/sec [  5]   2.00-3.00   sec  1.60 GBytes  13.8 Gbits/sec [  5]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec [  5]   4.00-5.00   sec  1.49 GBytes  12.8 Gbits/sec [  5]   5.00-6.00   sec  1.38 GBytes  11.9 Gbits/sec [  5]   6.00-7.00   sec  1.52 GBytes  13.1 Gbits/sec [  5]   7.00-8.00   sec  1.51 GBytes  13.0 Gbits/sec [  5]   8.00-9.00   sec  1.51 GBytes  13.0 Gbits/sec [  5]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec [  5]  10.00-10.03  sec  48.1 MBytes  11.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth [  5]   0.00-10.03  sec  0.00 Bytes  0.00 bits/sec sender [  5]   0.00-10.03  sec  15.5 GBytes  13.2 Gbits/sec                  receiver Client, Centos 7.6 Connecting to host 192.168.1.155, port 5201 [  4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201 [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd [  4]   0.00-1.00   sec  1.66 GBytes  14.3 Gbits/sec    0   1.41 MBytes [  4]   1.00-2.00   sec  1.58 GBytes  13.5 Gbits/sec    0   1.46 MBytes [  4]   2.00-3.00   sec  1.61 GBytes  13.9 Gbits/sec    0   1.91 MBytes [  4]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec    0   2.13 MBytes [  4]   4.00-5.00   sec  1.47 GBytes  12.6 Gbits/sec    0   2.49 MBytes [  4]   5.00-6.00   sec  1.39 GBytes  11.9 Gbits/sec    0   2.56 MBytes [  4]   6.00-7.00   sec  1.53 GBytes  13.1 Gbits/sec    0   2.71 MBytes [  4]   7.00-8.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.73 MBytes [  4]   8.00-9.00   sec  1.50 GBytes  12.9 Gbits/sec    0   2.81 MBytes [  4]   9.00-10.00  sec  1.51 GBytes  12.9 Gbits/sec    0   2.84 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bandwidth       Retr [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec 0 sender [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec                  receiver =============================== test4, Opensuse 15.1 -> OpenSuse 15.1 ================================ Server, Opensuse 15.1: [  5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334 [ ID] Interval           Transfer     Bitrate [  5]   0.00-1.00   sec  1.51 GBytes  13.0 Gbits/sec [  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec [  5]   2.00-3.00   sec  1.58 GBytes  13.6 Gbits/sec [  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec [  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec [  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec [  5]   6.00-7.00   sec  1.58 GBytes  13.5 Gbits/sec [  5]   7.00-8.00   sec  1.60 GBytes  13.8 Gbits/sec [  5]   8.00-9.00   sec  1.60 GBytes  13.7 Gbits/sec [  5]   9.00-10.00  sec  1.61 GBytes  13.8 Gbits/sec [  5]  10.00-10.04  sec  68.0 MBytes  14.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate [  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec                  receiver Client, OpenSuse 15.1: [  5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd [  5]   0.00-1.00   sec  1.58 GBytes  13.6 Gbits/sec    0    723 KBytes [  5]   1.00-2.00   sec  1.61 GBytes  13.8 Gbits/sec    0    805 KBytes [  5]   2.00-3.00   sec  1.57 GBytes  13.5 Gbits/sec    0    926 KBytes [  5]   3.00-4.00   sec  1.58 GBytes  13.6 Gbits/sec    0    926 KBytes [  5]   4.00-5.00   sec  1.56 GBytes  13.4 Gbits/sec    0    970 KBytes [  5]   5.00-6.00   sec  1.60 GBytes  13.7 Gbits/sec    0   1.05 MBytes [  5]   6.00-7.00   sec  1.57 GBytes  13.5 Gbits/sec    0   1.05 MBytes [  5]   7.00-8.00   sec  1.61 GBytes  13.8 Gbits/sec    0   1.05 MBytes [  5]   8.00-9.00   sec  1.59 GBytes  13.7 Gbits/sec    0   1.05 MBytes [  5]   9.00-10.00  sec  1.62 GBytes  13.9 Gbits/sec    0   1.05 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval           Transfer     Bitrate         Retr [  5]   0.00-10.00  sec  15.9 GBytes  13.7 Gbits/sec 0 sender [  5]   0.00-10.04  sec  15.9 GBytes  13.6 Gbits/sec                  receiver > I've noticed very slow networking speed between VM's with FreeBSD on the > same host (XCP-ng 7.6.0) for more recent FreeBSD-versions. > > > I've made some tests that show me that something happened from 10.4-RELEASE > to 11.0-RELEASE that had a huge impact on network performance, and > 12.0-RELEASE is even slower. > > > My test setup: > > Host: XCP-ng 7.6.0, managed with XenOrchestra. Open source. > > Network: Internal Private Network on the host (not connected to a PIF). > Each VM has only one VIF connected to this network. > > VM's: > > 2 x 12.0-RELEASE > > 2 x 11.0-RELEASE > > 2 x 10.4-RELEASE > > > All clean identical installs from XenOrchestra, only installed iperf on > each VM for testing. (xe-guest-utilities makes no difference in my tests, > I've tried with and without). > > > iperf -s on first server listed below, and iperf -c -r on the second > to test speed back and forth: > > > 12.0 <-> 12.0: 50Mbit as client and server > > 12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server) > > 12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server). > > 11.0 <-> 11.0: 219Mbit as client, 99Mbit as server > > 10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server > > > As I side note, not sure if related, but I've noticed that I can't run > iperf with -r flag on 10.4-RELEASE. I get this error message: > > > iperf -c 172.31.16.122 -r > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > write failed: Broken pipe > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec > > > > I can run iperf -s fine, and iperf -c from the other 10.4 VM though: > > > > iperf -c 172.31.16.122 > > ------------------------------------------------------------ > > Client connecting to 172.31.16.122, TCP port 5001 > > TCP window size: 32.5 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec > > > > What have I tried to solve this? > > I've tried to disable checksum offloading for the 12.0-RELEASE VIF's via > XCP-ng. Disabled basically everything without any difference in iperf > results: other-config (MRW): ethtool-sg: off; ethtool-tso: off; > ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off > > > Also tried disabling offloading in FreeBSD with ifconfig xn0 -txcsum > -rxcsum -tso -lro and no difference here either. > > > Any ideas of how to proceed now to find a solution for this? > > > Regards, > > Christian > -- mvh, Bjarne From owner-freebsd-xen@freebsd.org Mon Jun 24 20:24:24 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B656E15D8938 for ; Mon, 24 Jun 2019 20:24:23 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd30.google.com (mail-io1-xd30.google.com [IPv6:2607:f8b0:4864:20::d30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B3B5873B6 for ; Mon, 24 Jun 2019 20:24:22 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd30.google.com with SMTP id u19so3996092ior.9 for ; Mon, 24 Jun 2019 13:24:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Yn48fhCgEhx/16BISpbKwDTGsPo6khpuN3J3Vm6DWyg=; b=W0qdSrpCIJiS9WI4NQDp2meg7pHKrIfwz+BSvAmi2rQyo2jUIzhICs5a+j0eqkIRUN ymdgUvxuDIDm0jFlwUCSWYtEvevYWvEzCsYLFXeAvMZuGQCsxGrxTdLa3kfqR7QR/AXh 2kXJToH3XsgkrZ/Lf2Iv2ooWnrfoQlqVvPiQfRxIodjKAqbgzXHrmlH4yZOTn6PcnB0r LanlJzFzVJUgy1U1NpTiTExfn/lH/eC6I8Dyk0p8KIhNl3HlnqtMlAJSL64SXRb+KEud RCm8j6zM8dn+0uRDlk2iHSS6W2+r8IWtgcbpEImmtIrGRi7U1wkZLcbBtARl05ggeRLV uCuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Yn48fhCgEhx/16BISpbKwDTGsPo6khpuN3J3Vm6DWyg=; b=DUdTUPMoWxPSt6ZfDf6O+UmKnYrtxIosE5xd0jkXLQ9/cw2XzwtJ5+WcORwtplUoIb bE3Go4w8YXD3yyjuB06w3ZgYwMv1N70h/993pbY/mwXYXH6B7c0pkt5MHFNpfXJ1yXp2 6+0QbDhhZphH83IY/Y/DgE7tfyjPStiHnFKcoMqQdh3ruHzL8vELATv4MOYCOhC7eS21 TAtAGXAZmJJWQZ3VN6QTxGqqmcgJhU66RvGjSviAlI1hmmDrM3q+xU2MXq2fth/nrS3k Y/3r/3ri+weW6pZOhs14Vm+7fHrrkTM7jSExKVUCVYltnozIdnSWnLRh/QOyOCb3nxbF mU6A== X-Gm-Message-State: APjAAAUpFmXle3iOLvNcHupFAZ7YRCTf8ealvNE0OC8vMowY0JJdqV5h PWOPe8tIPuKKSvkEIAVOadgjCc8KkIfaY04KIX+gSf5Q X-Google-Smtp-Source: APXvYqx46TxzLo2I+NivkGv5gC5oIztTPhe9j/L24lVBibeIqsRFbEN8Iv9YWEhlElRxHPVZ/wxRFBqsDBR3rZ3t5zg= X-Received: by 2002:a6b:4107:: with SMTP id n7mr10962003ioa.12.1561407861218; Mon, 24 Jun 2019 13:24:21 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> In-Reply-To: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> From: Christian M Date: Mon, 24 Jun 2019 22:24:09 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: freebsd-xen@freebsd.org X-Rspamd-Queue-Id: 3B3B5873B6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=W0qdSrpC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d30 as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.82 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.97)[-0.965,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.95)[ip: (-9.20), ipnet: 2607:f8b0::/32(-3.15), asn: 15169(-2.33), country: US(-0.06)]; RCVD_IN_DNSWL_NONE(0.00)[0.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2019 20:24:24 -0000 Thanks for your input Bjarne. Previously I tested with iperf2, but I have made som tests with iperf3 now also and noticed there are a lot of "Retr" (TCP retries) in some cases. Went back here to your post and saw that your results also showed a lot of retries in some cases. My new tests showed similar results to yours (although not nearly as hi throughput as you have). When Linux is client the retries are 0, with FreeBSD as client the number is > 0. I'm not sure if our numbers are considered high though, and something that actually is a problem? Although, on a internal network with no external factors like interference, I feel this should always be 0 no matter what? What hypervisor were you running in your tests? Version? Regards, Christian Den m=C3=A5n 24 juni 2019 kl 17:56 skrev Bjarne : > Well, to add some datapoints, I just did a quick test om my private > homeserver, testing with iperf3 > -s / iperf3 -c > > I have not much knowledge of iperf3 network test and it's validity, but > there is definitely some > differences to be seen. > > Summary: > > FBSD <-> FBSD 3,5Gbit/sec > FBSD -> Centos: 3GBit/Sec > Centos -> FBSD: 12 Gbit/sec > > Centos <-> Centos: 12 Gbit/sec > OpenSuse <-> OpenSuse 13 GBit/sec > > So it ssems there is some fishy when Freebsd is used as client > > xen running on opensuse, 11 vm's, freebsd , openSuse and Centos. No vm > has been optimized in any > way, everything is out of the box. > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D > Test 1, Freebsd 12 -> freebsd 12 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D > > Server Freebsd 12: > ccepted connection from 192.168.1.100, port 39518 > [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.100 port 48518 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 374 MBytes 3.14 Gbits/sec > [ 5] 1.00-2.00 sec 444 MBytes 3.72 Gbits/sec > [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec > [ 5] 3.00-4.00 sec 438 MBytes 3.67 Gbits/sec > [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec > [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec > [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec > [ 5] 7.00-8.00 sec 432 MBytes 3.63 Gbits/sec > [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec > [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec > [ 5] 10.00-10.00 sec 332 KBytes 2.17 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate > [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec > receiver > > Client, freebsd 12: > Connecting to host 10.192.168.99, port 5201 > [ 5] local 192.168.1.100 port 48518 connected to 192.168.1.99 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 239 178 KBytes > [ 5] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec 834 74.1 KBytes > [ 5] 2.00-3.00 sec 411 MBytes 3.45 Gbits/sec 794 38.5 KBytes > [ 5] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 808 198 KBytes > [ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 777 67.0 KBytes > [ 5] 5.00-6.00 sec 417 MBytes 3.50 Gbits/sec 795 120 KBytes > [ 5] 6.00-7.00 sec 412 MBytes 3.45 Gbits/sec 763 72.7 KBytes > [ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 810 329 KBytes > [ 5] 8.00-9.00 sec 418 MBytes 3.51 Gbits/sec 777 219 KBytes > [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 688 304 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 7285 > sender > [ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec > receiver > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > test 2, centos 7.6 -> Freebsd 12 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > > Server Freebsd 12: > Accepted connection from 192.168.1.162, port 49180 > [ 5] local 192.168.1.99 port 5201 connected to 192.168.1.162 port 49182 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 1.17 GBytes 10.0 Gbits/sec > [ 5] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec > [ 5] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec > [ 5] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec > [ 5] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec > [ 5] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec > [ 5] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec > [ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec > [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec > [ 5] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate > [ 5] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec > receiver > > Client, centos 7.6: > Connecting to host 192.168.1.99, port 5201 > [ 4] local 192.168.1.162 port 49182 connected to 192.168.1.99 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 1.17 GBytes 10.1 Gbits/sec 0 822 KBytes > [ 4] 1.00-2.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.53 MBytes > [ 4] 2.00-3.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.57 MBytes > [ 4] 3.00-4.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.64 MBytes > [ 4] 4.00-5.00 sec 1.44 GBytes 12.4 Gbits/sec 0 1.70 MBytes > [ 4] 5.00-6.00 sec 1.29 GBytes 11.1 Gbits/sec 0 1.70 MBytes > [ 4] 6.00-7.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes > [ 4] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.70 MBytes > [ 4] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 0 1.70 MBytes > [ 4] 9.00-10.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.70 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec 0 sender > [ 4] 0.00-10.00 sec 13.4 GBytes 11.5 Gbits/sec > receiver > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > test 3, Freebsd 12 -> Centos 7.6 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > > Server, Centos 7.6: > Accepted connection from 192.168.1.100, port 20309 > [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.100 port 17917 > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec > [ 5] 1.00-2.00 sec 379 MBytes 3.18 Gbits/sec > [ 5] 2.00-3.00 sec 368 MBytes 3.08 Gbits/sec > [ 5] 3.00-4.00 sec 371 MBytes 3.11 Gbits/sec > [ 5] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec > [ 5] 5.00-6.00 sec 362 MBytes 3.04 Gbits/sec > [ 5] 6.00-7.00 sec 381 MBytes 3.19 Gbits/sec > [ 5] 7.00-8.00 sec 384 MBytes 3.22 Gbits/sec > [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec > [ 5] 9.00-10.00 sec 366 MBytes 3.07 Gbits/sec > [ 5] 10.00-10.13 sec 56.4 MBytes 3.52 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-10.13 sec 0.00 Bytes 0.00 bits/sec sender > [ 5] 0.00-10.13 sec 3.66 GBytes 3.10 Gbits/sec > receiver > > Client, Freebsd 12: > 5] local 192.168.1.100 port 17917 connected to 192.168.1.155 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 387 MBytes 3.25 Gbits/sec 83 309 KBytes > [ 5] 1.00-2.00 sec 380 MBytes 3.19 Gbits/sec 88 11.4 KBytes > [ 5] 2.00-3.00 sec 365 MBytes 3.06 Gbits/sec 83 54.2 KBytes > [ 5] 3.00-4.00 sec 372 MBytes 3.12 Gbits/sec 86 322 KBytes > [ 5] 4.00-5.00 sec 370 MBytes 3.11 Gbits/sec 83 2.85 KBytes > [ 5] 5.00-6.00 sec 366 MBytes 3.07 Gbits/sec 84 298 KBytes > [ 5] 6.00-7.00 sec 382 MBytes 3.20 Gbits/sec 87 356 KBytes > [ 5] 7.00-8.00 sec 380 MBytes 3.19 Gbits/sec 87 369 KBytes > [ 5] 8.00-9.00 sec 374 MBytes 3.14 Gbits/sec 87 29.9 KBytes > [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec 86 419 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec 854 sender > [ 5] 0.00-10.00 sec 3.66 GBytes 3.15 Gbits/sec > receiver > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > test 4, Centos 7.6 -> Centos 7.6 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > > Server: > Accepted connection from 192.168.1.162, port 47606 > [ 5] local 192.168.1.155 port 5201 connected to 192.168.1.162 port 47608 > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-1.00 sec 1.60 GBytes 13.7 Gbits/sec > [ 5] 1.00-2.00 sec 1.59 GBytes 13.7 Gbits/sec > [ 5] 2.00-3.00 sec 1.60 GBytes 13.8 Gbits/sec > [ 5] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec > [ 5] 4.00-5.00 sec 1.49 GBytes 12.8 Gbits/sec > [ 5] 5.00-6.00 sec 1.38 GBytes 11.9 Gbits/sec > [ 5] 6.00-7.00 sec 1.52 GBytes 13.1 Gbits/sec > [ 5] 7.00-8.00 sec 1.51 GBytes 13.0 Gbits/sec > [ 5] 8.00-9.00 sec 1.51 GBytes 13.0 Gbits/sec > [ 5] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec > [ 5] 10.00-10.03 sec 48.1 MBytes 11.9 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender > [ 5] 0.00-10.03 sec 15.5 GBytes 13.2 Gbits/sec > receiver > > Client, Centos 7.6 > Connecting to host 192.168.1.155, port 5201 > [ 4] local 192.168.1.162 port 47608 connected to 192.168.1.155 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.41 MBytes > [ 4] 1.00-2.00 sec 1.58 GBytes 13.5 Gbits/sec 0 1.46 MBytes > [ 4] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 1.91 MBytes > [ 4] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0 2.13 MBytes > [ 4] 4.00-5.00 sec 1.47 GBytes 12.6 Gbits/sec 0 2.49 MBytes > [ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 2.56 MBytes > [ 4] 6.00-7.00 sec 1.53 GBytes 13.1 Gbits/sec 0 2.71 MBytes > [ 4] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.73 MBytes > [ 4] 8.00-9.00 sec 1.50 GBytes 12.9 Gbits/sec 0 2.81 MBytes > [ 4] 9.00-10.00 sec 1.51 GBytes 12.9 Gbits/sec 0 2.84 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec 0 sender > [ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec > receiver > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > test4, Opensuse 15.1 -> OpenSuse 15.1 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > Server, Opensuse 15.1: > [ 5] local 192.168.1.158 port 5201 connected to 192.168.1.163 port 39334 > [ ID] Interval Transfer Bitrate > [ 5] 0.00-1.00 sec 1.51 GBytes 13.0 Gbits/sec > [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec > [ 5] 2.00-3.00 sec 1.58 GBytes 13.6 Gbits/sec > [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec > [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec > [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec > [ 5] 6.00-7.00 sec 1.58 GBytes 13.5 Gbits/sec > [ 5] 7.00-8.00 sec 1.60 GBytes 13.8 Gbits/sec > [ 5] 8.00-9.00 sec 1.60 GBytes 13.7 Gbits/sec > [ 5] 9.00-10.00 sec 1.61 GBytes 13.8 Gbits/sec > [ 5] 10.00-10.04 sec 68.0 MBytes 14.2 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate > [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec > receiver > > Client, OpenSuse 15.1: > [ 5] local 192.168.1.163 port 39334 connected to 192.168.1.158 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 1.58 GBytes 13.6 Gbits/sec 0 723 KBytes > [ 5] 1.00-2.00 sec 1.61 GBytes 13.8 Gbits/sec 0 805 KBytes > [ 5] 2.00-3.00 sec 1.57 GBytes 13.5 Gbits/sec 0 926 KBytes > [ 5] 3.00-4.00 sec 1.58 GBytes 13.6 Gbits/sec 0 926 KBytes > [ 5] 4.00-5.00 sec 1.56 GBytes 13.4 Gbits/sec 0 970 KBytes > [ 5] 5.00-6.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.05 MBytes > [ 5] 6.00-7.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.05 MBytes > [ 5] 7.00-8.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.05 MBytes > [ 5] 8.00-9.00 sec 1.59 GBytes 13.7 Gbits/sec 0 1.05 MBytes > [ 5] 9.00-10.00 sec 1.62 GBytes 13.9 Gbits/sec 0 1.05 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 15.9 GBytes 13.7 Gbits/sec 0 sender > [ 5] 0.00-10.04 sec 15.9 GBytes 13.6 Gbits/sec > receiver > > > > > > > From owner-freebsd-xen@freebsd.org Tue Jun 25 08:21:19 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51B3015C45A6 for ; Tue, 25 Jun 2019 08:21:19 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa5.hc3370-68.iphmx.com (esa5.hc3370-68.iphmx.com [216.71.155.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa5.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0146F77644 for ; Tue, 25 Jun 2019 08:21:17 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: KOrvZNNgOobr7c2fH4jrQK9OQ6R+IFOhdDLozf16eyPosMHlbQTgfOUbCpa1V9lUMEP6sxMFDo O8PUU23zrkKoTn5Vm1ccoci6uKh0KC3Ndzkeujwo5slNX4DG85108lTKBR91iYCSlQXPdRHcme Ju1g1jWSi3zsR68Zotq8PXhvUNcA2dMbe97ccq31p/FF3EZhJSbxXkbLRw0nFrO6vKDgethb3m wKoW0RzPBTBSrhEidp3YedbWR6RmHeOYm5VDNlEma+r1H06o7NtH8EZ/hZcq/0aQZD+69MODil k/M= X-SBRS: 2.7 X-MesageID: 2175570 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,415,1557201600"; d="scan'208";a="2175570" Date: Tue, 25 Jun 2019 10:21:03 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Christian M CC: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: 0146F77644 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.168 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-4.41 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:216.71.155.168]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[168.155.71.216.list.dnswl.org : 127.0.3.0]; NEURAL_HAM_SHORT(-0.97)[-0.967,0]; IP_SCORE(-1.14)[ip: (-2.80), ipnet: 216.71.154.0/23(-1.73), asn: 16417(-1.09), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 08:21:19 -0000 On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote: > Thanks for your input Bjarne. > > Previously I tested with iperf2, but I have made som tests with iperf3 now > also and noticed there are a lot of "Retr" (TCP retries) in some cases. > Went back here to your post and saw that your results also showed a lot of > retries in some cases. My new tests showed similar results to yours > (although not nearly as hi throughput as you have). When Linux is client > the retries are 0, with FreeBSD as client the number is > 0. Can you figure out what caused those retries? Wrong MTU, bad checksums? > I'm not sure > if our numbers are considered high though, and something that actually is a > problem? Although, on a internal network with no external factors like > interference, I feel this should always be 0 no matter what? If properly configured yes, there should be no retries when doing intra-VM connections. Thanks, Roger. From owner-freebsd-xen@freebsd.org Tue Jun 25 11:55:54 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DF7015C9B0E; Tue, 25 Jun 2019 11:55:54 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5984486F5E; Tue, 25 Jun 2019 11:55:53 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2e.google.com with SMTP id r185so2325797iod.6; Tue, 25 Jun 2019 04:55:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=0Q9cJyn5HIuRziOIKWB0r2lYgE6qyQIIlqM/s2ofC7U=; b=EMW1/ap4LKB1vtqJoXyizScKcZb3sNWRA0Pt818HrLweaJbq1OEAhNDGqazZOcl4iX 1/giIK10AnoDqJo8wDKzjYNLbW51MTQYIgwihibNgXKIP16EAJmQ/cpwIS9NAroUmgbj KmZQQVBrUjrmMXF6Q3vqArZlQbvWzl5w2q3XzHT/dBvrM24utu8QIlFHjginOS2+NnEv UDzwzkhNn5XOdc02nD9owU9FYKI3b/bMc4e8c3XFOgC/kxA5bBh4m30QcI6lP3TZ7Yie aSu+XI+KbXXHWs07l44qhVY69vcTN9Nh47WoK3YDFsTbPNsjgyTv+TvXUXv0e+VSie6v Nl+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=0Q9cJyn5HIuRziOIKWB0r2lYgE6qyQIIlqM/s2ofC7U=; b=DmJMLL8lrbABbSIa2ubRhPm6GntivQnyrkuGrunecL7n+3zHnZjz8DzSgr6Wf7vIuH 9WCIJre4s40lyZkhVaxsvfp6w0uF90nsMEuBI78l2BZz4cYhmBtmJ6xkJqoUwqPiogP/ ljCx5KXpw7szBKi83kBvIE90qOAXC3YYG05E1+wJkZbhwcc8ppXWtYy7xGjaFi9VbK2D cpGW/5YRWPGZXH4ZFYM1KJ6Fta2Se+F3oCGNi0R9TL3V3/YUMmkoD9S8l+GNa4Ox//AQ ly6c5Xce62+FSWAm0dCypyp05nsNVn8dn34igSJ+PdMQs+LojYN8+HQwO3u0s9Fm4M0p OxIg== X-Gm-Message-State: APjAAAX9cGl7RCGmMvNFI7ablseZEFaMlXkP9qZRiBOQGOPayvWt80ot odi7GWi7xr5YB8usprUar9uYLWCwl6lfT/CJAvck18Of7do= X-Google-Smtp-Source: APXvYqz5DtJrpGCwNyWRIU710boh8nn0Vl/3v0L3XmjIcu7M47d0nXdzWkbX3mvBGBzoxws+TaRppE17xSHaEXiHf2Q= X-Received: by 2002:a6b:4107:: with SMTP id n7mr1218546ioa.12.1561463751894; Tue, 25 Jun 2019 04:55:51 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> In-Reply-To: <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> From: Christian M Date: Tue, 25 Jun 2019 13:55:40 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 5984486F5E X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=EMW1/ap4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2e as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.66 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[e.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.84)[-0.841,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; IP_SCORE(-2.81)[ip: (-8.55), ipnet: 2607:f8b0::/32(-3.13), asn: 15169(-2.33), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 11:55:54 -0000 I've made two tests while running tcpdump on the xcp-ng host. I'm not at all qualified to interpret the .pcap files from tcpdump, but I've put them on Google Drive and linked them below the two tests. Perhaps someone more qualified could have a look for anything useful in there. Please note the extremely uneven throughput for test 2 below. It's almost like the throughput increased when running tcpdump simultaneously. Host: XCP-ng 7.6.0 Network: Private Network on host, not connected to any PIF. VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) On the host I listen with tcpdump on the VIF for VM1 in both tests. VM1 as client: On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap xcp-ng-vm1-client.pcap (80M): https://drive.google.com/open?id=3D1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU On VM1: iperf3 -c 172.31.16.126 On VM2: iperf3 -s VM1 iperf3 output: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 18182 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 8.00 MBytes 67.1 Mbits/sec 156 15.6 KBytes [ 5] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec 318 1.43 KBytes [ 5] 2.00-3.00 sec 7.51 MBytes 63.1 Mbits/sec 218 1.43 KBytes [ 5] 3.00-4.00 sec 8.29 MBytes 69.3 Mbits/sec 193 1.43 KBytes [ 5] 4.00-5.01 sec 10.7 MBytes 89.4 Mbits/sec 252 1.43 KBytes [ 5] 5.01-6.00 sec 13.6 MBytes 115 Mbits/sec 313 31.3 KBytes [ 5] 6.00-7.00 sec 8.41 MBytes 70.2 Mbits/sec 309 1.43 KBytes [ 5] 7.00-8.01 sec 12.6 MBytes 106 Mbits/sec 223 1.43 KBytes [ 5] 8.01-9.00 sec 12.5 MBytes 106 Mbits/sec 227 1.43 KBytes [ 5] 9.00-10.00 sec 13.5 MBytes 113 Mbits/sec 263 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 109 MBytes 91.6 Mbits/sec 2472 sender [ 5] 0.00-10.01 sec 109 MBytes 91.3 Mbits/sec receiver VM1 as server: On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-server.pcap xcp-ng-vm1-server.pcap: (337M): https://drive.google.com/open?id=3D1N3-GccXaBc6hlzFrgshi2vhcCt8vekxF On VM1: iperf3 -s On VM2: iperf3 -c 172.31.16.125 VM1 iperf3 output: ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 172.31.16.126, port 31523 [ 5] local 172.31.16.125 port 5201 connected to 172.31.16.126 port 34605 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 10.9 MBytes 91.6 Mbits/sec [ 5] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec [ 5] 2.00-3.01 sec 4.92 MBytes 41.0 Mbits/sec [ 5] 3.01-4.00 sec 10.4 MBytes 87.9 Mbits/sec [ 5] 4.00-5.00 sec 8.52 MBytes 71.2 Mbits/sec [ 5] 5.00-6.00 sec 22.0 MBytes 185 Mbits/sec [ 5] 6.00-7.00 sec 9.25 MBytes 77.8 Mbits/sec [ 5] 7.00-8.00 sec 70.8 MBytes 594 Mbits/sec [ 5] 8.00-9.00 sec 89.5 MBytes 752 Mbits/sec [ 5] 9.00-10.00 sec 105 MBytes 884 Mbits/sec [ 5] 10.00-10.00 sec 180 KBytes 1.19 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 346 MBytes 290 Mbits/sec receiver Den tis 25 juni 2019 kl 10:21 skrev Roger Pau Monn=C3=A9 : > On Mon, Jun 24, 2019 at 10:24:09PM +0200, Christian M wrote: > > Thanks for your input Bjarne. > > > > Previously I tested with iperf2, but I have made som tests with iperf3 > now > > also and noticed there are a lot of "Retr" (TCP retries) in some cases. > > Went back here to your post and saw that your results also showed a lot > of > > retries in some cases. My new tests showed similar results to yours > > (although not nearly as hi throughput as you have). When Linux is clien= t > > the retries are 0, with FreeBSD as client the number is > 0. > > Can you figure out what caused those retries? Wrong MTU, bad > checksums? > > > I'm not sure > > if our numbers are considered high though, and something that actually > is a > > problem? Although, on a internal network with no external factors like > > interference, I feel this should always be 0 no matter what? > > If properly configured yes, there should be no retries when doing > intra-VM connections. > > Thanks, Roger. > From owner-freebsd-xen@freebsd.org Tue Jun 25 12:08:19 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D05F15CA99A; Tue, 25 Jun 2019 12:08:19 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B2C5387841; Tue, 25 Jun 2019 12:08:07 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-xen@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x5PC7tfl096801 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 25 Jun 2019 19:07:55 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Christian M , freebsd-xen@freebsd.org, freebsd-net@freebsd.org References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> From: Eugene Grosbein Message-ID: Date: Tue, 25 Jun 2019 19:07:47 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: B2C5387841 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [0.27 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.76)[-0.762,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.13)[-0.134,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_SPAM_SHORT(0.71)[0.707,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(0.06)[asn: 29072(0.27), country: RU(0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 12:08:19 -0000 25.06.2019 18:55, Christian M wrote: > Please note the > extremely uneven throughput for test 2 below. It's almost like the > throughput increased when running tcpdump simultaneously. Generally this indicates broken "hardware" offload for some features like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc. Some of them may be automatically disabled while interface is put in promiscuous mode by running tcpdump and enabled back after tcpdump terminated. You should look at "options" line in the output of ifconfig for the interface and manually disable offloads (look for ifconfig(8) manual for details). They retry tests without tcpdump running. From owner-freebsd-xen@freebsd.org Tue Jun 25 12:40:43 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17F4215CB7EC; Tue, 25 Jun 2019 12:40:43 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd34.google.com (mail-io1-xd34.google.com [IPv6:2607:f8b0:4864:20::d34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA74588CE8; Tue, 25 Jun 2019 12:40:41 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd34.google.com with SMTP id e5so3356414iok.4; Tue, 25 Jun 2019 05:40:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CTbeuJEDDFN3dGYCFzDdWmUXc0xwIMcg1uqPYm4jGBI=; b=NM6mEBNgS0b98g9xzfd6SFaGRX16Y3gYlPjM4cFgQJuij3boYCGlYeiSBCSG1TK3RG m18E7CK5PDbarnowhslP0+baWcNj9vXmtYkWvlDYEdsyuYKC5aOJFfboXz56CVbp6J1M zK2TUtXGWGKNs+RGw40OeXJ5X9udneH6OlsSPdjoX6mM//CS9xpVil3Q/VDhPu280T5O hPgwEluvqw9agKYqDoMiavgiMti8LTWox/sGLVwJ7fEOcPpuBey66/H5HUb/m5wGyT1u jbXmOrBJtVMXM5K/X0S2CR2cbGsLrLxP4FbYzXkyfQS80iswyafNww1FsYi7Ug+Wo8Fv Bv2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CTbeuJEDDFN3dGYCFzDdWmUXc0xwIMcg1uqPYm4jGBI=; b=ny6k72LImxO8QDuA+01zBveNjo7h/quv1ysa6DmRBbEcCaqfFaDp28K0tjsmC57OQt wnn+rOqoC/K/n+gCV0UsWUeiCjDrnAOdRC0/xcNcMP2o6jed55ePUVq+Z1bd5xaTwuxJ W3iIP+7sW1zH2ra311/YO/DymcC40CQleEeM2YKmNRW8m+lDTIXusVcX0RVeTJeDBZ10 ELeLOxgJktSl3EJtANBb6QPNj57k7uebe2tkHAeQMAbQnIczVHU6HDFHqYeLJUDbW7q6 lX7Z3+F/qjLjK9Cqbi89n1ZU1tauqDwZZcwLTMSLG8KcozWRuiF1Xz9rfYIYyxiWWjnq OJSw== X-Gm-Message-State: APjAAAUS8XEnF1SizyUEQkfFU4EnUwS3zo2dS7FU3j9WxN7Pb4i4aW6n m4oNOwETiUyG9I7XoSrYbzg9mh0ljoniSxoJdjmFceIG X-Google-Smtp-Source: APXvYqyU2kj5bMuTpL1ByLV1DhYTPyEdE0/5svZ1kA2vDqGFVKuodC5XJfXk4cUku6NT0F/4EIdtFZxK52aIb59X2zQ= X-Received: by 2002:a5e:a710:: with SMTP id b16mr23053160iod.38.1561466440695; Tue, 25 Jun 2019 05:40:40 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> In-Reply-To: From: Christian M Date: Tue, 25 Jun 2019 14:40:29 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Eugene Grosbein Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: CA74588CE8 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=NM6mEBNg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d34 as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.54 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[4.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.60)[ip: (-7.50), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.33), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.93)[-0.926,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 12:40:43 -0000 I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0 -txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF via XCP-ng earlier. It made no difference unfortunately. Den tis 25 juni 2019 kl 14:08 skrev Eugene Grosbein : > 25.06.2019 18:55, Christian M wrote: > > > Please note the > > extremely uneven throughput for test 2 below. It's almost like the > > throughput increased when running tcpdump simultaneously. > > Generally this indicates broken "hardware" offload for some features > like txcsum/rxcsum/vlan_hwcsum/vlan_hwtagging etc. > > Some of them may be automatically disabled while interface is put in > promiscuous mode > by running tcpdump and enabled back after tcpdump terminated. > > You should look at "options" line in the output of ifconfig for the > interface > and manually disable offloads (look for ifconfig(8) manual for details). > They retry tests without tcpdump running. > > From owner-freebsd-xen@freebsd.org Tue Jun 25 14:58:06 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E2FAF15CEABD; Tue, 25 Jun 2019 14:58:05 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa4.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 454108E690; Tue, 25 Jun 2019 14:58:03 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa4.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: F+ZM2OCDeMS9HPxSOqBkcCbxLavjozwQSUSWl6QMHa9BxgRax+sCXgfrs8RZ4diyZ7C3X94ipP DTkqMQiCSG0ZV/s+LXgPKDcWulana2ijQiEH68Pad9ohWDBInB8k2qIdu9oamLoZHP9wl/uQcb wqcYn59yi0CztRAhJ3+qX5oK/oPUs4b0DHX9fVxH4gaudtPJCpqVZUT7j1frV5bI/uAGVxHfGU zhBxSK0W43c4QrAMGbQ0I3phnU/bqLDXPM4YJlwIunfWiRsJJZM7WZLZi5nuawo0FqqRGVt116 kEE= X-SBRS: 2.7 X-MesageID: 2229330 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,416,1557201600"; d="scan'208";a="2229330" Date: Tue, 25 Jun 2019 16:56:43 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Christian M CC: , Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <20190625145643.shasw7bcgu6s2hg7@MacBook-Air-de-Roger.local> References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: 454108E690 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.144 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-3.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:216.71.155.144]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mx1.hc3370-68.iphmx.com]; NEURAL_HAM_SHORT(-0.80)[-0.800,0]; RCVD_IN_DNSWL_NONE(0.00)[144.155.71.216.list.dnswl.org : 127.0.3.0]; IP_SCORE(-0.86)[ip: (-1.33), ipnet: 216.71.154.0/23(-1.80), asn: 16417(-1.10), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 14:58:06 -0000 On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote: > I've made two tests while running tcpdump on the xcp-ng host. I'm not at > all qualified to interpret the .pcap files from tcpdump, but I've put them > on Google Drive and linked them below the two tests. Perhaps someone more > qualified could have a look for anything useful in there. Please note the > extremely uneven throughput for test 2 below. It's almost like the > throughput increased when running tcpdump simultaneously. > > Host: XCP-ng 7.6.0 > Network: Private Network on host, not connected to any PIF. > VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) > VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) > > On the host I listen with tcpdump on the VIF for VM1 in both tests. > > VM1 as client: > > On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap Can you check the capabilities of vif42.0? (ie: whether csum offloading is actually disabled on the host?) > xcp-ng-vm1-client.pcap (80M): > https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU > On VM1: iperf3 -c 172.31.16.126 > On VM2: iperf3 -s I've taken a look at the dump and the checksum is wrong (or maybe missing) for all? packets. Packets with source 172.31.16.125 all have the TCP checksum set to 0x7f80 and all packets with source 172.31.16.125 have the TCP checksum set to 0x7960. FTR, can you also paste the ifconfig output of each interface before running iperf? Thanks, Roger. From owner-freebsd-xen@freebsd.org Tue Jun 25 15:36:45 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8345915CF9E9; Tue, 25 Jun 2019 15:36:45 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa2.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3EF858FE52; Tue, 25 Jun 2019 15:36:44 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: cFXwqpBxqVWw+gmt56iij5eI5grY18aP974hRWq2gJyHha9tud1D8f1sVBEcSpLOxU0846HSK9 FC9tvrIq9+wal61//Jt6MueOrc0cg3Ky5VDEfKLzZOOYoA9jb/kDWUKITEyMwqWd6ggCGFWOrK tnfeHwB0N0YFw5xhT4bSNBRlu02lJBGQ9vdP8DfxHSv4tq/R/66kpjpE2Id/4sgXlNqQn5sGGb 0PolhFeH2H/gQFP6r/smVHEn4jjvPP7suiM/tfQwVpiQiNGIXDqGyfZa0tDh9JiOQPUD1Mwn0r WuY= X-SBRS: 2.7 X-MesageID: 2192123 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,416,1557201600"; d="scan'208";a="2192123" Date: Tue, 25 Jun 2019 17:01:24 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Roger Pau =?utf-8?B?TW9ubsOp?= CC: Christian M , , Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <20190625150124.wiqikig623k3lzuv@MacBook-Air-de-Roger.local> References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <20190625145643.shasw7bcgu6s2hg7@MacBook-Air-de-Roger.local> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20190625145643.shasw7bcgu6s2hg7@MacBook-Air-de-Roger.local> User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: 3EF858FE52 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.145.153 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-3.69 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:216.71.145.153]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mx1.hc3370-68.iphmx.com]; NEURAL_HAM_SHORT(-0.83)[-0.825,0]; RCVD_IN_DNSWL_NONE(0.00)[153.145.71.216.list.dnswl.org : 127.0.3.0]; IP_SCORE(-0.55)[ip: (-1.33), ipnet: 216.71.145.0/24(-0.27), asn: 16417(-1.11), country: US(-0.06)]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 15:36:45 -0000 There's a mistake in my reply below. On Tue, Jun 25, 2019 at 04:56:43PM +0200, Roger Pau Monné wrote: > On Tue, Jun 25, 2019 at 01:55:40PM +0200, Christian M wrote: > > I've made two tests while running tcpdump on the xcp-ng host. I'm not at > > all qualified to interpret the .pcap files from tcpdump, but I've put them > > on Google Drive and linked them below the two tests. Perhaps someone more > > qualified could have a look for anything useful in there. Please note the > > extremely uneven throughput for test 2 below. It's almost like the > > throughput increased when running tcpdump simultaneously. > > > > Host: XCP-ng 7.6.0 > > Network: Private Network on host, not connected to any PIF. > > VM1: 12.0-RELEASE (1 VIF, 172.31.16.125) > > VM2: 12.0-RELEASE (1 VIF, 172.31.15.126) > > > > On the host I listen with tcpdump on the VIF for VM1 in both tests. > > > > VM1 as client: > > > > On XCP-ng: tcpdump -i vif42.0 -s 0 -w xcp-ng-vm1-client.pcap > > Can you check the capabilities of vif42.0? (ie: whether csum > offloading is actually disabled on the host?) > > > xcp-ng-vm1-client.pcap (80M): > > https://drive.google.com/open?id=1eR3fetvKRz3vFSXCxDKuJYFrQ3wLqjrU > > On VM1: iperf3 -c 172.31.16.126 > > On VM2: iperf3 -s > > I've taken a look at the dump and the checksum is wrong (or maybe > missing) for all? packets. > > Packets with source 172.31.16.125 all have the TCP checksum set to > 0x7f80 and all packets with source 172.31.16.125 have the TCP checksum ^ 172.31.16.126 From owner-freebsd-xen@freebsd.org Tue Jun 25 17:52:56 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3022B15D2981 for ; Tue, 25 Jun 2019 17:52:56 +0000 (UTC) (envelope-from bjb@c.dk) Received: from outgoing-yousee-3.gl-mut-gbl.as8677.net (outgoing-yousee-3.gl-mut-gbl.as8677.net [193.201.76.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B37196E4CF for ; Tue, 25 Jun 2019 17:52:53 +0000 (UTC) (envelope-from bjb@c.dk) Received: from filter.yousee.as8677.net (localhost [10.26.226.131]) by mwumf0305.yousee.as8677.net (SMTP Server) with ESMTP id 45YDJr6yyrzrF8 for ; Tue, 25 Jun 2019 19:52:44 +0200 (CEST) Received: from mwumf0305 (front03-smtp-abo-yousee.worldlinemail.net [10.26.226.131]) by mwumf0305.yousee.as8677.net (SMTP Server) with ESMTP id 45YDJr6SbDzrF2 for ; Tue, 25 Jun 2019 19:52:44 +0200 (CEST) Received: from mwumf0305 (front03-smtp-abo-yousee.worldlinemail.net [10.26.226.131]) (Authenticated sender: 120107804745) by mwumf0305.yousee.as8677.net (SMTP Server) with ESMTPA for ; Tue, 25 Jun 2019 19:52:44 +0200 (CEST) Subject: Re: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: freebsd-xen@freebsd.org References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> From: Bjarne Blichfeldt Organization: BlichSoft Message-ID: Date: Tue, 25 Jun 2019 19:52:44 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: X-yse-mailing: LEGIT X-yse-spamcause: OK, (0)(0000)gggruggvucftvghtrhhoucdtuddrgeduvddrudeggdduudeiucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuvfffvedpggftfgfpufgfuefuveftkfeugfdpqfgfvfenuceurghilhhouhhtmecufedttdenucenucfjughrpefuvfhfhfhokffffgggjggtgfesthekredttdefjeenucfhrhhomhepuehjrghrnhgvuceulhhitghhfhgvlhguthcuoegsjhgssegtrdgukheqnecukfhppedutddrvdeirddvvdeirddufedunecurfgrrhgrmhephhgvlhhopehmfihumhhftdeftdehpdhinhgvthepuddtrddviedrvddviedrudefuddpmhgrihhlfhhrohhmpegsjhgssegtrdgukhdprhgtphhtthhopehfrhgvvggsshguqdigvghnsehfrhgvvggsshgurdhorhhgnecuvehluhhsthgvrhfuihiivgeptd Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: B37196E4CF X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.65 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:193.201.76.63/32]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[fpo9.mail.dk]; DKIM_TRACE(0.00)[c.dk:+]; DMARC_POLICY_ALLOW(-0.50)[c.dk,quarantine]; NEURAL_HAM_SHORT(-0.99)[-0.986,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8677, ipnet:193.201.76.0/24, country:FR]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[c.dk:s=sela]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.65)[ipnet: 193.201.76.0/24(-1.81), asn: 8677(-1.45), country: FR(-0.01)] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 17:52:56 -0000 oh yes, sorry I am using: 0:~ # xl info host                   : x0 release                : 4.12.14-lp150.12.64-default version                : #1 SMP Mon Jun 17 16:53:50 UTC 2019 (3edfd41) machine                : x86_64 nr_cpus                : 4 max_cpu_id             : 3 nr_nodes               : 1 cores_per_socket       : 4 threads_per_core       : 1 cpu_mhz                : 3192 hw_caps                : bfebfbff:77faf3bf:2c100800:00000021:00000001:000027ab:00000000:00000100 virt_caps              : hvm hvm_directio total_memory           : 32461 free_memory            : 5319 sharing_freed_memory   : 0 sharing_used_memory    : 0 outstanding_claims     : 0 free_cpus              : 0 xen_major              : 4 xen_minor              : 10 xen_extra              : .3_04-lp150.2.1 xen_version            : 4.10.3_04-lp150.2.1 xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler          : credit xen_pagesize           : 4096 platform_params        : virt_start=0xffff800000000000 xen_changeset          : xen_commandline        : dom0_mem=3072M,max:3072M cc_compiler            : gcc (SUSE Linux) 7.4.0 cc_compile_by          : abuild cc_compile_domain      : suse.de cc_compile_date        : Thu May  9 16:10:08 UTC 2019 build_id               : bf77d035ff05eb55d9616e418f73cbcbd96f747b xend_config_format     : 4 Since you peaked my interest, I tryed playing around a little. Initially to day, the retransmits where between 4-15   retransmits. I then changed the network interface from hypervisor default to  E1000 (still xn0 in freebsd) and the retransmits jumped up to 700 to 800! After changing the network interface back to hypervisor default, the number of retranmits remains high in the 3-digit area.  Fooling around with mtu size did not change anything. Something is not quite right. On 6/24/19 10:24 PM, Christian M wrote: > Thanks for your input Bjarne. > > Previously I tested with iperf2, but I have made som tests with iperf3 now > also and noticed there are a lot of "Retr" (TCP retries) in some cases. > Went back here to your post and saw that your results also showed a lot of > retries in some cases. My new tests showed similar results to yours > (although not nearly as hi throughput as you have). When Linux is client > the retries are 0, with FreeBSD as client the number is > 0. I'm not sure > if our numbers are considered high though, and something that actually is a > problem? Although, on a internal network with no external factors like > interference, I feel this should always be 0 no matter what? > > What hypervisor were you running in your tests? Version? > > Regards, > Christian > > > Den mÃ¥n 24 juni 2019 kl 17:56 skrev Bjarne : > >> Well, to add some datapoints, I just did a quick test om my private >> homeserver, testing with iperf3 >> -s / iperf3 -c >> >> I have not much knowledge of iperf3 network test and it's validity, but >> there is definitely some >> differences to be seen. >> >> Summary: >> >> FBSD <-> FBSD 3,5Gbit/sec >> FBSD -> Centos: 3GBit/Sec >> Centos -> FBSD: 12 Gbit/sec >> :snip >> >> >> -- mvh, Bjarne From owner-freebsd-xen@freebsd.org Tue Jun 25 23:14:55 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D494F15D96D1; Tue, 25 Jun 2019 23:14:55 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A8CAB82B61; Tue, 25 Jun 2019 23:14:44 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-xen@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x5PNEYsm001645 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 26 Jun 2019 06:14:34 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Christian M References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org From: Eugene Grosbein Message-ID: <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> Date: Wed, 26 Jun 2019 06:14:26 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: A8CAB82B61 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-0.04 / 15.00]; MX_INVALID(0.50)[greylisted]; TO_DN_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.78)[-0.782,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.39)[0.389,0]; NEURAL_HAM_LONG(-0.10)[-0.105,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(0.06)[asn: 29072(0.26), country: RU(0.01)]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Jun 2019 23:14:56 -0000 25.06.2019 19:40, Christian M wrote: > I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig xn0 > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the VIF > via XCP-ng earlier. It made no difference unfortunately. Then you should try to bring the interface to promiscuous mode manually with "ifconfig xn0 promisc" any retry. If this does not make a difference, then you may be facing some timekeeping problem as tcpdump makes additional CPU load on the system that can force scheduler behaviour to change. As next step, you should check your kern.timecounter.* and kern.eventtimer.* sysctls. If default value of kern.eventtimer.periodic is 0 (it depends on detected "hardware"), you should change it kern.eventtimer.periodic=1 and retry the test. From owner-freebsd-xen@freebsd.org Wed Jun 26 08:12:02 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B50DE15BCB29; Wed, 26 Jun 2019 08:12:01 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 75861927DB; Wed, 26 Jun 2019 08:12:00 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd33.google.com with SMTP id k8so404157iot.1; Wed, 26 Jun 2019 01:12:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+QQPK/xM0PZMFb1Q6QCnAGmc92GXmGQPoAeIQtSEbk0=; b=ujvEmiZCsW/KoBzUvu0HgRbbEvfmKKKPbJ4svngI/Qje/FBfm1kWK749iSDqWtNqb1 gkd3d0yeSEK9QqV8rb7mVEetAQQWvWDm0z0VZJvH9s/R1q3TRmOKmegE6kWxQjhXChTX NRpFAUXccIy5coc2c7piKn2WjHmMofks1vtfRRHOG8HTskOyiWREdx1Bt08JDbMBljm+ BX5HvSX6ixoMJVmcCYgr0yrwebtNjl7i/xlGNpMvl8QiSAwzX9xYrALWv2DNyBqGw8rn CFiWGFD/Cbcq8bAHtUcEXP8UDXw8DV7zzfHredL5Qn9nivYvkPbBQRV4x1glCx8nzB75 S24A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+QQPK/xM0PZMFb1Q6QCnAGmc92GXmGQPoAeIQtSEbk0=; b=O9xNonA/p4VX+uMWOIKVYOTMk/ELUKeU2ehOODshcP1o7CIiAvAq4VeMGfrh3GcjZ/ DMVsy2M8u1URH2vES11M8zfvcb5PlEv8VFaWWDNm0e2/d9J8iqJHLMKBQLJrDMUtWGIE P6wWsictm958i1SpvvFfJP7onhluv5grGgqrPnakelQAwKYSC80QGTGjeqH6EJE7vr3v 1CBdkkZe3XjNaSoDpSdon2OTC/jaIj+A51+yHg631eqL2uvPBHrmfOSvdRgmCJlKFnP9 Udv+yjaateWOVwgn2+8jhtvODolJ+qlT1CjFcBf3E63zxEf1zTah/P3sV83SA8HQBheZ gNFQ== X-Gm-Message-State: APjAAAUrVQtaGiR31qaV3YLCqNxyhj1gLwKtS0rz3eCKYPAZwEAtE1in s491LkdCegocGCzf9hm/ouLf1aLjzTJS12Zg1/9ZZb/J+io= X-Google-Smtp-Source: APXvYqxGaVTUa+77kCobtJZ4vgJoG4icV5NNp0DrybYQvq5pwXUdGqYseUMV+rU0zwjDx0IYd57PuYxg3vII+DMIsys= X-Received: by 2002:a6b:2cc7:: with SMTP id s190mr3603681ios.29.1561536718778; Wed, 26 Jun 2019 01:11:58 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> In-Reply-To: <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> From: Christian M Date: Wed, 26 Jun 2019 10:11:47 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Eugene Grosbein Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 75861927DB X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ujvEmiZC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d33 as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.64 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[3.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.77)[ip: (-8.31), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.33), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.86)[-0.864,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 08:12:02 -0000 checksum offloading doe not seem to have any affect when done on the VM VIF's. Both 12.0-RELEASE VIF's now have: other-config (MRW): ethtool-rx: off; ethtool-tx: off; ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off ethtool -k vif54.0 Features for vif54.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Both vif's look the same as above. Running tcpdump on the host while running iperf3 between the 12.0 VM's results in a lot of incorrect cksum like this. tcpdump -i vif54.0 -v -nn| grep -i incorrect 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 ... Also noticed that iperf3 does not work to run every time, more than 50% of the tries with iperf3 -c I get: Connecting to host 172.31.16.126, port 5201 iperf3: error - unable to initialize stream: Socket is not connected Or nothing happens and I need to abort, then restart iperf3 -s on the other VM, and retry. The results from iperf3 are really inconsistent still, and I cant get a good feel of any pattern. Currently as I type this, a result looks like this: Host: tcpdump running VIF's: no promisc xn0 on both VM's: no promisc [ 5] local 172.31.16.125 port 47738 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 37.8 MBytes 316 Mbits/sec 176 1.43 KBytes [ 5] 1.00-2.00 sec 133 MBytes 1.11 Gbits/sec 1022 49.8 KBytes [ 5] 2.00-3.00 sec 179 MBytes 1.50 Gbits/sec 784 191 KBytes [ 5] 3.00-4.00 sec 132 MBytes 1.11 Gbits/sec 201 345 KBytes [ 5] 4.00-5.00 sec 95.3 MBytes 799 Mbits/sec 438 180 KBytes [ 5] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 1256 353 KBytes [ 5] 6.00-7.00 sec 144 MBytes 1.20 Gbits/sec 1242 275 KBytes [ 5] 7.00-8.00 sec 94.5 MBytes 792 Mbits/sec 593 167 KBytes [ 5] 8.00-9.00 sec 125 MBytes 1.05 Gbits/sec 326 169 KBytes [ 5] 9.00-10.00 sec 57.0 MBytes 478 Mbits/sec 1013 1.43 KBytes [ 5] 10.00-11.00 sec 31.3 MBytes 262 Mbits/sec 463 1.43 KBytes [ 5] 11.00-12.00 sec 42.3 MBytes 355 Mbits/sec 544 78.3 KBytes [ 5] 12.00-13.00 sec 41.3 MBytes 346 Mbits/sec 624 1.43 KBytes [ 5] 13.00-14.00 sec 52.7 MBytes 442 Mbits/sec 739 1.43 KBytes [ 5] 14.00-15.00 sec 20.4 MBytes 171 Mbits/sec 384 1.43 KBytes [ 5] 15.00-16.00 sec 74.5 MBytes 625 Mbits/sec 489 299 KBytes [ 5] 16.00-17.00 sec 40.1 MBytes 337 Mbits/sec 835 1.43 KBytes [ 5] 17.00-18.00 sec 43.2 MBytes 363 Mbits/sec 372 1.43 KBytes [ 5] 18.00-19.00 sec 58.9 MBytes 494 Mbits/sec 723 1.43 KBytes [ 5] 19.00-20.00 sec 21.2 MBytes 178 Mbits/sec 300 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec 12524 sender [ 5] 0.00-20.00 sec 1.51 GBytes 649 Mbits/sec receiver But please be sure, this can change dramatically without making any changes what so ever. Offloading is not turned off in FreeBSD (but on the VIF's as stated before), and tcpdump is running on the host. xn0: flags=8843 metric 0 mtu 1500 options=503 ether 12:ec:6a:5c:af:df inet 172.31.16.125 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29 xn0: flags=8843 metric 0 mtu 1500 options=503 ether aa:4f:f7:4c:a0:e6 inet 172.31.16.126 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=29 Absolutely nothing changed this time (restarted iperf3 -s though since I ran ifconfig do get the info above), tcpdump still running on host (never even aborted): Host: tcpdump still running VIF's: no promisc xn0 on both VM's: no promisc [ 5] local 172.31.16.125 port 30543 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.3 MBytes 86.8 Mbits/sec 165 38.4 KBytes [ 5] 1.00-2.00 sec 39.7 MBytes 333 Mbits/sec 175 167 KBytes [ 5] 2.00-3.00 sec 40.0 MBytes 335 Mbits/sec 437 54.1 KBytes [ 5] 3.00-4.00 sec 11.2 MBytes 93.6 Mbits/sec 320 1.43 KBytes [ 5] 4.00-5.00 sec 11.2 MBytes 94.2 Mbits/sec 305 1.43 KBytes [ 5] 5.00-6.00 sec 7.63 MBytes 64.1 Mbits/sec 179 2.84 KBytes [ 5] 6.00-7.00 sec 6.77 MBytes 56.7 Mbits/sec 191 1.43 KBytes [ 5] 7.00-8.00 sec 4.26 MBytes 35.9 Mbits/sec 145 21.3 KBytes [ 5] 8.00-9.00 sec 5.38 MBytes 45.1 Mbits/sec 176 1.43 KBytes [ 5] 9.00-10.00 sec 8.24 MBytes 69.1 Mbits/sec 225 1.43 KBytes [ 5] 10.00-11.00 sec 16.1 MBytes 135 Mbits/sec 422 1.43 KBytes [ 5] 11.00-12.00 sec 3.76 MBytes 31.5 Mbits/sec 113 1.43 KBytes [ 5] 12.00-13.00 sec 6.62 MBytes 55.5 Mbits/sec 161 1.43 KBytes [ 5] 13.00-14.00 sec 5.28 MBytes 44.3 Mbits/sec 156 18.4 KBytes [ 5] 14.00-15.00 sec 5.21 MBytes 43.7 Mbits/sec 146 1.43 KBytes [ 5] 15.00-16.00 sec 6.37 MBytes 53.4 Mbits/sec 199 1.43 KBytes [ 5] 16.00-17.00 sec 10.7 MBytes 89.5 Mbits/sec 193 1.43 KBytes [ 5] 17.00-18.00 sec 9.49 MBytes 79.6 Mbits/sec 390 1.43 KBytes [ 5] 18.00-19.00 sec 5.36 MBytes 44.9 Mbits/sec 125 1.43 KBytes [ 5] 19.00-20.00 sec 3.65 MBytes 30.6 Mbits/sec 58 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 217 MBytes 91.1 Mbits/sec 4281 sender [ 5] 0.00-20.00 sec 217 MBytes 90.8 Mbits/sec receiver Host: tcpdump still running VIF's: promisc xn0 on both VM's: promisc Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 61196 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 25.9 MBytes 217 Mbits/sec 271 90.5 KBytes [ 5] 1.00-2.00 sec 56.4 MBytes 473 Mbits/sec 94 255 KBytes [ 5] 2.00-3.00 sec 83.4 MBytes 699 Mbits/sec 872 201 KBytes [ 5] 3.00-4.00 sec 76.4 MBytes 641 Mbits/sec 684 176 KBytes [ 5] 4.00-5.00 sec 80.4 MBytes 674 Mbits/sec 742 1.43 KBytes [ 5] 5.00-6.00 sec 81.4 MBytes 684 Mbits/sec 248 333 KBytes [ 5] 6.00-7.00 sec 86.1 MBytes 722 Mbits/sec 826 160 KBytes [ 5] 7.00-8.00 sec 77.5 MBytes 650 Mbits/sec 209 253 KBytes [ 5] 8.00-9.00 sec 9.64 MBytes 80.9 Mbits/sec 433 12.8 KBytes [ 5] 9.00-10.00 sec 8.31 MBytes 69.6 Mbits/sec 210 1.43 KBytes [ 5] 10.00-11.00 sec 8.64 MBytes 72.5 Mbits/sec 199 1.43 KBytes [ 5] 11.00-12.00 sec 7.02 MBytes 58.9 Mbits/sec 240 1.43 KBytes [ 5] 12.00-13.00 sec 20.1 MBytes 169 Mbits/sec 579 1.43 KBytes [ 5] 13.00-14.00 sec 10.8 MBytes 90.4 Mbits/sec 277 22.7 KBytes [ 5] 14.00-15.00 sec 2.75 MBytes 23.1 Mbits/sec 107 1.43 KBytes [ 5] 15.00-16.00 sec 8.79 MBytes 73.8 Mbits/sec 203 12.8 KBytes [ 5] 16.00-17.00 sec 9.23 MBytes 77.3 Mbits/sec 158 1.43 KBytes [ 5] 17.00-18.00 sec 13.5 MBytes 114 Mbits/sec 381 1.43 KBytes [ 5] 18.00-19.00 sec 41.6 MBytes 349 Mbits/sec 331 99.5 KBytes [ 5] 19.00-20.00 sec 40.6 MBytes 340 Mbits/sec 70 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 749 MBytes 314 Mbits/sec 7134 sender [ 5] 0.00-20.00 sec 748 MBytes 314 Mbits/sec receiver Host: tcpdump NOT running VIF's: no promisc xn0: no promisc Run1: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 20600 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 28.4 MBytes 238 Mbits/sec 451 1.43 KBytes [ 5] 1.00-2.00 sec 21.5 MBytes 181 Mbits/sec 546 2.84 KBytes [ 5] 2.00-3.00 sec 26.7 MBytes 224 Mbits/sec 595 1.43 KBytes [ 5] 3.00-4.00 sec 15.9 MBytes 133 Mbits/sec 347 1.43 KBytes [ 5] 4.00-5.00 sec 4.45 MBytes 37.4 Mbits/sec 187 2.84 KBytes [ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec 239 66.5 KBytes [ 5] 6.00-7.00 sec 10.6 MBytes 89.1 Mbits/sec 366 1.43 KBytes [ 5] 7.00-8.00 sec 30.8 MBytes 258 Mbits/sec 510 1.43 KBytes [ 5] 8.00-9.00 sec 16.6 MBytes 140 Mbits/sec 332 2.84 KBytes [ 5] 9.00-10.00 sec 12.4 MBytes 104 Mbits/sec 376 1.43 KBytes [ 5] 10.00-11.00 sec 11.3 MBytes 94.9 Mbits/sec 223 1.43 KBytes [ 5] 11.00-12.00 sec 9.00 MBytes 75.5 Mbits/sec 206 1.43 KBytes [ 5] 12.00-13.00 sec 8.95 MBytes 75.1 Mbits/sec 201 1.43 KBytes [ 5] 13.00-14.00 sec 12.3 MBytes 103 Mbits/sec 179 1.43 KBytes [ 5] 14.00-15.00 sec 20.5 MBytes 172 Mbits/sec 445 1.43 KBytes [ 5] 15.00-16.00 sec 3.92 MBytes 32.9 Mbits/sec 163 1.43 KBytes [ 5] 16.00-17.00 sec 6.82 MBytes 57.2 Mbits/sec 122 2.84 KBytes [ 5] 17.00-18.00 sec 14.3 MBytes 120 Mbits/sec 238 1.43 KBytes [ 5] 18.00-19.00 sec 14.2 MBytes 119 Mbits/sec 226 1.43 KBytes [ 5] 19.00-20.00 sec 12.9 MBytes 108 Mbits/sec 297 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 297 MBytes 125 Mbits/sec 6249 sender [ 5] 0.00-20.00 sec 297 MBytes 124 Mbits/sec receiver Run2 right after a iperf3 -s restart on second VM: Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 40622 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.49 MBytes 54.3 Mbits/sec 163 1.43 KBytes [ 5] 1.00-2.00 sec 9.93 MBytes 83.3 Mbits/sec 263 1.43 KBytes [ 5] 2.00-3.00 sec 7.71 MBytes 64.7 Mbits/sec 173 1.43 KBytes [ 5] 3.00-4.00 sec 5.58 MBytes 47.0 Mbits/sec 83 1.43 KBytes [ 5] 4.00-5.00 sec 14.8 MBytes 124 Mbits/sec 347 1.43 KBytes [ 5] 5.00-6.00 sec 16.8 MBytes 141 Mbits/sec 349 1.43 KBytes [ 5] 6.00-7.00 sec 19.2 MBytes 161 Mbits/sec 205 1.43 KBytes [ 5] 7.00-8.00 sec 6.39 MBytes 53.6 Mbits/sec 79 1.43 KBytes [ 5] 8.00-9.00 sec 7.19 MBytes 60.3 Mbits/sec 133 1.43 KBytes [ 5] 9.00-10.00 sec 18.2 MBytes 153 Mbits/sec 265 1.43 KBytes [ 5] 10.00-11.00 sec 11.0 MBytes 92.6 Mbits/sec 220 1.43 KBytes [ 5] 11.00-12.00 sec 7.93 MBytes 66.6 Mbits/sec 172 1.43 KBytes [ 5] 12.00-13.00 sec 10.1 MBytes 84.8 Mbits/sec 198 1.43 KBytes [ 5] 13.00-14.00 sec 7.92 MBytes 66.5 Mbits/sec 196 1.43 KBytes [ 5] 14.00-15.00 sec 7.63 MBytes 64.0 Mbits/sec 144 1.43 KBytes [ 5] 15.00-16.00 sec 8.70 MBytes 73.0 Mbits/sec 267 1.43 KBytes [ 5] 16.00-17.00 sec 16.0 MBytes 134 Mbits/sec 296 1.43 KBytes [ 5] 17.00-18.00 sec 10.9 MBytes 91.5 Mbits/sec 261 1.43 KBytes [ 5] 18.00-19.00 sec 23.0 MBytes 193 Mbits/sec 444 1.43 KBytes [ 5] 19.00-20.00 sec 8.86 MBytes 74.3 Mbits/sec 224 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 224 MBytes 94.1 Mbits/sec 4482 sender [ 5] 0.00-20.00 sec 224 MBytes 93.9 Mbits/sec receiver As suggested before, I have set kern.eventtimer.periodic: 1, the above tests are done with that setting. I don't see any difference with it set or not (kern.eventtimer.periodic: 0 was default btw). There was also a suggestion to have a look at kern.timecouter.*, but I'm not sure what to look for here: kern.timecounter.tsc_shift: 1 kern.timecounter.smp_tsc_adjust: 0 kern.timecounter.smp_tsc: 0 kern.timecounter.invariant_tsc: 0 kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: XENTIMER(950) ACPI-fast(900) i8254(0) HPET(950) TSC-low(-100) dummy(-1000000) kern.timecounter.hardware: XENTIMER kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 0 kern.timecounter.tc.XENTIMER.quality: 950 kern.timecounter.tc.XENTIMER.frequency: 1000000000 kern.timecounter.tc.XENTIMER.counter: 3517394506 kern.timecounter.tc.XENTIMER.mask: 4294967295 kern.timecounter.tc.ACPI-fast.quality: 900 kern.timecounter.tc.ACPI-fast.frequency: 3579545 kern.timecounter.tc.ACPI-fast.counter: 1749326798 kern.timecounter.tc.ACPI-fast.mask: 4294967295 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.counter: 58432 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.HPET.quality: 950 kern.timecounter.tc.HPET.frequency: 62500000 kern.timecounter.tc.HPET.counter: 3606880145 kern.timecounter.tc.HPET.mask: 4294967295 kern.timecounter.tc.TSC-low.quality: -100 kern.timecounter.tc.TSC-low.frequency: 1200075192 kern.timecounter.tc.TSC-low.counter: 3248698599 kern.timecounter.tc.TSC-low.mask: 4294967295 Regards, Christian Den ons 26 juni 2019 kl 01:14 skrev Eugene Grosbein : > 25.06.2019 19:40, Christian M wrote: > > > I've disabled them all on both VM's with FreeBSD 12.0-RELEASE (ifconfig > xn0 > > -txcsum -rxcsum -lro -tso), and also tried disabling everything for the > VIF > > via XCP-ng earlier. It made no difference unfortunately. > > Then you should try to bring the interface to promiscuous mode manually > with "ifconfig xn0 promisc" any retry. If this does not make a difference, > then you may be facing some timekeeping problem as tcpdump makes additional > CPU load on the system that can force scheduler behaviour to change. > > As next step, you should check your kern.timecounter.* and > kern.eventtimer.* sysctls. > If default value of kern.eventtimer.periodic is 0 (it depends on detected > "hardware"), > you should change it kern.eventtimer.periodic=1 and retry the test. > > From owner-freebsd-xen@freebsd.org Wed Jun 26 08:37:24 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C129515C082D; Wed, 26 Jun 2019 08:37:24 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C55879360D; Wed, 26 Jun 2019 08:37:13 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-xen@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x5Q8aoVt005684 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 26 Jun 2019 15:36:50 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Christian M References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org From: Eugene Grosbein Message-ID: Date: Wed, 26 Jun 2019 15:36:42 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: C55879360D X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-0.01 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.80)[-0.801,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.15)[-0.150,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_SPAM_SHORT(0.48)[0.485,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(0.05)[asn: 29072(0.26), country: RU(0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 08:37:24 -0000 26.06.2019 15:11, Christian M wrote: > Running tcpdump on the host while running iperf3 between the 12.0 VM's > results in a lot of incorrect cksum like this. > > tcpdump -i vif54.0 -v -nn| grep -i incorrect > 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 > (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options > [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 Incorrect checksum shown by tcpdump is normal and expected for outgoing packets if txcsum offload is not disabled because networking stack skips checksum calculation offloading it to "hardware". It still indicates a bug at sending side if seen for outgoing packets when txcsum is disabled. Incorrect checksum shown by tcpdump for incoming packets always indicates a problem at sending side or intermediate network (if any). You should check output of "netstat -sp ip" and "netstat -sp tcp" for non-zero and growing checksum error counters at receiving side. From owner-freebsd-xen@freebsd.org Wed Jun 26 10:26:22 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 108FB15C36CB; Wed, 26 Jun 2019 10:26:22 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1D04E96E09; Wed, 26 Jun 2019 10:26:21 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2c.google.com with SMTP id s7so836132iob.11; Wed, 26 Jun 2019 03:26:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ce/Y75hv8f3Z1k/tyNNQe2EEt3ZrAZL81QwAPF/GOZg=; b=hNtj5RHucXSHeD2mRQr0MCuFfmG77Z3hqo1LfuPiRHvBakc6p9zTB9/caCEGBvPsst 4SLzpzCVW03+9E8tt6Y6Z3Mf3DbGFSIO3sARF+jt2XfmRF1buyVEWqOCJ3dlTCRwLbPk A2OZjh8aXGdCwx4I/61IEO4ozODOa/XrTSrWWQ5Uen3nsM7UPlPoUyx+CYz1jlJhlcwS sCWQ4lDawSrUYHqj7g8HgPqcoG90fTEQszgFvzU8koUPuLymO7uXtBZuPZ1f44XQGuJG dYHicwCz0bGUxVmtNX0QyvwQxbhAWZG6wuDpKRHL3KdJXkW8PP9d6JdlnUIJCtR/Zp5I NqWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ce/Y75hv8f3Z1k/tyNNQe2EEt3ZrAZL81QwAPF/GOZg=; b=qyfFB7zgrM2HkUy726aNTD7Jl0F24MEI1USUpjqldzuLwqXp5+7lrGcP6TRcx07adV j+LSzuunq+D9f4+EwJBKmNh33Ew9mkKx00+bGkTU6i0Mf1/EliFc7w1CrAUjIdvXkvWm IVE9lcwQs5a9e7xQrAmg3RekweoRRlCEnAghNKN7Diccd9Ec7yyg2kjwIXcdFKI/MvzN FG188hPRv0uQANqcykv+2+xlCHe8/M8FsdSnYSu31QOdNpJs7SOIK+Dm3c57HY5UFEQV MKF1cjJRxZfmuBp//mUGOT/igMnrXXAQwLv5n+oa2LLHFX039doD1kkV2XbvBe8Rkz0X LnXw== X-Gm-Message-State: APjAAAXjs/IVxpVacMAHx2sDNnkk1jPYkSYIn3yG45tiB7KeeORQEg8i LQE9IPlndh0ez1jU/7YMZNKtF/HKczbJ0uB1dwkg4DH74mE= X-Google-Smtp-Source: APXvYqxYuNv+jAyU4CQEEy6CShH4JoHVAbX98FuEgUS5Hx5oFhMGJoZeAH/+qjzceQ4V8oLVK8xrdk5dQyA/8uMUBPA= X-Received: by 2002:a5e:a710:: with SMTP id b16mr4481180iod.38.1561544780160; Wed, 26 Jun 2019 03:26:20 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> In-Reply-To: From: Christian M Date: Wed, 26 Jun 2019 12:26:09 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Eugene Grosbein Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 1D04E96E09 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=hNtj5RHu; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2c as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.86 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[c.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.90)[ip: (-8.97), ipnet: 2607:f8b0::/32(-3.13), asn: 15169(-2.33), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.95)[-0.954,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 10:26:22 -0000 I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum disappeared. netstat -sp ip for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): ip: 3664084 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 3664080 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 4 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 7003245 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header ip: 8217827 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 8217827 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 4300027 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header netstat -sp tcp for VM1 (172.31.16.125, running iperf3 -c always) and VM2 (172.31.16.126, running iperf3 -s always): tcp: 7002523 packets sent 6866670 data packets (9939548357 bytes) 109393 data packets (101233068 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 27039 ack-only packets (90 delayed) 0 URG only packets 0 window probe packets 0 window update packets 125 control packets 3664099 packets received 3387212 acks (for 9939333715 bytes) 270936 duplicate acks 0 acks for unsent data 915 packets (36547 bytes) received in-sequence 4887 completely duplicate packets (208 bytes) 4853 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 5694 window update packets 6 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 88 connection requests 2 connection accepts 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 89 connections established (including accepts) 85 times used RTT from hostcache 85 times used RTT variance from hostcache 85 times used slow-start threshold from hostcache 90 connections closed (including 57 drops) 51 connections updated cached RTT on close 51 connections updated cached RTT variance on close 23 connections updated cached ssthresh on close 1 embryonic connection dropped 3387212 segments updated rtt (of 3313837 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 759139 correct ACK header predictions 70 correct data packet header predictions 2 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 2 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 2 cookies sent 0 cookies received 1 hostcache entry added 0 bucket overflow 5602 SACK recovery episodes 109329 segment rexmits in SACK recovery episodes 101663360 byte rexmits in SACK recovery episodes 249723 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole detection failures TCP connection count by state: 0 connections in CLOSED state 3 connections in LISTEN state 0 connections in SYN_SENT state 0 connections in SYN_RCVD state 2 connections in ESTABLISHED state 0 connections in CLOSE_WAIT state 0 connections in FIN_WAIT_1 state 0 connections in CLOSING state 0 connections in LAST_ACK state 0 connections in FIN_WAIT_2 state 0 connections in TIME_WAIT state tcp: 4299563 packets sent 1510 data packets (132487 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 4297568 ack-only packets (2610 delayed) 0 URG only packets 0 window probe packets 392 window update packets 93 control packets 8217839 packets received 1623 acks (for 132605 bytes) 150324 duplicate acks 0 acks for unsent data 7770336 packets (11249613126 bytes) received in-sequence 126189 completely duplicate packets (117973252 bytes) 4853 old duplicate packets 194 packets with some dup. data (59900 bytes duped) 290196 out-of-order packets (415026177 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 25 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 6 connection requests 102 connection accepts 0 bad connection attempts 0 listen queue overflows 1 ignored RSTs in the window 108 connections established (including accepts) 95 times used RTT from hostcache 95 times used RTT variance from hostcache 0 times used slow-start threshold from hostcache 163 connections closed (including 47 drops) 29 connections updated cached RTT on close 29 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 0 embryonic connections dropped 1623 segments updated rtt (of 1314 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 9 keepalive timeouts 8 keepalive probes sent 1 connection dropped by keepalive 593 correct ACK header predictions 7769705 correct data packet header predictions 102 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 102 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 102 cookies sent 0 cookies received 3 hostcache entries added 0 bucket overflow 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 293795 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole detection failures TCP connection count by state: 0 connections in CLOSED state 4 connections in LISTEN state 0 connections in SYN_SENT state 0 connections in SYN_RCVD state 2 connections in ESTABLISHED state 0 connections in CLOSE_WAIT state 0 connections in FIN_WAIT_1 state 0 connections in CLOSING state 0 connections in LAST_ACK state 0 connections in FIN_WAIT_2 state 0 connections in TIME_WAIT state The data collected is after a number of runs with iperf3. I'm really not qualified to interpret any of this, so my hopes are that someone else sees something unusual here. Thanks, Christian Den ons 26 juni 2019 kl 10:37 skrev Eugene Grosbein : > 26.06.2019 15:11, Christian M wrote: > > > Running tcpdump on the host while running iperf3 between the 12.0 VM's > > results in a lot of incorrect cksum like this. > > > > tcpdump -i vif54.0 -v -nn| grep -i incorrect > > 172.31.16.125.63013 > 172.31.16.126.5201: Flags [.], cksum 0x7f08 > > (incorrect -> 0x030f), seq 223153882:223155330, ack 1, win 1026, options > > [nop,nop,TS val 4104002274 ecr 3926764642], length 1448 > > Incorrect checksum shown by tcpdump is normal and expected for outgoing > packets > if txcsum offload is not disabled because networking stack skips checksum > calculation > offloading it to "hardware". It still indicates a bug at sending side if > seen for outgoing packets > when txcsum is disabled. > > Incorrect checksum shown by tcpdump for incoming packets always indicates > a problem > at sending side or intermediate network (if any). > > You should check output of "netstat -sp ip" and "netstat -sp tcp" for > non-zero > and growing checksum error counters at receiving side. > > From owner-freebsd-xen@freebsd.org Wed Jun 26 10:36:15 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C335815C3D65; Wed, 26 Jun 2019 10:36:15 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C2CA97592; Wed, 26 Jun 2019 10:36:13 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-xen@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x5QAa4B9006778 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 26 Jun 2019 17:36:04 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Christian M References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org From: Eugene Grosbein Message-ID: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> Date: Wed, 26 Jun 2019 17:35:55 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 2C2CA97592 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [0.32 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.83)[-0.834,0]; MX_INVALID(0.50)[cached]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.20)[-0.203,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_SPAM_SHORT(0.91)[0.906,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(0.05)[asn: 29072(0.25), country: RU(0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 10:36:15 -0000 26.06.2019 17:26, Christian M wrote: > I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum disappeared. [skip] Data looks good. Did it run better this time? From owner-freebsd-xen@freebsd.org Wed Jun 26 10:44:25 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5942C15C44AB; Wed, 26 Jun 2019 10:44:25 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC76897BB5; Wed, 26 Jun 2019 10:44:23 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2f.google.com with SMTP id j6so2240418ioa.5; Wed, 26 Jun 2019 03:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CTKlhNC5OBShmKv3cBuFmnH3IJULuk4ihG5KKBo5chk=; b=Qx5izRjfvgeG4LdSip1g7O+pJ0FhpJmJ2NOxqRq5xYf/UwQ9hev5RF0qWQy4iLe5uj 4A3zUky0DBdszEY5qoGYVzqahpDeomPLYp3xM0DaxOmzBwhb+jwb4Idz3HKFyyHf+PAb YoCi5nJ5hMccJXDH0RPJWlUE1nDkZJqYRr7ZFUQC2V8idBJJstWV8RIFmWdSeUgUHC1h UvChQch7OWd9JTTzdCee8tXBWZGyl+BKH4goLUgimeayhJb3QsKAHUAiljPIB3TLeDfm 1C5tyTmw3ioKVfJ4o+2Kw5Y7d29Ec7ZjLlVydKMDJEjVb0C6AgkJ2ZM2K7VXGoJDH6cB hz7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CTKlhNC5OBShmKv3cBuFmnH3IJULuk4ihG5KKBo5chk=; b=Bhl8OHFBdTQYbXMqOrJQI8kbd09N0YJPccpX/pDkbONB5pNs9zjcWMxV964FlHZilQ Ld2OqT/ohboktQdX7syONv0b+r0e5ZqB3P73+9CWzh2Sesi2z+9qYOASfKJ7EilEOxOb xInnIFWrPaCYmys5mqgJz+qoo1fEkSTgYxKL5wSKKVWjk02YD6sg13py9nKAQYnON1Xa ozR5ji63u+NMsHUuyJJ+2b/NmcNLcK3H9/UF31OL9a6vRrMFBrAxuTQzy1OucvfxZdad p0IZuxZPWO+iMn7oL8wSdjw2Zc0rn6VeCxz0+MzGbovQ3XRraRrXnHu4x5r8CQL1VMyF Qzeg== X-Gm-Message-State: APjAAAVyh3UHvINbDsKLQ/3pJ29b47Beu5JYhf9phRF7u91eo2/edhzC S03JFU/ukRhvJ1xKZOFTBx5zRYhGR9ryZgYQC8TSwo+Xv6g= X-Google-Smtp-Source: APXvYqyzB2lttv0WUaA/YYKAYVnHRS/5OcaEAqRG9zsT1/gzugfgHZWaIVc71H7awnVVm3ai8k+gzf5jwhP9vsRocTY= X-Received: by 2002:a6b:4107:: with SMTP id n7mr3815693ioa.12.1561545863080; Wed, 26 Jun 2019 03:44:23 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> In-Reply-To: <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> From: Christian M Date: Wed, 26 Jun 2019 12:44:12 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Eugene Grosbein Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: CC76897BB5 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Qx5izRjf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2f as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[f.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.86)[ip: (-8.80), ipnet: 2607:f8b0::/32(-3.13), asn: 15169(-2.33), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.95)[-0.949,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 10:44:25 -0000 Sorry to say, but no. Nothing changed :( The iperf issue concerns me a bit also: root@:/usr/home/asdf # iperf3 -c 172.31.16.126 iperf3: error - unable to receive control message: Connection reset by peer root@:/usr/home/asdf # iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 iperf3: error - unable to initialize stream: Socket is not connected root@:/usr/home/asdf # iperf3 -c 172.31.16.126 ^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr iperf3: interrupt - the client has terminated root@:/usr/home/asdf # iperf3 -c 172.31.16.126 Connecting to host 172.31.16.126, port 5201 [ 5] local 172.31.16.125 port 23884 connected to 172.31.16.126 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 6.00 MBytes 50.2 Mbits/sec 217 1.43 KBytes [ 5] 1.00-2.00 sec 26.7 MBytes 224 Mbits/sec 566 1.43 KBytes [ 5] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec 445 1.43 KBytes [ 5] 3.00-4.00 sec 13.5 MBytes 113 Mbits/sec 453 1.43 KBytes [ 5] 4.00-5.00 sec 13.7 MBytes 115 Mbits/sec 512 1.43 KBytes [ 5] 5.00-6.00 sec 6.51 MBytes 54.6 Mbits/sec 201 1.43 KBytes [ 5] 6.00-7.00 sec 13.5 MBytes 114 Mbits/sec 366 1.43 KBytes [ 5] 7.00-8.00 sec 16.9 MBytes 142 Mbits/sec 325 1.43 KBytes [ 5] 8.00-9.00 sec 24.2 MBytes 203 Mbits/sec 275 1.43 KBytes [ 5] 9.00-10.00 sec 38.8 MBytes 325 Mbits/sec 277 1.43 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 175 MBytes 147 Mbits/sec 3637 sender [ 5] 0.00-10.00 sec 175 MBytes 147 Mbits/sec receiver iperf Done. root@:/usr/home/asdf # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- iperf3: error - unable to receive parameters from client: Connection reset by peer ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 172.31.16.125, port 21052 ^Ciperf3: interrupt - the server has terminated root@:/usr/home/asdf # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 172.31.16.125, port 29365 [ 5] local 172.31.16.126 port 5201 connected to 172.31.16.125 port 23884 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 5.85 MBytes 49.0 Mbits/sec [ 5] 1.00-2.00 sec 26.6 MBytes 223 Mbits/sec [ 5] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec [ 5] 3.00-4.00 sec 13.5 MBytes 113 Mbits/sec [ 5] 4.00-5.00 sec 13.7 MBytes 115 Mbits/sec [ 5] 5.00-6.00 sec 6.51 MBytes 54.6 Mbits/sec [ 5] 6.00-7.00 sec 13.5 MBytes 114 Mbits/sec [ 5] 7.00-8.00 sec 16.9 MBytes 142 Mbits/sec [ 5] 8.00-9.00 sec 24.1 MBytes 202 Mbits/sec [ 5] 9.00-10.00 sec 38.8 MBytes 325 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 175 MBytes 147 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Den ons 26 juni 2019 kl 12:36 skrev Eugene Grosbein : > 26.06.2019 17:26, Christian M wrote: > > > I ran ifconfig xn0 -txcsum on both test VM's, and all incorrect checksum > disappeared. > > [skip] > > Data looks good. Did it run better this time? > > From owner-freebsd-xen@freebsd.org Wed Jun 26 10:50:39 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4B2C15C476D; Wed, 26 Jun 2019 10:50:39 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 28B8697E8E; Wed, 26 Jun 2019 10:50:28 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-xen@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x5QAoLp4006846 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 26 Jun 2019 17:50:21 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Christian M References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org From: Eugene Grosbein Message-ID: Date: Wed, 26 Jun 2019 17:50:13 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 28B8697E8E X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [0.39 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.81)[-0.809,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.16)[-0.158,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_SPAM_SHORT(0.91)[0.908,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(0.05)[asn: 29072(0.25), country: RU(0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 10:50:40 -0000 26.06.2019 17:44, Christian M wrote: > Sorry to say, but no. Nothing changed :( > > The iperf issue concerns me a bit also: Try starting ntpd inside VM guests, wait an hour then post contents of /var/db/ntpd.drift This is to verify quality of time source provided to VMs by hypervisor. From owner-freebsd-xen@freebsd.org Wed Jun 26 13:17:48 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38A2C15C82B6; Wed, 26 Jun 2019 13:17:48 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 973626DC1A; Wed, 26 Jun 2019 13:17:46 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd36.google.com with SMTP id i10so3135851iol.13; Wed, 26 Jun 2019 06:17:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3T1PcP0+UJUsc3L6TfDmVYakwdq/7Dz24T7TXvn4ePE=; b=MLdBWa78lvim60jiqtTK2FPYmdnfYp7CG4+d0I2+lgktKcWz/6U4IM7kiugTDdGrck Y7uVXKVdFJbclRaoGCTxc6L/PToqY9nmiONjw3ynaengGK0Q6rMHpKA5WXO+AIuW5xXp kDznf+StpdtE35BA4t5PmWS3jSAYVvFRXJJzx8WH8D/ewC/YEcLEdRR2xC+w/TOWsbM3 CurIGAUQi8f3iviRHSP96f1inYwp7eJSfSxCJQjODfE4QDkpxHoSxiwqXsOYp66qX00S SXTPUHx91eke4XYe5cTuTDr1z0yuh3PYmTRTCXOnEJ4okcVU6Q/k9eQYGxNsmcQep83P WN0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3T1PcP0+UJUsc3L6TfDmVYakwdq/7Dz24T7TXvn4ePE=; b=LiBJyz6FA1jRwP+hQY3M3CuLGhzKDVG9wUto0J6H6JIvID+Yn3gYjYVFX8H4Jq6bRN E1REeHm/U1l0y3nICMzq9iIqB+4TCi1IKY2K2UiV+17BQhGThEQ8mj+d6JEuQOBmhJtl xpy0s1vi7up6xm62IzFIZl6IpHNB16ySZGfXkCn2SPf3DZxJF29B0BJ72gZzZhO7ERyF eJEoB+xCngnoL6nq1AKoE69N7b9hD56KnQDy7Yf9hrZXA+tRtt/Zp9sISx+/s8eAbdkO Aqi8P+4NpkjO4tuXMhaNkuaaJQturrGKMhwaA8hXI4uro/6WIW007+bETJjbRaedeiCP DjdA== X-Gm-Message-State: APjAAAVgHk+Hze2YLPNaVRcxVxOFY4tp+65OtSnTMqWpZNQK6HAojwbO 3dUuL2YR+H8pZZMaQKcN/SM6G/3Xt4/5+U4ehh9yNe7U X-Google-Smtp-Source: APXvYqyifxMvyiq4OIfuME/VXPsxsG8RXyv2/2UFhCHJnRt5n85HWwfNbLNTNtfXGnAgDMNB4C/BMg48DKSAyZNafAQ= X-Received: by 2002:a6b:5b01:: with SMTP id v1mr4992645ioh.120.1561555065771; Wed, 26 Jun 2019 06:17:45 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> In-Reply-To: From: Christian M Date: Wed, 26 Jun 2019 15:17:34 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Eugene Grosbein Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 973626DC1A X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=MLdBWa78; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d36 as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[6.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.93)[ip: (-9.12), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.33), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.90)[-0.902,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 13:17:48 -0000 Started ntpd in the two main 12-0-RELEASE vm's I've been testing with. drift files show -0.262 and -1.144 after about two hours. Den ons 26 juni 2019 kl 12:50 skrev Eugene Grosbein : > 26.06.2019 17:44, Christian M wrote: > > > Sorry to say, but no. Nothing changed :( > > > > The iperf issue concerns me a bit also: > > Try starting ntpd inside VM guests, wait an hour then post contents of > /var/db/ntpd.drift > This is to verify quality of time source provided to VMs by hypervisor. > > From owner-freebsd-xen@freebsd.org Thu Jun 27 03:33:41 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 256BC15D97EF; Thu, 27 Jun 2019 03:33:41 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "eg.sd.rdtc.ru", Issuer "eg.sd.rdtc.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E112C6C5D1; Thu, 27 Jun 2019 03:33:29 +0000 (UTC) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-xen@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x5R3XI28017063 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 27 Jun 2019 10:33:18 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Christian M References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org From: Eugene Grosbein Message-ID: <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> Date: Thu, 27 Jun 2019 10:33:09 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: E112C6C5D1 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [0.14 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.836,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.40)[-0.404,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_SPAM_SHORT(0.92)[0.925,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(0.05)[asn: 29072(0.24), country: RU(0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:29072, ipnet:2a03:3100::/32, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2019 03:33:41 -0000 26.06.2019 20:17, Christian M wrote: > Started ntpd in the two main 12-0-RELEASE vm's I've been testing with. > drift files show -0.262 and -1.144 after about two hours. This is very good, so it should not be timekeeping problem. There were many fixes to FreeBSD12 after the release. Can you update to 12.0-STABLE? From owner-freebsd-xen@freebsd.org Thu Jun 27 10:14:46 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B701F15B3823; Thu, 27 Jun 2019 10:14:46 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B2DC277245; Thu, 27 Jun 2019 10:14:45 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2d.google.com with SMTP id u13so3566425iop.0; Thu, 27 Jun 2019 03:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KQbBJuYPKvFcxPLYETvyKZRoVLnzi/O5m8MlPrkCS1o=; b=UABh6ZB3si/LOB0B4UJ5S8Plb9nZAy4xauheMnAwLRvg0AyxUDoIMWDsmerQ+8PyDA +pw3/AmSMbtYmhF804oAmZ2JJfhvgja+uZUoCiajd+rt9uLvhfSZ+fFxXgS5B/wp+Zr9 HVJ+UvyqaGsXpt7hYuIb3JL4QUV+pbrEIgmTcx9sLC/sBwv6UbrUBZOGWyp6mBWNqtKR IVPM/eal5lN1zDgOciVMcXr3XiT4bCYG6ZXxUMJIM7sQvnK6cytFubbfmiBjLNws/JbZ G/jiUTTDDGBaSJbkj4XXC3I/20iqNN0Z1Nqb+uBim2rLVqDbX4vdb0gfCp7Fw+G+CvEz +NqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KQbBJuYPKvFcxPLYETvyKZRoVLnzi/O5m8MlPrkCS1o=; b=lHnmH4GXtU6Y/X2sk4N3MUzkQT21G8SJ9BuFaZYFHwjR6hWWhIBzvqdB+EoOpe/OGc 2Iuh/N373Io753XCh9Bk0zs5vK4TwocV8W/7b07qCSQMtVuI5ysOdOBgiUGqwVxJoLPE Q/Qi6gEtnikzzK7+ARMEsvqBRXOn4zjv8rRfiIA3IG+BnTRiygcEp9N8+UZUlthH0YfZ 0VpdUD/dAb5gSKA0fMv0F0ucVMFsTMa13gHA6OiM+e8ofwrrGQG9loKlqtUJVKN5Xamn wbLQezIGUzBR8oNJRRd4nwqzYF1MGZwUQN4DcaOf/Ae3QlTe30OHb7u3XyGus1FnYZuc c63w== X-Gm-Message-State: APjAAAVz3xzYXUJpg07TLPvSiLxCbk/e/ZYPgWrrg6IwtPCPeAul+lvb ZPj4TfThV386Q6lLYV4rzah2JW8MC+yReFBGWQLWxiG8 X-Google-Smtp-Source: APXvYqyfAz3p6XhkrgNjM1VyEHLHzgbTWX2e/JaI/HG+CSKuybXxiKbvw9JG+tD9poE7s7miv5pnoqXSGzVBZVN8pa0= X-Received: by 2002:a6b:b7d5:: with SMTP id h204mr3547070iof.188.1561630484693; Thu, 27 Jun 2019 03:14:44 -0700 (PDT) MIME-Version: 1.0 References: <12994df1-f847-ec92-aae8-43a32e59385f@darco.dk> <20190625082103.qiiz2cikrauaqlas@MacBook-Air-de-Roger.local> <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> In-Reply-To: <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> From: Christian M Date: Thu, 27 Jun 2019 12:14:33 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: Eugene Grosbein Cc: freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: B2DC277245 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UABh6ZB3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2d as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.56 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.71)[-0.706,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[d.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.95)[ip: (-9.19), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.34), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2019 10:14:47 -0000 I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF cheksum offloading is turned off, and -txcsum for xn0 for both VM's. I feel the throughput is more consistent now, not all over the place as before, even between runs. But the Retr column (tcp retries) in iperf3 has jumped up considerably from hundreds/s to thousands/s. Just a reminder, I have tested this with 11.0-RELEASE also, where the issue appeared first for me. 10.4-RELEASE is as fast as I could expect it to be, and 0 retries. 12.0-STABLE: Connecting to host 172.31.16.128, port 5201 [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 sender [ 5] 0.00-20.00 sec 2.05 GBytes 881 Mbits/sec receiver 10.3-RELEASE (offloading enabled everywhere for these VM's VIF, and for xn0): Connecting to host 172.31.16.122, port 5201 [ 5] local 172.31.16.121 port 20712 connected to 172.31.16.122 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 832 MBytes 6.98 Gbits/sec 0 580 KBytes [ 5] 1.00-2.00 sec 1.24 GBytes 10.6 Gbits/sec 0 1.04 MBytes [ 5] 2.00-3.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.57 MBytes [ 5] 3.00-4.00 sec 1.41 GBytes 12.1 Gbits/sec 0 1.76 MBytes [ 5] 4.00-5.00 sec 1.37 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 5.00-6.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.77 MBytes [ 5] 6.00-7.00 sec 1.37 GBytes 11.8 Gbits/sec 0 1.77 MBytes [ 5] 7.00-8.00 sec 1.39 GBytes 12.0 Gbits/sec 0 1.77 MBytes [ 5] 8.00-9.00 sec 1.35 GBytes 11.6 Gbits/sec 0 1.77 MBytes [ 5] 9.00-10.00 sec 1.42 GBytes 12.2 Gbits/sec 0 1.77 MBytes [ 5] 10.00-11.00 sec 1.34 GBytes 11.5 Gbits/sec 0 1.77 MBytes [ 5] 11.00-12.00 sec 1.32 GBytes 11.4 Gbits/sec 0 1.77 MBytes [ 5] 12.00-13.00 sec 1.34 GBytes 11.6 Gbits/sec 0 1.77 MBytes [ 5] 13.00-14.00 sec 1.49 GBytes 12.8 Gbits/sec 0 1.77 MBytes [ 5] 14.00-15.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 15.00-16.00 sec 1.37 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 16.00-17.00 sec 1.48 GBytes 12.7 Gbits/sec 0 1.77 MBytes [ 5] 17.00-18.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes [ 5] 18.00-19.00 sec 1.47 GBytes 12.6 Gbits/sec 0 1.77 MBytes [ 5] 19.00-20.00 sec 1.36 GBytes 11.7 Gbits/sec 0 1.77 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-20.00 sec 26.9 GBytes 11.6 Gbits/sec 0 sender [ 5] 0.00-20.01 sec 26.9 GBytes 11.6 Gbits/sec receiver From owner-freebsd-xen@freebsd.org Thu Jun 27 10:19:18 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A79BD15B3A3B; Thu, 27 Jun 2019 10:19:18 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa6.hc3370-68.iphmx.com (esa6.hc3370-68.iphmx.com [216.71.155.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa6.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 92A63773B8; Thu, 27 Jun 2019 10:19:16 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received-SPF: None (esa6.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa6.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa6.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: klK2fiCCg2V4c0CSNpWZDbjxnXCowAICkvvkv05lN/A84M2pcCEBq7xulR/L9JMmgrNSrUM+Zn CPDvsbIjE2CNbN6xiCwgfxqpwun/1Qkz7WAk7eHDwFG+90jyT1k6/mIhVF5VjULhNmO8kcI+gv b/Fwdc5l2nOMCGrurdu0wr8INPzs4upSpvwPqImOwaG0AFLSvIc9NVZNN8IcTQXSJkCL2LuyXj V5MIc1CvYRbM/EgtvnJYTy3DYyuwFZYzOJRMzx4j5JHBgP2PrQK5KOlxVRNfJr9pxBJsnSEzz0 sZs= X-SBRS: 2.7 X-MesageID: 2313737 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,423,1557201600"; d="scan'208";a="2313737" Date: Thu, 27 Jun 2019 12:19:01 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Christian M CC: Eugene Grosbein , , Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer Message-ID: <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> References: <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: 92A63773B8 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of roger.pau@citrix.com designates 216.71.155.175 as permitted sender) smtp.mailfrom=roger.pau@citrix.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:216.71.155.175]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[citrix.com]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com,mx1.hc3370-68.iphmx.com,mx2.hc3370-68.iphmx.com]; NEURAL_HAM_SHORT(-0.82)[-0.821,0]; RCVD_IN_DNSWL_NONE(0.00)[175.155.71.216.list.dnswl.org : 127.0.3.0]; IP_SCORE(-0.87)[ip: (-1.33), ipnet: 216.71.154.0/23(-1.86), asn: 16417(-1.11), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16417, ipnet:216.71.154.0/23, country:US]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2019 10:19:19 -0000 On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. VIF > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > I feel the throughput is more consistent now, not all over the place as > before, even between runs. But the Retr column (tcp retries) in iperf3 has > jumped up considerably from hundreds/s to thousands/s. > > Just a reminder, I have tested this with 11.0-RELEASE also, where the issue > appeared first for me. 10.4-RELEASE is as fast as I could expect it to be, > and 0 retries. > > 12.0-STABLE: > > Connecting to host 172.31.16.128, port 5201 > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KBytes > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KBytes > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KBytes > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KBytes > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KBytes > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KBytes > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KBytes > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KBytes > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KBytes > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KBytes > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KBytes > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KBytes > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KBytes > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KBytes > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KBytes > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KBytes > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KBytes > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KBytes > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KBytes > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 Can you paste the output of ifconfig for both the interfaces used in the test? Are you sure all hardware offloading capabilities are turned off on both interfaces? Can you check what's causing those retries? Either using tcpdump, whireshark or some other tool to analyze the network traffic and detect the errors that cause such retries? Thanks, Roger. From owner-freebsd-xen@freebsd.org Thu Jun 27 10:31:48 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F41715B41CA; Thu, 27 Jun 2019 10:31:48 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 533D277B7F; Thu, 27 Jun 2019 10:31:47 +0000 (UTC) (envelope-from christian.marcos@gmail.com) Received: by mail-io1-xd2b.google.com with SMTP id h6so3630316ioh.3; Thu, 27 Jun 2019 03:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KflgItLl2Jr30IONB3DvSNvbwHNnqkoi9nPf8ylLLtA=; b=Bm8KGipJibVkXcid/+U9nLyLgFySVktUVAHrN0kWF0JvZ8NuOLqzqI69GU2rBkOWez 6GMrWVeIvobAGpbMj9EPqkQfXDD27v7l5/3bLW0IfJSYb4toEvoeAOPMYDTg+3Hv0a9N fYCu1aHTbngS82FvR2sLmW5uYZ877SaI446KXg+LKYfA5r1ksCsOEiqBmKycgX/ubO6K IOFlzEEF6MZmYrB9GCPGSmJleWbWuRYpVASa0P90qhjTxyNjVqVtf4vnW5TjvTqIjqKI HCx1AVhFcRgPO7bO+K3sfNKDDToT88L3t8fXEoSYjU1mhMYfoE8a1qhJoqCdUPYk1oO1 sPYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KflgItLl2Jr30IONB3DvSNvbwHNnqkoi9nPf8ylLLtA=; b=cJX4lPaghwrBXtaNB09vft3+Mtm2FRxPqE8g009S3Y0tnZaj42i69gIsKo3BzSNb3n iVr3RMGUxA14KlF1KwBgtvfuWOaWWpSzYfUEZwBkC1D80sMZj7sWeKNR+BMbOKIyyzG1 mX1zLxj0V57FtivwwervULWbpFspnASy4nKywtpf5WbGc0j1mOYGpEWCGigDSh/Mz9kX a0yvkOv4Urrs+gB6qyTgozEmYOIjFVMsEad+BJLuZFjzB61BAt74joMnMxKIh9OXxct7 GDARjIPOEt0TUcaMEV2crtZyI8max60skDnPcVHNB7j+K8tdjJzKS5EeaKJuyB2492Es aEQg== X-Gm-Message-State: APjAAAX88ar6c04s4WX53RKaLrZ57Rs0y6wtl7W78CJr33iAlEVVCTrD DIOfRaOR/21L51F4erHuazZZTRtOywXnZbO0JMjfkcDfu10= X-Google-Smtp-Source: APXvYqwDL7LXjJoUiz+8sNJyuatbnsMtnENby80aFUMM5H9kabXUWFPlZQ4azV8Mt2HIsW+uANQI0I6RZp/PvLjEKms= X-Received: by 2002:a02:cb4b:: with SMTP id k11mr3617601jap.109.1561631506366; Thu, 27 Jun 2019 03:31:46 -0700 (PDT) MIME-Version: 1.0 References: <8ec84f75-2100-ff8a-6337-3c955f31da62@grosbein.net> <9c377fe7-5546-9e0b-fac9-076dd094cbac@grosbein.net> <47bb1bfe-ac39-78d3-c9b9-17ff1896e0ec@grosbein.net> <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> In-Reply-To: <20190627101901.3szaaxhp4xzjiay5@MacBook-Air-de-Roger.local> From: Christian M Date: Thu, 27 Jun 2019 12:31:35 +0200 Message-ID: Subject: Re: Very slow and inconsistent internal network speed (between, VM's on the same host) for FreeBSD 11.0+ as guest on, XCP-ng/XenServer To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: Eugene Grosbein , freebsd-xen@freebsd.org, freebsd-net@freebsd.org X-Rspamd-Queue-Id: 533D277B7F X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Bm8KGipJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of christianmarcos@gmail.com designates 2607:f8b0:4864:20::d2b as permitted sender) smtp.mailfrom=christianmarcos@gmail.com X-Spamd-Result: default: False [-6.52 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[b.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; IP_SCORE(-2.80)[ip: (-8.45), ipnet: 2607:f8b0::/32(-3.14), asn: 15169(-2.34), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.71)[-0.707,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2019 10:31:48 -0000 Den tors 27 juni 2019 kl 12:19 skrev Roger Pau Monn=C3=A9 : > On Thu, Jun 27, 2019 at 12:14:33PM +0200, Christian M wrote: > > I've installed 12.0-STABLE on two new VM's now. 172.31.16.127 and .128. > VIF > > cheksum offloading is turned off, and -txcsum for xn0 for both VM's. > > > > I feel the throughput is more consistent now, not all over the place as > > before, even between runs. But the Retr column (tcp retries) in iperf3 > has > > jumped up considerably from hundreds/s to thousands/s. > > > > Just a reminder, I have tested this with 11.0-RELEASE also, where the > issue > > appeared first for me. 10.4-RELEASE is as fast as I could expect it to > be, > > and 0 retries. > > > > 12.0-STABLE: > > > > Connecting to host 172.31.16.128, port 5201 > > [ 5] local 172.31.16.127 port 16833 connected to 172.31.16.128 port 52= 01 > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 96.3 MBytes 808 Mbits/sec 2401 2.85 KByte= s > > > > [ 5] 1.00-2.00 sec 118 MBytes 991 Mbits/sec 3120 17.0 KByte= s > > > > [ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 3203 69.8 KByte= s > > > > [ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 3126 15.6 KByte= s > > > > [ 5] 4.00-5.00 sec 110 MBytes 921 Mbits/sec 2890 15.6 KByte= s > > > > [ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec 3308 17.0 KByte= s > > > > [ 5] 6.00-7.00 sec 104 MBytes 869 Mbits/sec 3046 48.2 KByte= s > > > > [ 5] 7.00-8.00 sec 98.9 MBytes 830 Mbits/sec 2845 2.85 KByte= s > > > > [ 5] 8.00-9.00 sec 104 MBytes 874 Mbits/sec 2711 86.8 KByte= s > > > > [ 5] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 2696 14.2 KByte= s > > > > [ 5] 10.00-11.00 sec 103 MBytes 864 Mbits/sec 2660 31.3 KByte= s > > > > [ 5] 11.00-12.00 sec 98.8 MBytes 828 Mbits/sec 2476 19.9 KByte= s > > > > [ 5] 12.00-13.00 sec 99.9 MBytes 838 Mbits/sec 2857 11.3 KByte= s > > > > [ 5] 13.00-14.00 sec 107 MBytes 894 Mbits/sec 2685 24.1 KByte= s > > > > [ 5] 14.00-15.00 sec 114 MBytes 953 Mbits/sec 2321 25.5 KByte= s > > > > [ 5] 15.00-16.00 sec 93.1 MBytes 781 Mbits/sec 2427 48.3 KByte= s > > > > [ 5] 16.00-17.00 sec 107 MBytes 895 Mbits/sec 2219 29.8 KByte= s > > > > [ 5] 17.00-18.00 sec 92.5 MBytes 776 Mbits/sec 2441 12.8 KByte= s > > > > [ 5] 18.00-19.00 sec 116 MBytes 976 Mbits/sec 2840 38.2 KByte= s > > > > [ 5] 19.00-20.00 sec 102 MBytes 853 Mbits/sec 2573 43.9 KByte= s > > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-20.00 sec 2.05 GBytes 882 Mbits/sec 54845 > > Can you paste the output of ifconfig for both the interfaces used in > the test? > > Are you sure all hardware offloading capabilities are turned off on > both interfaces? > > Can you check what's causing those retries? > > Either using tcpdump, whireshark or some other tool to analyze the > network traffic and detect the errors that cause such retries? > > Thanks, Roger. > 172.31.16.127 (12.0-STABLE): lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 xn0: flags=3D8843 metric 0 mtu 1500 options=3D501 ether 6e:83:99:ed:ce:f7 inet 172.31.16.127 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=3D29 ethtool -k vif68.0 Features for vif68.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] 172.31.16.128 (12.0-STABLE): lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 xn0: flags=3D8843 metric 0 mtu 1500 options=3D501 ether 32:a7:9f:cc:94:8a inet 172.31.16.128 netmask 0xffffff00 broadcast 172.31.16.255 media: Ethernet manual status: active nd6 options=3D29 ethtool -k vif67.0 Features for vif67.0: rx-checksumming: on [fixed] tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] Should I run tcpdump on xen host, or on FreeBSD while running iperf3? As I side note, I installed XCP-ng 7.6.0 on a older server I had laying around just to see how 12.0-RELEASE behaved there, and It was the same thing, super slow between VM's. From owner-freebsd-xen@freebsd.org Fri Jun 28 12:03:06 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 189AB15B812C for ; Fri, 28 Jun 2019 12:03:06 +0000 (UTC) (envelope-from knight.neo@gmail.com) Received: from mail-io1-xd42.google.com (mail-io1-xd42.google.com [IPv6:2607:f8b0:4864:20::d42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E84C6F378 for ; Fri, 28 Jun 2019 12:03:05 +0000 (UTC) (envelope-from knight.neo@gmail.com) Received: by mail-io1-xd42.google.com with SMTP id k8so11993881iot.1 for ; Fri, 28 Jun 2019 05:03:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=QOdhiWmfxCgNvS5jotS4p7JItux3pvLwDFXXeJTcwlE=; b=QJqcvs3aTrATZJ4KKlKEgRSsLTE8EI+080wBvSHSkPGiT0GibEGs+gBE/ZAzyWTaO4 J0hRVCBFzeIEvH6+jgSpNX3PhC6+gayTBDU3FHRkHLD4B4wqgiyeLdEw4xPERPSBKe/8 82nrdRlgGB08WJbOqs5DwAiPc8IVCofNVMQEjai1pRXJ9v6rXwj7yqLY5NUjjqJsyI2w wm6XWUFDg/w98TOnadMPXPJg6AeTkQEru406zs0NWMI9QUlN6TadULt0ziMbW1wFlpWE O9Gp7dGB8zzpCDZEErgqhBG+hnYs3QzoSJ3Z4R3soztqdwU+k35nsFCLS7Q7+Wi7K6J+ eZpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=QOdhiWmfxCgNvS5jotS4p7JItux3pvLwDFXXeJTcwlE=; b=iZ2c5TOt4xtyL1cXwDIoNfat+kdM5PSTbrbMQcr00/DK0MCGxD++s+htcb/QmCoaPw b5k2OglQGhoENRh2/b6lGb1kP/l7nbNu48AKZ7u/mT/Av9yi9//dq9/MqVvRI7cJUg25 diZtzE8sl9s0/fApIPRWxq7UNFFhlull+XkfxnTXi1RbGRuuwztpkFgRWFB/c3nu+QKZ 8h4mkO71Nv7+66qCzVJlDek6srwoEjD9Mxq5zkTOZqEuIRhZJcE5ZZ+aHqMVEsFQ8Hlm iWIZc4cPYX2rQxYiruydiUCoZlhheGGDieSXa4mJ+V72/J/52K147JcCntIaVghQaAFT aD/g== X-Gm-Message-State: APjAAAViewCv587FdY+MxJmc3VU3kV80A+kEzPNDLwdg6sF69cSYrDxo hmLFW0DRLeLXsIYcxqelUBiLvPRXZPJKiEBAjRicxjo5 X-Google-Smtp-Source: APXvYqxFXvzXOHoYG9Phwxaa1AUlSfJ5IM9Ar775iww3DrtD86TuhRpV54R3jscc0fATYSE5T8KaRdwpTxuQIno0reY= X-Received: by 2002:a02:a38d:: with SMTP id y13mr10389478jak.68.1561723384245; Fri, 28 Jun 2019 05:03:04 -0700 (PDT) MIME-Version: 1.0 From: "R0me0 ***" Date: Fri, 28 Jun 2019 12:02:37 +0000 Message-ID: Subject: Xen 4.12 - systemrescuecd-6.0.3.iso - Baremetal reboots To: freebsd-xen@freebsd.org X-Rspamd-Queue-Id: 2E84C6F378 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=QJqcvs3a; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of knightneo@gmail.com designates 2607:f8b0:4864:20::d42 as permitted sender) smtp.mailfrom=knightneo@gmail.com X-Spamd-Result: default: False [-4.75 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.95)[-0.954,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.79)[ip: (1.59), ipnet: 2607:f8b0::/32(-3.13), asn: 15169(-2.33), country: US(-0.06)]; RCVD_IN_DNSWL_NONE(0.00)[2.4.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2019 12:03:06 -0000 Guys, I'am using FreeBSD 12.0-RELEASE-p6 FreeBSD 12.0-RELEASE-p6 GENERIC amd64 When booting systemrescuecd-6.0.3.iso ( http://www.system-rescue-cd.org ) , my system just hangs and reboot. Just tried an older version and it's ok. I don't have any log to help. If you guys want and some help, I can provide more information. Bests, Packages installed: # pkg info argp-standalone-1.3_3 Standalone version of arguments parsing functions from GLIBC bareos17-client-17.2.7_3 Backup archiving recovery open sourced (client) ca_root_nss-3.44.1 Root certificate bundle from the Mozilla Project curl-7.65.1 Command line tool and library for transferring data with URLs doas-6.0p3 Simple sudo alternative to run commands as another user gettext-runtime-0.20.1 GNU gettext runtime libraries and programs glib-2.56.3_5,1 Some useful routines of C programming (current stable version) indexinfo-0.3.1 Utility to regenerate the GNU info page index jansson-2.12 C library for encoding, decoding, and manipulating JSON data libffi-3.2.1_3 Foreign Function Interface libiconv-1.14_11 Character set conversion library libnghttp2-1.39.1 HTTP/2.0 C Library libxml2-2.9.9 XML parser library for GNOME lzo2-2.10_1 Portable speedy, lossless data compression library mtr-nox11-0.92_1 Traceroute and ping in a single network diagnostic tool p5-Path-Tiny-0.108 File path utility pcre-8.43_1 Perl Compatible Regular Expressions library perl5-5.28.2 Practical Extraction and Report Language pixman-0.34.0_1 Low-level pixel manipulation library pkg-1.11.1 Package manager python27-2.7.16_1 Interpreted object-oriented programming language python36-3.6.8_2 Interpreted object-oriented programming language qemu-utils-3.0.1_1 QEMU userland utilities readline-8.0.0 Library for editing command lines as they are typed seabios-1.11.0_2 Open source implementation of a 16bit X86 BIOS strongswan-5.8.0 Open Source IKEv2 IPsec-based VPN solution vim-console-8.1.1439 Improved version of the vi editor (console only) xen-kernel-4.12.0_2 Hypervisor using a microkernel design xen-tools-4.12.0_1 Xen management tools yajl-2.1.0 Portable JSON parsing and serialization library in ANSI C And here the relevant dmesg: ---<>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-RELEASE-p6 GENERIC amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) VT(vga): text 80x25 XEN: Hypervisor version 4.12 detected. CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz (3500.03-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4 Features=0x1fc3fbff Features2=0xffba2203 AMD Features=0x2c100800 AMD Features2=0x1 Structured Extended Features=0x281 Structured Extended Features3=0x9c000000 XSAVE Features=0x1 AMD Extended Feature Extensions ID EBX=0x1000 TSC: P-state invariant Hypervisor: Origin = "XenVMMXenVMM" real memory = 21522096128 (20525 MB) avail memory = 18692644864 (17826 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: < > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) random: unblocking device. ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard Launching APs: 1 random: entropy device external interface netmap: loaded module [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0xffffffff810f9a00, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd1 at kbdmux0 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard aesni0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 hpet0: HPET never increments, disabling device_attach: hpet0 attach returned 6 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 27 at device 1.0 on pci0 pci1: on pcib1 pcib2: irq 27 at device 1.1 on pci0 pci2: on pcib2 pcib3: irq 33 at device 2.0 on pci0 pci3: on pcib3 mfi0: port 0xe000-0xe0ff mem 0xfba60000-0xfba63fff,0xfba00000-0xfba3ffff irq 32 at device 0.0 on pci3 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23 mfi0: FW MaxCmds = 1008, limiting to 128 mfi0: 20431 (614968582s/0x0020/info) - Shutdown command received from host mfi0: 20432 (614968582s/0x0002/info) - Unexpected sense: PD 05(e0xfc/s0) Path 4433221103000000, CDB: 2a 00 16 40 fd 02 00 01 00 00, Sense: 2/04/02 mfi0: 20433 (614968582s/0x0002/info) - Power state change on PD 05(e0xfc/s0) from POWERSAVE(1) to TRANSITION(ff) mfi0: 20434 (614968582s/0x0002/info) - Unexpected sense: PD 05(e0xfc/s0) Path 4433221103000000, CDB: 2a 00 16 40 fe 02 00 01 00 00, Sense: 2/04/02 mfi0: 20435 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9260/1000) mfi0: 20436 (boot + 3s/0x0020/info) - Firmware version 2.130.403-4660 mfi0: 20437 (boot + 5s/0x0020/info) - Package version 12.15.0-0239 mfi0: 20438 (boot + 5s/0x0020/info) - Board Revision 86C mfi0: 20439 (boot + 24s/0x0002/info) - Inserted: PD 04(e0xff/s1) mfi0: 20440 (boot + 24s/0x0002/info) - Inserted: PD 04(e0xff/s1) Info: enclPd=ffff, scsiType=0, portMap=02, sasAddr=4433221102000000,0000000000000000 mfi0: 20441 (boot + 24s/0x0002/info) - Inserted: PD 05(e0xff/s0) mfi0: 20442 (boot + 24s/0x0002/info) - Inserted: PD 05(e0xff/s0) Info: enclPd=ffff, scsiType=0, portMap=01, sasAddr=4433221103000000,0000000000000000 mfi0: 20443 (boot + 24s/0x0002/info) - Inserted: PD 06(e0xff/s3) mfi0: 20444 (boot + 24s/0x0002/info) - Inserted: PD 06(e0xff/s3) Info: enclPd=ffff, scsiType=0, portMap=00, sasAddr=4433221100000000,0000000000000000 mfi0: 20445 (boot + 24s/0x0002/info) - Inserted: PD 07(e0xff/s2) mfi0: 20446 (boot + 24s/0x0002/info) - Inserted: PD 07(e0xff/s2) Info: enclPd=ffff, scsiType=0, portMap=03, sasAddr=4433221101000000,0000000000000000 mfi0: 20447 (boot + 25s/0x0001/info) - Policy change on VD 00/0 to [ID=00,dcp=21,ccp=20,ap=0,dc=0] from [ID=00,dcp=21,ccp=21,ap=0,dc=0] mfi0: 20448 (boot + 25s/0x0001/info) - Policy change on VD 01/1 to [ID=01,dcp=21,ccp=20,ap=0,dc=0] from [ID=01,dcp=21,ccp=21,ap=0,dcpcib4: =0] mfi0: 20449 (boot + 25s/0x0001/info) - Policy change on VD 02/2 to [ID=02,dcp=21,ccp=20,ap=0,dc=0] from [ID=02,dcp=21,ccp=21,ap=0,dc=0] mfi0: 20450 (614968626s/0x0020/info) - Time established as 06/27/19 16:37:06; (35 seconds since power on) mfi0: 20451 (614968723s/0x0020/info) - Host driver is loaded and operational irq 41 at device 3.0 on pci0 pci4: on pcib4 igb0: mem 0xfb900000-0xfb91ffff,0xfb920000-0xfb923fff irq 40 at device 0.0 on pci4 igb0: attach_pre capping queues at 8 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 8 igb0: pxm cpus: 2 queue msgs: 9 admincnt: 1 igb0: using 2 rx queues 2 tx queues igb0: Using MSIX interrupts with 3 vectors igb0: allocated for 2 tx_queues igb0: allocated for 2 rx_queues igb0: Ethernet address: 0c:c4:7a:4d:bc:a8 igb0: netmap queues/slots: TX 2/1024, RX 2/1024 pcib5: irq 41 at device 3.2 on pci0 pci5: on pcib5 pcib6: irq 16 at device 17.0 on pci0 pci6: on pcib6 pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xfbb23000-0xfbb233ff irq 16 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 usbus0: 480Mbps High Speed USB v2.0 ehci1: mem 0xfbb22000-0xfbb223ff irq 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 usbus1: 480Mbps High Speed USB v2.0 pcib7: at device 30.0 on pci0 pci7: on pcib7 vgapci0: mem 0xfa000000-0xfaffffff,0xfb800000-0xfb803fff,0xfb000000-0xfb7fffff irq 16 at device 4.0 on pci7 vgapci0: Boot video device isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xfbb21000-0xfbb217ff irq 18 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahciem0: on ahci0 pcib8: on acpi0 pci8: on pcib8 pci8: at device 14.1 (no driver attached) pci8: at device 19.1 (no driver attached) pci8: at device 19.5 (no driver attached) acpi_button0: on acpi0 acpi_syscontainer0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 xenpv0: on motherboard granttable0: on xenpv0 gntdev0: on xenpv0 debug0: on xenpv0 xc0: on xenpv0 xen_et0: on xenpv0 Event timer "XENTIMER" frequency 1000000000 Hz quality 950 Timecounter "XENTIMER" frequency 1000000000 Hz quality 950 xen_et0: registered as a time-of-day clock, resolution 0.000001s xenstore0: on xenpv0 xsd_dev0: on xenpv0 evtchn0: on xenpv0 privcmd0: on xenpv0 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xce7ff,0xce800-0xcf7ff pnpid ORM0000 on isa0 vga0: at port 0x3d0-0x3db iomem 0xb8000-0xbffff pnpid PNP0900 on isa0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 10.000 msec Timecounters tick every 10.000 msec mfid0 on mfi0 mfid0: 3814912MB (7812939776 sectors) RAID volume (no label) is optimal mfid1 on mfi0 mfid1: 228416MB (467795968 sectors) RAID volume (no label) is optimal mfid2 on mfi0 mfid2: 228416MB (467795968 sectors) RAID volume (no label) is optimal ugen1.1: at usbus1 ugen0.1: at usbus0 uhub0: on usbus1 uhub1: on usbus0 ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device Trying to mount root from zfs:fbroot/ROOT/default []... Root mount waiting for: usbus1 usbus0 uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered Root mount waiting for: usbus1 usbus0 ugen1.2: at usbus1 uhub2 on uhub0 uhub2: on usbus1 ugen0.2: at usbus0 uhub3 on uhub1 uhub3: on usbus0 Root mount waiting for: usbus1 usbus0 uhub3: 6 ports with 6 removable, self powered uhub2: 8 ports with 8 removable, self powered ugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 ioat0: mem 0xfbb1c000-0xfbb1ffff irq 31 at device 4.0 on pci0 ioat0: Capabilities: 2f7 ioat1: mem 0xfbb18000-0xfbb1bfff irq 39 at device 4.1 on pci0 ioat1: Capabilities: 2f7 ioat2: mem 0xfbb14000-0xfbb17fff irq 31 at device 4.2 on pci0 ioat2: Capabilities: f7 ioat3: mem 0xfbb10000-0xfbb13fff irq 39 at device 4.3 on pci0 ioat3: Capabilities: f7 ioat4: mem 0xfbb0c000-0xfbb0ffff irq 31 at device 4.4 on pci0 ioat4: Capabilities: f7 ioat5: mem 0xfbb08000-0xfbb0bfff irq 39 at device 4.5 on pci0 ioat5: Capabilities: f7 ioat6: mem 0xfbb04000-0xfbb07fff irq 31 at device 4.6 on pci0 ioat6: Capabilities: f7 ioat7: mem 0xfbb00000-0xfbb03fff irq 39 at device 4.7 on pci0 ioat7: Capabilities: f7 ums0 on uhub3 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 pflog0: promiscuous mode enabled xenbusb_front0: on xenstore0 xenbusb_back0: on xenstore0 xenballoon0: on xenstore0 xctrl0: on xenstore0 xs_dev0: on xenstore0