From owner-freebsd-net@freebsd.org Mon Feb 19 09:58:37 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75358F14FE9; Mon, 19 Feb 2018 09:58:37 +0000 (UTC) (envelope-from laurence.pawling@globalsign.com) Received: from APC01-PU1-obe.outbound.protection.outlook.com (mail-pu1apc01on0108.outbound.protection.outlook.com [104.47.126.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2DC306AB59; Mon, 19 Feb 2018 09:58:35 +0000 (UTC) (envelope-from laurence.pawling@globalsign.com) Received: from HK2PR0302MB2545.apcprd03.prod.outlook.com (10.170.152.14) by HK2PR0302MB2420.apcprd03.prod.outlook.com (10.170.151.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.527.6; Mon, 19 Feb 2018 09:58:31 +0000 Received: from HK2PR0302MB2545.apcprd03.prod.outlook.com ([fe80::997e:7c67:c760:d087]) by HK2PR0302MB2545.apcprd03.prod.outlook.com ([fe80::997e:7c67:c760:d087%2]) with mapi id 15.20.0527.012; Mon, 19 Feb 2018 09:58:30 +0000 From: Laurence Pawling To: "freebsd-xen@freebsd.org" , "freebsd-virtualization@freebsd.org" , "freebsd-net@freebsd.org" CC: David King , Vlad Galu Subject: multi-vCPU networking issues as client OS under Xen Thread-Topic: multi-vCPU networking issues as client OS under Xen Thread-Index: AQHTqWgyLMPz8qfMa0GcAhXNzUcYuA== Date: Mon, 19 Feb 2018 09:58:30 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [89.197.152.162] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HK2PR0302MB2420; 7:Jy+0LdTqz0VYXfNf2pZl3s/C/jZ7SQsE14zFpVs2YO7URYRr3lv64eqTlJ5Ilz2qWeeh1PFJjOMxA8yU88jfPy5t0GuT+KTTCQUMPSbpbDixZq8p4UpVVLsHp5rShIaw3PA/AvOQSQCsM1P3t/73JmFpJjDHvh8zeiJzcNPjYu6Lb4PFwTjltzb/NqVRRMflzoXYMdFY88MVMSYH82N4ubFyuTttyrznO13sa9zgG3HZgUO3dbIa0rgIPPSEyMYV x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-forefront-antispam-report: SFV:SKI; SCL:-1; SFV:NSPM; SFS:(10019020)(6019001)(7966004)(376002)(366004)(39850400004)(39380400002)(396003)(346002)(269900001)(189003)(199004)(53754006)(3280700002)(102836004)(105586002)(99936001)(6506007)(68736007)(55236004)(316002)(26005)(6116002)(3846002)(3660700001)(110136005)(36756003)(186003)(54906003)(4326008)(450100002)(6306002)(54896002)(6512007)(53936002)(82746002)(6436002)(6486002)(107886003)(106356001)(2501003)(5250100002)(5660300001)(2906002)(25786009)(2900100001)(33656002)(97736004)(66066001)(8936002)(83716003)(86362001)(508600001)(8676002)(81166006)(81156014)(7736002)(99286004)(14454004)(2201001)(966005); DIR:OUT; SFP:1102; SCL:1; SRVR:HK2PR0302MB2420; H:HK2PR0302MB2545.apcprd03.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; x-ms-office365-filtering-correlation-id: 2dd732c0-f776-41c5-28b3-08d5777f54b4 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(4534165)(7168020)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(3008032)(2017052603307)(7153060)(49563074)(7193020); SRVR:HK2PR0302MB2420; x-ms-traffictypediagnostic: HK2PR0302MB2420: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(158342451672863)(21748063052155); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415395)(6040501)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231101)(944501161)(10201501046)(3002001)(6041288)(20161123562045)(20161123564045)(20161123560045)(20161123558120)(201703131423095)(201703011903075)(201702281528075)(20161123555045)(201703061421075)(6072148)(201708071742011); SRVR:HK2PR0302MB2420; BCL:0; PCL:0; RULEID:; SRVR:HK2PR0302MB2420; x-forefront-prvs: 0588B2BD96 received-spf: None (protection.outlook.com: globalsign.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: 6f5xtPZRz6fb2IJcO2+F9eGVS6+UNiCiVRflUTJOaoFdG9F1sDQ1zLt52Z4PY+zILxt63HYDDis3FzwZkscoueHEW4+vOk5k3jAcN77A4PM5+EPR+YDQSDGrwfbcKiLrmG2emJQ1F8zPLRgd1FJUv1d5pMlhw6qTGcPWI+IfL9I= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha256; boundary="B_3601879107_1679118586" MIME-Version: 1.0 X-OriginatorOrg: globalsign.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2dd732c0-f776-41c5-28b3-08d5777f54b4 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Feb 2018 09:58:30.7188 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8fff67c1-8281-4635-b62f-93106cb7a9a8 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HK2PR0302MB2420 X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Feb 2018 09:58:37 -0000 --B_3601879107_1679118586 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable Hi all, =20 I=E2=80=99m wondering if anyone here has seen this issue before, I=E2=80=99ve spent the= last couple of days troubleshooting: =20 Platform: Host: XenServer 7.0 running on 2 x E2660-v4, 256GB RAM Server VM: FreeBSD 11 (tested on 11.0-p15 and 11.1-p6), 2GB RAM (also teste= d with 32GB RAM), 1x50GB HDD, 1 x NIC, 2 or more vCPUs in any combination (2= sockets x 1 core, 1 socket x 2 cores, =E2=80=A6) Client VM: FreeBSD 11, any configuration of vCPUs, RAM and HDD. =20 Behaviour: Sporadic interruption of TCP sessions when utilising the above machine as a= =E2=80=9Cserver=E2=80=9D with =E2=80=9Cclients=E2=80=9D connecting. Looking into the communication = with pcap/Wireshark, you see a TCP Dup Ack sent from both ends, followed by = the client sending an RST packet, terminating the TCP session. We have also = seen evidence of the client sending a Keepalive packet, which is ACK=E2=80=99d by = the server before the RST is sent from the client end. =20 To recreate: On the above VM, perform a vanilla install of nginx: pkg install nginx service nginx onestart Then on a client VM (currently only tested with FreeBSD), run the following= (or similar): for i in {1..10000}; do if [ $(curl -s -o /dev/null -w "%{http_code}" http:= //10.2.122.71) !=3D 200 ] ; then echo "error"; fi; done When vCPUs=3D1 on the server, I get no errors, when vCPUs>1 I get errors repo= rted. The frequency of errors *seems* to be proportional to the number of vC= PUs, but they are sporadic with no clear periodicity or pattern, so that is = just anecdotal. Also, the problem seems by far the most prevalent when commu= nicating between two VMs on the same host, in the same VLAN. Xen still sends= packets via the switch rather than bridging internally between the interfac= es. Note that we have not had a chance to investigate the effect of different n= umbers of CPUs on the *client* end, however it does seem to be governed enti= rely by the server end. =20 We cannot recreate this issue using the same FreeBSD image and same configu= ration, but using KVM as a hypervisor. =20 Has anyone met this before? =20 Thanks, =20 Laurence --B_3601879107_1679118586 Content-type: application/pkcs7-signature; name="smime.p7s" Content-transfer-encoding: base64 Content-disposition: attachment; filename="smime.p7s" MIIP9AYJKoZIhvcNAQcCoIIP5TCCD+ECAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0B BwGggg21MIIFYjCCBEqgAwIBAgIMSX2V7ElwCIrABZuQMA0GCSqGSIb3DQEBCwUAMF0xCzAJ BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTMwMQYDVQQDEypHbG9iYWxT aWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2IC0gRzMwHhcNMTYwODI2MTQzNjA4WhcN MTkwODI3MTQzNjA4WjCBnDELMAkGA1UEBhMCR0IxDTALBgNVBAgTBEtlbnQxEjAQBgNVBAcT CU1haWRzdG9uZTEfMB0GA1UEChMWR01PIEdsb2JhbFNpZ24gTGltaXRlZDEZMBcGA1UEAxMQ TGF1cmVuY2UgUGF3bGluZzEuMCwGCSqGSIb3DQEJARYfbGF1cmVuY2UucGF3bGluZ0BnbG9i YWxzaWduLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALBcjCf6ajlJBRGN ptXjlTf3Wwv5TuzE6iW5j3YOxxGE2uRSoisepRd5pHkQ3ffF1u4Rjfc3aXcHvlmipxf3pA3e TrGJGrlLhvpnY2Zg7ZXDYHEqZIfgASxl/5EXeshGqVi9uIoCjHng+oyfXU0GWuKPgqh88oZN Wm45QoeH4t96TRdlY/dpFjK8ea6vFvZApvbum8bYJZGlJOtgPl0j8/uF1akM8SNxciDQfT39 2quhOd6LxK0LeX9qIe5lXjbhojmyb0IUf4HKZanSuFMer5OY9MnsFgMsUdLs/D5LWw2dsqcF YfQgGs2FEolsnBwmogblGogYxpqy8qIK2JZwJp0CAwEAAaOCAeAwggHcMA4GA1UdDwEB/wQE AwIFoDCBngYIKwYBBQUHAQEEgZEwgY4wTQYIKwYBBQUHMAKGQWh0dHA6Ly9zZWN1cmUuZ2xv YmFsc2lnbi5jb20vY2FjZXJ0L2dzcGVyc29uYWxzaWduMnNoYTJnM29jc3AuY3J0MD0GCCsG AQUFBzABhjFodHRwOi8vb2NzcDIuZ2xvYmFsc2lnbi5jb20vZ3NwZXJzb25hbHNpZ24yc2hh MmczME0GA1UdIARGMEQwQgYKKwYBBAGgMgEoCjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3 dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAJBgNVHRMEAjAAMEQGA1UdHwQ9MDswOaA3 oDWGM2h0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vZ3NwZXJzb25hbHNpZ24yc2hhMmczLmNy bDAqBgNVHREEIzAhgR9sYXVyZW5jZS5wYXdsaW5nQGdsb2JhbHNpZ24uY29tMB0GA1UdJQQW MBQGCCsGAQUFBwMCBggrBgEFBQcDBDAdBgNVHQ4EFgQU5SIdmwI3kfJf98Y87lFl62amcP0w HwYDVR0jBBgwFoAUaXKCYjFnlUSFd5GAxAQ2SZ17C2EwDQYJKoZIhvcNAQELBQADggEBACMg 3mWAnD3BBOGpqvrPNem5D3G0GMKlVWSsRax251QsJwogpkoFDgbyX4QtHmkooPRGNydJGwhV TjvWu5BMle9UuZbcDw6zno3/hLD7kDIUlNJOuU+IgRhWtZwp6ELxaSvIBkiDKPRmpue3P73X gMTq2VnDwUx5Sxk9FkZcyFNfZ1wvZGWA03sZAZXj1UkmBrsGZ/dOpfsLr1Z0N4AR2ZVx+H4C MyWzVGJzp5Lvqw54WDY4KL4XRmkw8SrdujgIA5SsoAl4wEmSl1GZiXdb88koxbQ7yME+roNa q86YT7L4n4cWSBsf5lQTodjeYLWN1mmlBGzBxcv2ceTcy541lbwwggToMIID0KADAgECAg5I G2oJE72AJMpdIvAU8zANBgkqhkiG9w0BAQsFADBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJv b3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAe Fw0xNjA2MTUwMDAwMDBaFw0yNDA2MTUwMDAwMDBaMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQK ExBHbG9iYWxTaWduIG52LXNhMTMwMQYDVQQDEypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAy IENBIC0gU0hBMjU2IC0gRzMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC2lmiT Zf0sAexiow1Uv4vLpEORopqvsYPytW1v2fDq3M8We9cZ44QDhfnGH3CPH2xJxWkZHnCRODV+ AkhdOWsYRKZqpch09F31hD5jH3FEggg+7cmn9uluJkEpgMlJuv/ZdqCjuxv2ed9LKNNKe5xR wg9lUKGfqEqd5UYEjYNP3LLIPc+YIQmYTMDxj3qpPzcmZUfYZo9JJsLDQL5mPWz/Oq0pRvAT nOHy65mni8LTX1Btog5vxwaXOC9OoY5HArSDANik47pBB2Dl3Tda8gfBO6ecl2gut++pSDa8 6WmomapH6cf2UdL5sSy2xUm1mJ5TU9r7cvN8D/hxPtcD+mfDAgMBAAGjggG1MIIBsTAOBgNV HQ8BAf8EBAMCAQYwagYDVR0lBGMwYQYIKwYBBQUHAwIGCCsGAQUFBwMEBggrBgEFBQcDCQYK KwYBBAGCNxQCAgYKKwYBBAGCNwoDBAYJKwYBBAGCNxUGBgorBgEEAYI3CgMMBggrBgEFBQcD BwYIKwYBBQUHAxEwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUaXKCYjFnlUSFd5GA xAQ2SZ17C2EwHwYDVR0jBBgwFoAUj/BLf6guRSSuTVD6Y5qL3uLdG7wwPgYIKwYBBQUHAQEE MjAwMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcDIuZ2xvYmFsc2lnbi5jb20vcm9vdHIzMDYG A1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vcm9vdC1yMy5jcmww ZwYDVR0gBGAwXjALBgkrBgEEAaAyASgwDAYKKwYBBAGgMgEoCjBBBgkrBgEEAaAyAV8wNDAy BggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9yeS8wDQYJ KoZIhvcNAQELBQADggEBAKidzTLMfGfiC1DXpVxwo2biJe/qtLZTMG6HEjdcM+LCKFbjk71F lfNY2BVxTPPkgokUvv6lzEe96wZUgj7mv7716oj1ecQoIguMevYOC+MqrkmaDpvCJ/JsthtV SgG2GeFoUHRYvBJFGE+u3l4bEzDnVSY0gKL+FIoEqweEYVIRolAAtnLgcvQRZ24TogtgCNfo FJdEO0cV5Q911vjp/kd/mvMhMYuyf0Eimg5WuBLzvw7gmd9RZCLb3IF+fvkdqOJ8W88L66qf 6txWe+ukuCws5gb/riRZf8VEfz6aFV76ZxJffGxSlzEr2r28tNW2uhT/IlSYQLg5wfl1Cxsc GqMwggNfMIICR6ADAgECAgsEAAAAAAEhWFMIojANBgkqhkiG9w0BAQsFADBMMSAwHgYDVQQL ExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE AxMKR2xvYmFsU2lnbjAeFw0wOTAzMTgxMDAwMDBaFw0yOTAzMTgxMDAwMDBaMEwxIDAeBgNV BAsTF0dsb2JhbFNpZ24gUm9vdCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYD VQQDEwpHbG9iYWxTaWduMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzCV2kHkG eCIW9cCDtoTKKJ79BXYRxa2IcvxGAkPHsoqdBF8kyy5L4WCCRuFSqwyBR3Bs3WTR6/Usow+C PQwrrpfXthSGEHm7OxOAd4wI4UnSamIvH176lmjfiSeVOJ8G1z7JyyZZDXPesMjpJg6DFcbv W4vSBGDKSaYo9mk79svIKJHlnYphVzesdBTcdOA67nIvLpz70Lu/9T0A4QYz6IIrrlOmOhZz jN1BDiA6wLSnoemyT5AuMmDpV8u5BJJoaOU4JmB1sp93/5EU764gSfytQBVI0QIxYRleuJfv rXe3ZJp6v1/BE++bYvsNbOBUaRapA9pu6YOTcXbGaYWCFwIDAQABo0IwQDAOBgNVHQ8BAf8E BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUj/BLf6guRSSuTVD6Y5qL3uLdG7ww DQYJKoZIhvcNAQELBQADggEBAEtA28BQqv7IDO/3llRFSbuWAAlBrLMThoYoBzPKa+Z0uboA La6kCtP18fEPir9zZ0qDx0R7eOCvbmxvAymOMzlFw47kuVdsqvwSluxTxi3kJGy5lGP73FNo Z1Y+g7jPNSHDyWj+ztrCU6rMkIrp8F1GjJXdelgoGi8d3s0AN0GP7URt11Mol37zZwQeFdeK lrTT3kwnpEwbc3N29BeZwh96DuMtCK0KHCz/PKtVDg+Rfjbrw1dJvuEuLXxgi8NBURMjnc73 MmuUAaiZ5ywzHzo7JdKGQM47LIZ4yWEvFLru21Vv34TuBQlNvSjYcs7TYlBlHuuSl4Mx2bO1 ykdYP18xggIDMIIB/wIBATBtMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWdu IG52LXNhMTMwMQYDVQQDEypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2 IC0gRzMCDEl9lexJcAiKwAWbkDANBglghkgBZQMEAgEFAKBpMC8GCSqGSIb3DQEJBDEiBCB1 hdN+15YgrzdHOYlGIDgH1Z393+n2atHqAdmV2odHvTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0xODAyMTkwOTU4MjdaMA0GCSqGSIb3DQEBAQUABIIBAIYJ rroL/O9xwtOdTUhkbvXH+oYvFTtPzrlg1QvLkCQn+IC6BUF4840hpt7MrQOeU5vASAlo54t6 Y1GJiOj/kfMU3p+u0GCPf7LsiRWGfFjbpzX2nHaDisT7PGFDp9beMSXQIjZiO9ElElSbTpPF Dr4ofT21umsiuzZevY9GV2+CPEmt9ijuFCcDG1eEBppgEBlwjG+BkoYv2sEu7DiqKGf10znh Vk9aHngQtVuBi67P8LBCEsNKlGKW1Y1yH9hgT8G7CMM1jfqbCC/NkzVwba/F394z8LZOv5eI St5DPdqs5HzDpF3vjXuGz312Nx1IWV5VfAFYWflGgCVIch0H6mg= --B_3601879107_1679118586--