Date: Mon, 19 Feb 2018 09:58:30 +0000 From: Laurence Pawling <laurence.pawling@globalsign.com> To: "freebsd-xen@freebsd.org" <freebsd-xen@freebsd.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Cc: David King <david.king@globalsign.com>, Vlad Galu <vlad.galu@globalsign.com> Subject: multi-vCPU networking issues as client OS under Xen Message-ID: <D1AF75E7-49F9-4628-8B26-3ACB64994C97@globalsign.com>
next in thread | raw e-mail | index | archive | help
--B_3601879107_1679118586
Content-type: text/plain;
charset="UTF-8"
Content-transfer-encoding: quoted-printable
Hi all,
=20
I=E2=80=99m wondering if anyone here has seen this issue before, I=E2=80=99ve spent the=
last couple of days troubleshooting:
=20
Platform:
Host: XenServer 7.0 running on 2 x E2660-v4, 256GB RAM
Server VM: FreeBSD 11 (tested on 11.0-p15 and 11.1-p6), 2GB RAM (also teste=
d with 32GB RAM), 1x50GB HDD, 1 x NIC, 2 or more vCPUs in any combination (2=
sockets x 1 core, 1 socket x 2 cores, =E2=80=A6)
Client VM: FreeBSD 11, any configuration of vCPUs, RAM and HDD.
=20
Behaviour:
Sporadic interruption of TCP sessions when utilising the above machine as a=
=E2=80=9Cserver=E2=80=9D with =E2=80=9Cclients=E2=80=9D connecting. Looking into the communication =
with pcap/Wireshark, you see a TCP Dup Ack sent from both ends, followed by =
the client sending an RST packet, terminating the TCP session. We have also =
seen evidence of the client sending a Keepalive packet, which is ACK=E2=80=99d by =
the server before the RST is sent from the client end.
=20
To recreate:
On the above VM, perform a vanilla install of nginx:
pkg install nginx
service nginx onestart
Then on a client VM (currently only tested with FreeBSD), run the following=
(or similar):
for i in {1..10000}; do if [ $(curl -s -o /dev/null -w "%{http_code}" http:=
//10.2.122.71) !=3D 200 ] ; then echo "error"; fi; done
When vCPUs=3D1 on the server, I get no errors, when vCPUs>1 I get errors repo=
rted. The frequency of errors *seems* to be proportional to the number of vC=
PUs, but they are sporadic with no clear periodicity or pattern, so that is =
just anecdotal. Also, the problem seems by far the most prevalent when commu=
nicating between two VMs on the same host, in the same VLAN. Xen still sends=
packets via the switch rather than bridging internally between the interfac=
es.
Note that we have not had a chance to investigate the effect of different n=
umbers of CPUs on the *client* end, however it does seem to be governed enti=
rely by the server end.
=20
We cannot recreate this issue using the same FreeBSD image and same configu=
ration, but using KVM as a hypervisor.
=20
Has anyone met this before?
=20
Thanks,
=20
Laurence
--B_3601879107_1679118586
Content-type: application/pkcs7-signature; name="smime.p7s"
Content-transfer-encoding: base64
Content-disposition: attachment;
filename="smime.p7s"
MIIP9AYJKoZIhvcNAQcCoIIP5TCCD+ECAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0B
BwGggg21MIIFYjCCBEqgAwIBAgIMSX2V7ElwCIrABZuQMA0GCSqGSIb3DQEBCwUAMF0xCzAJ
BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTMwMQYDVQQDEypHbG9iYWxT
aWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2IC0gRzMwHhcNMTYwODI2MTQzNjA4WhcN
MTkwODI3MTQzNjA4WjCBnDELMAkGA1UEBhMCR0IxDTALBgNVBAgTBEtlbnQxEjAQBgNVBAcT
CU1haWRzdG9uZTEfMB0GA1UEChMWR01PIEdsb2JhbFNpZ24gTGltaXRlZDEZMBcGA1UEAxMQ
TGF1cmVuY2UgUGF3bGluZzEuMCwGCSqGSIb3DQEJARYfbGF1cmVuY2UucGF3bGluZ0BnbG9i
YWxzaWduLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALBcjCf6ajlJBRGN
ptXjlTf3Wwv5TuzE6iW5j3YOxxGE2uRSoisepRd5pHkQ3ffF1u4Rjfc3aXcHvlmipxf3pA3e
TrGJGrlLhvpnY2Zg7ZXDYHEqZIfgASxl/5EXeshGqVi9uIoCjHng+oyfXU0GWuKPgqh88oZN
Wm45QoeH4t96TRdlY/dpFjK8ea6vFvZApvbum8bYJZGlJOtgPl0j8/uF1akM8SNxciDQfT39
2quhOd6LxK0LeX9qIe5lXjbhojmyb0IUf4HKZanSuFMer5OY9MnsFgMsUdLs/D5LWw2dsqcF
YfQgGs2FEolsnBwmogblGogYxpqy8qIK2JZwJp0CAwEAAaOCAeAwggHcMA4GA1UdDwEB/wQE
AwIFoDCBngYIKwYBBQUHAQEEgZEwgY4wTQYIKwYBBQUHMAKGQWh0dHA6Ly9zZWN1cmUuZ2xv
YmFsc2lnbi5jb20vY2FjZXJ0L2dzcGVyc29uYWxzaWduMnNoYTJnM29jc3AuY3J0MD0GCCsG
AQUFBzABhjFodHRwOi8vb2NzcDIuZ2xvYmFsc2lnbi5jb20vZ3NwZXJzb25hbHNpZ24yc2hh
MmczME0GA1UdIARGMEQwQgYKKwYBBAGgMgEoCjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3
dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAJBgNVHRMEAjAAMEQGA1UdHwQ9MDswOaA3
oDWGM2h0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vZ3NwZXJzb25hbHNpZ24yc2hhMmczLmNy
bDAqBgNVHREEIzAhgR9sYXVyZW5jZS5wYXdsaW5nQGdsb2JhbHNpZ24uY29tMB0GA1UdJQQW
MBQGCCsGAQUFBwMCBggrBgEFBQcDBDAdBgNVHQ4EFgQU5SIdmwI3kfJf98Y87lFl62amcP0w
HwYDVR0jBBgwFoAUaXKCYjFnlUSFd5GAxAQ2SZ17C2EwDQYJKoZIhvcNAQELBQADggEBACMg
3mWAnD3BBOGpqvrPNem5D3G0GMKlVWSsRax251QsJwogpkoFDgbyX4QtHmkooPRGNydJGwhV
TjvWu5BMle9UuZbcDw6zno3/hLD7kDIUlNJOuU+IgRhWtZwp6ELxaSvIBkiDKPRmpue3P73X
gMTq2VnDwUx5Sxk9FkZcyFNfZ1wvZGWA03sZAZXj1UkmBrsGZ/dOpfsLr1Z0N4AR2ZVx+H4C
MyWzVGJzp5Lvqw54WDY4KL4XRmkw8SrdujgIA5SsoAl4wEmSl1GZiXdb88koxbQ7yME+roNa
q86YT7L4n4cWSBsf5lQTodjeYLWN1mmlBGzBxcv2ceTcy541lbwwggToMIID0KADAgECAg5I
G2oJE72AJMpdIvAU8zANBgkqhkiG9w0BAQsFADBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJv
b3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAe
Fw0xNjA2MTUwMDAwMDBaFw0yNDA2MTUwMDAwMDBaMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQK
ExBHbG9iYWxTaWduIG52LXNhMTMwMQYDVQQDEypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAy
IENBIC0gU0hBMjU2IC0gRzMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC2lmiT
Zf0sAexiow1Uv4vLpEORopqvsYPytW1v2fDq3M8We9cZ44QDhfnGH3CPH2xJxWkZHnCRODV+
AkhdOWsYRKZqpch09F31hD5jH3FEggg+7cmn9uluJkEpgMlJuv/ZdqCjuxv2ed9LKNNKe5xR
wg9lUKGfqEqd5UYEjYNP3LLIPc+YIQmYTMDxj3qpPzcmZUfYZo9JJsLDQL5mPWz/Oq0pRvAT
nOHy65mni8LTX1Btog5vxwaXOC9OoY5HArSDANik47pBB2Dl3Tda8gfBO6ecl2gut++pSDa8
6WmomapH6cf2UdL5sSy2xUm1mJ5TU9r7cvN8D/hxPtcD+mfDAgMBAAGjggG1MIIBsTAOBgNV
HQ8BAf8EBAMCAQYwagYDVR0lBGMwYQYIKwYBBQUHAwIGCCsGAQUFBwMEBggrBgEFBQcDCQYK
KwYBBAGCNxQCAgYKKwYBBAGCNwoDBAYJKwYBBAGCNxUGBgorBgEEAYI3CgMMBggrBgEFBQcD
BwYIKwYBBQUHAxEwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUaXKCYjFnlUSFd5GA
xAQ2SZ17C2EwHwYDVR0jBBgwFoAUj/BLf6guRSSuTVD6Y5qL3uLdG7wwPgYIKwYBBQUHAQEE
MjAwMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcDIuZ2xvYmFsc2lnbi5jb20vcm9vdHIzMDYG
A1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vcm9vdC1yMy5jcmww
ZwYDVR0gBGAwXjALBgkrBgEEAaAyASgwDAYKKwYBBAGgMgEoCjBBBgkrBgEEAaAyAV8wNDAy
BggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9yeS8wDQYJ
KoZIhvcNAQELBQADggEBAKidzTLMfGfiC1DXpVxwo2biJe/qtLZTMG6HEjdcM+LCKFbjk71F
lfNY2BVxTPPkgokUvv6lzEe96wZUgj7mv7716oj1ecQoIguMevYOC+MqrkmaDpvCJ/JsthtV
SgG2GeFoUHRYvBJFGE+u3l4bEzDnVSY0gKL+FIoEqweEYVIRolAAtnLgcvQRZ24TogtgCNfo
FJdEO0cV5Q911vjp/kd/mvMhMYuyf0Eimg5WuBLzvw7gmd9RZCLb3IF+fvkdqOJ8W88L66qf
6txWe+ukuCws5gb/riRZf8VEfz6aFV76ZxJffGxSlzEr2r28tNW2uhT/IlSYQLg5wfl1Cxsc
GqMwggNfMIICR6ADAgECAgsEAAAAAAEhWFMIojANBgkqhkiG9w0BAQsFADBMMSAwHgYDVQQL
ExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE
AxMKR2xvYmFsU2lnbjAeFw0wOTAzMTgxMDAwMDBaFw0yOTAzMTgxMDAwMDBaMEwxIDAeBgNV
BAsTF0dsb2JhbFNpZ24gUm9vdCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYD
VQQDEwpHbG9iYWxTaWduMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzCV2kHkG
eCIW9cCDtoTKKJ79BXYRxa2IcvxGAkPHsoqdBF8kyy5L4WCCRuFSqwyBR3Bs3WTR6/Usow+C
PQwrrpfXthSGEHm7OxOAd4wI4UnSamIvH176lmjfiSeVOJ8G1z7JyyZZDXPesMjpJg6DFcbv
W4vSBGDKSaYo9mk79svIKJHlnYphVzesdBTcdOA67nIvLpz70Lu/9T0A4QYz6IIrrlOmOhZz
jN1BDiA6wLSnoemyT5AuMmDpV8u5BJJoaOU4JmB1sp93/5EU764gSfytQBVI0QIxYRleuJfv
rXe3ZJp6v1/BE++bYvsNbOBUaRapA9pu6YOTcXbGaYWCFwIDAQABo0IwQDAOBgNVHQ8BAf8E
BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUj/BLf6guRSSuTVD6Y5qL3uLdG7ww
DQYJKoZIhvcNAQELBQADggEBAEtA28BQqv7IDO/3llRFSbuWAAlBrLMThoYoBzPKa+Z0uboA
La6kCtP18fEPir9zZ0qDx0R7eOCvbmxvAymOMzlFw47kuVdsqvwSluxTxi3kJGy5lGP73FNo
Z1Y+g7jPNSHDyWj+ztrCU6rMkIrp8F1GjJXdelgoGi8d3s0AN0GP7URt11Mol37zZwQeFdeK
lrTT3kwnpEwbc3N29BeZwh96DuMtCK0KHCz/PKtVDg+Rfjbrw1dJvuEuLXxgi8NBURMjnc73
MmuUAaiZ5ywzHzo7JdKGQM47LIZ4yWEvFLru21Vv34TuBQlNvSjYcs7TYlBlHuuSl4Mx2bO1
ykdYP18xggIDMIIB/wIBATBtMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWdu
IG52LXNhMTMwMQYDVQQDEypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2
IC0gRzMCDEl9lexJcAiKwAWbkDANBglghkgBZQMEAgEFAKBpMC8GCSqGSIb3DQEJBDEiBCB1
hdN+15YgrzdHOYlGIDgH1Z393+n2atHqAdmV2odHvTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN
AQcBMBwGCSqGSIb3DQEJBTEPFw0xODAyMTkwOTU4MjdaMA0GCSqGSIb3DQEBAQUABIIBAIYJ
rroL/O9xwtOdTUhkbvXH+oYvFTtPzrlg1QvLkCQn+IC6BUF4840hpt7MrQOeU5vASAlo54t6
Y1GJiOj/kfMU3p+u0GCPf7LsiRWGfFjbpzX2nHaDisT7PGFDp9beMSXQIjZiO9ElElSbTpPF
Dr4ofT21umsiuzZevY9GV2+CPEmt9ijuFCcDG1eEBppgEBlwjG+BkoYv2sEu7DiqKGf10znh
Vk9aHngQtVuBi67P8LBCEsNKlGKW1Y1yH9hgT8G7CMM1jfqbCC/NkzVwba/F394z8LZOv5eI
St5DPdqs5HzDpF3vjXuGz312Nx1IWV5VfAFYWflGgCVIch0H6mg=
--B_3601879107_1679118586--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D1AF75E7-49F9-4628-8B26-3ACB64994C97>