Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 19 Feb 2018 09:58:30 +0000
From:      Laurence Pawling <laurence.pawling@globalsign.com>
To:        "freebsd-xen@freebsd.org" <freebsd-xen@freebsd.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>
Cc:        David King <david.king@globalsign.com>, Vlad Galu <vlad.galu@globalsign.com>
Subject:   multi-vCPU networking issues as client OS under Xen
Message-ID:  <D1AF75E7-49F9-4628-8B26-3ACB64994C97@globalsign.com>
next in thread | raw e-mail | index | archive | help 

[-- Attachment #1 --]
Hi all,

 

I’m wondering if anyone here has seen this issue before, I’ve spent the last couple of days troubleshooting:

 

Platform:

Host: XenServer 7.0 running on 2 x E2660-v4, 256GB RAM

Server VM: FreeBSD 11 (tested on 11.0-p15 and 11.1-p6), 2GB RAM (also tested with 32GB RAM), 1x50GB HDD, 1 x NIC, 2 or more vCPUs in any combination (2 sockets x 1 core, 1 socket x 2 cores, …)

Client VM: FreeBSD 11, any configuration of vCPUs, RAM and HDD.

 

Behaviour:

Sporadic interruption of TCP sessions when utilising the above machine as a “server” with “clients” connecting. Looking into the communication with pcap/Wireshark, you see a TCP Dup Ack sent from both ends, followed by the client sending an RST packet, terminating the TCP session. We have also seen evidence of the client sending a Keepalive packet, which is ACK’d by the server before the RST is sent from the client end.

 

To recreate:

On the above VM, perform a vanilla install of nginx:

pkg install nginx

service nginx onestart

Then on a client VM (currently only tested with FreeBSD), run the following (or similar):

for i in {1..10000}; do if [ $(curl -s -o /dev/null -w "%{http_code}" http://10.2.122.71) != 200 ] ; then echo "error"; fi; done

When vCPUs=1 on the server, I get no errors, when vCPUs>1 I get errors reported. The frequency of errors *seems* to be proportional to the number of vCPUs, but they are sporadic with no clear periodicity or pattern, so that is just anecdotal. Also, the problem seems by far the most prevalent when communicating between two VMs on the same host, in the same VLAN. Xen still sends packets via the switch rather than bridging internally between the interfaces.

Note that we have not had a chance to investigate the effect of different numbers of CPUs on the *client* end, however it does seem to be governed entirely by the server end.

 

We cannot recreate this issue using the same FreeBSD image and same configuration, but using KVM as a hypervisor.

 

Has anyone met this before?

 

Thanks,

 

Laurence


[-- Attachment #2 --]
0	*H
010
	`He0	*H

0b0JI}Ip0
	*H
0]10	UBE10U
GlobalSign nv-sa1301U*GlobalSign PersonalSign 2 CA - SHA256 - G30
160826143608Z
190827143608Z010	UGB1
0UKent10U	Maidstone10U
GMO GlobalSign Limited10ULaurence Pawling1.0,	*H
	laurence.pawling@globalsign.com0"0
	*H
0
\'j9I7[N%vR+yy7iwY
NKgcf``q*d,ezFXy]MZ⏂|MZn9BzMeci2y@%$`>]#թ#qr }=ګ9ދĭyj!e^69oBeҸS,Q>K[
a ͅl&ƚ
ؖp&00U0+00M+0Ahttp://secure.globalsign.com/cacert/gspersonalsign2sha2g3ocsp.crt0=+01http://ocsp2.globalsign.com/gspersonalsign2sha2g30MU F0D0B
+2(
0402+&https://www.globalsign.com/repository/0	U00DU=0;09753http://crl.globalsign.com/gspersonalsign2sha2g3.crl0*U#0!laurence.pawling@globalsign.com0U%0++0U"7_<Qefp0U#0irb1gDw6I{a0
	*H
# e=ᩪ5q¥UdEvT,'
 J_-i(F7'IUN;ֻLT2NOV)Bi+H(f?׀YLyK=F\S_g\/de{I&gNVt7ٕq~3%TbsxX68(Fi0*ݺ8	xIQw[(Ŵ;>ZΘOHT`ilq˞500РHj	$]"0
	*H
0L1 0UGlobalSign Root CA - R310U

GlobalSign10U
GlobalSign0
160615000000Z
240615000000Z0]10	UBE10U
GlobalSign nv-sa1301U*GlobalSign PersonalSign 2 CA - SHA256 - G30"0
	*H
0
he,b
TˤCmo{plIip85~H]9kDjt]>cqD>ɧn&A)IvyK(J{QePJFOܲ=Ϙ!	Lz?7&eGfI&@f=l:)F뙧_Pmo8/NGؤA`7Z;h.H6iGQ,ISSr|q>g00U0jU%c0a+++	
+7
+7
	+7
+7
++0U00Uirb1gDw6I{a0U#0K.E$MPc0>+2000.+0"http://ocsp2.globalsign.com/rootr306U/0-0+)'%http://crl.globalsign.com/root-r3.crl0gU `0^0	+2(0
+2(
0A	+2_0402+&https://www.globalsign.com/repository/0
	*H
2|gPץ\pf%괶S0n7\3(V㓽EXqL䂉GT>濾y("z*I'lUJhPtXEO^0U&4aRPrrgn`D;GuG!1A"VQd"܁~~|[몟V{뤸,,$YD>^g_|lR1+ڽն"T@9u0_0G!XS0
	*H
0L1 0UGlobalSign Root CA - R310U

GlobalSign10U
GlobalSign0
090318100000Z
290318100000Z0L1 0UGlobalSign Root CA - R310U

GlobalSign10U
GlobalSign0"0
	*H
0
%vyx"(vŭrFCDz_$.K`FRGpld,=+׶y;wIjb/^h߉'8>&Y
sް&[`I(i;(坊aW7tt:r/.л=3+S:sA :O.2`W˹hh8&`uw I@H1a^wdz_b
lTin郓qviB0@0U0U00UK.E$MPc0
	*H
K@PTEI	A(3kt-
sgJD{xnlo)39EÎWlS-$lcShgV>5!hS̐]FzX(/7ADmS(~g׊L'Lssvz-
,<U~6WI.-|`AQ#2k,3:;%҆@;,xa/Uo߄	M(rbPe뒗1ٳGX?_100m0]10	UBE10U
GlobalSign nv-sa1301U*GlobalSign PersonalSign 2 CA - SHA256 - G3I}Ip0
	`Hei0/	*H
	1" u~ז 7G9F 8՝jٕڇG0	*H
	1	*H
0	*H
	1
180219095827Z0
	*H
	qӝMHdn/;Oι`ː$'Ax!̭SH	hzcQޟ`|Xۧ5v<aC1%"6b;%TN(}=k"6^FWo<I('W`po/.8(g9VOZx[BJbՍr`O5/͓5pm3NJC=ڬ|ä]{}v7HY^U|XYF%Hrh

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D1AF75E7-49F9-4628-8B26-3ACB64994C97>