Date: Thu, 12 Jan 2012 00:35:11 +0200 From: Ivan Ivanyuk <ivan.ivanyuk@gmail.com> To: freebsd-xen@freebsd.org Subject: [FreeBSD 8.2 amd64 XENHVM] Incorrect behavior of xen networking drivers: unexpected tcp reassembling Message-ID: <CAD_Den-o6rN4uarRrRpPrdzjX30kSJjrO%2BquTmZQ6cqokTrYFw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi All, I've been running FreeBSD 8.2-RELEASE amd64 with XENHVM kernel with PV drivers for a couple of months but when trying to use it as router I get huge problems with network performance. I can get about 500Mb/s from virtual machine to external host, but only 200Kb/s from any internal PC to the same external host through virtual machine router. Closest description I found in archives is this: http://lists.freebsd.org/pipermail/freebsd-xen/2011-May/000902.html. With further investigation by tcpdump I found a couple of problems with packets handling. My setup is like this: Dom0 is Debian with =A0xen-hypervisor-4.0-amd64 4.0.1-2. DomU is FreeBSD 8.2-RELEASE amd64 XENHVM kernel with "device xenpci" option --------- =A0 =A0 =A0 --------------- =A0 =A0 =A0 -------------- ----= ----------- |Internet|<--->|eth3 =A0 =A0vif14.0|<--->|xn0 =A0=A0 =A0 =A0 xn1|<--->|vif1= 4.1 eth2|<--->|some PC ----------=A0 =A0 | =A0 inetbr0 | =A0 =A0 | =A0 Freebsd =A0 | =A0 = =A0| =A0 localbr0 =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0| =A0 =A0 Dom0 =A0 =A0 | =A0 =A0 | =A0 =A0 Do= mU =A0=A0 =A0| =A0 =A0 |=A0 =A0 =A0Dom0 | --------------- -------------- --------------- bridge bridge So I can see couple of packets with TCP data from Internets coming to eth3, then the same packets are seen on vif14.0. And then on xn0 I see only one packet with reassembled TCP payload. While these big (2976 bytes, 4464 bytes, 8928 bytes, etc) packets are addressed to DomU - all works. When we try to route them elsewhere - we get ICMP need fragmentation message sent to origin of these packets. That's because original (small) TCP packets have DF flag set. And resulting big TCP packet has DF flag as well. So it seems to me that something in the chain "vif14.0<->xn0" is reassembling TCP packets. Is there some sysctl or other settings to control this behavior? (I tried turning off all offloads on vif14.0 in Dom0, tried to change fragmentation settings in FreeBSD, nothing changed) Now I set "scrub no-df" in pf to permit fragmentation of this big packets but get another strange problem, which still prevent expected network performance, but this will wait until first problem is solved. I can provide tcpdumps from eth3, vif14.0, xn0 for investigation. Regards, Ivan Ivanyuk
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD_Den-o6rN4uarRrRpPrdzjX30kSJjrO%2BquTmZQ6cqokTrYFw>