From owner-freebsd-xen@FreeBSD.ORG Wed Jan 11 23:04:26 2012 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 007BC1065678 for ; Wed, 11 Jan 2012 23:04:26 +0000 (UTC) (envelope-from ivan.ivanyuk@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id C1C0B8FC15 for ; Wed, 11 Jan 2012 23:04:25 +0000 (UTC) Received: by obbwd18 with SMTP id wd18so2336085obb.13 for ; Wed, 11 Jan 2012 15:04:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=xr8FpCtofHvv7vRVlhWBrRf6ptnVrCyZ/ZeSQTsY+tY=; b=Ha1S3JDOzKLJJbK0TRBo4/kxB6jQyu/pUS2eaB++W/UW0sPWUuJggt8scp/nNa/iZ0 VYjdWx2FsJI4OqgImxWSZv6UG0Iu9DAnWXRh0ZjvnS9rKzJNWT7ChnOV9IfZrbfPIb5w JO53s6eCp4DNk1jPBUWJdokClSpZsy9FYT+BU= Received: by 10.182.197.104 with SMTP id it8mr1012553obc.17.1326321352254; Wed, 11 Jan 2012 14:35:52 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.45.137 with HTTP; Wed, 11 Jan 2012 14:35:11 -0800 (PST) From: Ivan Ivanyuk Date: Thu, 12 Jan 2012 00:35:11 +0200 Message-ID: To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: [FreeBSD 8.2 amd64 XENHVM] Incorrect behavior of xen networking drivers: unexpected tcp reassembling X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2012 23:04:26 -0000 Hi All, I've been running FreeBSD 8.2-RELEASE amd64 with XENHVM kernel with PV drivers for a couple of months but when trying to use it as router I get huge problems with network performance. I can get about 500Mb/s from virtual machine to external host, but only 200Kb/s from any internal PC to the same external host through virtual machine router. Closest description I found in archives is this: http://lists.freebsd.org/pipermail/freebsd-xen/2011-May/000902.html. With further investigation by tcpdump I found a couple of problems with packets handling. My setup is like this: Dom0 is Debian with =A0xen-hypervisor-4.0-amd64 4.0.1-2. DomU is FreeBSD 8.2-RELEASE amd64 XENHVM kernel with "device xenpci" option --------- =A0 =A0 =A0 --------------- =A0 =A0 =A0 -------------- ----= ----------- |Internet|<--->|eth3 =A0 =A0vif14.0|<--->|xn0 =A0=A0 =A0 =A0 xn1|<--->|vif1= 4.1 eth2|<--->|some PC ----------=A0 =A0 | =A0 inetbr0 | =A0 =A0 | =A0 Freebsd =A0 | =A0 = =A0| =A0 localbr0 =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0| =A0 =A0 Dom0 =A0 =A0 | =A0 =A0 | =A0 =A0 Do= mU =A0=A0 =A0| =A0 =A0 |=A0 =A0 =A0Dom0 | --------------- -------------- --------------- bridge bridge So I can see couple of packets with TCP data from Internets coming to eth3, then the same packets are seen on vif14.0. And then on xn0 I see only one packet with reassembled TCP payload. While these big (2976 bytes, 4464 bytes, 8928 bytes, etc) packets are addressed to DomU - all works. When we try to route them elsewhere - we get ICMP need fragmentation message sent to origin of these packets. That's because original (small) TCP packets have DF flag set. And resulting big TCP packet has DF flag as well. So it seems to me that something in the chain "vif14.0<->xn0" is reassembling TCP packets. Is there some sysctl or other settings to control this behavior? (I tried turning off all offloads on vif14.0 in Dom0, tried to change fragmentation settings in FreeBSD, nothing changed) Now I set "scrub no-df" in pf to permit fragmentation of this big packets but get another strange problem, which still prevent expected network performance, but this will wait until first problem is solved. I can provide tcpdumps from eth3, vif14.0, xn0 for investigation. Regards, Ivan Ivanyuk