From owner-freebsd-xen@FreeBSD.ORG Thu Jan 12 18:43:32 2012 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDD54106566C for ; Thu, 12 Jan 2012 18:43:31 +0000 (UTC) (envelope-from ivan.ivanyuk@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7ECD78FC23 for ; Thu, 12 Jan 2012 18:43:31 +0000 (UTC) Received: by obbta17 with SMTP id ta17so1677515obb.13 for ; Thu, 12 Jan 2012 10:43:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=vAzYXrsLz9cNOHPXYC8pE5sled+BlXgvsNwJgKoZMAw=; b=w6O3d6msIhfhQW4KRczmDrVQfgujMbNomN1GSOPZpthdd/BBjXscrRnSfVR3tAkgLt cPpXujGZkMxFeXUDn3/wt6+4ssjeFJLicWg4hI/Vhm5k0PhV+Abyn7oQ72/uYZrhngBB UxGTx0y6EeTtCtQD0ds3xIc+aBdcQcJcYTlBw= Received: by 10.50.222.193 with SMTP id qo1mr5478776igc.22.1326393810516; Thu, 12 Jan 2012 10:43:30 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.45.137 with HTTP; Thu, 12 Jan 2012 10:42:49 -0800 (PST) In-Reply-To: References: From: Ivan Ivanyuk Date: Thu, 12 Jan 2012 20:42:49 +0200 Message-ID: To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Re: [FreeBSD 8.2 amd64 XENHVM] Incorrect behavior of xen networking drivers: unexpected tcp reassembling X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2012 18:43:32 -0000 On 12 January 2012 00:35, Ivan Ivanyuk wrote: > Hi All, > I can get about 500Mb/s from virtual machine to external host, but > only 200Kb/s from any internal PC to the same external host through > virtual machine router. > > Closest description I found in archives is this: > http://lists.freebsd.org/pipermail/freebsd-xen/2011-May/000902.html. > > My setup is like this: > Dom0 is Debian with =A0xen-hypervisor-4.0-amd64 4.0.1-2. > DomU is FreeBSD 8.2-RELEASE amd64 XENHVM kernel with "device xenpci" opti= on > > > --------- =A0 =A0 =A0 --------------- =A0 =A0 =A0 -------------- =A0 =A0 = =A0 --------------- > |Internet|<--->|eth3 =A0 =A0vif14.0|<--->|xn0 =A0=A0 =A0 =A0 xn1|<--->|vi= f14.1 > eth2|<--->|some PC > ----------=A0 =A0 =A0| =A0 inetbr0 =A0 =A0 | =A0 =A0 | =A0 Freebsd =A0 = =A0| =A0=A0 =A0| =A0 localbr0 =A0 =A0| > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0| =A0 =A0 Dom0 =A0 =A0=A0 | =A0 =A0 | =A0 = =A0 DomU =A0=A0 =A0| =A0 =A0 |=A0 =A0 =A0Dom0 =A0 =A0 =A0| > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0--------------- =A0 =A0 =A0 --------------= =A0 =A0 =A0 --------------- > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 bridge =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0bridge > > So I can see couple of packets with TCP data from Internets coming to > eth3, then the same packets are seen on vif14.0. And then on xn0 I see > only one packet with reassembled TCP payload. > While these big (2976 bytes, 4464 bytes, 8928 bytes, etc) packets are > addressed to DomU - all works. When we try to route them elsewhere - > we get ICMP need fragmentation message sent to origin of these > packets. That's because original (small) TCP packets have DF flag set. > And resulting big TCP packet has DF flag as well. > > So it seems to me that something in the chain "vif14.0<->xn0" is > reassembling TCP packets. > > Is there some sysctl or other settings to control this behavior? (I > tried turning off all offloads on vif14.0 in Dom0, tried to change > fragmentation settings in FreeBSD, nothing changed) Somewhat successful follow-up: There is still some occurrences of reordered packets but original issue was resolved. Turned out this mechanism can be turned off by "ifconfig -lro" in FreeBSD itself. I'm not sure if turning LRO on virtual driver by default is good idea at all, but obviously LRO implementation for xn device doesn't consider DF flag implication on resulting packets. I'm not familiar with other LRO algorithms which used in physical NIC's. Can someone comment if such problem is normal for it? And what is your opinion on using tso, rxcsum and txcsum offloads in virtual envirnoment? Regards, Ivan Ivanyuk