From owner-freebsd-xen@FreeBSD.ORG Mon Jan 9 10:38:06 2012 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD5AB106566B for ; Mon, 9 Jan 2012 10:38:06 +0000 (UTC) (envelope-from krichy@tvnetwork.hu) Received: from smtp-b.tvnetwork.hu (smtp-b.tvnetwork.hu [109.61.0.52]) by mx1.freebsd.org (Postfix) with SMTP id 30A498FC14 for ; Mon, 9 Jan 2012 10:38:05 +0000 (UTC) Received: (qmail 4475 invoked by uid 1001); 9 Jan 2012 11:38:03 +0100 Received: from 109.61.101.194 by smtp-b.tvnetwork.hu (envelope-from , uid 64011) with qmail-scanner-1.25st (clamdscan: 0.88.1/1396. spamassassin: 3.0.3. perlscan: 1.25st. Clear:RC:1(109.61.101.194):SA:0(-2.9/5.0):. Processed in 6.00812 secs); 09 Jan 2012 10:38:03 -0000 X-Spam-Status: No, hits=-2.9 required=5.0 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp-b.tvnetwork.hu X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.3.1 X-Spam-Report: * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] X-Envelope-From: krichy@tvnetwork.hu Received: from unknown (HELO krichy.tvnetwork.hu) (109.61.101.194) by smtp-b.tvnetwork.hu with SMTP; 9 Jan 2012 11:37:57 +0100 Received: by krichy.tvnetwork.hu (Postfix, from userid 1000) id 807F93F1E; Mon, 9 Jan 2012 11:37:57 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by krichy.tvnetwork.hu (Postfix) with ESMTP id 789ED1079 for ; Mon, 9 Jan 2012 11:37:57 +0100 (CET) Date: Mon, 9 Jan 2012 11:37:57 +0100 (CET) From: Richard Kojedzinszky To: freebsd-xen@freebsd.org Message-ID: User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: 9.0 PV under amd64 xen 4.0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2012 10:38:06 -0000 Hi all, I;ve tried to boot a freshly built 9.0 XEN PV kernel under amd64 xen 4.0 hypervisor, but got this error message: xc_dom_compat_check: guest type xen-3.0-x86_32 not supported by xen kernel, sorry The configuration is the included XEN configuration in the source tree, with the only difference that modules are not built. On the host, xm info gives: # xm info host : host release : 2.6.32-5-xen-amd64 version : #1 SMP Thu Nov 3 05:42:31 UTC 2011 machine : x86_64 nr_cpus : 4 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2660 hw_caps : bfebfbff:20100800:00000000:00000940:0000e3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 4094 free_memory : 1997 node_to_cpu : node0:0-3 node_to_memory : node0:1997 node_to_dma32_mem : node0:1997 max_node_id : 0 xen_major : 4 xen_minor : 0 xen_extra : .1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable xen_commandline : dom0_mem=512M dom0_max_vcpus=1 dom0_vcpus_pin cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8) cc_compile_by : waldi cc_compile_domain : debian.org cc_compile_date : Sun Nov 6 09:42:15 CET 2011 xend_config_format : 4 Within this host a 8.2 PV guest boots well. Any advise? Regards, Kojedzinszky Richard Euronet Magyarorszag Informatikai Zrt. From owner-freebsd-xen@FreeBSD.ORG Mon Jan 9 11:07:19 2012 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6ED2106564A for ; Mon, 9 Jan 2012 11:07:19 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8A63D8FC1D for ; Mon, 9 Jan 2012 11:07:19 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q09B7JVT042391 for ; Mon, 9 Jan 2012 11:07:19 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q09B7IwR042389 for freebsd-xen@FreeBSD.org; Mon, 9 Jan 2012 11:07:18 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 9 Jan 2012 11:07:18 GMT Message-Id: <201201091107.q09B7IwR042389@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-xen@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-xen@FreeBSD.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2012 11:07:19 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/162677 xen [xen] FreeBSD not compatible with "Current Stable Xen" o kern/161318 xen [xen] sysinstall crashes with floating point exception o kern/155468 xen [xen] Xen PV i386 multi-kernel CPU system is not worki o kern/155353 xen [xen] [patch] put "nudging TOD" message under boot_ver o kern/154833 xen [xen]: xen 4.0 - DomU freebsd8.2RC3 i386, XEN kernel. o kern/154473 xen [xen] xen 4.0 - DomU freebsd8.1 i386, XEN kernel. Not o kern/154472 xen [xen] xen 4.0 - DomU freebsd8.1 i386 xen kernel reboot o kern/154428 xen [xen] xn0 network interface and PF - Massive performan o kern/153674 xen [xen] i386/XEN idle thread shows wrong percentages o kern/153672 xen [xen] [panic] i386/XEN panics under heavy fork load o kern/153620 xen [xen] Xen guest system clock drifts in AWS EC2 (FreeBS o kern/153477 xen [xen] XEN pmap code abuses vm page queue lock o kern/153150 xen [xen] xen/ec2: disable checksum offloading on interfac o kern/152228 xen [xen] [panic] Xen/PV panic with machdep.idle_mwait=1 o kern/144629 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143398 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143340 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor f kern/143069 xen [xen] [panic] Xen Kernel Panic - Memory modified after f kern/135667 xen ufs filesystem corruption on XEN DomU system f kern/135421 xen [xen] FreeBSD Xen PVM DomU network failure - netfronc. f kern/135178 xen [xen] Xen domU outgoing data transfer stall when TSO i p kern/135069 xen [xen] FreeBSD-current/Xen SMP doesn't function at all f i386/124516 xen [xen] FreeBSD-CURRENT Xen Kernel Segfaults when config o kern/118734 xen [xen] FreeBSD 6.3-RC1 and FreeBSD 7.0-BETA 4 fail to b 24 problems total. From owner-freebsd-xen@FreeBSD.ORG Wed Jan 11 23:04:26 2012 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 007BC1065678 for ; Wed, 11 Jan 2012 23:04:26 +0000 (UTC) (envelope-from ivan.ivanyuk@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id C1C0B8FC15 for ; Wed, 11 Jan 2012 23:04:25 +0000 (UTC) Received: by obbwd18 with SMTP id wd18so2336085obb.13 for ; Wed, 11 Jan 2012 15:04:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=xr8FpCtofHvv7vRVlhWBrRf6ptnVrCyZ/ZeSQTsY+tY=; b=Ha1S3JDOzKLJJbK0TRBo4/kxB6jQyu/pUS2eaB++W/UW0sPWUuJggt8scp/nNa/iZ0 VYjdWx2FsJI4OqgImxWSZv6UG0Iu9DAnWXRh0ZjvnS9rKzJNWT7ChnOV9IfZrbfPIb5w JO53s6eCp4DNk1jPBUWJdokClSpZsy9FYT+BU= Received: by 10.182.197.104 with SMTP id it8mr1012553obc.17.1326321352254; Wed, 11 Jan 2012 14:35:52 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.45.137 with HTTP; Wed, 11 Jan 2012 14:35:11 -0800 (PST) From: Ivan Ivanyuk Date: Thu, 12 Jan 2012 00:35:11 +0200 Message-ID: To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: [FreeBSD 8.2 amd64 XENHVM] Incorrect behavior of xen networking drivers: unexpected tcp reassembling X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2012 23:04:26 -0000 Hi All, I've been running FreeBSD 8.2-RELEASE amd64 with XENHVM kernel with PV drivers for a couple of months but when trying to use it as router I get huge problems with network performance. I can get about 500Mb/s from virtual machine to external host, but only 200Kb/s from any internal PC to the same external host through virtual machine router. Closest description I found in archives is this: http://lists.freebsd.org/pipermail/freebsd-xen/2011-May/000902.html. With further investigation by tcpdump I found a couple of problems with packets handling. My setup is like this: Dom0 is Debian with =A0xen-hypervisor-4.0-amd64 4.0.1-2. DomU is FreeBSD 8.2-RELEASE amd64 XENHVM kernel with "device xenpci" option --------- =A0 =A0 =A0 --------------- =A0 =A0 =A0 -------------- ----= ----------- |Internet|<--->|eth3 =A0 =A0vif14.0|<--->|xn0 =A0=A0 =A0 =A0 xn1|<--->|vif1= 4.1 eth2|<--->|some PC ----------=A0 =A0 | =A0 inetbr0 | =A0 =A0 | =A0 Freebsd =A0 | =A0 = =A0| =A0 localbr0 =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0| =A0 =A0 Dom0 =A0 =A0 | =A0 =A0 | =A0 =A0 Do= mU =A0=A0 =A0| =A0 =A0 |=A0 =A0 =A0Dom0 | --------------- -------------- --------------- bridge bridge So I can see couple of packets with TCP data from Internets coming to eth3, then the same packets are seen on vif14.0. And then on xn0 I see only one packet with reassembled TCP payload. While these big (2976 bytes, 4464 bytes, 8928 bytes, etc) packets are addressed to DomU - all works. When we try to route them elsewhere - we get ICMP need fragmentation message sent to origin of these packets. That's because original (small) TCP packets have DF flag set. And resulting big TCP packet has DF flag as well. So it seems to me that something in the chain "vif14.0<->xn0" is reassembling TCP packets. Is there some sysctl or other settings to control this behavior? (I tried turning off all offloads on vif14.0 in Dom0, tried to change fragmentation settings in FreeBSD, nothing changed) Now I set "scrub no-df" in pf to permit fragmentation of this big packets but get another strange problem, which still prevent expected network performance, but this will wait until first problem is solved. I can provide tcpdumps from eth3, vif14.0, xn0 for investigation. Regards, Ivan Ivanyuk From owner-freebsd-xen@FreeBSD.ORG Thu Jan 12 18:43:32 2012 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDD54106566C for ; Thu, 12 Jan 2012 18:43:31 +0000 (UTC) (envelope-from ivan.ivanyuk@gmail.com) Received: from mail-tul01m020-f182.google.com (mail-tul01m020-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7ECD78FC23 for ; Thu, 12 Jan 2012 18:43:31 +0000 (UTC) Received: by obbta17 with SMTP id ta17so1677515obb.13 for ; Thu, 12 Jan 2012 10:43:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=vAzYXrsLz9cNOHPXYC8pE5sled+BlXgvsNwJgKoZMAw=; b=w6O3d6msIhfhQW4KRczmDrVQfgujMbNomN1GSOPZpthdd/BBjXscrRnSfVR3tAkgLt cPpXujGZkMxFeXUDn3/wt6+4ssjeFJLicWg4hI/Vhm5k0PhV+Abyn7oQ72/uYZrhngBB UxGTx0y6EeTtCtQD0ds3xIc+aBdcQcJcYTlBw= Received: by 10.50.222.193 with SMTP id qo1mr5478776igc.22.1326393810516; Thu, 12 Jan 2012 10:43:30 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.45.137 with HTTP; Thu, 12 Jan 2012 10:42:49 -0800 (PST) In-Reply-To: References: From: Ivan Ivanyuk Date: Thu, 12 Jan 2012 20:42:49 +0200 Message-ID: To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Re: [FreeBSD 8.2 amd64 XENHVM] Incorrect behavior of xen networking drivers: unexpected tcp reassembling X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2012 18:43:32 -0000 On 12 January 2012 00:35, Ivan Ivanyuk wrote: > Hi All, > I can get about 500Mb/s from virtual machine to external host, but > only 200Kb/s from any internal PC to the same external host through > virtual machine router. > > Closest description I found in archives is this: > http://lists.freebsd.org/pipermail/freebsd-xen/2011-May/000902.html. > > My setup is like this: > Dom0 is Debian with =A0xen-hypervisor-4.0-amd64 4.0.1-2. > DomU is FreeBSD 8.2-RELEASE amd64 XENHVM kernel with "device xenpci" opti= on > > > --------- =A0 =A0 =A0 --------------- =A0 =A0 =A0 -------------- =A0 =A0 = =A0 --------------- > |Internet|<--->|eth3 =A0 =A0vif14.0|<--->|xn0 =A0=A0 =A0 =A0 xn1|<--->|vi= f14.1 > eth2|<--->|some PC > ----------=A0 =A0 =A0| =A0 inetbr0 =A0 =A0 | =A0 =A0 | =A0 Freebsd =A0 = =A0| =A0=A0 =A0| =A0 localbr0 =A0 =A0| > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0| =A0 =A0 Dom0 =A0 =A0=A0 | =A0 =A0 | =A0 = =A0 DomU =A0=A0 =A0| =A0 =A0 |=A0 =A0 =A0Dom0 =A0 =A0 =A0| > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0--------------- =A0 =A0 =A0 --------------= =A0 =A0 =A0 --------------- > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 bridge =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0bridge > > So I can see couple of packets with TCP data from Internets coming to > eth3, then the same packets are seen on vif14.0. And then on xn0 I see > only one packet with reassembled TCP payload. > While these big (2976 bytes, 4464 bytes, 8928 bytes, etc) packets are > addressed to DomU - all works. When we try to route them elsewhere - > we get ICMP need fragmentation message sent to origin of these > packets. That's because original (small) TCP packets have DF flag set. > And resulting big TCP packet has DF flag as well. > > So it seems to me that something in the chain "vif14.0<->xn0" is > reassembling TCP packets. > > Is there some sysctl or other settings to control this behavior? (I > tried turning off all offloads on vif14.0 in Dom0, tried to change > fragmentation settings in FreeBSD, nothing changed) Somewhat successful follow-up: There is still some occurrences of reordered packets but original issue was resolved. Turned out this mechanism can be turned off by "ifconfig -lro" in FreeBSD itself. I'm not sure if turning LRO on virtual driver by default is good idea at all, but obviously LRO implementation for xn device doesn't consider DF flag implication on resulting packets. I'm not familiar with other LRO algorithms which used in physical NIC's. Can someone comment if such problem is normal for it? And what is your opinion on using tso, rxcsum and txcsum offloads in virtual envirnoment? Regards, Ivan Ivanyuk