From owner-freebsd-xen@FreeBSD.ORG Mon May 9 11:07:20 2011 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFC461065670 for ; Mon, 9 May 2011 11:07:20 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 84D648FC15 for ; Mon, 9 May 2011 11:07:20 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p49B7KnS070831 for ; Mon, 9 May 2011 11:07:20 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p49B7JcN070829 for freebsd-xen@FreeBSD.org; Mon, 9 May 2011 11:07:19 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 9 May 2011 11:07:19 GMT Message-Id: <201105091107.p49B7JcN070829@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-xen@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-xen@FreeBSD.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 May 2011 11:07:20 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/155468 xen [xen] Xen PV i386 multi-kernel CPU system is not worki o kern/155353 xen [xen] [patch] put "nudging TOD" message under boot_ver o kern/154833 xen [xen]: xen 4.0 - DomU freebsd8.2RC3 i386, XEN kernel. o kern/154473 xen [xen] xen 4.0 - DomU freebsd8.1 i386, XEN kernel. Not o kern/154472 xen [xen] xen 4.0 - DomU freebsd8.1 i386 xen kernel reboot o kern/154428 xen [xen] xn0 network interface and PF - Massive performan o kern/154302 xen [xen] [panic] [patch] xn0: Error 2 parsing device/vif/ o kern/153789 xen [xen] [regression] FreeBSD 8.2-RC1 crashes under Solar o kern/153674 xen [xen] i386/XEN idle thread shows wrong percentages o kern/153672 xen [xen] [panic] i386/XEN panics under heavy fork load o kern/153620 xen [xen] Xen guest system clock drifts in AWS EC2 (FreeBS o kern/153477 xen [xen] XEN pmap code abuses vm page queue lock o kern/153150 xen [xen] xen/ec2: disable checksum offloading on interfac o kern/152228 xen [xen] [panic] Xen/PV panic with machdep.idle_mwait=1 o kern/144629 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143398 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143340 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor f kern/143069 xen [xen] [panic] Xen Kernel Panic - Memory modified after f kern/135667 xen ufs filesystem corruption on XEN DomU system f kern/135421 xen [xen] FreeBSD Xen PVM DomU network failure - netfronc. f kern/135178 xen [xen] Xen domU outgoing data transfer stall when TSO i p kern/135069 xen [xen] FreeBSD-current/Xen SMP doesn't function at all f i386/124516 xen [xen] FreeBSD-CURRENT Xen Kernel Segfaults when config o kern/118734 xen [xen] FreeBSD 6.3-RC1 and FreeBSD 7.0-BETA 4 fail to b 24 problems total. From owner-freebsd-xen@FreeBSD.ORG Thu May 12 14:45:30 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B15021065673 for ; Thu, 12 May 2011 14:45:30 +0000 (UTC) (envelope-from laurent.cligny@steadinet.fr) Received: from smtp.steadinet.fr (steadinet.fr [88.191.77.166]) by mx1.freebsd.org (Postfix) with ESMTP id 461658FC13 for ; Thu, 12 May 2011 14:45:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.steadinet.fr (Postfix) with SMTP id 152ED1D4A78 for ; Thu, 12 May 2011 16:29:55 +0200 (CEST) Received: from [192.168.1.10] (ARennes-555-1-203-34.w2-10.abo.wanadoo.fr [2.10.234.34]) by smtp.steadinet.fr (Postfix) with ESMTPSA id 198FA1D4A6E for ; Thu, 12 May 2011 16:29:53 +0200 (CEST) Message-ID: <4DCBEEE0.9060705@steadinet.fr> Date: Thu, 12 May 2011 16:29:52 +0200 From: Laurent Cligny Organization: Steadinet User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110402 Iceowl/1.0b2 Icedove/3.1.9 MIME-Version: 1.0 To: freebsd-xen@freebsd.org Content-Type: multipart/mixed; boundary="------------000208000907060105050905" X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu May 12 16:29:55 2011 X-DSPAM-Confidence: 0.5000 X-DSPAM-Probability: 0.5000 X-DSPAM-Signature: 64,4dcbeee37249238915244 X-DSPAM-Factors: 27, the, 0.50000, the, 0.50000, a, 0.50000, a, 0.50000, and, 0.50000, and, 0.50000, with, 0.50000, with, 0.50000, is, 0.50000, is, 0.50000, I, 0.50000, I, 0.50000, to, 0.50000, to, 0.50000, FreeBSD, 0.50000, FreeBSD, 0.50000, on, 0.50000, on, 0.50000, NAT, 0.50000, NAT, 0.50000, gateway, 0.50000, gateway, 0.50000, DomU, 0.50000, DomU, 0.50000, NAT+gateway, 0.50000, NAT+gateway, 0.50000, in, 0.50000 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: laurent.cligny@steadinet.fr List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 14:45:30 -0000 This is a multi-part message in MIME format. --------------000208000907060105050905 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hello all, I want to setup a FreeBSD DomU VM as a network NAT gateway in order to provide Internet access to other FreeBSD and Linux DomU VMs. My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores, 16GiB RAM and one NIC. All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a Paravirtualized Debian amd64. The NAT gateway is a two-NIC box, with one (xn0) configured with a public Internet address and the other (xn1) configured with a class A private address (10.0.0.254). The Internet traffic is very good from my FreeBSD NAT gateway to the Internet (~ 50MiB/s) and also between other VMs on the private network (~ 70MiB/s trough scp) after applying the patch here (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html) fixing the "too many frags" problem. Here is a schema of my setup with the observed net speed on it: ---------- --------------- ---------------------- |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs| ---------- --------------- ---------------------- <-------> <-------> 50MiB/s 70MiB/s <-------------------------------> 0KiB/s --------------000208000907060105050905-- From owner-freebsd-xen@FreeBSD.ORG Thu May 12 15:17:51 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEC52106564A for ; Thu, 12 May 2011 15:17:51 +0000 (UTC) (envelope-from tobias@ntelecom.com.br) Received: from mail.ntelecom.com.br (mail2.ntelecom.com.br [189.1.176.233]) by mx1.freebsd.org (Postfix) with ESMTP id B06078FC08 for ; Thu, 12 May 2011 15:17:51 +0000 (UTC) Received: from [172.16.16.100] (mailgw.ntelecom.com.br [189.1.176.249]) by mail.ntelecom.com.br (Postfix) with ESMTPA id B798525886AC for ; Thu, 12 May 2011 12:02:25 -0300 (BRT) Message-ID: <4DCBF681.7070106@ntelecom.com.br> Date: Thu, 12 May 2011 12:02:25 -0300 From: "Tobias P. Santos" User-Agent: Thunderbird 2.0.0.18 (X11/20081105) MIME-Version: 1.0 To: freebsd-xen@freebsd.org References: <4DCBEEE0.9060705@steadinet.fr> In-Reply-To: <4DCBEEE0.9060705@steadinet.fr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97 at mail.ntelecom.com.br X-Virus-Status: Clean Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 15:17:52 -0000 Try this: ifconfig xn1 -txcsum Best regards, Tobias. Laurent Cligny wrote: > Hello all, > > I want to setup a FreeBSD DomU VM as a network NAT gateway in order to > provide Internet access to other FreeBSD and Linux DomU VMs. > My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores, > 16GiB RAM and one NIC. > > All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a > Paravirtualized Debian amd64. > > The NAT gateway is a two-NIC box, with one (xn0) configured with a > public Internet address and the other (xn1) configured with a class A > private address (10.0.0.254). > The Internet traffic is very good from my FreeBSD NAT gateway to the > Internet (~ 50MiB/s) and also between other VMs on the private network > (~ 70MiB/s trough scp) after applying the patch here > (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html) > fixing the "too many frags" problem. > > Here is a schema of my setup with the observed net speed on it: > > ---------- --------------- ---------------------- > |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs| > ---------- --------------- ---------------------- > <-------> <-------> > 50MiB/s 70MiB/s > > <-------------------------------> > 0KiB/s > [...] From owner-freebsd-xen@FreeBSD.ORG Thu May 12 15:19:28 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 770AD106566C for ; Thu, 12 May 2011 15:19:28 +0000 (UTC) (envelope-from gs@omegacube.fr) Received: from slow3-v.mail.gandi.net (slow3-v.mail.gandi.net [217.70.178.89]) by mx1.freebsd.org (Postfix) with ESMTP id B750A8FC13 for ; Thu, 12 May 2011 15:19:27 +0000 (UTC) X-WhiteListed: mail was accepted with no delay X-WhiteListed: mail was accepted with no delay Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by slow3-v.mail.gandi.net (Postfix) with ESMTP id E49C786274 for ; Thu, 12 May 2011 17:01:02 +0200 (CEST) X-Originating-IP: 217.70.178.131 Received: from mfilter11-d.gandi.net (mfilter11-d.gandi.net [217.70.178.131]) by relay3-d.mail.gandi.net (Postfix) with ESMTP id 1AE1FA80AE for ; Thu, 12 May 2011 17:00:51 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at undef13-d.mgt.gandi.net Received: from relay3-d.mail.gandi.net ([217.70.183.195]) by mfilter11-d.gandi.net (mfilter11-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id J9qCWg-qGFOC for ; Thu, 12 May 2011 17:00:48 +0200 (CEST) X-Originating-IP: 209.85.212.54 Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) (Authenticated sender: gs@omegacube.fr) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 5D69AA80B3 for ; Thu, 12 May 2011 17:00:48 +0200 (CEST) Received: by vws18 with SMTP id 18so1596849vws.13 for ; Thu, 12 May 2011 08:00:47 -0700 (PDT) Received: by 10.220.187.76 with SMTP id cv12mr89731vcb.128.1305212447196; Thu, 12 May 2011 08:00:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.203.137 with HTTP; Thu, 12 May 2011 08:00:27 -0700 (PDT) In-Reply-To: <4DCBEEE0.9060705@steadinet.fr> References: <4DCBEEE0.9060705@steadinet.fr> From: Guillaume Seigneuret Date: Thu, 12 May 2011 17:00:27 +0200 Message-ID: To: laurent.cligny@steadinet.fr Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-xen@freebsd.org Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 15:19:28 -0000 Maybe you should have a look on the tcp window size values. I had such a behavior with two gateways who was not playing the same way with the TCP window size which made my network damn slow. If you have a linux gateway in front of your FreeBSD VM, it should be this. Command Description sysctl net.inet.tcp.rfc1323=3D1 Activate window scaling and timestamp optio= ns according to RFC 1323.sysctl ipc.maxsockbuf=3D*[sbmax]* Set maximum size of TCP window.sysctl net.inet.tcp.recvspace=3D*[wstd]* Set default size of TCP receive window.sysctl net.inet.tcp.sendspace=3D*[wstd]* Set default size of TCP transmit window.sysctl kern.ipc.nmbclusters View maximum number of mbuf clusters. Used for storage of data packets to/from the network interface. Can only be set att boot time - see above. sysctl net.inet.tcp.liondmask=3D= 7Used to enable the "netlion" patch above. Cordialement, *Guillaume Seigneuret* *Network and System Security Architect* *Web* : http://www.omegacube.fr *Address* : H=F4tel Technologique Marseille Innovation - BP 100 45 rue Frederic Joliot Curie Technop=F4le de Ch=E2teau Gombert 13382 Marseille Cedex 13 2011/5/12 Laurent Cligny > Hello all, > > I want to setup a FreeBSD DomU VM as a network NAT gateway in order to > provide Internet access to other FreeBSD and Linux DomU VMs. > My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores, > 16GiB RAM and one NIC. > > All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a > Paravirtualized Debian amd64. > > The NAT gateway is a two-NIC box, with one (xn0) configured with a > public Internet address and the other (xn1) configured with a class A > private address (10.0.0.254). > The Internet traffic is very good from my FreeBSD NAT gateway to the > Internet (~ 50MiB/s) and also between other VMs on the private network > (~ 70MiB/s trough scp) after applying the patch here > (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html) > fixing the "too many frags" problem. > > Here is a schema of my setup with the observed net speed on it: > > ---------- --------------- ---------------------- > |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs| > ---------- --------------- ---------------------- > <-------> <-------> > 50MiB/s 70MiB/s > > <-------------------------------> > 0KiB/s > I tried doing NAT on the FBSD gateway with PF and also with NATd after > shutting down PF and unloaded the kernel modules pf.ko and pflog.ko, but > I have the same perf problems with both solutions. Also I tried > disabling net.inet.tcp.tso and did ifconfig -tso on all NICs on NAT > gateway and other VMs without change. > > I don't see anything on the error console or in /var/log/message. > > When i'm trying going though NAT with a Linux paravirt DomU, the network > perf is slightly better but don't go over 5KiB/s. > > Finally I put my Linux paravirt DomU VM as a NAT gateway in place of the > FreeBSD one, and the network perf is now normal through it (50MiB/s) > from the FreeBSD DomU VMs. > > As I'm very happy with FreeBSD for my environment I'd like to keep as > much as this lovely OS for all my needs, so if anyone have an idea where > the problem can lies, and what to test next to have a clue of what's > happening, I will be very thankfull. > > Thanks in advance. > > -- > /Regards,/ > > *Laurent Cligny > /Founder/* > 02 97 37 71 14 > 06 25 40 86 48 > laurent.cligny@steadinet.fr > > Steadinet > > > _______________________________________________ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" > > From owner-freebsd-xen@FreeBSD.ORG Thu May 12 15:48:43 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6BC4106566C for ; Thu, 12 May 2011 15:48:43 +0000 (UTC) (envelope-from snabb@epipe.com) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6CE0E8FC1C for ; Thu, 12 May 2011 15:48:43 +0000 (UTC) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by tiktik.epipe.com (8.14.4/8.14.4) with ESMTP id p4CFmgs4001569 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 12 May 2011 15:48:42 GMT (envelope-from snabb@epipe.com) X-DKIM: Sendmail DKIM Filter v2.8.3 tiktik.epipe.com p4CFmgs4001569 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=epipe.com; s=default; t=1305215322; x=1305820122; bh=kJ2I1JcZpMywrVGKHPTBsske48KOiuTXrmAsjT50L74=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=kD4jW9hVmevblyyiROK+BQmppv+hT2SNktQOw/4jFE5znuS4wzMvrutsU0h0e5c4U Ne+wVIkH5PHqbTDRYDMDl+H2m1eL1t393cnBGYVB/BLzYPPxPhdaIa4eZuLCekrA9T Fd5ilI47GJyQBUxMomKW6UR0xnwsFctzUdQZ1tjg= Date: Thu, 12 May 2011 15:48:42 +0000 (UTC) From: Janne Snabb To: Laurent Cligny In-Reply-To: <4DCBEEE0.9060705@steadinet.fr> Message-ID: References: <4DCBEEE0.9060705@steadinet.fr> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (tiktik.epipe.com [IPv6:2001:1828:0:3::2]); Thu, 12 May 2011 15:48:42 +0000 (UTC) Cc: freebsd-xen@freebsd.org Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 15:48:43 -0000 On Thu, 12 May 2011, Laurent Cligny wrote: > All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a > Paravirtualized Debian amd64. My suggestion would be to try out the same setup with GENERIC kernel and the rtl driver (or even better e1000 if your Xen allows it) which is easy to do to make a simple comparison. In one of my recent benchmarks the FreeBSD Xen PV network driver performed very well in one direction, but very badly in the other. Using rtl or e1000 gave much better TCP throughput if you care equally about both directions. This benchmark was done only for internal TCP traffic between a Linux dom0 and FreeBSD dumU, the traffic never entered a real NIC. This might or might not help in your situation... -- Janne Snabb / EPIPE Communications snabb@epipe.com - http://epipe.com/ From owner-freebsd-xen@FreeBSD.ORG Thu May 12 16:49:18 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3613C1065676 for ; Thu, 12 May 2011 16:49:18 +0000 (UTC) (envelope-from laurent.cligny@steadinet.fr) Received: from smtp.steadinet.fr (steadinet.fr [88.191.77.166]) by mx1.freebsd.org (Postfix) with ESMTP id 9D0EB8FC08 for ; Thu, 12 May 2011 16:49:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.steadinet.fr (Postfix) with SMTP id 7EF131D4A78 for ; Thu, 12 May 2011 18:49:16 +0200 (CEST) Received: from [192.168.1.10] (ARennes-555-1-203-34.w2-10.abo.wanadoo.fr [2.10.234.34]) by smtp.steadinet.fr (Postfix) with ESMTPSA id EE9A81D4A6E for ; Thu, 12 May 2011 18:49:13 +0200 (CEST) Message-ID: <4DCC0F89.2060707@steadinet.fr> Date: Thu, 12 May 2011 18:49:13 +0200 From: Laurent Cligny Organization: Steadinet User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110402 Iceowl/1.0b2 Icedove/3.1.9 MIME-Version: 1.0 To: freebsd-xen@freebsd.org References: <4DCBEEE0.9060705@steadinet.fr> In-Reply-To: Content-Type: multipart/mixed; boundary="------------060905080406010700050607" X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu May 12 18:49:16 2011 X-DSPAM-Confidence: 0.5132 X-DSPAM-Probability: 0.4868 X-DSPAM-Signature: 64,4dcc0f8c7243218941358 X-DSPAM-Factors: 27, net+inet, 0.49951, net+inet, 0.49951, inet+tcp, 0.49951, inet+tcp, 0.49951, on+the, 0.49951, on+the, 0.49951, is+a, 0.49951, is+a, 0.49951, NAT+gateway, 0.49951, NAT+gateway, 0.49951, the+network, 0.49951, the+network, 0.49951, <+>, 0.49951, <+>, 0.49951, Laurent+Cligny, 0.49951, Laurent+Cligny, 0.49951, gateway+in, 0.49951, gateway+in, 0.49951, and+the, 0.49951, and+the, 0.49951, cligny+steadinet, 0.49951, cligny+steadinet, 0.49951, have+a, 0.49951, have+a, 0.49951, with+a, 0.49951, with+a, 0.49951, the+Linux, 0.49951 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: laurent.cligny@steadinet.fr List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 16:49:18 -0000 This is a multi-part message in MIME format. --------------060905080406010700050607 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks for your quick reply. > Maybe you should have a look on the tcp window size values. > I had such a behavior with two gateways who was not playing the same wa= y > with the TCP window size which made my network damn slow. > If you have a linux gateway in front of your FreeBSD VM, it should be t= his. Well I compared the TCP default values between the FreeBSD Vm and the Linux PV VM and changed the net.inet.tcp.recvspace and net.inet.tcp.sendpace to 124928 wich are the values on the Linux VM, but there's no change on the network performance. Nice idea, much appreciated= . > Command > Description > sysctl net.inet.tcp.rfc1323=3D1 Activate window scaling and timestamp o= ptions > according to RFC 1323.sysctl ipc.maxsockbuf=3D*[sbmax]* Set maximum siz= e of > TCP window.sysctl net.inet.tcp.recvspace=3D*[wstd]* Set default size of= TCP > receive window.sysctl net.inet.tcp.sendspace=3D*[wstd]* Set default siz= e of > TCP transmit window.sysctl kern.ipc.nmbclusters View maximum number of = mbuf > clusters. Used for storage of data packets to/from the network interfac= e. > Can only be set att boot time - see above. sysctl net.inet.tcp.liondmas= k=3D7Used > to enable the "netlion" patch above. > > Cordialement, > *Guillaume Seigneuret* > > > *Network and System Security Architect* > *Web* : http://www.omegacube.fr > *Address* : > H=F4tel Technologique Marseille Innovation - BP 100 > 45 rue Frederic Joliot Curie > Technop=F4le de Ch=E2teau Gombert > 13382 Marseille Cedex 13 > > > > 2011/5/12 Laurent Cligny > >> Hello all, >> >> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to >> provide Internet access to other FreeBSD and Linux DomU VMs. >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores= , >> 16GiB RAM and one NIC. >> >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a >> Paravirtualized Debian amd64. >> >> The NAT gateway is a two-NIC box, with one (xn0) configured with a >> public Internet address and the other (xn1) configured with a class A >> private address (10.0.0.254). >> The Internet traffic is very good from my FreeBSD NAT gateway to the >> Internet (~ 50MiB/s) and also between other VMs on the private network >> (~ 70MiB/s trough scp) after applying the patch here >> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html) >> fixing the "too many frags" problem. >> >> Here is a schema of my setup with the observed net speed on it: >> >> ---------- --------------- ---------------------- >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs| >> ---------- --------------- ---------------------- >> <-------> <-------> >> 50MiB/s 70MiB/s >> >> <-------------------------------> >> 0KiB/s > >> I tried doing NAT on the FBSD gateway with PF and also with NATd after >> shutting down PF and unloaded the kernel modules pf.ko and pflog.ko, b= ut >> I have the same perf problems with both solutions. Also I tried >> disabling net.inet.tcp.tso and did ifconfig -tso on all NICs on NAT >> gateway and other VMs without change. >> >> I don't see anything on the error console or in /var/log/message. >> >> When i'm trying going though NAT with a Linux paravirt DomU, the netwo= rk >> perf is slightly better but don't go over 5KiB/s. >> >> Finally I put my Linux paravirt DomU VM as a NAT gateway in place of t= he >> FreeBSD one, and the network perf is now normal through it (50MiB/s) >> from the FreeBSD DomU VMs. >> >> As I'm very happy with FreeBSD for my environment I'd like to keep as >> much as this lovely OS for all my needs, so if anyone have an idea whe= re >> the problem can lies, and what to test next to have a clue of what's >> happening, I will be very thankfull. >> >> Thanks in advance. >> >> -- >> /Regards,/ >> >> *Laurent Cligny >> /Founder/* >> 02 97 37 71 14 >> 06 25 40 86 48 >> laurent.cligny@steadinet.fr >> >> Steadinet >> >> >> _______________________________________________ >> freebsd-xen@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-xen >> To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" >> >> > _______________________________________________ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" --------------060905080406010700050607-- From owner-freebsd-xen@FreeBSD.ORG Thu May 12 17:22:01 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A6E3106566B for ; Thu, 12 May 2011 17:22:01 +0000 (UTC) (envelope-from laurent.cligny@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id E69D98FC0A for ; Thu, 12 May 2011 17:22:00 +0000 (UTC) Received: by wwc33 with SMTP id 33so1973627wwc.31 for ; Thu, 12 May 2011 10:21:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=Sz1jq0k1mh4IkGQmvt0NoM2YUfzZ88/Eiupmq98q4zQ=; b=XiyQGD9noOS7nJ7wlZkm4vOr4RymEdW1ihqTnEUbTkSKeJb7WY+SHY9hnux9LyshW+ cqomVtBS56IUD2T35Av3CbSnoBUBQhCzX342c34EZagvOxFw/4KXv0h/R1/eFfM4blOS KHB3ZM7rvl737UWzA/tmy2SAY4C9hhAs1yyxU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=a3lN+UOtSBnnxzR+GLqcbNboYseeo4kG6AmteGTBkUdTAqs0Mn2CSRQ392vnIw4ja6 5Ll24OF3VmDJM0MmkdzrsLt8ca7p4nMa/yElqhGwOKuM9+6J2E5y9SyI9SvZrvl5Ennf niD1rW9H+91H8FaUzxCxtHUlEXVYkPBibuRn4= Received: by 10.227.11.148 with SMTP id t20mr416553wbt.98.1305219226104; Thu, 12 May 2011 09:53:46 -0700 (PDT) Received: from [192.168.1.10] (ARennes-555-1-203-34.w2-10.abo.wanadoo.fr [2.10.234.34]) by mx.google.com with ESMTPS id ed10sm852472wbb.66.2011.05.12.09.53.44 (version=SSLv3 cipher=OTHER); Thu, 12 May 2011 09:53:45 -0700 (PDT) Message-ID: <4DCC1097.4080003@gmail.com> Date: Thu, 12 May 2011 18:53:43 +0200 From: "laurent.cligny@gmail.com" User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110402 Iceowl/1.0b2 Icedove/3.1.9 MIME-Version: 1.0 To: freebsd-xen@freebsd.org References: <4DCBEEE0.9060705@steadinet.fr> <4DCBF681.7070106@ntelecom.com.br> In-Reply-To: <4DCBF681.7070106@ntelecom.com.br> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 17:22:01 -0000 Le 12/05/2011 17:02, Tobias P. Santos a écrit : > Try this: > > ifconfig xn1 -txcsum > > Best regards, > Tobias. Hello Tobias, Unfortunately ifconfig xn1 -txcsum didn't give me lot of result. Doing this my network perf through NAT is about 7KiB/s (that is better than before, but still unworkable). Thanks for the idea, maybe we are not that far away from the solution. > > > Laurent Cligny wrote: >> Hello all, >> >> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to >> provide Internet access to other FreeBSD and Linux DomU VMs. >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores, >> 16GiB RAM and one NIC. >> >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a >> Paravirtualized Debian amd64. >> >> The NAT gateway is a two-NIC box, with one (xn0) configured with a >> public Internet address and the other (xn1) configured with a class A >> private address (10.0.0.254). >> The Internet traffic is very good from my FreeBSD NAT gateway to the >> Internet (~ 50MiB/s) and also between other VMs on the private network >> (~ 70MiB/s trough scp) after applying the patch here >> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html) >> fixing the "too many frags" problem. >> >> Here is a schema of my setup with the observed net speed on it: >> >> ---------- --------------- ---------------------- >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs| >> ---------- --------------- ---------------------- >> <-------> <-------> >> 50MiB/s 70MiB/s >> >> <-------------------------------> >> 0KiB/s > >> [...] > _______________________________________________ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" From owner-freebsd-xen@FreeBSD.ORG Thu May 12 17:59:53 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D4401065676 for ; Thu, 12 May 2011 17:59:53 +0000 (UTC) (envelope-from mikemacleod@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id EB25D8FC19 for ; Thu, 12 May 2011 17:59:52 +0000 (UTC) Received: by yxl31 with SMTP id 31so786852yxl.13 for ; Thu, 12 May 2011 10:59:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=EdUz331IdO4e8Zl57arzjCeaukJmAYoYRlF+lt8JDa0=; b=Xl6VMcS51P4Md03lCf6P2GYl2tq7qQbztZTvXd9ShmDLzjqghqRqrm4eNZqTpxhAMK e3EJWjhHFz8gMgaPNX0/wZjehQP9hfepFUnWWpN8uM+QZZBiqxSr5SM9YBJQdT6lL77I 30yitibrlP+qaVX0JAsa82cEHqsNe2/PjqE+I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=W5gygFz0IlVINogQcv7Q2LS0ar10PiciMILqdulMkC0Ag+2TbEXEHkzWyLFHDqzO0z MXb4MrPbMZbhuCb2AMt4nFA3pqWeMcGqWow0YJ1bXTdNLj2r2GTFJc9tBmUVNWOI2IHN /2S/b3kSN+K3aw/QmroHxrqUI0K0aUI9qGGo4= Received: by 10.91.35.33 with SMTP id n33mr416910agj.126.1305221820121; Thu, 12 May 2011 10:37:00 -0700 (PDT) MIME-Version: 1.0 Received: by 10.90.93.18 with HTTP; Thu, 12 May 2011 10:36:40 -0700 (PDT) In-Reply-To: <4DCC1097.4080003@gmail.com> References: <4DCBEEE0.9060705@steadinet.fr> <4DCBF681.7070106@ntelecom.com.br> <4DCC1097.4080003@gmail.com> From: Michael MacLeod Date: Thu, 12 May 2011 13:36:40 -0400 Message-ID: To: "laurent.cligny@gmail.com" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-xen@freebsd.org Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 17:59:53 -0000 Try -rxcsum as well. I have both disabled on my domU. On Thu, May 12, 2011 at 12:53 PM, laurent.cligny@gmail.com < laurent.cligny@gmail.com> wrote: > Le 12/05/2011 17:02, Tobias P. Santos a =E9crit : > > Try this: > > > > ifconfig xn1 -txcsum > > > > Best regards, > > Tobias. > Hello Tobias, > > Unfortunately ifconfig xn1 -txcsum didn't give me lot of result. > Doing this my network perf through NAT is about 7KiB/s (that is better > than before, but still unworkable). > > Thanks for the idea, maybe we are not that far away from the solution. > > > > > > > Laurent Cligny wrote: > >> Hello all, > >> > >> I want to setup a FreeBSD DomU VM as a network NAT gateway in order to > >> provide Internet access to other FreeBSD and Linux DomU VMs. > >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores= , > >> 16GiB RAM and one NIC. > >> > >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a > >> Paravirtualized Debian amd64. > >> > >> The NAT gateway is a two-NIC box, with one (xn0) configured with a > >> public Internet address and the other (xn1) configured with a class A > >> private address (10.0.0.254). > >> The Internet traffic is very good from my FreeBSD NAT gateway to the > >> Internet (~ 50MiB/s) and also between other VMs on the private network > >> (~ 70MiB/s trough scp) after applying the patch here > >> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html) > >> fixing the "too many frags" problem. > >> > >> Here is a schema of my setup with the observed net speed on it: > >> > >> ---------- --------------- ---------------------- > >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs| > >> ---------- --------------- ---------------------- > >> <-------> <-------> > >> 50MiB/s 70MiB/s > >> > >> <-------------------------------> > >> 0KiB/s >> > >> [...] > > _______________________________________________ > > freebsd-xen@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" > From owner-freebsd-xen@FreeBSD.ORG Thu May 12 18:20:52 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8D5F106564A for ; Thu, 12 May 2011 18:20:52 +0000 (UTC) (envelope-from gs@omegacube.fr) Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by mx1.freebsd.org (Postfix) with ESMTP id 488148FC12 for ; Thu, 12 May 2011 18:20:52 +0000 (UTC) X-Originating-IP: 217.70.178.137 Received: from mfilter8-d.gandi.net (mfilter8-d.gandi.net [217.70.178.137]) by relay4-d.mail.gandi.net (Postfix) with ESMTP id AFE8B172081 for ; Thu, 12 May 2011 20:20:40 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter8-d.gandi.net Received: from relay4-d.mail.gandi.net ([217.70.183.196]) by mfilter8-d.gandi.net (mfilter8-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id WrpRNFtFwpqU for ; Thu, 12 May 2011 20:20:38 +0200 (CEST) X-Originating-IP: 209.85.220.182 Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) (Authenticated sender: gs@omegacube.fr) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 65C45172095 for ; Thu, 12 May 2011 20:20:38 +0200 (CEST) Received: by vxc34 with SMTP id 34so1819818vxc.13 for ; Thu, 12 May 2011 11:20:37 -0700 (PDT) Received: by 10.220.193.69 with SMTP id dt5mr152150vcb.93.1305224437209; Thu, 12 May 2011 11:20:37 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.203.137 with HTTP; Thu, 12 May 2011 11:20:17 -0700 (PDT) In-Reply-To: References: <4DCBEEE0.9060705@steadinet.fr> <4DCBF681.7070106@ntelecom.com.br> <4DCC1097.4080003@gmail.com> From: Guillaume Seigneuret Date: Thu, 12 May 2011 20:20:17 +0200 Message-ID: To: Michael MacLeod Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-xen@freebsd.org Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 18:20:53 -0000 I think you should try totally disable the window sizing both parts to isolate it doesn't come from here : On your FreeBSD VM : sysctl net.inet.tcp.rfc1323=3D0 On your Linux box : sysctl net.ipv4.tcp_window_scaling=3D0 Then if the rate is hugely better, try some settings to increase the compatibility. You can also try an *iperf *test with TCP and UDP packets to see if the behavior is generalized, and see what latency you have both part of the network. Cordialement, *Guillaume Seigneuret* *Network and System Security Architect Mobile : +33.6.15.37.35.67 Web : http://www.omegacube.fr Address :* H=F4tel Technologique Marseille Innovation - BP 100 45 rue Frederic Joliot Curie Technop=F4le de Ch=E2teau Gombert 13382 Marseille Cedex 13 2011/5/12 Michael MacLeod > Try -rxcsum as well. I have both disabled on my domU. > > On Thu, May 12, 2011 at 12:53 PM, laurent.cligny@gmail.com < > laurent.cligny@gmail.com> wrote: > > > Le 12/05/2011 17:02, Tobias P. Santos a =E9crit : > > > Try this: > > > > > > ifconfig xn1 -txcsum > > > > > > Best regards, > > > Tobias. > > Hello Tobias, > > > > Unfortunately ifconfig xn1 -txcsum didn't give me lot of result. > > Doing this my network perf through NAT is about 7KiB/s (that is better > > than before, but still unworkable). > > > > Thanks for the idea, maybe we are not that far away from the solution. > > > > > > > > > > > Laurent Cligny wrote: > > >> Hello all, > > >> > > >> I want to setup a FreeBSD DomU VM as a network NAT gateway in order = to > > >> provide Internet access to other FreeBSD and Linux DomU VMs. > > >> My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 > cores, > > >> 16GiB RAM and one NIC. > > >> > > >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is = a > > >> Paravirtualized Debian amd64. > > >> > > >> The NAT gateway is a two-NIC box, with one (xn0) configured with a > > >> public Internet address and the other (xn1) configured with a class = A > > >> private address (10.0.0.254). > > >> The Internet traffic is very good from my FreeBSD NAT gateway to the > > >> Internet (~ 50MiB/s) and also between other VMs on the private netwo= rk > > >> (~ 70MiB/s trough scp) after applying the patch here > > >> (http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html) > > >> fixing the "too many frags" problem. > > >> > > >> Here is a schema of my setup with the observed net speed on it: > > >> > > >> ---------- --------------- ---------------------- > > >> |Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs| > > >> ---------- --------------- ---------------------- > > >> <-------> <-------> > > >> 50MiB/s 70MiB/s > > >> > > >> <-------------------------------> > > >> 0KiB/s > >> > > >> [...] > > > _______________________________________________ > > > freebsd-xen@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > > > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org= " > > > > _______________________________________________ > > freebsd-xen@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" > From owner-freebsd-xen@FreeBSD.ORG Thu May 12 18:22:31 2011 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1273F106566C for ; Thu, 12 May 2011 18:22:31 +0000 (UTC) (envelope-from tobias@ntelecom.com.br) Received: from mail.ntelecom.com.br (mail2.ntelecom.com.br [189.1.176.233]) by mx1.freebsd.org (Postfix) with ESMTP id C86808FC17 for ; Thu, 12 May 2011 18:22:30 +0000 (UTC) Received: from [172.16.16.100] (mailgw.ntelecom.com.br [189.1.176.249]) by mail.ntelecom.com.br (Postfix) with ESMTPA id A54E225887D9 for ; Thu, 12 May 2011 15:22:28 -0300 (BRT) Message-ID: <4DCC2564.8090002@ntelecom.com.br> Date: Thu, 12 May 2011 15:22:28 -0300 From: "Tobias P. Santos" User-Agent: Thunderbird 2.0.0.18 (X11/20081105) MIME-Version: 1.0 To: freebsd-xen@freebsd.org References: <4DCBEEE0.9060705@steadinet.fr> <4DCBF681.7070106@ntelecom.com.br> <4DCC1097.4080003@gmail.com> In-Reply-To: <4DCC1097.4080003@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.97 at mail.ntelecom.com.br X-Virus-Status: Clean Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 18:22:31 -0000 laurent.cligny@gmail.com wrote: > Le 12/05/2011 17:02, Tobias P. Santos a écrit : > >> Try this: >> >> ifconfig xn1 -txcsum >> >> Best regards, >> Tobias. >> > Hello Tobias, > > Unfortunately ifconfig xn1 -txcsum didn't give me lot of result. > Doing this my network perf through NAT is about 7KiB/s (that is better > than before, but still unworkable). > > Thanks for the idea, maybe we are not that far away from the solution. > > We had a problem where traffic between VMs was too slow (about 5Mbps on a Gigabit network) and disabling TX checksum solved the problem (actually, speed increased to more than 1Gbps). Your problem is different as you have a nice throughput between VMs on xn1, sorry that I missed that on my first message. Did you try to transfer data from something connected to xn0 to one of yours DomU servers connected to xn1 without NAT? You also mentioned pf. Did you make any test using natd or anything else? Something tells me that's not a nic problem, but that's just a guess. Anyway, on the coming weeks I will have to set up a similar environment to yours, so maybe I'll face the same problem (although we use natd daemon instead of pf). Good luck! From owner-freebsd-xen@FreeBSD.ORG Thu May 12 20:34:30 2011 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B519106564A for ; Thu, 12 May 2011 20:34:30 +0000 (UTC) (envelope-from gibbs@FreeBSD.org) Received: from aslan.scsiguy.com (mail.scsiguy.com [70.89.174.89]) by mx1.freebsd.org (Postfix) with ESMTP id 3CFE28FC1B for ; Thu, 12 May 2011 20:34:30 +0000 (UTC) Received: from Justins-MacBook-Pro.local ([192.75.139.252]) (authenticated bits=0) by aslan.scsiguy.com (8.14.4/8.14.4) with ESMTP id p4CKEYji047549 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 12 May 2011 14:14:35 -0600 (MDT) (envelope-from gibbs@FreeBSD.org) Message-ID: <4DCC3FAA.9030608@FreeBSD.org> Date: Thu, 12 May 2011 16:14:34 -0400 From: "Justin T. Gibbs" Organization: The FreeBSD Project User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10 MIME-Version: 1.0 To: Janne Snabb References: <4DCBEEE0.9060705@steadinet.fr> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (aslan.scsiguy.com [70.89.174.89]); Thu, 12 May 2011 14:14:35 -0600 (MDT) Cc: freebsd-xen@FreeBSD.org, Laurent Cligny Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gibbs@FreeBSD.org List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2011 20:34:30 -0000 On 5/12/11 11:48 AM, Janne Snabb wrote: > On Thu, 12 May 2011, Laurent Cligny wrote: > >> All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a >> Paravirtualized Debian amd64. > My suggestion would be to try out the same setup with GENERIC kernel > and the rtl driver (or even better e1000 if your Xen allows it) > which is easy to do to make a simple comparison. > > In one of my recent benchmarks the FreeBSD Xen PV network driver > performed very well in one direction, but very badly in the other. > Using rtl or e1000 gave much better TCP throughput if you care > equally about both directions. This benchmark was done only for > internal TCP traffic between a Linux dom0 and FreeBSD dumU, the > traffic never entered a real NIC. Do you recall which path was slow (rx or tx from the perspective of the FreeBSD driver) and what the relative difference in performance was between the two approaches? -- Justin From owner-freebsd-xen@FreeBSD.ORG Fri May 13 05:55:20 2011 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE81C1065674; Fri, 13 May 2011 05:55:20 +0000 (UTC) (envelope-from snabb@epipe.com) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by mx1.freebsd.org (Postfix) with ESMTP id B0B8B8FC13; Fri, 13 May 2011 05:55:20 +0000 (UTC) Received: from tiktik.epipe.com (tiktik.epipe.com [IPv6:2001:1828:0:3::2]) by tiktik.epipe.com (8.14.4/8.14.4) with ESMTP id p4D5tJPx083515 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 13 May 2011 05:55:19 GMT (envelope-from snabb@epipe.com) X-DKIM: Sendmail DKIM Filter v2.8.3 tiktik.epipe.com p4D5tJPx083515 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=epipe.com; s=default; t=1305266120; x=1305870920; bh=P5w76SbKwrWhOpXY3KFza5tSs6qvhBb6uB/K1bhvSTg=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=PGrspSAybnwzR89IulgPNlntVDZWnbCyW52mVj7+U47mcnwESq2WapMLCvGUWItdJ vzQma5YgF9CoWG5A6Ul9vUXZTNgNlMjrAJPje9gzXqeTFoRm6YeXPuuHA++EQtHm74 +KPsLqL5j2jpZogui6oM92Y76Uc5zQjSIpwcyu98= Date: Fri, 13 May 2011 05:55:19 +0000 (UTC) From: Janne Snabb To: "Justin T. Gibbs" In-Reply-To: <4DCC3FAA.9030608@FreeBSD.org> Message-ID: References: <4DCBEEE0.9060705@steadinet.fr> <4DCC3FAA.9030608@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (tiktik.epipe.com [IPv6:2001:1828:0:3::2]); Fri, 13 May 2011 05:55:20 +0000 (UTC) Cc: freebsd-xen@FreeBSD.org, Laurent Cligny Subject: Re: [FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 May 2011 05:55:21 -0000 On Thu, 12 May 2011, Justin T. Gibbs wrote: > Do you recall which path was slow (rx or tx from the perspective of > the FreeBSD driver) and what the relative difference in performance > was between the two approaches? No, which is really stupid (that is why I haven't published these numbers before). The numbers are as follows, for simple iperf TCP test, with all networking, kernel, etc. settings on default values: input (Mbit/s) output (Mbit/s) Xen FreeBSD (rtl) 724 200 Xen FreeBSD (xn) 44 1700 Xen Linux (xn) 8570 2340 KVM FreeBSD (e1000) 475 495 KVM FreeBSD (rtl) 1100 85 KVM Linux (e1000) 785 890 KVM Linux (virtio) 585 715 dom0 <-> dom0 18500 18500 The table above will probably not render correctly, but hopefully it is somewhat readable. The first column indicates the hypervisor used, the guest OS and the network driver in the guest. dom0 was running Debian's 2.6.32-5-amd64 Linux kernel. Xen was Debian's 4.0.1. Linux guests were running Debian's 2.6.32-5-amd64. FreeBSD was 8.2 amd64 with the "too many frags" and "panic: do something smart" patches to make it usable with Xen at all. I did not test multiple concurrent connections, multiple virtual machines transferring data simultaneously, mixtures of different kinds of data, CPU load nor anything like that which is also relevant. Only the raw TCP speed was measured to figure out if there is any significant difference: and yes there was as can be seen from the numbers above. Unfortunately I have no recollection and forgot to write down if input was input from dom0's perspective or domU's perspective. They should be the same way around for all the tests though. My lesson was that it does make sense to select your network drivers carefully in a virtualized environment. I also tried two different virtio patches for FreeBSD with KVM, but one of them did not work at all and the another one gave worse results than any sensible emulated hardware. -- Janne Snabb / EPIPE Communications snabb@epipe.com - http://epipe.com/