From owner-freebsd-stable@FreeBSD.ORG Tue Aug 27 04:26:11 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 082D7B5F; Tue, 27 Aug 2013 04:26:11 +0000 (UTC) (envelope-from bryanv@daemoninthecloset.org) Received: from torment.daemoninthecloset.org (torment.daemoninthecloset.org [94.242.209.234]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8E07F28B4; Tue, 27 Aug 2013 04:26:10 +0000 (UTC) Received: from sage.daemoninthecloset.org (unknown [70.114.209.60]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "sage.daemoninthecloset.org", Issuer "daemoninthecloset.org" (verified OK)) by torment.daemoninthecloset.org (Postfix) with ESMTPS id 9865C42C2632; Tue, 27 Aug 2013 06:24:02 +0200 (CEST) X-Virus-Scanned: amavisd-new at daemoninthecloset.org X-Virus-Scanned: amavisd-new at daemoninthecloset.org Date: Mon, 26 Aug 2013 23:18:37 -0500 (CDT) From: Bryan Venteicher To: Harald Schmalzbauer Message-ID: <1117188271.19176.1377577117350.JavaMail.root@daemoninthecloset.org> In-Reply-To: <5214D37F.5000307@omnilan.de> References: <601099152.721.1375661537866.JavaMail.root@daemoninthecloset.org> <5214D37F.5000307@omnilan.de> Subject: Re: [CFT] VMware vmxnet3 ethernet driver MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [192.168.10.20] X-Mailer: Zimbra 8.0.2_GA_5569 (ZimbraWebClient - GC20 ([unknown])/8.0.2_GA_5569) Thread-Topic: VMware vmxnet3 ethernet driver Thread-Index: wVc2YFjrnum8ylaZxy8pLzcODD7JCg== Cc: FreeBSD Stable , current@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Aug 2013 04:26:11 -0000 ----- Original Message ----- > Bez=C3=BCglich Bryan Venteicher's Nachricht vom 05.08.2013 02:12 (localti= me): > > Hi, > > > > I've ported the OpenBSD vmxnet3 ethernet driver to FreeBSD. I did a > > lot of cleanup, bug fixes, new features, etc (+2000 new lines) along > > the way so there is not much of a resemblance left. > > > > The driver is in good enough shape I'd like additional testers. A patch > > against -CURRENT is at [1]. Alternatively, the driver and a Makefile is > > at [2]; this should compile at least as far back as 9.1. I can look at > > 8-STABLE if there is interest. > > > > Obviously, besides reports of 'it works', I'm interested performance vs > > the emulated e1000, and (for those using it) the VMware tools vmxnet3 > > driver. Hopefully it is no worse :) >=20 > Hello Bryan, >=20 > thanks a lot for your hard work! >=20 > It seems if_vmx doesn't support jumbo frames. If I set mtu 9000, I get > =C2=BBvmx0: cannot populate Rx queue 0=C2=AB, I have no problems using ju= mbo > frames with vmxnet3. >=20 This could fail for two reasons - could not allocate an mbuf cluster, or the call to bus_dmamap_load_mbuf_sg() failed. For the former, you should check vmstat -z. For the later, the behavior of bus_dmamap_load_mbuf= _sg() changed between 9.1 and 9.2, and I know it was broken for awhile. I don't recall exactly when I fixed it (I think shortly after I made the original announcement). Could you retry with the files from HEAD @ [1]? Also, there are new sysctl oids (dev.vmx.X.mbuf_load_failed & dev.vmx.X.mgetcl_failed) for these errors. I just compiled the driver on 9.2-RC2 with the sources from HEAD and was able to change the MTU to 9000. [1]- http://svnweb.freebsd.org/base/head/sys/dev/vmware/vmxnet3/ > I took a oldish host (4x2,8GHz Core2[LGA775]) with recent software: ESXi > 5.1U1 and FreeBSD-9.2-RC2 > Two guests are connected to one MTU9000 "VMware Software Switch". >=20 I've got a few performance things to still look at. What's the sysctl=20 dev.vmx.X output for the if_vmx<->if_vmx tests? > Simple iperf (standard TCP) results: >=20 > vmxnet3jumbo <-> vmxnet3jumbo > 5.3Gbits/sec, load: 40-60%Sys 0.5-2%Intr >=20 > vmxnet3 <-> vmxnet3 > 1.85 GBits/sec, load: 60-80%Sys 0-0.8%Intr >=20 >=20 > if_vmx <-> if_vmx > 1.51 GBits/sec, load: 10-45%Sys 40-48%Intr > !!! > if_vmxjumbo <-> if_vmxjumbo not possible >=20 >=20 > if_em(e1000) <-> if_em(e1000) > 1.23 GBits/sec, load: 80-60%Sys 0.5-8%Intr >=20 > if_em(e1000)jumbo <-> if_em(e1000)jumbo > 2.27Gbits/sec, load: 40-30%Sys 0.5-5%Intr >=20 >=20 > if_igb(e1000e)junmbo <-> if_igb(e1000e)jumbo > 5.03 Gbits/s, load: 70-60%Sys 0.5%Intr >=20 > if_igb(e1000e) <-> if_igb(e1000e) > 1.39 Gbits/s, load: 60-80%Sys 0.5%Intr >=20 >=20 > f_igb(e1000e) <-> if_igb(e1000e), both hw.em.[rt]xd=3D4096 > 1.66 Gbits/s, load: 65-90%Sys 0.5%Intr >=20 > if_igb(e1000e)junmbo <-> if_igb(e1000e)jumbo, both hw.em.[rt]xd=3D4096 > 4.81 Gbits/s, load: 65%Sys 0.5%Intr >=20 > Conclusion: > if_vmx performs well compared to the regular emulated nics and standard > MTU, but it's behind tuned e1000e nic emulation and can't reach vmxnet3 > performance with regular mtu. If one needs throughput, the missing jumbo > frame support in if_vmx is a show stopper. >=20 > e1000e is preferable over e1000, even if not officially choosable with > "FreeBSD"-selection as guest (edit .vmx and alter ethernet0.virtualDev = =3D > "e1000e", and dont forget to set hw.em.enable_msix=3D0 in loader.conf, > although the driver e1000e attaches is if_igb!) >=20 > Thanks, >=20 > -Harry >=20 >