From owner-freebsd-net@freebsd.org Mon Dec 21 15:57:08 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B100DA4E8F2 for ; Mon, 21 Dec 2015 15:57:08 +0000 (UTC) (envelope-from pradosh.datta@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 973101E87 for ; Mon, 21 Dec 2015 15:57:08 +0000 (UTC) (envelope-from pradosh.datta@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 94780A4E8F1; Mon, 21 Dec 2015 15:57:08 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A1D3A4E8F0 for ; Mon, 21 Dec 2015 15:57:08 +0000 (UTC) (envelope-from pradosh.datta@gmail.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44CFA1E86 for ; Mon, 21 Dec 2015 15:57:08 +0000 (UTC) (envelope-from pradosh.datta@gmail.com) Received: by mail-oi0-x234.google.com with SMTP id o124so93786949oia.1 for ; Mon, 21 Dec 2015 07:57:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=v3uRldngIPOz7sxAEsTM9kWIwhVFPRH/rHRnY45KjuM=; b=GhPG1ncZXWAjKt6D/GHMII/n6X/TRLFnQ4gR0AIzt0CbxrUajB9F9itoTKDB/vJe6c LUqUWqKpiGGrMCzkAEgum+3hY/BrCJLG/ZYnm/sR/+hBUYZZy1r6f4a+5vljhyCCc3ET vMo/2xxcCqHacFIl0ck4vrD5v9t2jHWwkbLVytLFtAzP9BPw21n3ZtaokQgwHecxUE9m we+7MwjTGbHNDdy/5+nC28jjH3vF6s5/coavRCKjmpxHNgtkinVilcpAGEv2qZhyjK1K eWHVxTTsKFnJF+nlSfdiGpnY1TYkwrUpzvW90O2sWq4uqCDHK7T25LsMaLI/vWsTtHlo 9rlg== MIME-Version: 1.0 X-Received: by 10.202.178.135 with SMTP id b129mr6646106oif.86.1450713427404; Mon, 21 Dec 2015 07:57:07 -0800 (PST) Received: by 10.76.177.162 with HTTP; Mon, 21 Dec 2015 07:57:07 -0800 (PST) In-Reply-To: References: Date: Mon, 21 Dec 2015 21:27:07 +0530 Message-ID: Subject: Re: Netmap with VMWare From: Pradosh Datta To: Luigi Rizzo Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Dec 2015 15:57:08 -0000 Hi Luigi, Thank you for your response and the reference to the paper. After receiving your comment, for the time being, I shifted my focus to PCI-Passthrough because without modifying the host, it will not be possible to achieve line rate between guest-to-guest. With PCI-Passthrough enabled in ESXi, I am able to test with ixgbe (device: 82599) with netmap. But here again I am reaching maximum of ~5.5Gbps (with 64byte pkt, single CPU 2666.761MHz), the pkt-gen is using ~73%. I looked into another paper related to ptnetmap, but that again needs hypervisor change (and I do not want to go into that). I played with different values with 'ethtool --coalesce rx-usecs' - nothing is helping. The flow-control is also disabled. Are there any specific CONFIG(s) I need to enable/disable while compiling the linux-kernel (3.10 x86_64), or compiling the ixgbe driver (I am using 3.13.10)? Also, the sections 3.X in the paper (Speeding Up Packet I/O in Virtual Machines - Our Works) describes many ways of improvement - are those need to be taken care in case of PCI-Passthrough as well? If you are already aware of any experiment, is it possible to achieve line rate using PCI-Passthrough with netmap? Again thanks for your time and help. Best Regards, Pradosh (NB: one typo correction in my last mail, with e1000 ...throughput is _not_ close to line rate) On Fri, Dec 18, 2015 at 10:04 PM, Luigi Rizzo wrote: > On Fri, Dec 18, 2015 at 5:14 AM, Pradosh Datta > wrote: > > Hi, > > I am trying to use the netmap on vmware in CentOS guests, I am able to > use > > netmap with the e1000 driver on vmware (though the throughput is close to > > line speed). > > > > But is there any way to use the netmap with vmxnet3? Is there any patch > > available to support that? Or any suggestion on how can I achieve > > high-packet-rate between vm-to-vm using netmap? > > The vm-to-vm bottleneck is typically in the hypervisor (vmware) and the > switch (in this case it may be the linux bridge in centos). > > I am afraid just using netmap in the guest won't help you much in this > respect. > See our ANCS'13 paper > http://info.iet.unipi.it/~luigi/papers/20130903-rizzo-ancs.pdf > for what you can expect (look at the unmodified hypervisor/linux bridge > case). > > This said, you can always use netmap (in emulated mode) on any network > interface > in the guest. > > cheers > luigi > > > > > Best Regards, > > Pradosh > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- >