From owner-freebsd-net@freebsd.org Mon Jun 5 14:06:52 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3F5EAF9CC7 for ; Mon, 5 Jun 2017 14:06:52 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 960037F540 for ; Mon, 5 Jun 2017 14:06:52 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-oi0-x22f.google.com with SMTP id o65so134190474oif.1 for ; Mon, 05 Jun 2017 07:06:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=XO+m8BoxO4Pcmtgsl4oYmUQagvk2WAR6FGUFPe330ew=; b=EdriPRsKJMdg+GnB7zNd4q3dtucV8MomqhGUyNsNmspNerkIGf2jel08aBYo3BEB2p PHik7me53ubUidorTDb2E36ITv81L8h+bDSHmzWTxHAwvAr5CgxEVtwqkZgEfhop43Ps 9XDmYZfAA9FRP3EfgSo8I2bN+/TPwlrp44E1nF22DSEzkIw8xjaNW90QP3S8IxtnSQg6 flztMOBsHzpIDOaZOG6vMgEZZuqDYFkasjuq89scW7eeEGckyTiddvOIrkCrvTHy6g11 aAwam4s+FrqgN1UYtuYjOay++LPZ9sSxVrhvH2JriOwQD991pls1EHDZtaNlm1sSsA2H sVlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=XO+m8BoxO4Pcmtgsl4oYmUQagvk2WAR6FGUFPe330ew=; b=jqGqObqfXUjJyNhSkoD0jm4c4OZSLv22q0aBU3HZrZMP/t37vj7UMOOBMA3YXP/VQU BTvF3r4jLc+KC5Pt2SSMXxQhF/FpC6peCGJ+tvDNYbuHUK70vC2Dgz1XXJlPmRygBF1y B1rHwPbXugW7CQ/4zaiZ9VucP2mckRxVpNZ544G1PHS2zqQNQroZrbQcBEWEDrXGBH8o Qb3ka2FOvluHz1fuiWVGYAsqPZoN+b4lufQb6FWUVIJCjGHG90fCXdfXJO4uZS8n1dDB NB+/PL1Ny6hGJlIxo1O4B1y7YQ2M3OtHVaF7HdLrcdn/c/6qXJmL2xxqzjXuDscfXDkB eHBQ== X-Gm-Message-State: AODbwcAYixJTP1il5Cy+C2N5lWBRzk1vmpSZlo0eyogzkXSQ9hQbbyzI s4bXN/23o1qiGsJqTje0aMgRYNZ37tFV X-Received: by 10.157.58.102 with SMTP id j93mr2729811otc.147.1496671611528; Mon, 05 Jun 2017 07:06:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.17.49 with HTTP; Mon, 5 Jun 2017 07:06:51 -0700 (PDT) In-Reply-To: <592FC60A.1030308@omnilan.de> References: <5926FFDB.7040900@omnilan.de> <592F20A0.4020702@omnilan.de> <592FC60A.1030308@omnilan.de> From: Vincenzo Maffione Date: Mon, 5 Jun 2017 16:06:51 +0200 Message-ID: Subject: Re: ovs-netmap forgotten? To: Harry Schmalzbauer Cc: freebsd-net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jun 2017 14:06:52 -0000 Hi Harry, I've done some investigation on this issue (just for fun) , and I think I may have found the issue. When using vlan interfaces, netmap use the emulated adapter, as the "vlan" driver is not netmap-enabled (and it cannot be). To intercept RX packets, netmap replaces the "if_input" function pointer field in the kernel "struct ifnet" (the struct representing a network interface). Note that you have an instance of "struct ifnet" for em0 (physical NIC), and a different instance for each VLAN cloned interface (e.g. "vlan100") on em0. If you put vlan100 in netmap mode, netmap will replace the if_input of vlan100, and not the if_input of em0. So far, this is an expected behaviour= . Unfortunately, I see in the code here https://github.com/freebsd/freebsd/blob/master/sys/net/if_vlan.c#L1244-L124= 5 that when VLAN driver intercepts the RX packet coming from the underlying interface (e.g. em0 in our example), the em0 if_input is used rather than the vlan100 if_input. In terms of code, we have (*ifp->if_input)(ifv->ifv_ifp, m); rather than (*ifv->ifv_ifp->if_input)(ifv->ifv_ifp, m); Since em0 if_input is not replaced, netmap does not intercept it and you don't see it in your application, e.g. # pkt-gen -i vlan100 -f rx will see nothing. Now, I think that normally ifv->ifv_ifp->if_input =3D=3D ifp->if_input, so = this may explain why the code is written like that (to avoid the additional pointer dereferencing). This is not the case for netmap, where ifv->ifv_ifp->if_input !=3D ifp->if_input when em0 xor vlan100 are in netmap mode. You may try to recompile the kernel with that change and see if you can see packets coming on vlan100 with pkt-gen. I recommend you always doing tests with pkt-gen before trying to use vale-ctl -a. Cheers, Vincenzo 2017-06-01 9:45 GMT+02:00 Harry Schmalzbauer : > Bez=C3=BCglich Vincenzo Maffione's Nachricht vom 01.06.2017 00:39 (localt= ime): > > Hi Harry, > > OVS integration with netmap is very patchy and Linux only. Most > > importantly, it is not the right way to go, for a number of reasons. > > The real solution would be to integrate netmap into OVS would be to > > follow the DPDK-OVS approach: this means implementing the switching > > logic completely in userspace, in this case using the netmap API. This > > has not been implemented nor sketched so far. > > > > `vale-ctl -n valeXXX:YYY` just creates a persistent VALE port (YYY) > > attached to the VALE switch XXX. > > There is no difference with an ephemeral VALE port, apart for the fact > > that the persistent one is visible with ifconfig. > > > > It does not really make sense to attach a VLAN interface to VALE, since > > the VLAN driver does not have netmap support, so you lose all the > > advantage of using netmap and VALE. > > In your case the best solution I see is to write a custom netmap > > application that forwards the packets between a netmap-supported NIC an= d > > one or more VMs, doing the VLAN stripping in software. > > Thanks again, Vincenzo, for your highly appreciated support! > > I can only concur to your proposed solution. Problem is, I don't speak > any programmin language well (besides sh maybe) and have abosuletely no > budget/time to do any work out on my C skills (which I'd love to do) ;-) > > So ovs-netmap wasn't the right direction, but the difference between > em0+if_brdige+vmnet|virtio-net+vtnet and vale:em0|vale:guest+vtnet is > noticable. I haven't done any measuring, but just performing typical > admin jobs via cli (ssh into the bhyve-guest, whith resorces via NFS > (v4)) behave completely different =E2=80=93 human-noticable much more sna= ppy in > the vale:guest case! > I don't think this enormous efficiency advantge is soleley caused by > em0-netmap/ring connection; More important, in the > vale:em0|vale:guest+vtnet case, I gain excellent inter-vm efficiency > (and much higher attainable performance of course, which is not crucial > at the moment; but efficiency is!). > Now replacing vale:em0 by vale:vlan0 will surely destroy one big > efficiency advantage, but I still benefit from excellent inter-vm > efficiency and most likely some small efficiency advantage left over the > if_bridge picture. > Also, ptnet is a very interisting area of optimization which is easy to > explore with the vale:vlan scenario. > > In another post I described that the vale:vlan path doesn't work, while > vale:em0 (the parent) with everything else untouched does work. > Dou you think it's possible to fix the vale:vlan coupling without netmap > experts setting up a test environment? > > Thanks, > > -harry > > --=20 Vincenzo Maffione