From owner-freebsd-net@freebsd.org Tue Nov 24 02:57:32 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EC45947585D; Tue, 24 Nov 2020 02:57:32 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-oo1-xc36.google.com (mail-oo1-xc36.google.com [IPv6:2607:f8b0:4864:20::c36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cg7wq64Hrz3Nl4; Tue, 24 Nov 2020 02:57:31 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-oo1-xc36.google.com with SMTP id i30so2226527ooh.9; Mon, 23 Nov 2020 18:57:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WRXU77dkmi57J0rG1+wUZrz0LA4hqhm12oPa4o7IzPM=; b=IjcJwV2urbE1myPtBrAROcDlwFSpLv7lAlgrjVf15Bh7MSW7l+lnCk8juUbjOCH5s4 WeypV6wWQ9DUNYQNSS9O//6l6OYl9mkxy3WSQehHxz3CXwZndwyOJccgRAPY+X5wtb92 J45FwvwRVHqFcZa6iorBc9cUyDAN6esG/Fg+/B/LQOQTzoorux6TnHIQHx5lqSMh15IZ QgFx7uhifOsDfFHZv2wE9nj7UhrHza4cvbyNz40MW5+Qqk9B+aRWRKz4bXzx8zt1pmtE k+9QpVA7ILz5WSWfi2gh3BC7kXVMhThwjaH/y+bX3P67qjvbe4QXkE1Kt1oa2GBhge0O NVtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WRXU77dkmi57J0rG1+wUZrz0LA4hqhm12oPa4o7IzPM=; b=k2Ko+2QoxWqF8++erHAZC19XwoQn/g0vn9FDOirSde8fvKZUsWJ1B5KON3YnkfkQCc kWuN3cEwt7Ug10xwxgNo6sPM6recH+v7WzRNTejR6spbfmAGjp2mbi0dP8bpKFe1MbVf mCEXly05ygACNxCP0h+bUFU1jCg54LAsHM/IEcGMizOg03YgrX2JglUUBlkDw+HFbwty fn7skSDYtcO0UVZwyn7id1Z1ABJYID/oc5o4ZSUdkVh7MRIb4QIUj3HM2UiPV2aa+kFI 8Uy5BGG+AYFpEaQXB6EoD2JmBDi2RT442HUIuKiduCVrbx9sNqShAadGtRZXaXPTP2zm Q2gQ== X-Gm-Message-State: AOAM5339p+BAs3iWRaUfps9HyNCrzWs2xHeGNHQifmDSc2KqZDSniAXe WyoQTFmYbxo3RmYghOaBHFtafRah2AkhFdpWtWn9nwSa X-Google-Smtp-Source: ABdhPJzJefqHmzbSPYcL841Ux/jdCMnvE5ZxNc8W4v3NmD4X+cxvlfB/UpnyYmsUZS2/ZPNdI6Epx2TjrBQtjtnL/6I= X-Received: by 2002:a4a:9711:: with SMTP id u17mr1694821ooi.57.1606186650160; Mon, 23 Nov 2020 18:57:30 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Rajesh Kumar Date: Tue, 24 Nov 2020 08:27:19 +0530 Message-ID: Subject: Re: Netmap bridge not working with 10G Ethernet ports To: Vincenzo Maffione Cc: "freebsd-net@freebsd.org" , FreeBSD Hackers , stephan.dewt@yahoo.co.uk X-Rspamd-Queue-Id: 4Cg7wq64Hrz3Nl4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=IjcJwV2u; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2607:f8b0:4864:20::c36 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::c36:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::c36:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c36:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net,freebsd-hackers]; FREEMAIL_CC(0.00)[freebsd.org,yahoo.co.uk] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Nov 2020 02:57:33 -0000 Hi Vincenzo, Thanks for pointing this. On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione wrote: > # ifconfig ix0 promisc > # ifconfig ix1 promisc > > This is an additional requirement when using netmap bridge, because that > is not done automatically (differently from what happens with if_bridge(4)). > If promisc is not enabled, the NIC will drop any unicast packet that is > not directed to the NIC's address (e.g. the ARP reply in your case). > Broadcast packets will of course pass (e.g. the ARP request). This explains > the absence of IRQs and the head/tail pointers not being updated. > So no bugs AFAIK. > Setting the interfaces in promiscous mode makes things to work properly. I tried the same with AMD Ports and it's still not working. I believe this is something specific to if_axp driver. I will see what is going wrong with packet forwarding with AMD ports. Thanks for pointing this out. I figured it out the hard way, but it was actually also documented on the > github (https://github.com/luigirizzo/netmap#receiver-does-not-receive). > I will add it to the netmap bridge man page. > That would be helpful. Thanks. > Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione < > vmaffione@freebsd.org> ha scritto: > >> >> >> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar >> ha scritto: >> >>> Hi Vincenzo, >>> >>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione >>> wrote: >>> >>>> >>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) >>>> uses iflib(4). This means that actually if_axp(4) has native netmap >>>> support, because iflib(4) has native netmap support. >>>> >>>> >>> It means that the driver has some modifications to allow netmap to >>>> directly program the NIC rings. These modifications are mostly the >>>> per-driver txsync and rxsyng routines. >>>> In case of iflib(4) drivers, these modifications are provided directly >>>> within the iflib(4) code, and therefore any driver using iflib will have >>>> native netmap support. >>>> >>> >>> Thanks for clarifying on the Native Netmap support. >>> >>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you >>>> are basically hitting the same issue of if_axp(4) >>>> At this point I must think that there is still some issue with the >>>> interaction between iflib(4) and netmap(4). >>>> >>> >>> Ok. Let me know if any more debug info needed in this part. >>> >>> I see. This info may be useful. Have you tried to look at interrupts >>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing >>>> ARP replies)? >>>> >>> >>> It's interesting here. When I try with Intel NIC card. I see atleast 1 >>> interrupt raised. But not sure whether that is for ARP reply. Because, >>> when I try to dump the packet from "bridge"(modified) utility, I don't see >>> any ARP reply packet getting dumped. >>> >>> >>> *irq59: ix0:rxq0 1 0 (only 1 interrupt >>> on the opposite side)*irq67: ix0:aq 2 >>> 0 >>> >>> *irq68: ix1:rxq0 3 0 (you can see 3 >>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq >>> 2 0 >>> >>> The same experiment, when I try with AMD inbuilt ports, I don't see that >>> 1 interrupt also raised. >>> >>> irq81: ax0:dev_irq 16 0 >>> irq83: ax0 2541 4 >>> irq93: ax1:dev_irq 27 0 >>> irq95: ax1 2371 3 >>> *irq97: ax1:rxq0 3 0 (you can see 3 >>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from >>> "ax0:rxq0" for ARP reply from System 2)* >>> >>> I will do some more testing to see whether this behavior is consistent >>> or intermittent. >>> >>> Also the igb(4) driver is using iflib(4). So the involved netmap code is >>>> the same as ix(4) and if_axp(4). >>>> This is something that I'm not able to understand right now. >>>> It does not look like something related to offloads. >>>> >>>> Next week I will try to see if I can reproduce your issue with em(4), >>>> and report back. That's still an Intel driver using iflib(4). >>>> >>> >>> The "igb(4)" driver, with which things are working now is related to >>> em(4) driver (may be for newer hardware version). Initially we faced >>> similar issue with igb(4) driver as well. After reverting the following >>> commits, things started to work. Thanks to Stephan Dewt (copied) for >>> pointing this. But it still fails with ix(4) driver and if_axp(4) driver. >>> >>> >>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf >>> >>> Thanks for providing your inputs on this issue Vincenzo. Let me know >>> for any more details that you need. >>> >>> >> I was able to reproduce your issue on FreeBSD-CURRENT running within a >> QEMU VM, with two em(4) devices and the netmap bridge running between them. >> I see the ARP request packet received on em0 (with associated IRQ), and >> forwarded on em1. However, the ARP reply coming on em1 does not trigger an >> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as >> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird, >> and lets me think that the issue is more likely driver-related than >> netmap/iflib-related. >> In any case, would you mind filing the issue on the bugzilla, so that we >> can properly track this issue? >> >> Thanks, >> Vincenzo >> >> >>> Thanks, >>> Rajesh. >>> >>