From owner-freebsd-hackers@freebsd.org Tue Nov 24 15:24:05 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F111A2E9088; Tue, 24 Nov 2020 15:24:05 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CgSVF6PLZz4qt6; Tue, 24 Nov 2020 15:24:05 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id BADEA2C3A2; Tue, 24 Nov 2020 15:24:05 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-pg1-f182.google.com with SMTP id w4so17733737pgg.13; Tue, 24 Nov 2020 07:24:05 -0800 (PST) X-Gm-Message-State: AOAM530i4eTKD+7E1lgD8gu04toS2gKCbhfZ0cxa3T6SPe90YjvDQMYB 1NMM5ekDAikZbbOntU2GUl+DM6PCQCoHMD8qD0s= X-Google-Smtp-Source: ABdhPJwTppd1BntVpeEhMOFCPkbYmaWEU/cg1q8YKeKtQJvJrXsDJOrQEw5zwNVv6WK4UCfriJJKCr/6pEjOdTh4h3Y= X-Received: by 2002:a17:90a:c084:: with SMTP id o4mr5402621pjs.150.1606231444645; Tue, 24 Nov 2020 07:24:04 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Vincenzo Maffione Date: Tue, 24 Nov 2020 16:23:53 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Netmap bridge not working with 10G Ethernet ports To: Rajesh Kumar Cc: "freebsd-net@freebsd.org" , FreeBSD Hackers , stephan.dewt@yahoo.co.uk Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Nov 2020 15:24:06 -0000 Il giorno mar 24 nov 2020 alle ore 03:57 Rajesh Kumar ha scritto: > Hi Vincenzo, > > Thanks for pointing this. > > On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione > wrote: > >> # ifconfig ix0 promisc >> # ifconfig ix1 promisc >> >> This is an additional requirement when using netmap bridge, because that >> is not done automatically (differently from what happens with if_bridge(4)). >> If promisc is not enabled, the NIC will drop any unicast packet that is >> not directed to the NIC's address (e.g. the ARP reply in your case). >> Broadcast packets will of course pass (e.g. the ARP request). This explains >> the absence of IRQs and the head/tail pointers not being updated. >> So no bugs AFAIK. >> > > Setting the interfaces in promiscous mode makes things to work properly. > > I tried the same with AMD Ports and it's still not working. I believe > this is something specific to if_axp driver. I will see what is going wrong > with packet forwarding with AMD ports. Thanks for pointing this out. > Yeah, it's weird because axgbe also uses iflib(4). If the driver exposes NIC head/tail pointers (sysctl) it may be useful to check what happens there. It may be that the NIC is dropping these packets for some reason. > > I figured it out the hard way, but it was actually also documented on the >> github (https://github.com/luigirizzo/netmap#receiver-does-not-receive). >> I will add it to the netmap bridge man page. >> > > That would be helpful. Thanks. > Done in r367920. Cheers, Vincenzo > > >> Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione < >> vmaffione@freebsd.org> ha scritto: >> >>> >>> >>> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar >>> ha scritto: >>> >>>> Hi Vincenzo, >>>> >>>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione < >>>> vmaffione@freebsd.org> wrote: >>>> >>>>> >>>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) >>>>> uses iflib(4). This means that actually if_axp(4) has native netmap >>>>> support, because iflib(4) has native netmap support. >>>>> >>>>> >>>> It means that the driver has some modifications to allow netmap to >>>>> directly program the NIC rings. These modifications are mostly the >>>>> per-driver txsync and rxsyng routines. >>>>> In case of iflib(4) drivers, these modifications are provided directly >>>>> within the iflib(4) code, and therefore any driver using iflib will have >>>>> native netmap support. >>>>> >>>> >>>> Thanks for clarifying on the Native Netmap support. >>>> >>>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you >>>>> are basically hitting the same issue of if_axp(4) >>>>> At this point I must think that there is still some issue with the >>>>> interaction between iflib(4) and netmap(4). >>>>> >>>> >>>> Ok. Let me know if any more debug info needed in this part. >>>> >>>> I see. This info may be useful. Have you tried to look at interrupts >>>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing >>>>> ARP replies)? >>>>> >>>> >>>> It's interesting here. When I try with Intel NIC card. I see atleast 1 >>>> interrupt raised. But not sure whether that is for ARP reply. Because, >>>> when I try to dump the packet from "bridge"(modified) utility, I don't see >>>> any ARP reply packet getting dumped. >>>> >>>> >>>> *irq59: ix0:rxq0 1 0 (only 1 interrupt >>>> on the opposite side)*irq67: ix0:aq 2 >>>> 0 >>>> >>>> *irq68: ix1:rxq0 3 0 (you can see 3 >>>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq >>>> 2 0 >>>> >>>> The same experiment, when I try with AMD inbuilt ports, I don't see >>>> that 1 interrupt also raised. >>>> >>>> irq81: ax0:dev_irq 16 0 >>>> irq83: ax0 2541 4 >>>> irq93: ax1:dev_irq 27 0 >>>> irq95: ax1 2371 3 >>>> *irq97: ax1:rxq0 3 0 (you can see 3 >>>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from >>>> "ax0:rxq0" for ARP reply from System 2)* >>>> >>>> I will do some more testing to see whether this behavior is consistent >>>> or intermittent. >>>> >>>> Also the igb(4) driver is using iflib(4). So the involved netmap code >>>>> is the same as ix(4) and if_axp(4). >>>>> This is something that I'm not able to understand right now. >>>>> It does not look like something related to offloads. >>>>> >>>>> Next week I will try to see if I can reproduce your issue with em(4), >>>>> and report back. That's still an Intel driver using iflib(4). >>>>> >>>> >>>> The "igb(4)" driver, with which things are working now is related to >>>> em(4) driver (may be for newer hardware version). Initially we faced >>>> similar issue with igb(4) driver as well. After reverting the following >>>> commits, things started to work. Thanks to Stephan Dewt (copied) for >>>> pointing this. But it still fails with ix(4) driver and if_axp(4) driver. >>>> >>>> >>>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf >>>> >>>> Thanks for providing your inputs on this issue Vincenzo. Let me know >>>> for any more details that you need. >>>> >>>> >>> I was able to reproduce your issue on FreeBSD-CURRENT running within a >>> QEMU VM, with two em(4) devices and the netmap bridge running between them. >>> I see the ARP request packet received on em0 (with associated IRQ), and >>> forwarded on em1. However, the ARP reply coming on em1 does not trigger an >>> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as >>> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird, >>> and lets me think that the issue is more likely driver-related than >>> netmap/iflib-related. >>> In any case, would you mind filing the issue on the bugzilla, so that we >>> can properly track this issue? >>> >>> Thanks, >>> Vincenzo >>> >>> >>>> Thanks, >>>> Rajesh. >>>> >>>