From owner-freebsd-hackers@freebsd.org Sat Nov 21 16:06:38 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7F3E8469F70; Sat, 21 Nov 2020 16:06:38 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CddZk374yz4rNL; Sat, 21 Nov 2020 16:06:38 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id 54D032B4FD; Sat, 21 Nov 2020 16:06:38 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-pg1-f174.google.com with SMTP id f17so1610276pge.6; Sat, 21 Nov 2020 08:06:38 -0800 (PST) X-Gm-Message-State: AOAM533b+Kbh1yUf1skzGncrjEfmBk4erE3Joim6OBRQCQAqwCplwbeK cQgLjEFKUqJYuDiLHz6OWd5JzBRVFcNu8q2YaVM= X-Google-Smtp-Source: ABdhPJzTruSs1RR5ioxUncuty5+Oj6xQF8Y425PD0r6hdVbKpr9Qmdp+sFyesrCvahiWP82P5cGKRBAq4PjlHA2bD/I= X-Received: by 2002:a17:90a:e615:: with SMTP id j21mr11741439pjy.74.1605974797327; Sat, 21 Nov 2020 08:06:37 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Vincenzo Maffione Date: Sat, 21 Nov 2020 17:06:26 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Netmap bridge not working with 10G Ethernet ports To: Rajesh Kumar Cc: "freebsd-net@freebsd.org" , FreeBSD Hackers , stephan.dewt@yahoo.co.uk Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2020 16:06:38 -0000 Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar ha scritto: > Hi Vincenzo, > > On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione > wrote: > >> >> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses >> iflib(4). This means that actually if_axp(4) has native netmap support, >> because iflib(4) has native netmap support. >> >> > It means that the driver has some modifications to allow netmap to >> directly program the NIC rings. These modifications are mostly the >> per-driver txsync and rxsyng routines. >> In case of iflib(4) drivers, these modifications are provided directly >> within the iflib(4) code, and therefore any driver using iflib will have >> native netmap support. >> > > Thanks for clarifying on the Native Netmap support. > > Ok, this makes sense, because also ix(4) uses iflib, and therefore you are >> basically hitting the same issue of if_axp(4) >> At this point I must think that there is still some issue with the >> interaction between iflib(4) and netmap(4). >> > > Ok. Let me know if any more debug info needed in this part. > > I see. This info may be useful. Have you tried to look at interrupts (e.g. >> `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP >> replies)? >> > > It's interesting here. When I try with Intel NIC card. I see atleast 1 > interrupt raised. But not sure whether that is for ARP reply. Because, > when I try to dump the packet from "bridge"(modified) utility, I don't see > any ARP reply packet getting dumped. > > > *irq59: ix0:rxq0 1 0 (only 1 interrupt on > the opposite side)*irq67: ix0:aq 2 0 > > *irq68: ix1:rxq0 3 0 (you can see 3 > interrupts for 3 ARP requests from System 1)*irq76: ix1:aq > 2 0 > > The same experiment, when I try with AMD inbuilt ports, I don't see that 1 > interrupt also raised. > > irq81: ax0:dev_irq 16 0 > irq83: ax0 2541 4 > irq93: ax1:dev_irq 27 0 > irq95: ax1 2371 3 > *irq97: ax1:rxq0 3 0 (you can see 3 > interrupts for 3 ARP requests from System 1, but no interrupt is seen from > "ax0:rxq0" for ARP reply from System 2)* > > I will do some more testing to see whether this behavior is consistent or > intermittent. > > Also the igb(4) driver is using iflib(4). So the involved netmap code is >> the same as ix(4) and if_axp(4). >> This is something that I'm not able to understand right now. >> It does not look like something related to offloads. >> >> Next week I will try to see if I can reproduce your issue with em(4), and >> report back. That's still an Intel driver using iflib(4). >> > > The "igb(4)" driver, with which things are working now is related to em(4) > driver (may be for newer hardware version). Initially we faced similar > issue with igb(4) driver as well. After reverting the following commits, > things started to work. Thanks to Stephan Dewt (copied) for pointing > this. But it still fails with ix(4) driver and if_axp(4) driver. > > > https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf > > Thanks for providing your inputs on this issue Vincenzo. Let me know for > any more details that you need. > > I was able to reproduce your issue on FreeBSD-CURRENT running within a QEMU VM, with two em(4) devices and the netmap bridge running between them. I see the ARP request packet received on em0 (with associated IRQ), and forwarded on em1. However, the ARP reply coming on em1 does not trigger an IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird, and lets me think that the issue is more likely driver-related than netmap/iflib-related. In any case, would you mind filing the issue on the bugzilla, so that we can properly track this issue? Thanks, Vincenzo > Thanks, > Rajesh. >