Date: Tue, 24 Nov 2020 16:23:53 +0100 From: Vincenzo Maffione <vmaffione@freebsd.org> To: Rajesh Kumar <rajfbsd@gmail.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, stephan.dewt@yahoo.co.uk Subject: Re: Netmap bridge not working with 10G Ethernet ports Message-ID: <CA%2B_eA9jOOFTYmv_bmxcaXz26=-uqG_DnTx7=%2Bg1QM_%2BftJXkMQ@mail.gmail.com> In-Reply-To: <CAAO%2BANMT_So=kVk2qcs6Jf2Js2tnbdF8cbZdfkE6oFGC=DpK=g@mail.gmail.com> References: <CAAO%2BANOg5MEfHf9bV5x4L_QXNY2O9vQk0s%2BJrD7yzeXCQfHt8w@mail.gmail.com> <CA%2B_eA9hR8ysiFGj-iriMpqXcDbc4X_h_C1sgNoO05KoLy5orCA@mail.gmail.com> <CAAO%2BANP3PcqKo1nUfZTB92uKoJ40VA9YLo9MMqSN9AkMhq55tw@mail.gmail.com> <CA%2B_eA9ixySZtvJ8C9mwPj6q2fAQmZJicVgLHKkpb398-u_PaJw@mail.gmail.com> <CAAO%2BANM_zUgViGfFRuxZo7bTTNd7w9Xco=ptZweYHSGgx_xXsg@mail.gmail.com> <CA%2B_eA9i4i5txARoW-cLc=nzY28r9QCDw1%2B58pjAZBj=LMTV1og@mail.gmail.com> <CA%2B_eA9iiZ_CLuKxfZzz3AwVs=ZQPeq-g3ezkYTeXcowmRxHnSA@mail.gmail.com> <CAAO%2BANMT_So=kVk2qcs6Jf2Js2tnbdF8cbZdfkE6oFGC=DpK=g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Il giorno mar 24 nov 2020 alle ore 03:57 Rajesh Kumar <rajfbsd@gmail.com> ha scritto: > Hi Vincenzo, > > Thanks for pointing this. > > On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione <vmaffione@freebsd.org> > wrote: > >> # ifconfig ix0 promisc >> # ifconfig ix1 promisc >> >> This is an additional requirement when using netmap bridge, because that >> is not done automatically (differently from what happens with if_bridge(4)). >> If promisc is not enabled, the NIC will drop any unicast packet that is >> not directed to the NIC's address (e.g. the ARP reply in your case). >> Broadcast packets will of course pass (e.g. the ARP request). This explains >> the absence of IRQs and the head/tail pointers not being updated. >> So no bugs AFAIK. >> > > Setting the interfaces in promiscous mode makes things to work properly. > > I tried the same with AMD Ports and it's still not working. I believe > this is something specific to if_axp driver. I will see what is going wrong > with packet forwarding with AMD ports. Thanks for pointing this out. > Yeah, it's weird because axgbe also uses iflib(4). If the driver exposes NIC head/tail pointers (sysctl) it may be useful to check what happens there. It may be that the NIC is dropping these packets for some reason. > > I figured it out the hard way, but it was actually also documented on the >> github (https://github.com/luigirizzo/netmap#receiver-does-not-receive). >> I will add it to the netmap bridge man page. >> > > That would be helpful. Thanks. > Done in r367920. Cheers, Vincenzo > > >> Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione < >> vmaffione@freebsd.org> ha scritto: >> >>> >>> >>> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar <rajfbsd@gmail.com> >>> ha scritto: >>> >>>> Hi Vincenzo, >>>> >>>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione < >>>> vmaffione@freebsd.org> wrote: >>>> >>>>> >>>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) >>>>> uses iflib(4). This means that actually if_axp(4) has native netmap >>>>> support, because iflib(4) has native netmap support. >>>>> >>>>> >>>> It means that the driver has some modifications to allow netmap to >>>>> directly program the NIC rings. These modifications are mostly the >>>>> per-driver txsync and rxsyng routines. >>>>> In case of iflib(4) drivers, these modifications are provided directly >>>>> within the iflib(4) code, and therefore any driver using iflib will have >>>>> native netmap support. >>>>> >>>> >>>> Thanks for clarifying on the Native Netmap support. >>>> >>>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you >>>>> are basically hitting the same issue of if_axp(4) >>>>> At this point I must think that there is still some issue with the >>>>> interaction between iflib(4) and netmap(4). >>>>> >>>> >>>> Ok. Let me know if any more debug info needed in this part. >>>> >>>> I see. This info may be useful. Have you tried to look at interrupts >>>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing >>>>> ARP replies)? >>>>> >>>> >>>> It's interesting here. When I try with Intel NIC card. I see atleast 1 >>>> interrupt raised. But not sure whether that is for ARP reply. Because, >>>> when I try to dump the packet from "bridge"(modified) utility, I don't see >>>> any ARP reply packet getting dumped. >>>> >>>> >>>> *irq59: ix0:rxq0 1 0 (only 1 interrupt >>>> on the opposite side)*irq67: ix0:aq 2 >>>> 0 >>>> >>>> *irq68: ix1:rxq0 3 0 (you can see 3 >>>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq >>>> 2 0 >>>> >>>> The same experiment, when I try with AMD inbuilt ports, I don't see >>>> that 1 interrupt also raised. >>>> >>>> irq81: ax0:dev_irq 16 0 >>>> irq83: ax0 2541 4 >>>> irq93: ax1:dev_irq 27 0 >>>> irq95: ax1 2371 3 >>>> *irq97: ax1:rxq0 3 0 (you can see 3 >>>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from >>>> "ax0:rxq0" for ARP reply from System 2)* >>>> >>>> I will do some more testing to see whether this behavior is consistent >>>> or intermittent. >>>> >>>> Also the igb(4) driver is using iflib(4). So the involved netmap code >>>>> is the same as ix(4) and if_axp(4). >>>>> This is something that I'm not able to understand right now. >>>>> It does not look like something related to offloads. >>>>> >>>>> Next week I will try to see if I can reproduce your issue with em(4), >>>>> and report back. That's still an Intel driver using iflib(4). >>>>> >>>> >>>> The "igb(4)" driver, with which things are working now is related to >>>> em(4) driver (may be for newer hardware version). Initially we faced >>>> similar issue with igb(4) driver as well. After reverting the following >>>> commits, things started to work. Thanks to Stephan Dewt (copied) for >>>> pointing this. But it still fails with ix(4) driver and if_axp(4) driver. >>>> >>>> >>>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf >>>> >>>> Thanks for providing your inputs on this issue Vincenzo. Let me know >>>> for any more details that you need. >>>> >>>> >>> I was able to reproduce your issue on FreeBSD-CURRENT running within a >>> QEMU VM, with two em(4) devices and the netmap bridge running between them. >>> I see the ARP request packet received on em0 (with associated IRQ), and >>> forwarded on em1. However, the ARP reply coming on em1 does not trigger an >>> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as >>> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird, >>> and lets me think that the issue is more likely driver-related than >>> netmap/iflib-related. >>> In any case, would you mind filing the issue on the bugzilla, so that we >>> can properly track this issue? >>> >>> Thanks, >>> Vincenzo >>> >>> >>>> Thanks, >>>> Rajesh. >>>> >>>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B_eA9jOOFTYmv_bmxcaXz26=-uqG_DnTx7=%2Bg1QM_%2BftJXkMQ>