Date: Tue, 24 Nov 2020 08:27:19 +0530 From: Rajesh Kumar <rajfbsd@gmail.com> To: Vincenzo Maffione <vmaffione@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, stephan.dewt@yahoo.co.uk Subject: Re: Netmap bridge not working with 10G Ethernet ports Message-ID: <CAAO%2BANMT_So=kVk2qcs6Jf2Js2tnbdF8cbZdfkE6oFGC=DpK=g@mail.gmail.com> In-Reply-To: <CA%2B_eA9iiZ_CLuKxfZzz3AwVs=ZQPeq-g3ezkYTeXcowmRxHnSA@mail.gmail.com> References: <CAAO%2BANOg5MEfHf9bV5x4L_QXNY2O9vQk0s%2BJrD7yzeXCQfHt8w@mail.gmail.com> <CA%2B_eA9hR8ysiFGj-iriMpqXcDbc4X_h_C1sgNoO05KoLy5orCA@mail.gmail.com> <CAAO%2BANP3PcqKo1nUfZTB92uKoJ40VA9YLo9MMqSN9AkMhq55tw@mail.gmail.com> <CA%2B_eA9ixySZtvJ8C9mwPj6q2fAQmZJicVgLHKkpb398-u_PaJw@mail.gmail.com> <CAAO%2BANM_zUgViGfFRuxZo7bTTNd7w9Xco=ptZweYHSGgx_xXsg@mail.gmail.com> <CA%2B_eA9i4i5txARoW-cLc=nzY28r9QCDw1%2B58pjAZBj=LMTV1og@mail.gmail.com> <CA%2B_eA9iiZ_CLuKxfZzz3AwVs=ZQPeq-g3ezkYTeXcowmRxHnSA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Vincenzo, Thanks for pointing this. On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione <vmaffione@freebsd.org> wrote: > # ifconfig ix0 promisc > # ifconfig ix1 promisc > > This is an additional requirement when using netmap bridge, because that > is not done automatically (differently from what happens with if_bridge(4)). > If promisc is not enabled, the NIC will drop any unicast packet that is > not directed to the NIC's address (e.g. the ARP reply in your case). > Broadcast packets will of course pass (e.g. the ARP request). This explains > the absence of IRQs and the head/tail pointers not being updated. > So no bugs AFAIK. > Setting the interfaces in promiscous mode makes things to work properly. I tried the same with AMD Ports and it's still not working. I believe this is something specific to if_axp driver. I will see what is going wrong with packet forwarding with AMD ports. Thanks for pointing this out. I figured it out the hard way, but it was actually also documented on the > github (https://github.com/luigirizzo/netmap#receiver-does-not-receive). > I will add it to the netmap bridge man page. > That would be helpful. Thanks. > Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione < > vmaffione@freebsd.org> ha scritto: > >> >> >> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar <rajfbsd@gmail.com> >> ha scritto: >> >>> Hi Vincenzo, >>> >>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione <vmaffione@freebsd.org> >>> wrote: >>> >>>> >>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) >>>> uses iflib(4). This means that actually if_axp(4) has native netmap >>>> support, because iflib(4) has native netmap support. >>>> >>>> >>> It means that the driver has some modifications to allow netmap to >>>> directly program the NIC rings. These modifications are mostly the >>>> per-driver txsync and rxsyng routines. >>>> In case of iflib(4) drivers, these modifications are provided directly >>>> within the iflib(4) code, and therefore any driver using iflib will have >>>> native netmap support. >>>> >>> >>> Thanks for clarifying on the Native Netmap support. >>> >>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you >>>> are basically hitting the same issue of if_axp(4) >>>> At this point I must think that there is still some issue with the >>>> interaction between iflib(4) and netmap(4). >>>> >>> >>> Ok. Let me know if any more debug info needed in this part. >>> >>> I see. This info may be useful. Have you tried to look at interrupts >>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing >>>> ARP replies)? >>>> >>> >>> It's interesting here. When I try with Intel NIC card. I see atleast 1 >>> interrupt raised. But not sure whether that is for ARP reply. Because, >>> when I try to dump the packet from "bridge"(modified) utility, I don't see >>> any ARP reply packet getting dumped. >>> >>> >>> *irq59: ix0:rxq0 1 0 (only 1 interrupt >>> on the opposite side)*irq67: ix0:aq 2 >>> 0 >>> >>> *irq68: ix1:rxq0 3 0 (you can see 3 >>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq >>> 2 0 >>> >>> The same experiment, when I try with AMD inbuilt ports, I don't see that >>> 1 interrupt also raised. >>> >>> irq81: ax0:dev_irq 16 0 >>> irq83: ax0 2541 4 >>> irq93: ax1:dev_irq 27 0 >>> irq95: ax1 2371 3 >>> *irq97: ax1:rxq0 3 0 (you can see 3 >>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from >>> "ax0:rxq0" for ARP reply from System 2)* >>> >>> I will do some more testing to see whether this behavior is consistent >>> or intermittent. >>> >>> Also the igb(4) driver is using iflib(4). So the involved netmap code is >>>> the same as ix(4) and if_axp(4). >>>> This is something that I'm not able to understand right now. >>>> It does not look like something related to offloads. >>>> >>>> Next week I will try to see if I can reproduce your issue with em(4), >>>> and report back. That's still an Intel driver using iflib(4). >>>> >>> >>> The "igb(4)" driver, with which things are working now is related to >>> em(4) driver (may be for newer hardware version). Initially we faced >>> similar issue with igb(4) driver as well. After reverting the following >>> commits, things started to work. Thanks to Stephan Dewt (copied) for >>> pointing this. But it still fails with ix(4) driver and if_axp(4) driver. >>> >>> >>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf >>> >>> Thanks for providing your inputs on this issue Vincenzo. Let me know >>> for any more details that you need. >>> >>> >> I was able to reproduce your issue on FreeBSD-CURRENT running within a >> QEMU VM, with two em(4) devices and the netmap bridge running between them. >> I see the ARP request packet received on em0 (with associated IRQ), and >> forwarded on em1. However, the ARP reply coming on em1 does not trigger an >> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as >> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird, >> and lets me think that the issue is more likely driver-related than >> netmap/iflib-related. >> In any case, would you mind filing the issue on the bugzilla, so that we >> can properly track this issue? >> >> Thanks, >> Vincenzo >> >> >>> Thanks, >>> Rajesh. >>> >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAO%2BANMT_So=kVk2qcs6Jf2Js2tnbdF8cbZdfkE6oFGC=DpK=g>