From owner-freebsd-net@freebsd.org Fri Nov 20 13:31:12 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AD1F22EA877; Fri, 20 Nov 2020 13:31:12 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ccy9r34vsz3NBf; Fri, 20 Nov 2020 13:31:12 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-ot1-x330.google.com with SMTP id y22so8677253oti.10; Fri, 20 Nov 2020 05:31:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4en2qv4i/arwzRLCSEaR6mvJfko42IYgg5QLnPVzn+Y=; b=LzpZ+ReOQ1nRLgTUmfUuVzleNT4/R7tgvNN5z/7AS000hkZVC135Hf8lK9kUwZiFAz 7UG13xgoLwBfANmLkBHpqtwPF0YXAp4Bn9hi5T6JpU6OEnK94fanCUm8a5tyfPiT9yQ+ BidDwZXgZIoQinkUE+iGyqP3TErn3JxB7/46MubihcSQiePROT8eX3Ows7lLO+Rv61uq TtELQj0zjCr18+DC1anxa8zA5754JUs875eYYZlUiIADhmSF96JuBfmqkNGl2c3xATtb jd8Y0ihPSZefHCgwwpgCza8V6d12uNFGKt+Qan/lcbUiEUxlNlqZaxlUqFym7WXt0qgF Dy9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4en2qv4i/arwzRLCSEaR6mvJfko42IYgg5QLnPVzn+Y=; b=o2aTL6vabfKt1bW2zuPNAVucwCrAHB50dLJlj5d2Dc6HIbFUMt5IZzWdtLDAM1L7gs 9whXWsrMBQk0XP4zLJXPZaVdJljwT6i1I0iuwmd10riRHASP7Ie1tOvXsREw69u0VOq/ zTLL2wSlbbMPlurWBqEFIhFKlHnBFcA2ShqC36hHfrxn1Oi8mBKD10TxXw6CVSaqPa8X o3YgQcRIz34Yd5D0ez+OXL4LqfcAavAbS+/TvFnc0V7C1gwxlyvWuudC23qUrKiGdlI6 +4DCT6Y8nKVGeKoFn9LvTci73z7w1T19KA8m6Rf8o2hG+Vebrws6netpeJFKf8/i+mSS bd5g== X-Gm-Message-State: AOAM530HFhDtQ3JOV8H8nV9lHN6TVyw2CQXV2bxZGP4fm3I4H7mb5ojM 6f0cD51XK+fBnLrcZZvMILYHV+wwAT8K1bQ6LsSBJdUjq7s= X-Google-Smtp-Source: ABdhPJzwqeIlDwhfyFPO56tpkRBAKLM2hglPfoTqRWxboSnFq5Wvqv7RoKcnkds94ft9dZ9HOrOvPphodzczyXhMYvU= X-Received: by 2002:a9d:76d7:: with SMTP id p23mr14042591otl.180.1605879071060; Fri, 20 Nov 2020 05:31:11 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Rajesh Kumar Date: Fri, 20 Nov 2020 19:00:59 +0530 Message-ID: Subject: Re: Netmap bridge not working with 10G Ethernet ports To: Vincenzo Maffione Cc: "freebsd-net@freebsd.org" , FreeBSD Hackers , stephan.dewt@yahoo.co.uk X-Rspamd-Queue-Id: 4Ccy9r34vsz3NBf X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 13:31:12 -0000 Hi Vincenzo, On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione wrote: > > Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses > iflib(4). This means that actually if_axp(4) has native netmap support, > because iflib(4) has native netmap support. > > It means that the driver has some modifications to allow netmap to directly > program the NIC rings. These modifications are mostly the per-driver txsync > and rxsyng routines. > In case of iflib(4) drivers, these modifications are provided directly > within the iflib(4) code, and therefore any driver using iflib will have > native netmap support. > Thanks for clarifying on the Native Netmap support. Ok, this makes sense, because also ix(4) uses iflib, and therefore you are > basically hitting the same issue of if_axp(4) > At this point I must think that there is still some issue with the > interaction between iflib(4) and netmap(4). > Ok. Let me know if any more debug info needed in this part. I see. This info may be useful. Have you tried to look at interrupts (e.g. > `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP > replies)? > It's interesting here. When I try with Intel NIC card. I see atleast 1 interrupt raised. But not sure whether that is for ARP reply. Because, when I try to dump the packet from "bridge"(modified) utility, I don't see any ARP reply packet getting dumped. *irq59: ix0:rxq0 1 0 (only 1 interrupt on the opposite side)*irq67: ix0:aq 2 0 *irq68: ix1:rxq0 3 0 (you can see 3 interrupts for 3 ARP requests from System 1)*irq76: ix1:aq 2 0 The same experiment, when I try with AMD inbuilt ports, I don't see that 1 interrupt also raised. irq81: ax0:dev_irq 16 0 irq83: ax0 2541 4 irq93: ax1:dev_irq 27 0 irq95: ax1 2371 3 *irq97: ax1:rxq0 3 0 (you can see 3 interrupts for 3 ARP requests from System 1, but no interrupt is seen from "ax0:rxq0" for ARP reply from System 2)* I will do some more testing to see whether this behavior is consistent or intermittent. Also the igb(4) driver is using iflib(4). So the involved netmap code is > the same as ix(4) and if_axp(4). > This is something that I'm not able to understand right now. > It does not look like something related to offloads. > > Next week I will try to see if I can reproduce your issue with em(4), and > report back. That's still an Intel driver using iflib(4). > The "igb(4)" driver, with which things are working now is related to em(4) driver (may be for newer hardware version). Initially we faced similar issue with igb(4) driver as well. After reverting the following commits, things started to work. Thanks to Stephan Dewt (copied) for pointing this. But it still fails with ix(4) driver and if_axp(4) driver. https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf Thanks for providing your inputs on this issue Vincenzo. Let me know for any more details that you need. Thanks, Rajesh.