From owner-freebsd-net@freebsd.org Thu Nov 19 11:28:56 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 85A9E2EFFB1; Thu, 19 Nov 2020 11:28:56 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CcHWD3Bk7z4YBV; Thu, 19 Nov 2020 11:28:56 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-ot1-x334.google.com with SMTP id n89so4944999otn.3; Thu, 19 Nov 2020 03:28:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=mdN8Uo/HeiG8juRtBFCcYjEG+o6U53xrDKKFwZvI9T0=; b=uua+E2PKC6+RGSRusA9CHyGAfEzYh0Av6dRl5CwhN3KHgESf01b16A2FLeV19RTUH8 une2eyVgA79x6Gg9rEzgi2XKv9VZgCLTB2FexnqyxrIJmH6fH9aoXqsXeuuWFsOaeXi0 Pky7MSdTBbjwgNiW+aF94sPJB+GkldrrlvllmOga+0Ne1iM+RdbKIKaZUpM+sVnXAzeS e04fexwrqVFhCLqbJYqPRxQzoKY7+uLzRUYk0Db9rSB5vmJec1hcpSR/9fEepxuV7Uci rPZ7cWIWNW580am+in9FKAtLugOBVamEdqqs8RmElvYNdoJy/QVFnkf0mENqNxQbWKQu moCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=mdN8Uo/HeiG8juRtBFCcYjEG+o6U53xrDKKFwZvI9T0=; b=FJ56MakSDefmIY1I8nnQoQVpxPyiafHr5whLOYA7kfUaToHqotDYwmViaHMJTWoge7 sLCjp8As2AJFs5dhWeiUBOg0UkDgxRQwHVWCiVUDO344EKYaD4vA+dslbuerYufyPDfx 2CNYecDkD0lCc4ga+vg39QFZNK7Ou1Tw4t9esJoDBQpQpCQ7flr40XKYE9g6bjgRTqYH PgfARbdM7RXJaZQxRmBY9LnYgFjpr7qSQm/J6DsV3tYcvlUaX1QXjBgT5KHChy8j5/fK dr9SwtLfiUVQpUp3JKQHrhLlDaqLBoYSoN/paVxsm6AzUnLi1wjesXUh/B0dHrnDckz9 +Rrg== X-Gm-Message-State: AOAM530hxO1wieBKapU0q/+l2f+danQU6w8mC7QeYkqM3QBy+TmlLxdH L0J0XiTvTaxD2V0pi/QRUgA53r747Ga79dj9owcAdKzP1x8= X-Google-Smtp-Source: ABdhPJwavgMaWO4FJQyrT+L7nx8QmJ1QVCwQa70LGnucvWntFrdFK91+Y5X5Zq3phHPAJ5B7TKmXCLuWcBy8M3NHu4g= X-Received: by 2002:a9d:76d7:: with SMTP id p23mr9967605otl.180.1605785334727; Thu, 19 Nov 2020 03:28:54 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Rajesh Kumar Date: Thu, 19 Nov 2020 16:58:42 +0530 Message-ID: Subject: Re: Netmap bridge not working with 10G Ethernet ports To: Vincenzo Maffione Cc: "freebsd-net@freebsd.org" , FreeBSD Hackers X-Rspamd-Queue-Id: 4CcHWD3Bk7z4YBV X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Nov 2020 11:28:56 -0000 Hi Vincenzo, Thanks for your reply. On Thu, Nov 19, 2020 at 3:16 AM Vincenzo Maffione wrote: > > This looks like if_axe(4) driver, and therefore there's no native netmap > support, which means you are falling back on > the emulated netmap adapter. Are these USB dongles? If so, how can they be > 10G? > The Driver I am working with is "if_axp" (sys/dev/axgbe). This is AMD 10Gigabit Ethernet Driver. This is recently committed upstream. Yes, it doesn't have a Native netmap support, but uses the netmap stack which is existing already. These are inbuilt SFP ports with our test board and not USB dongles. Does Native netmap mean the hardware capability which needs to be programmed appropriately from driver side? Any generic documentation regarding the same? > In this kind of configuration it is mandatory to disable all the NIC > offloads, because netmap does not program the NIC > to honor them, e.g.: > > # ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 > # ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 > Earlier, I haven't tried disabling the Offload capabilities. But I tried now, but it still behaves the same way. ARP replies doesn't seem to reach the bridge (or dropped) to be forwarded. I will collect the details for AMD driver. Tried the same test with another 10G card (Intel "ix" driver) also exhibits similar behavior. Details below. > a) I tried with another vendor 10G NIC card. It behaves the same way. So >> this issue doesn't seem to be generic and not hardware specific. >> > > Which driver are those NICs using? That makes the difference. I guess it's > still a driver with no native netmap support, hence > you are using the same emulated adapter > I am using the "ix" driver (Intel 10G NIC adapter). I guess this driver also doesn't support Native Netmap. Please correct me if I am wrong. I tried disabling the offload capabilities with this device/driver and tested and still observed the netmap bridging fails. root@fbsd_cur# sysctl dev.ix.0 | grep tx_packets dev.ix.0.queue7.tx_packets: 0 dev.ix.0.queue6.tx_packets: 0 dev.ix.0.queue5.tx_packets: 0 dev.ix.0.queue4.tx_packets: 0 dev.ix.0.queue3.tx_packets: 0 dev.ix.0.queue2.tx_packets: 0 dev.ix.0.queue1.tx_packets: 0 *dev.ix.0.queue0.tx_packets: 3* root@fbsd_cur# sysctl dev.ix.0 | grep rx_packets dev.ix.0.queue7.rx_packets: 0 dev.ix.0.queue6.rx_packets: 0 dev.ix.0.queue5.rx_packets: 0 dev.ix.0.queue4.rx_packets: 0 dev.ix.0.queue3.rx_packets: 0 dev.ix.0.queue2.rx_packets: 0 dev.ix.0.queue1.rx_packets: 0 dev.ix.0.queue0.rx_packets: 0 root@fbsd_cur # sysctl dev.ix.1 | grep tx_packets dev.ix.1.queue7.tx_packets: 0 dev.ix.1.queue6.tx_packets: 0 dev.ix.1.queue5.tx_packets: 0 dev.ix.1.queue4.tx_packets: 0 dev.ix.1.queue3.tx_packets: 0 dev.ix.1.queue2.tx_packets: 0 dev.ix.1.queue1.tx_packets: 0 dev.ix.1.queue0.tx_packets: 0 root@fbsd_cur # sysctl dev.ix.1 | grep rx_packets dev.ix.1.queue7.rx_packets: 0 dev.ix.1.queue6.rx_packets: 0 dev.ix.1.queue5.rx_packets: 0 dev.ix.1.queue4.rx_packets: 0 dev.ix.1.queue3.rx_packets: 0 dev.ix.1.queue2.rx_packets: 0 dev.ix.1.queue1.rx_packets: 0 *dev.ix.1.queue0.rx_packets: 3* You can see "ix1" received 3 packets (ARP requests) from system 1 and transmitted 3 packets to system 2 via "ix0". But ARP reply from system 2 is not captured or forwared properly. You can see the checksum features disabled (except VLAN_HWCSIM) on both interfaces. And you can see both interfaces active and Link up. root@fbsd_cur # ifconfig -a ix0: flags=8862 metric 0 mtu 1500 options=48538b8 ether a0:36:9f:a5:49:90 media: Ethernet autoselect (100baseTX ) status: active nd6 options=29 ix1: flags=8862 metric 0 mtu 1500 options=48538b8 ether a0:36:9f:a5:49:92 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 > > b) Trying with another vendor 1G NIC card, things are working. So not >> sure, what makes a difference here. The ports in System 1 and System 2 >> are >> USB attached Ethernet capable of maximum speed of 1G. So does connecting >> 1G to 10G bridge ports is having any impact? >> > > I don't think so. On each p2p link the NICs will negotiate 1G speed. > In any case, what driver was this one? > This is "igb" driver. Intel 1G NIC Card. Thanks, Rajesh.