From owner-freebsd-net@freebsd.org Wed Nov 18 21:46:10 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1C767472DD9; Wed, 18 Nov 2020 21:46:10 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CbxFt0Gc0z4hrg; Wed, 18 Nov 2020 21:46:10 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id EA9DF2C9A5; Wed, 18 Nov 2020 21:46:09 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-pf1-f177.google.com with SMTP id a18so2413021pfl.3; Wed, 18 Nov 2020 13:46:09 -0800 (PST) X-Gm-Message-State: AOAM532GSNJZo0efBBU41tldn3rvVA075bcigFQPiHamd/JqC4x/eTwD If+gojiP2SzFDpDKJ0lHVVopfybc2Uf21he94+U= X-Google-Smtp-Source: ABdhPJwd5LIx8MTO2po4EDgctCDYy5Tp8Keb4XL9jqsR9Z4TT6BcWak7AbyNskdpICc9+ezpsY5cMhPHoRrfhRURKU8= X-Received: by 2002:a17:90a:e016:: with SMTP id u22mr1087315pjy.54.1605735968876; Wed, 18 Nov 2020 13:46:08 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Vincenzo Maffione Date: Wed, 18 Nov 2020 22:45:57 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Netmap bridge not working with 10G Ethernet ports To: Rajesh Kumar Cc: "freebsd-net@freebsd.org" , FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2020 21:46:10 -0000 Hi, Il giorno mer 18 nov 2020 alle ore 08:13 Rajesh Kumar ha scritto: > Hi, > > I am testing a 10G Network driver with Netmap "bridge" utility, where it > doesn't seem to work. Here is my setup details. > > *System under Test:* Running FreeBSD CURRENT. Has two inbuilt 10G NIC > ports. > *System 1:* Running Ubuntu, whose network port is connected to Port1 of > System Under Test > *System 2:* Running FreeBSD CURRENT, whose network port is connected to > Port 0 of System Under Test. > > Bridged the Port0 and Port1 of System Under Test using the Netmap "bridge" > utility. Able to see interfaces coming up active and Link UP. > # bridge -c -v -i netmap:ax0 -i netmap:ax1 > > This looks like if_axe(4) driver, and therefore there's no native netmap support, which means you are falling back on the emulated netmap adapter. Are these USB dongles? If so, how can they be 10G? > Then tried pinging from System 1 to System 2. It fails. > > *Observations:* > 1. ARP request from System 1 goes to bridge port 1 (netmap_rxsync) and then > forwarded to port 0 (netmap_txsync) > 2. ARP request is received in System 2 (via bridge port 0) and ARP reply is > being sent from System 2. > 3. ARP reply from System 2 seems to be not reaching bridge port 0 to get > forwarded to bridge 1 and hence to System 1. > 4. Above 3 steps happen 3 times for ARP resolution cycle and then fails. > Hence the ping fails. > > On Debugging, when the ARP reply is being sent from System 2, I don't see > any interrupt triggered on the bridge port 0 in system under test. > > In this kind of configuration it is mandatory to disable all the NIC offloads, because netmap does not program the NIC to honor them, e.g.: # ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 # ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 > Netstat in system under test, doesn't show any receive or drop counters > incremented. But as I understand netstat capture the stats above the netmap > stack. Hence not reflecting the counts. > Correct. > > *Note:* > a) I tried with another vendor 10G NIC card. It behaves the same way. So > this issue doesn't seem to be generic and not hardware specific. > Which driver are those NICs using? That makes the difference. I guess it's still a driver with no native netmap support, hence you are using the same emulated adapter. > b) Trying with another vendor 1G NIC card, things are working. So not > sure, what makes a difference here. The ports in System 1 and System 2 are > USB attached Ethernet capable of maximum speed of 1G. So does connecting > 1G to 10G bridge ports is having any impact? > I don't think so. On each p2p link the NICs will negotiate 1G speed. In any case, what driver was this one? > c) We have verified the same 10G driver with pkt-gen utility and things are > working. Facing issue only when using "bridge" utility. > That may be because pkt-gen does not care about checksums, whereas the TCP/IP stack does. Hence the need to disable offloads (see above). Cheers, Vincenzo > So, wondering how the ARP reply packet is getting lost here. Any ideas to > debug? > > Thanks, > Rajesh. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >