From nobody Mon Mar 17 18:25:02 2025 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZGk0r53gFz5qTky for ; Mon, 17 Mar 2025 18:25:16 +0000 (UTC) (envelope-from sm@codenetworks.net) Received: from relayout02-q03.dominioabsoluto.net (relayout02-q03.dominioabsoluto.net [217.116.26.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZGk0q75Blz3rPy for ; Mon, 17 Mar 2025 18:25:15 +0000 (UTC) (envelope-from sm@codenetworks.net) Authentication-Results: mx1.freebsd.org; none Received: from relayout02-redir.dominioabsoluto.net (relayout02-redir.dominioabsoluto.net [217.116.26.75]) by relayout02.dominioabsoluto.net (Postfix) with ESMTP id 4ZGk0n4R60zlWTG; Mon, 17 Mar 2025 19:25:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codenetworks.net; s=domabs; t=1742235913; bh=iIRYSBo6Kw+zs10oKYqMOTByPzLHKT3K7JQzlArbEl4=; h=From:Subject:Date:References:Cc:In-Reply-To:To:From; b=Mz8THXYnx5VobqEZ3annQcV/KV0Ugs9wYl45I7gkX6slaP97xt148XiTdEtMy2Cij bHQ8pw+7lpgzXzpMHVdrwctvDV+yaPTz3MAQk46+P0EkQ4spKdimzXFtu7V5MgY39n 6GcbQnWVsisBqDqR2o5p7nmz+4wEjrr8tvYDFWQk= Received: from smtpclient.apple (unknown [188.241.98.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: sm.codenetworks.net) by relayout02-dsp.dominioabsoluto.net (Postfix) with ESMTPSA id 4ZGk0n10vxzlWTG; Mon, 17 Mar 2025 19:25:13 +0100 (CET) Content-Type: multipart/alternative; boundary=Apple-Mail-F8EEAA0E-4FA5-4658-8DD3-FBC6BA419194 Content-Transfer-Encoding: 7bit From: Santiago Martinez List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: issue while bridging VF on mellanox Date: Mon, 17 Mar 2025 19:25:02 +0100 Message-Id: <1FC405CE-31D7-4CF8-9EB6-65B1C631385F@codenetworks.net> References: <4AC80160-388D-4E85-8349-018FC5D47562@enki-multimedia.eu> Cc: freebsd-net@freebsd.org In-Reply-To: <4AC80160-388D-4E85-8349-018FC5D47562@enki-multimedia.eu> To: benoitc@enki-multimedia.eu X-Mailer: iPhone Mail (22D72) X-PostalOut-Country: IP: 188.241.98.123 | Country: ES X-PostalOut-Information: AntiSPAM and AntiVIRUS on relayout02 X-PostalOut-MsgID: 4ZGk0n10vxzlWTG.A9CEE X-PostalOut-SpamCheck: no es spam, clean X-PostalOut-From: sm@codenetworks.net X-PostalOut-Watermark: 1742840713.4556@fki0QKX2rDFB/OFpIWAoWg X-Spam-Status: No X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16371, ipnet:217.116.24.0/21, country:ES] X-Rspamd-Queue-Id: 4ZGk0q75Blz3rPy X-Spamd-Bar: ---- --Apple-Mail-F8EEAA0E-4FA5-4658-8DD3-FBC6BA419194 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hey, haven=E2=80=99t tested with SR-IOV but can test it later if you want. Santi > On 17 Mar 2025, at 17:50, benoitc@enki-multimedia.eu wrote: >=20 > =EF=BB=BFOh, i need to check if the vxlan works correctly indeed... >=20 >> So you are also using sr-iov ?=20 >=20 > Beno=C3=AEt >=20 >> On 17 Mar 2025, at 17:08, Santiago Martinez wrote: >>=20 >> Hi Benoit, one more thing, Im not sure whats the current support for v6, s= pecially for ND. >>=20 >> Best regards. >>=20 >> Santi >>=20 >>=20 >>=20 >> On 3/17/25 16:48, Santiago Martinez wrote: >>> Hi Benoit, >>>=20 >>> I have a similar setup to yours (not using MLX4), and it works. >>>=20 >>> I have two VMs acting as routers (R-A and R-B), each with two interfaces= . >>>=20 >>> On Router-A: >>>=20 >>> VTNET1 =E2=86=92 P2P to Router-B. >>> VTNET0 =E2=86=92 Sub-interface (VLAN 333) - Client-A. >>> VXLAN0 =E2=86=92 Towards Router-B. >>> BR-TEST =E2=86=92 VXLAN0 and VTNET0.333 as members. >>> On Router-B: >>>=20 >>> VTNET1 =E2=86=92 P2P to Router-A. >>> VTNET0 =E2=86=92 Untagged - Client-B. >>> VXLAN0 =E2=86=92 Towards Router-A. >>> BR-TEST =E2=86=92 VXLAN0 and VTNET0 as members. >>> Traffic passes. >>>=20 >>> One thing to note=E2=80=94VXLAN0 will appear as UP/Active even if it=E2=80= =99s not reaching the other side. >>>=20 >>> Best regards, >>> Santi >>>=20 >>>=20 >>>=20 >>> On 3/17/25 15:32, Benoit Chesneau wrote: >>>> Hi all, >>>>=20 >>>> I'm trying to make some kind of cross connect between a vlan and a vxla= n, but as far as I understand the mac address are not learnt from the vlan W= hen I listen using tcpdump on the vlan i can see the arp request but not in t= he in the vxlan. ANy idea how should I fix this? Is there any other way to d= o a cross conect or shoud I go back to not use sriov when i need to build br= idges? >>>>=20 >>>>=20 >>>> Any idea is welcome. >>>>=20 >>>> Benoit >>>>=20 >>>> More details follow >>>>=20 >>>> The vlan is created over a VF device in a bhyve vm: >>>>=20 >>>> vlan601: flags=3D1008943 metric 0 mtu 1356 >>>> options=3D18080000 >>>> ether 4a:75:f1:f5:a5:0a >>>> inet6 fe80::4875:f1ff:fef5:a50a%vlan601 prefixlen 64 scopeid 0x= 5 >>>> groups: vlan >>>> vlan: 601 vlanproto: 802.1q vlanpcp: 0 parent interface: mce0 >>>> media: Ethernet 25GBase-SR >>>> status: active >>>> nd6 options=3D23 >>>>=20 >>>> The vxlan is >>>>=20 >>>> vxlan0: flags=3D1008903 metric 0 mtu 1356 >>>> options=3D80020 >>>> ether 58:9c:fc:10:12:01 >>>> inet6 fe80::5a9c:fcff:fe10:1201%vxlan0 prefixlen 64 tentative s= copeid 0x7 >>>> groups: vxlan >>>> vxlan vni 400 local [2001:db8:1000::1]:4789 remote [2001:db8:10= 00::2]:4789 >>>> media: Ethernet autoselect (autoselect ) >>>> status: active >>>> nd6 options=3D23 >>>>=20 >>>>=20 >>>> And the bridge: >>>>=20 >>>> # ifconfig bridge0 >>>> bridge0: flags=3D1028943 metric 0 mtu 1356 >>>> options=3D0 >>>> ether 58:9c:fc:10:ff:d3 >>>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >>>> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >>>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >>>> member: vlan601 flags=3D143= >>>> ifmaxaddr 0 port 5 priority 128 path cost 800 >>>> member: vxlan0 flags=3D143 >>>> ifmaxaddr 0 port 7 priority 128 path cost 55 >>>> groups: bridge >>>> nd6 options=3D3 >>>>=20 >>>> The PF on the machine is a Mellanox [ConnectX-4 Lx >>>>=20 >>>> # pciconf -lvbce mlx5_core0 >>>> mlx5_core0@pci0:17:0:0: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vend= or=3D0x15b3 device=3D0x1015 subvendor=3D0x1590 subdevice=3D0x00d4 >>>> vendor =3D 'Mellanox Technologies' >>>> device =3D 'MT27710 Family [ConnectX-4 Lx]' >>>> class =3D network >>>> subclass =3D ethernet >>>> bar [10] =3D type Prefetchable Memory, range 64, base 0xe4000000,= size 33554432, enabled >>>> cap 10[60] =3D PCI-Express 2 endpoint max data 256(512) FLR RO NS >>>> max read 4096 >>>> link x8(x8) speed 8.0(8.0) ASPM L1(L1) >>>> cap 03[48] =3D VPD >>>> cap 11[9c] =3D MSI-X supports 64 messages, enabled >>>> Table in map 0x10[0x2000], PBA in map 0x10[0x3000] >>>> cap 09[c0] =3D vendor (length 24) >>>> cap 01[40] =3D powerspec 3 supports D0 D3 current D0 >>>> ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected >>>> ecap 000e[150] =3D ARI 1 >>>> ecap 0010[180] =3D SR-IOV 1 IOV enabled, Memory Space enabled, ARI e= nabled >>>> 8 VFs configured out of 16 supported >>>> First VF RID Offset 0x0002, VF RID Stride 0x0001 >>>> VF Device ID 0x1016 >>>> Page Sizes: 4096 (enabled), 8192, 16384, 32768, 65= 536, 131072, 262144, 524288, 1048576, 2097152, 4194304 >>>> iov bar [1a4] =3D type Prefetchable Memory, range 64, base 0xc7ffe= 000000, size 1048576, enabled >>>> ecap 0019[1c0] =3D PCIe Sec 1 lane errors 0 >>>> ecap 000d[230] =3D ACS 1 Source Validation unavailable, Translation= Blocking unavailable >>>> P2P Req Redirect unavailable, P2P Cmpl Redirect un= available >>>> P2P Upstream Forwarding unavailable, P2P Egress Co= ntrol unavailable >>>> P2P Direct Translated unavailable, Enhanced Capabi= lity unavailable >>>> PCI-e errors =3D Correctable Error Detected >>>> Unsupported Request Detected >>>> Corrected =3D Advisory Non-Fatal Error >>>>=20 >>>> Other side is the same except that the card is an intel one: >>>>=20 >>>> # pciconf -lvbce ixl0 >>>> ixl0@pci0:23:0:0: class=3D0x020000 rev=3D0x02 hdr=3D0x00 vendor=3D0x8= 086 device=3D0x1572 subvendor=3D0x15d9 subdevice=3D0x093b >>>> vendor =3D 'Intel Corporation' >>>> device =3D 'Ethernet Controller X710 for 10GbE SFP+' >>>> class =3D network >>>> subclass =3D ethernet >>>> bar [10] =3D type Prefetchable Memory, range 64, base 0xc4800000,= size 8388608, enabled >>>> bar [1c] =3D type Prefetchable Memory, range 64, base 0xc5808000,= size 32768, enabled >>>> cap 01[40] =3D powerspec 3 supports D0 D3 current D0 >>>> cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks >>>> cap 11[70] =3D MSI-X supports 129 messages, enabled >>>> Table in map 0x1c[0x0], PBA in map 0x1c[0x1000] >>>> cap 10[a0] =3D PCI-Express 2 endpoint max data 256(2048) FLR RO >>>> max read 512 >>>> link x8(x8) speed 8.0(8.0) >>>> cap 03[e0] =3D VPD >>>> ecap 0001[100] =3D AER 2 0 fatal 0 non-fatal 1 corrected >>>> ecap 0003[140] =3D Serial 1 b8f678ffff55c27c >>>> ecap 000e[150] =3D ARI 1 >>>> ecap 0010[160] =3D SR-IOV 1 IOV enabled, Memory Space enabled, ARI e= nabled >>>> 16 VFs configured out of 64 supported >>>> First VF RID Offset 0x0010, VF RID Stride 0x0001 >>>> VF Device ID 0x154c >>>> Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1= 048576, 4194304 >>>> iov bar [184] =3D type Prefetchable Memory, range 64, base 0xc5000= 000, size 65536, enabled >>>> iov bar [190] =3D type Prefetchable Memory, range 64, base 0xc5100= 000, size 16384, enabled >>>> ecap 0017[1a0] =3D TPH Requester 1 >>>> ecap 000d[1b0] =3D ACS 1 Source Validation unavailable, Translation= Blocking unavailable >>>> P2P Req Redirect unavailable, P2P Cmpl Redirect un= available >>>> P2P Upstream Forwarding unavailable, P2P Egress Co= ntrol unavailable >>>> P2P Direct Translated unavailable, Enhanced Capabi= lity unavailable >>>> ecap 0019[1d0] =3D PCIe Sec 1 lane errors 0 >>>> PCI-e errors =3D Correctable Error Detected >>>> Unsupported Request Detected >>>> Corrected =3D Advisory Non-Fatal Error >=20 --Apple-Mail-F8EEAA0E-4FA5-4658-8DD3-FBC6BA419194 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hey= , haven=E2=80=99t tested with SR-IOV but can test it later if you want.
Santi

On 17 Mar 2025, at 17:50, benoitc@enki-multimedia= .eu wrote:

=EF=BB=BFOh, i need to check if the vxlan works correctly indeed...
So you are also using sr-iov ? <= /span>

Beno=C3=AEt

On 17 Mar 2025, at 17:08, Santiago Martinez <= sm@codenetworks.net> wrote:

=
=20 =20

Hi Benoit, one more thing, Im not sure whats the current support for v6, specially for ND.

Best regards.

Santi


On 3/17/25 16:48, Santiago Martinez wrote:

Hi Benoit,

I have a similar setup to yours (not using MLX4), and it works.

I have two VMs acting as routers (R-A and R-B), each with two interfaces.

On Router-A:

  • VTNET1 =E2=86=92 P2P to Rout= er-B.
  • VTNET0 =E2=86=92 Sub-interfa= ce (VLAN 333) - Client-A.
  • VXLAN0 =E2=86=92 Towards Rou= ter-B.
  • BR-TEST =E2=86=92 VXLAN0 and= VTNET0.333 as members.

On Router-B:

  • VTNET1 =E2=86=92 P2P to Rout= er-A.
  • VTNET0 =E2=86=92 Untagged - Client-B.
  • VXLAN0 =E2=86=92 Towards Rou= ter-A.
  • BR-TEST =E2=86=92 VXLAN0 and= VTNET0 as members.

Traffic passes.

One thing to note=E2=80=94VXLAN0 will appear as UP/Active even if it=E2=80=99s not reaching the other side= .

Best regards,
Santi


On 3/17/25 15:32, Benoit Chesneau wrote:
Hi all,

I'm trying to make some kind of cross connect between a vlan and a vxlan, bu=
t as far as I  understand the mac address are not learnt from the vlan When I=
 listen using tcpdump on the vlan i can see the arp request but not in the i=
n the vxlan. ANy idea how should I fix this? Is there any other way to do a c=
ross conect or shoud I go back to not use sriov when i need to build bridges=
?


Any idea is welcome.

Benoit

More details follow

The vlan is created over a VF device in  a bhyve vm:

vlan601: flags=3D1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,L=
OWER_UP> metric 0 mtu 1356
        options=3D18080000<LINKSTATE,TXTLS4,TXTLS6>
        ether 4a:75:f1:f5:a5:0a
        inet6 fe80::4875:f1ff:fef5:a50a%vlan601 prefixlen 64 scopeid 0x5
        groups: vlan
        vlan: 601 vlanproto: 802.1q vlanpcp: 0 parent interface: mce0
        media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause>
        status: active
        nd6 options=3D23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

The vxlan is

vxlan0: flags=3D1008903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST,LOWER_UP&g=
t; metric 0 mtu 1356
        options=3D80020<JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:10:12:01
        inet6 fe80::5a9c:fcff:fe10:1201%vxlan0 prefixlen 64 tentative scopei=
d 0x7
        groups: vxlan
        vxlan vni 400 local [2001:db8:1000::1]:4789 remote [2001:db8:1000::2=
]:4789
        media: Ethernet autoselect (autoselect <full-duplex>)
        status: active
        nd6 options=3D23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


And the bridge:

# ifconfig bridge0
bridge0: flags=3D1028943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,P=
PROMISC,LOWER_UP> metric 0 mtu 1356
        options=3D0
        ether 58:9c:fc:10:ff:d3
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan601 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>=
;
                ifmaxaddr 0 port 5 priority 128 path cost 800
        member: vxlan0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>=

                ifmaxaddr 0 port 7 priority 128 path cost 55
        groups: bridge
        nd6 options=3D3<PERFORMNUD,ACCEPT_RTADV>

The PF on the machine is a Mellanox [ConnectX-4 Lx

#  pciconf -lvbce mlx5_core0
mlx5_core0@pci0:17:0:0:	class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1=
5b3 device=3D0x1015 subvendor=3D0x1590 subdevice=3D0x00d4
    vendor     =3D 'Mellanox Technologies'
    device     =3D 'MT27710 Family [ConnectX-4 Lx]'
    class      =3D network
    subclass   =3D ethernet
    bar   [10] =3D type Prefetchable Memory, range 64, base 0xe4000000, size=
 33554432, enabled
    cap 10[60] =3D PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 4096
                 link x8(x8) speed 8.0(8.0) ASPM L1(L1)
    cap 03[48] =3D VPD
    cap 11[9c] =3D MSI-X supports 64 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x3000]
    cap 09[c0] =3D vendor (length 24)
    cap 01[40] =3D powerspec 3  supports D0 D3  current D0
    ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 000e[150] =3D ARI 1
    ecap 0010[180] =3D SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabl=
ed
                     8 VFs configured out of 16 supported
                     First VF RID Offset 0x0002, VF RID Stride 0x0001
                     VF Device ID 0x1016
                     Page Sizes: 4096 (enabled), 8192, 16384, 32768, 65536, 1=
31072, 262144, 524288, 1048576, 2097152, 4194304
    iov bar  [1a4] =3D type Prefetchable Memory, range 64, base 0xc7ffe00000=
0, size 1048576, enabled
    ecap 0019[1c0] =3D PCIe Sec 1 lane errors 0
    ecap 000d[230] =3D ACS 1 Source Validation unavailable, Translation Bloc=
king unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavail=
able
                     P2P Upstream Forwarding unavailable, P2P Egress Control=
 unavailable
                     P2P Direct Translated unavailable, Enhanced Capability u=
navailable
  PCI-e errors =3D Correctable Error Detected
                 Unsupported Request Detected
     Corrected =3D Advisory Non-Fatal Error

Other side is the same except that the card is an intel one:

#  pciconf -lvbce ixl0
ixl0@pci0:23:0:0:	class=3D0x020000 rev=3D0x02 hdr=3D0x00 vendor=3D0x8=
086 device=3D0x1572 subvendor=3D0x15d9 subdevice=3D0x093b
    vendor     =3D 'Intel Corporation'
    device     =3D 'Ethernet Controller X710 for 10GbE SFP+'
    class      =3D network
    subclass   =3D ethernet
    bar   [10] =3D type Prefetchable Memory, range 64, base 0xc4800000, size=
 8388608, enabled
    bar   [1c] =3D type Prefetchable Memory, range 64, base 0xc5808000, size=
 32768, enabled
    cap 01[40] =3D powerspec 3  supports D0 D3  current D0
    cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks
    cap 11[70] =3D MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] =3D PCI-Express 2 endpoint max data 256(2048) FLR RO
                 max read 512
                 link x8(x8) speed 8.0(8.0)
    cap 03[e0] =3D VPD
    ecap 0001[100] =3D AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] =3D Serial 1 b8f678ffff55c27c
    ecap 000e[150] =3D ARI 1
    ecap 0010[160] =3D SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabl=
ed
                     16 VFs configured out of 64 supported
                     First VF RID Offset 0x0010, VF RID Stride 0x0001
                     VF Device ID 0x154c
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 104857=
6, 4194304
    iov bar  [184] =3D type Prefetchable Memory, range 64, base 0xc5000000, s=
ize 65536, enabled
    iov bar  [190] =3D type Prefetchable Memory, range 64, base 0xc5100000, s=
ize 16384, enabled
    ecap 0017[1a0] =3D TPH Requester 1
    ecap 000d[1b0] =3D ACS 1 Source Validation unavailable, Translation Bloc=
king unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavail=
able
                     P2P Upstream Forwarding unavailable, P2P Egress Control=
 unavailable
                     P2P Direct Translated unavailable, Enhanced Capability u=
navailable
    ecap 0019[1d0] =3D PCIe Sec 1 lane errors 0
  PCI-e errors =3D Correctable Error Detected
                 Unsupported Request Detected
     Corrected =3D Advisory Non-Fatal Error

= --Apple-Mail-F8EEAA0E-4FA5-4658-8DD3-FBC6BA419194--