Date: Tue, 05 Dec 2023 14:21:21 +0000 From: Benoit Chesneau <benoitc@enki-multimedia.eu> To: Kristof Provost <kp@FreeBSD.org> Cc: freebsd-net@freebsd.org Subject: Re: vxlan with IPv6 underlay ? Message-ID: <Q4lNxBr17sBg-pHCFTyIugCDvSpd0oXXH6UBv5yEkJtq2lJKIXG_K4MSDarhwkFt7PAFvaLTLdm-rj6MR-ix7B8BfG-7aINXUg5h2a5VkPg=@enki-multimedia.eu> In-Reply-To: <793EA8DF-BD26-414A-A12D-866356D339DB@FreeBSD.org> References: <bmkPdpqYkXzCqs5CYfBIZO6sXwmR1hU38pCr5cRlBeTxM4VIbzTj_A0YU6yFP2CbVxXHoOLeqHpdBY1reGJS_VvNBesk3J1HBWblpjijl2o=@enki-multimedia.eu> <793EA8DF-BD26-414A-A12D-866356D339DB@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Changing the mtu wasn't enough. To make it works i had to disable vxlan offload features:
```
ifconfig mce0 -vxlanhwcsum -vxlanhwtso
```
Then suddenly all started to work. The card is the following:
```
mlx5_core1@pci0:17:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx]'
class = network
subclass = ethernet
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
```
Which is odd. Wonder why it's needed. Thoughts?
Benoît
On Tuesday, December 5th, 2023 at 13:11, Kristof Provost <kp@FreeBSD.org> wrote:
> On 4 Dec 2023, at 11:25, Benoit Chesneau wrote:
>
> > Hi,
> >
> > Is IPv6 underlay fully supported with FreebBSD ? I have created the a tunnel and associated an Ipv6 address to each side. I'm able to ping between each devicesl. But when I want to curl from the remote side it timeout. Locally on the remote side it is OK. Is this expected ? Should I rather create a bridge with vxlan as a member and bind nginx to it ?
> >
> > ```
> > $ ifconfig vxlan0 create vxlanid 108 vxlanlocal AAAA:BBBB:110b:102::100 vxlanremote AAAA:BBBB:110b:102::12
> > $ ifconfig vxlan0
> > vxlan0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1430
> > options=80020<JUMBO_MTU,LINKSTATE>
> > ether 58:9c:fc:10:ff:eb
> > groups: vxlan
> > vxlan vni 108 local [AAAA:BBBB:102::100]:4789 remote [AAAA:BBBB:110b:102::12]:4789
> > media: Ethernet autoselect (autoselect <full-duplex>)
> > status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > $ ifconfig vxlan0 inet6 AAAA:BBBB:110b:300::1/64
> > ````
> >
> > Ping from remote is ok:
> > `ifconfig vxlan0 vxlan0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1430 options=680323<RXCSUM,TXCSUM,JUMBO_MTU,TSO4,TSO6,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 58:9c:fc:10:df:1f inet6 fe80::5a9c:fcff:fe10:df1f%vxlan0 prefixlen 64 scopeid 0xf inet6 AAAA:BBBB:110b:300::2 prefixlen 64 groups: vxlan vxlan vni 108 local [AAAA:BBBB110b:102::12]:4789 remote [AAAA:BBBB:110b:102::100]:4789 media: Ethernet autoselect (autoselect <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> $ ping6 AAAA:BBBB:110b:300::1 PING6(56=40+8+8 bytes) AAAA:BBBB:110b:300::2 --> AAAA:BBBB::110b:300::1 16 bytes from 2a0e:e701:110b:300::1, icmp_seq=0 hlim=64 time=0.071 ms 16 bytes from 2a0e:e701:110b:300::1, icmp_seq=1 hlim=64 time=0.078 ms 16 bytes from 2a0e:e701:110b:300::1, icmp_seq=2 hlim=64 time=0.076 ms 16 bytes from 2a0e:e701:110b:300::1, icmp_seq=3 hlim=64 time=0.104 ms 16 bytes from 2a0e:e701:110b:300::1, icmp_seq=4 hlim=64 time=0.077 ms^C`
> >
> > But when I run `curl -6 -v 'http://[AAAA:BBBB:110b:300::1]'` it timeout.
>
> That smells like an MTU issue. Make sure you’re not blocking ICMPv6, and check with larger ping messages.
>
> Best regards,
> Kristof
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Q4lNxBr17sBg-pHCFTyIugCDvSpd0oXXH6UBv5yEkJtq2lJKIXG_K4MSDarhwkFt7PAFvaLTLdm-rj6MR-ix7B8BfG-7aINXUg5h2a5VkPg=>