From nobody Thu Jan 25 09:06:59 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TLFML6YF4z587js for ; Thu, 25 Jan 2024 09:07:10 +0000 (UTC) (envelope-from freebsd-net@c0decafe.de) Received: from mail.c0decafe.de (mail.c0decafe.de [IPv6:2a01:4f8:222:100a::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.c0decafe.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TLFML2njcz4B7S; Thu, 25 Jan 2024 09:07:10 +0000 (UTC) (envelope-from freebsd-net@c0decafe.de) Authentication-Results: mx1.freebsd.org; none Received: from [172.17.30.254] (unknown [172.17.30.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.c0decafe.de (Postfix) with ESMTPSA id 180A9E592F; Thu, 25 Jan 2024 10:07:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=c0decafe.de; s=c0decafe.de; t=1706173620; bh=R4PL6kCcSY3GnlRQmjXTMNZ7nss9LHbvPWqL3sojKy8=; h=Date:Subject:To:References:From:Cc:In-Reply-To; b=JbCM5YXHIJGVi77+INS9BHsOoLyKv96mqk2rcaKJcH9lKJoEtOJb89GLnHmnc5PFG Vt6dZMo38p8rNYGfxftPUJvvzsj4q6M85QblOoJufOPGgxzkOdLlJjljI2giFtFgrz CfpWBo8wo9dV0n0M7imfuMsZf0gH6qyg2xzEgrJI= Content-Type: multipart/alternative; boundary="------------1W4CUCK5105E1AOz0qmc0SmV" Message-ID: <401ae00d-730c-4ac7-a18c-a2b1b75b3edf@c0decafe.de> Date: Thu, 25 Jan 2024 10:06:59 +0100 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: problem with ixl(4) and vlans Content-Language: en-US To: Zhenlei Huang References: <3779d6c1-48de-4941-9444-36d69890be26@c0decafe.de> <40DB5F45-6507-4274-BF28-6B9A5F45CAE9@FreeBSD.org> From: Daniel Autocrypt: addr=mail@c0decafe.de; keydata= xsFNBFuKxS8BEADGswc2TX/b65QzcDw+b/W30LGgJXEn/GUnV8SqTNI5G3LoJLzJkVZtXh21 ng3wkc13JBD7Vb1yC6NRrmFUf77Gq3uyDVnqaKreMmZRgds9/uLFHiYM8NNEm/IjdY3nbhlk +WW/Reae8PVk7lOrO0VNcy+rwm/hJF8hFGzzBCx4tEyVZw5O4FPGAiD/SkM+FD5veupcuzI8 qkuuaInuuP8oZ4fTqdJTd1JvkRLMvytqHBx744v2Pg9Oos0ucxXYpTMXVIYO1S3KxJFyiRuO sxc+jtEft/2VwMNVp2wFssboHIMc9rKziJVHadfqBY4LxqVPhgbExjK3f65RmvZ/d/dEsF3I mTgyr4sIi1OBFLapxLdzzH1QYc6HVDVMDnndckS/4spHCKnCsPsKkeEx+L2lwAlr6PwE9xMi BtM+2NjMXN4g6u9SmIxoIHcIDOQwpmtFp70UB3SBZlIParjmZysIN9fSOtzBksuSUFGJ7PP+ lo/96PW1u6OBDocRBdtTD5710d9tsGluU4S86BVdytn/u/QAoEviae29arUYIGsMR9oeJhJc wEqRS4KfEuMhzjhJQDyJ4fLE7MKB5rGPdvhIhY7XqjKVTBIoRABZzYaPeHDvLyP83G0qye3l mwE6E8xsDPBZ4cFgZPl6xkLWugHiRVkE8VsY/RM3+v323ZvT9QARAQABzRlEYW5pZWwgPG1h aWxAYzBkZWNhZmUuZGU+wsGUBBMBCgA+FiEEw5F1cA3yVgaiOsGl9pDM4NUKhiIFAluKxS8C GyMFCRLMAwAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQ9pDM4NUKhiKRLw//XhDaN9QY o5lotBPWJSosFCdiEJBgMKfIWFmg8HgKbRVbxPV025zFXYU2WEMcymMw0cZEMCS3KtsvsyTv M04Qi5U5fxGs8RWTi8VuclUZQ96wJjWu9yDIL0jfOyzibq2Gi4zE2DE5zGd27Zpe/mHNlpmO 6wojZur5H5VaP+Mfeb5vq7D8oJJwLI7/qtZgesJzIv2rDy5og8CyCjM9uZKAAWTxCV6j6qUX K9DNanx16NBT3bTGA1AxqkvuogRmbY+GWqb+Jjtz0uOdUY4BQhW3rM30kOZybtmResaHwtaT EQm6YVKns2rAFzVS4Vcn9rYB4RJ5ovUCjk90tBXZrR2XvWJenckC+9oOkc4uoYF28lOzFoBa C/P4DzlVag5AiJTF7R2mHCnOj3wBKQqV1VhyHnqHGpVbwPEAbm0RX97lAn5cSLJ9ZFCY+Hin 1Ssh1hjA0McQNs9uFSyNj8HSfwenn5wqA6zwJzCaYONPNWZs3ejajCOOHyMoSswbkC/uupPA GvFaZdb34n6C3ShuUfr6DqbTbp+xER2sl/QQnrZ5cpPwEo4bG4aaEi0T0LbLcCZgVLQzC3iw BfjOHKjx9AIEOENbVjw5aTAEA6IPPJVvqMIjTy9lbTO/dteOpHfQ4lbeMyenYMVmHeiTeNA3 yV9KKWkrdr0C1VvpDM9/Z9uHMWHOwU0EW4rFLwEQALPCXxjWvvbQca6wmu8GzBZ0ymUd2eDq B7CTJmfR7FwvAIzNslGKH+kUIF18EnfDYiTBlsJCrCLaRsrGtUfNOmti0qVi/fjfllasicpp S+oxC2yBcGvGhm3/HH9g9HkezvEM4QOjsPjKLZMi+suVYRipaSJCf3RGRdCh9vxmN0MLRMoe 1jHVr94BvgQo4ktaFn8ACl2Z5351uOmedL1y3/LgmUyCm/vCa9z5YRb9sp3h5YQyEupFae2x i8NoDrm0StHcKAGucz04DwV2hC2NWL9UfJKhqC1l8gx7NkcohE8nEbMmRcuc9vhGUPquIMm3 ejvd5XGetN7lKC+5YVeG0e7kiCPIwNnON22dEMWv1bCuYXUoLDkejlKCqX+n8xC8Ddao+5t1 7oLXYQJqTaFKz67kxKJG/gXRBDXWrkdQ/7tE7qZbHTqaREEXRA1SYLxJGCNSUzt4kkq0kuvB AI6f72EYj5GZPKENIRZAbHv6pDy9qd9f6cB+qj1Cz7zqfDPLpN228nV88T0Fq5QH4s9KLXSs vT4nyKz5p0I+u4IydOKLwEqcjlqMNwIaBEZTjD8UFNnd8mCJcwn7uZiKWobjU2jJRU2SmROP /WjDHEff2SmJws7FVVmoyPl/FRD4QYDtuu9qwqh1tU1ut73PXxMoB/IHNkjBe5g++R1tvXWE fF0ZABEBAAHCwXwEGAEKACYWIQTDkXVwDfJWBqI6waX2kMzg1QqGIgUCW4rFLwIbDAUJEswD AAAKCRD2kMzg1QqGIoofD/9tB2wD6G7nAL9mWEC79Kusioh2HNn7HqUnB0HcGb6jD9coD8BK 1Io/1Z//slcw9+2FFwP4AXv0DTmYhb/zNX/SPrTQqSP2TPgaecVyIHzK5r5hqNE6nponvUVe jJfIzTJ5r+eKxuuMi/BkltZC98IuQV0PRLHlrVRKHWgRH/YGyFZ/OXRGeoxwZhkFE3ftwr+o ORz3ZZJIGjJpQGK3ujZ328qiswfNN3g65GoW2HU9LlWul6UgM8lFdIfOmvZqzGPnHEvPzidi q+/dezdTWFJtifQHjdpWYNnO2141N+1fU+tH7dt4TxdN1KP600BXvD29jWxPnnA6fSOyAxYT wdZOrDwrftWMF97dOIVrvhnC3Sm7PwtK639ksP2eCHWn532S4A3Ikb7xoPnuFeAXM9o9/9In yqKixsI+JodUY+wpTJsqkvhzPtrnzH+rwPSB9mpnbajRqVZ0qR0n2z8IkYhvqhIOwD5BBj// RjfZNvxaQ94UFxyqbEWOnBv5osYbprO1eZwJnNvYuf0bCaYKJX/UM87GbgTimDp1jolmJ0Nw KZRkdfs+a1j480Xt/Pyzr/muuGmFRk7/gj1tRkxAeKSdsUW88il88ZXPEwDT0BRzXo51tiiu dSHhtMS24Qzevf3cP93tAanM4Xni6Tu10ZveOALJYC5T161VfZAur0yJBA== Cc: freebsd-net@freebsd.org In-Reply-To: <40DB5F45-6507-4274-BF28-6B9A5F45CAE9@FreeBSD.org> X-Rspamd-Queue-Id: 4TLFML2njcz4B7S X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] This is a multi-part message in MIME format. --------------1W4CUCK5105E1AOz0qmc0SmV Content-Type: text/plain; charset=UTF-8; format=flowed X-Clacks-Overhead: GNU Terry Pratchett Content-Transfer-Encoding: 8bit Hi, thanks for your suggestion. Turns out, when i unplug the vlan interface from the bridge and put the ip address on the vlan interface, as you suggested, things start to work, e.g. arp resolves. as soon as i put the ip and the vlan interface back on the bridge, things stop again. so where does this lead me? the problem is not in the vlan handling, but on the bridge? I started playing with the net.link.bridge sysctls and indeed, when i set # sysctl net.link.bridge.inherit_mac=1 and then recreate the bridge # ifconfig bridge0 deletem ixl3.15 deletem vnet0.1 # ifconfig bridge0 addm ixl3.15 addm vnet0.1 with the ip address still on bridge0 and ixl3, ixl3.15 and bridge0 all sharing the same mac address, arp starts resolving. but only for requests sent from the bridge0 interface. inside of the jail things still don't work (as the vnet interface again has another mac address). # ifconfig ixl3 ixl3: flags=28963 metric 0 mtu 1500 options=4a500b9         ether a4:bf:01:76:ef:9d         media: Ethernet autoselect (10Gbase-SR )         status: active         nd6 options=29 # ifconfig ixl3.15 ixl3.15: flags=8943 metric 0 mtu 1500         options=4200001         ether a4:bf:01:76:ef:9d         groups: vlan         vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3         media: Ethernet autoselect (10Gbase-SR )         status: active         nd6 options=29 # ifconfig bridge0 bridge0: flags=8843 metric 0 mtu 1500         ether a4:bf:01:76:ef:9d         inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255         id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15         maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200         root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0         member: vnet0.1 flags=143                 ifmaxaddr 0 port 9 priority 128 path cost 2000         member: ixl3.15 flags=143                 ifmaxaddr 0 port 8 priority 128 path cost 2000         groups: bridge         nd6 options=9 # ping 192.168.55.1 PING 192.168.55.1 (192.168.55.1): 56 data bytes ^C --- 192.168.55.1 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss [! yes, the host does not answer on ICMP, but that is to be expected !] # arp -an ? (192.168.55.20) at a4:bf:01:76:ef:9d on bridge0 permanent [bridge] ? (192.168.55.1) at b8:27:eb:47:8f:43 on bridge0 expires in 1197 seconds [bridge] [...] [! into the jail !] JAIL # ifconfig epair0b epair0b: flags=8863 metric 0 mtu 1500         options=8         ether ac:16:2d:bd:b7:34         hwaddr 02:51:73:d1:33:0b         inet 192.168.55.10 netmask 0xffffff00 broadcast 192.168.55.255         inet6 fe80::ae16:2dff:febd:b734%epair0b prefixlen 64 scopeid 0x2         groups: epair         media: Ethernet 10Gbase-T (10Gbase-T )         status: active         nd6 options=21 JAIL # ping 192.168.55.1 PING 192.168.55.1 (192.168.55.1): 56 data bytes ^C --- 192.168.55.1 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss JAIL # arp -an ? (192.168.55.10) at ac:16:2d:bd:b7:34 on epair0b permanent [ethernet] ? (192.168.55.1) at (incomplete) on epair0b expired [ethernet] I conclude that there must be some mac address filtering going on in the data path, whether its on ixl or the bridge. In dmesg I also see: >> bridge0: can't disable some capabilities on ixl3.15: 0x400 but as of /usr/src/sys/net/if.h:233 this maps to IFCAP_LRO which afaik should not have any influence on L2 filtering. Have to say, I'm out of ideas again. Never had something like this. So far just 'throwing interfaces on a bridge' worked in the past. Any ideas where to look next? Thanks a lot & best Daniel On 1/25/24 08:22, Zhenlei Huang wrote: > > >> On Jan 23, 2024, at 11:03 PM, Daniel wrote: >> >> Hi List, >> >> >> just recently I discovered a problem with the ixl(4) driver. >> Hopefully someone here can help me. my setup is as follows: >> >> >> Network ----- ixl3 interface ----- ixl3.15 vlan interface ----- >> bridge0 ----- vnet0.1 to jail >> >> >> the problem now is that the jail can send data out (arp requests), i >> do see the responses on the ixl3 interface of the host, but they >> never make their way up to the ixl3.15 vlan interface (even though >> they are tagged correctly). To rule out that my config or the network >> is the cruel pit i did test the same setup with a cheap usb-ethernet >> adapter and there everything works as expected. I'm on FreeBSD >> 13.2-RELEASE-p8 and I did test both, the in kernel driver and the >> driver from ports intel-ixl-kmod-1.13.4_1. >> > I would encourage you to do plain VLAN tests, i.e. plug ixl3.15 out > from bridge0 > > ``` > # ifconfig bridge0 deletem ixl3.15 > # ifconfig bridge0 inet 192.168.55.20/24 delete ### to prevent confusion > # ifconfig ixl3.15 inet 192.168.55.x/24 > # ping -c1 192.168.55.1 > ``` > >> >> Here is a bit of information on my environment: >> >> # uname -a >> FreeBSD mimir 13.2-RELEASE-p8 FreeBSD 13.2-RELEASE-p8 GENERIC amd64 >> >> # pciconf -lBbcevV pci0:25:0:3 >> ixl3@pci0:25:0:3:       class=0x020000 rev=0x09 hdr=0x00 >> vendor=0x8086 device=0x37d3 subvendor=0x8086 subdevice=0x35d5 >>     vendor     = 'Intel Corporation' >>     device     = 'Ethernet Connection X722 for 10GbE SFP+' >>     class      = network >>     subclass   = ethernet >>     bar   [10] = type Prefetchable Memory, range 64, base 0xb0000000, >> size 16777216, enabled >>     bar   [1c] = type Prefetchable Memory, range 64, base 0xb5000000, >> size 32768, enabled >>     cap 01[40] = powerspec 3  supports D0 D3  current D0 >>     cap 05[50] = MSI supports 1 message, 64 bit, vector masks >>     cap 11[70] = MSI-X supports 129 messages, enabled >>                  Table in map 0x1c[0x0], PBA in map 0x1c[0x1000] >>     cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO >>                  max read 512 >>                  link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) >>     cap 03[e0] = VPD >>     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected >>     ecap 0003[140] = Serial 1 9aef76ffff01bfa4 >>     ecap 000e[150] = ARI 1 >>     ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, >> ARI disabled >>                      0 VFs configured out of 32 supported >>                      First VF RID Offset 0x006d, VF RID Stride 0x0001 >>                      VF Device ID 0x37cd >>                      Page Sizes: 4096 (enabled), 8192, 65536, 262144, >> 1048576, 4194304 >>     ecap 0017[1a0] = TPH Requester 1 >>     ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation >> Blocking unavailable >>                      P2P Req Redirect unavailable, P2P Cmpl Redirect >> unavailable >>                      P2P Upstream Forwarding unavailable, P2P Egress >> Control unavailable >>                      P2P Direct Translated unavailable, Enhanced >> Capability unavailable >>   PCI-e errors = Correctable Error Detected >>                  Unsupported Request Detected >>      Corrected = Advisory Non-Fatal Error >>     VPD ident  = 'Example VPD' >> >> # ifconfig >> [...] >> ixl3: flags=8963 >> metric 0 mtu 1500 >> options=4a500b9 >>         ether a4:bf:01:76:ef:9d >>         media: Ethernet autoselect (10Gbase-SR ) >>         status: active >>         nd6 options=29 >> ixl3.15: flags=8942 >> metric 0 mtu 1500 >>         options=4200001 >>         ether a4:bf:01:76:ef:9d >>         groups: vlan >>         vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3 >>         media: Ethernet autoselect (10Gbase-SR ) >>         status: active >>         nd6 options=29 >> bridge0: flags=8843 metric 0 >> mtu 1500 >>         ether 58:9c:fc:10:dd:05 >>         inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255 >>         id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >>         maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >>         root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >>         member: vnet0.1 flags=143 >>                 ifmaxaddr 0 port 9 priority 128 path cost 2000 >>         member: ixl3.15 flags=143 >>                 ifmaxaddr 0 port 8 priority 128 path cost 55 >>         groups: bridge >>         nd6 options=9 >> [...] >> >> >> >> # cat /etc/rc.conf >> [...] >> ifconfig_ixl3="up" >> vlans_ixl3="15" >> cloned_interfaces="bridge0" >> ifconfig_bridge0="addm ixl3.15 up" >> [...] >> >> >> >> # dmesg | grep ixl >> ixl0: mem >> 0xb3000000-0xb3ffffff,0xb5018000-0xb501ffff at device 0.0 numa-domain >> 0 on pci6 >> ixl0: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 >> ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared >> ixl0: Using 1024 TX descriptors and 1024 RX descriptors >> ixl0: Using 12 RX queues 12 TX queues >> ixl0: Using MSI-X interrupts with 13 vectors >> ixl0: Ethernet address: a4:bf:01:76:ef:9a >> ixl0: Allocating 16 queues for PF LAN VSI; 12 queues active >> ixl0: SR-IOV ready >> ixl0: netmap queues/slots: TX 12/1024, RX 12/1024 >> ixl1: mem >> 0xb2000000-0xb2ffffff,0xb5010000-0xb5017fff at device 0.1 numa-domain >> 0 on pci6 >> ixl1: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 >> ixl1: PF-ID[1]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared >> ixl1: Using 1024 TX descriptors and 1024 RX descriptors >> ixl1: Using 12 RX queues 12 TX queues >> ixl1: Using MSI-X interrupts with 13 vectors >> ixl1: Ethernet address: a4:bf:01:76:ef:9b >> ixl1: Allocating 16 queues for PF LAN VSI; 12 queues active >> ixl1: SR-IOV ready >> ixl1: netmap queues/slots: TX 12/1024, RX 12/1024 >> ixl2: >> mem 0xb1000000-0xb1ffffff,0xb5008000-0xb500ffff at device 0.2 >> numa-domain 0 on pci6 >> ixl2: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 >> ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C >> ixl2: Using 1024 TX descriptors and 1024 RX descriptors >> ixl2: Using 12 RX queues 12 TX queues >> ixl2: Using MSI-X interrupts with 13 vectors >> ixl2: Ethernet address: a4:bf:01:76:ef:9c >> ixl2: Allocating 16 queues for PF LAN VSI; 12 queues active >> ixl2: ixl_set_link: Error getting phy capabilities -7, aq error: 5 >> ixl2: SR-IOV ready >> ixl2: netmap queues/slots: TX 12/1024, RX 12/1024 >> ixl3: >> mem 0xb0000000-0xb0ffffff,0xb5000000-0xb5007fff at device 0.3 >> numa-domain 0 on pci6 >> ixl3: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 >> ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C >> ixl3: Using 1024 TX descriptors and 1024 RX descriptors >> ixl3: Using 12 RX queues 12 TX queues >> ixl3: Using MSI-X interrupts with 13 vectors >> ixl3: Ethernet address: a4:bf:01:76:ef:9d >> ixl3: Allocating 16 queues for PF LAN VSI; 12 queues active >> ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5 >> ixl3: SR-IOV ready >> ixl3: netmap queues/slots: TX 12/1024, RX 12/1024 >> ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, >> Negotiated FEC: None, Autoneg: False, Flow Control: None >> ixl2: link state changed to UP >> ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, >> Negotiated FEC: None, Autoneg: False, Flow Control: None >> ixl3: link state changed to UP >> bridge0: can't disable some capabilities on ixl3.15: 0x400 >> ixl3: promiscuous mode enabled >> ixl3.15: promiscuous mode enabled >> >> >> from my packet traces: >> >> # tcpdump -vvv -i ixl3 >> > May you please add the option -e to tcpdump, so that the link-level > header can be printed out. > > ``` > # tcpdump -nvei ixl3 > ``` > >> tcpdump: listening on ixl3, link-type EN10MB (Ethernet), capture size >> 262144 bytes >> [...] >> 13:36:20.155843 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has >> 192.168.55.1 tell 192.168.55.10, length 28 >> 13:36:20.156285 ARP, Ethernet (len 6), IPv4 (len 4), Reply >> 192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46 >> 13:36:21.169003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has >> 192.168.55.1 tell 192.168.55.10, length 28 >> 13:36:21.169538 ARP, Ethernet (len 6), IPv4 (len 4), Reply >> 192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46 >> >> Here the answer can be see, its tagged with 802.1q tag 15 >> >> >> # tcpdump -vvv -i ixl3.15 >> tcpdump: listening on ixl3.15, link-type EN10MB (Ethernet), capture >> size 262144 bytes >> 14:14:37.255429 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has >> 192.168.55.1 tell 192.168.55.10, length 28 >> 14:14:42.263475 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has >> 192.168.55.1 tell 192.168.55.10, length 28 >> 14:15:02.556311 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has >> 192.168.55.1 tell 192.168.55.10, length 28 >> 14:15:07.557644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has >> 192.168.55.1 tell 192.168.55.10, length 28 >> >> The answer cannot be seen on the VLAN interface ): >> >> >> I hope the list can help me out here, as I am lost. >> >> >> Thanks & best >> >> >> Daniel >> > > Best regards, > Zhenlei > --------------1W4CUCK5105E1AOz0qmc0SmV Content-Type: text/html; charset=UTF-8 X-Clacks-Overhead: GNU Terry Pratchett Content-Transfer-Encoding: 8bit

Hi,

thanks for your suggestion. Turns out, when i unplug the vlan interface from the bridge and put the ip address on the vlan interface, as you suggested, things start to work, e.g. arp resolves.

as soon as i put the ip and the vlan interface back on the bridge, things stop again. so where does this lead me? the problem is not in the vlan handling, but on the bridge?

I started playing with the net.link.bridge sysctls and indeed, when i set


# sysctl net.link.bridge.inherit_mac=1


and then recreate the bridge


# ifconfig bridge0 deletem ixl3.15 deletem vnet0.1
# ifconfig bridge0 addm ixl3.15 addm vnet0.1


with the ip address still on bridge0 and ixl3, ixl3.15 and bridge0 all sharing the same mac address, arp starts resolving. but only for requests sent from the bridge0 interface. inside of the jail things still don't work (as the vnet interface again has another mac address).


# ifconfig ixl3
ixl3: flags=28963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
        ether a4:bf:01:76:ef:9d
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# ifconfig ixl3.15
ixl3.15: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP>
        ether a4:bf:01:76:ef:9d
        groups: vlan
        vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a4:bf:01:76:ef:9d
        inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: ixl3.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
# ping 192.168.55.1
PING 192.168.55.1 (192.168.55.1): 56 data bytes
^C
--- 192.168.55.1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

[! yes, the host does not answer on ICMP, but that is to be expected !]

# arp -an
? (192.168.55.20) at a4:bf:01:76:ef:9d on bridge0 permanent [bridge]
? (192.168.55.1) at b8:27:eb:47:8f:43 on bridge0 expires in 1197 seconds [bridge]
[...]

[! into the jail !]

JAIL # ifconfig epair0b
epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether ac:16:2d:bd:b7:34
        hwaddr 02:51:73:d1:33:0b
        inet 192.168.55.10 netmask 0xffffff00 broadcast 192.168.55.255
        inet6 fe80::ae16:2dff:febd:b734%epair0b prefixlen 64 scopeid 0x2
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
JAIL # ping 192.168.55.1
PING 192.168.55.1 (192.168.55.1): 56 data bytes
^C
--- 192.168.55.1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
JAIL # arp -an
? (192.168.55.10) at ac:16:2d:bd:b7:34 on epair0b permanent [ethernet]
? (192.168.55.1) at (incomplete) on epair0b expired [ethernet]


I conclude that there must be some mac address filtering going on in the data path, whether its on ixl or the bridge.

In dmesg I also see:

>> bridge0: can't disable some capabilities on ixl3.15: 0x400

but as of /usr/src/sys/net/if.h:233 this maps to IFCAP_LRO which afaik should not have any influence on L2 filtering.


Have to say, I'm out of ideas again. Never had something like this. So far just 'throwing interfaces on a bridge' worked in the past. Any ideas where to look next?


Thanks a lot & best


Daniel


On 1/25/24 08:22, Zhenlei Huang wrote:


On Jan 23, 2024, at 11:03 PM, Daniel <freebsd-net@c0decafe.de> wrote:

Hi List,


just recently I discovered a problem with the ixl(4) driver. Hopefully someone here can help me. my setup is as follows:


Network ----- ixl3 interface ----- ixl3.15 vlan interface ----- bridge0 ----- vnet0.1 to jail


the problem now is that the jail can send data out (arp requests), i do see the responses on the ixl3 interface of the host, but they never make their way up to the ixl3.15 vlan interface (even though they are tagged correctly). To rule out that my config or the network is the cruel pit i did test the same setup with a cheap usb-ethernet adapter and there everything works as expected. I'm on FreeBSD 13.2-RELEASE-p8 and I did test both, the in kernel driver and the driver from ports intel-ixl-kmod-1.13.4_1.

I would encourage you to do plain VLAN tests, i.e. plug ixl3.15 out from bridge0

```
# ifconfig bridge0 deletem ixl3.15
# ifconfig bridge0 inet 192.168.55.20/24 delete ### to prevent confusion
# ifconfig ixl3.15 inet 192.168.55.x/24
# ping -c1 192.168.55.1
``` 


Here is a bit of information on my environment:

# uname -a
FreeBSD mimir 13.2-RELEASE-p8 FreeBSD 13.2-RELEASE-p8 GENERIC amd64

# pciconf -lBbcevV pci0:25:0:3
ixl3@pci0:25:0:3:       class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x37d3 subvendor=0x8086 subdevice=0x35d5
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X722 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xb0000000, size 16777216, enabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xb5000000, size 32768, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 9aef76ffff01bfa4
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 32 supported
                     First VF RID Offset 0x006d, VF RID Stride 0x0001
                     VF Device ID 0x37cd
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Example VPD'

# ifconfig
[...]
ixl3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
        ether a4:bf:01:76:ef:9d
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ixl3.15: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP>
        ether a4:bf:01:76:ef:9d
        groups: vlan
        vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:dd:05
        inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: ixl3.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
[...]



# cat /etc/rc.conf
[...]
ifconfig_ixl3="up"
vlans_ixl3="15"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm ixl3.15 up"
[...]



# dmesg | grep ixl
ixl0: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem 0xb3000000-0xb3ffffff,0xb5018000-0xb501ffff at device 0.0 numa-domain 0 on pci6
ixl0: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared
ixl0: Using 1024 TX descriptors and 1024 RX descriptors
ixl0: Using 12 RX queues 12 TX queues
ixl0: Using MSI-X interrupts with 13 vectors
ixl0: Ethernet address: a4:bf:01:76:ef:9a
ixl0: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 12/1024, RX 12/1024
ixl1: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem 0xb2000000-0xb2ffffff,0xb5010000-0xb5017fff at device 0.1 numa-domain 0 on pci6
ixl1: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl1: PF-ID[1]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared
ixl1: Using 1024 TX descriptors and 1024 RX descriptors
ixl1: Using 12 RX queues 12 TX queues
ixl1: Using MSI-X interrupts with 13 vectors
ixl1: Ethernet address: a4:bf:01:76:ef:9b
ixl1: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl1: SR-IOV ready
ixl1: netmap queues/slots: TX 12/1024, RX 12/1024
ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xb1000000-0xb1ffffff,0xb5008000-0xb500ffff at device 0.2 numa-domain 0 on pci6
ixl2: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl2: Using 1024 TX descriptors and 1024 RX descriptors
ixl2: Using 12 RX queues 12 TX queues
ixl2: Using MSI-X interrupts with 13 vectors
ixl2: Ethernet address: a4:bf:01:76:ef:9c
ixl2: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl2: ixl_set_link: Error getting phy capabilities -7, aq error: 5
ixl2: SR-IOV ready
ixl2: netmap queues/slots: TX 12/1024, RX 12/1024
ixl3: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xb0000000-0xb0ffffff,0xb5000000-0xb5007fff at device 0.3 numa-domain 0 on pci6
ixl3: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0
ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl3: Using 1024 TX descriptors and 1024 RX descriptors
ixl3: Using 12 RX queues 12 TX queues
ixl3: Using MSI-X interrupts with 13 vectors
ixl3: Ethernet address: a4:bf:01:76:ef:9d
ixl3: Allocating 16 queues for PF LAN VSI; 12 queues active
ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5
ixl3: SR-IOV ready
ixl3: netmap queues/slots: TX 12/1024, RX 12/1024
ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl2: link state changed to UP
ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl3: link state changed to UP
bridge0: can't disable some capabilities on ixl3.15: 0x400
ixl3: promiscuous mode enabled
ixl3.15: promiscuous mode enabled


from my packet traces:

# tcpdump -vvv -i ixl3

May you please add the option -e to tcpdump, so that the link-level header can be printed out.

```
# tcpdump -nvei ixl3
```

tcpdump: listening on ixl3, link-type EN10MB (Ethernet), capture size 262144 bytes
[...]
13:36:20.155843 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28
13:36:20.156285 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46
13:36:21.169003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28
13:36:21.169538 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46

Here the answer can be see, its tagged with 802.1q tag 15


# tcpdump -vvv -i ixl3.15
tcpdump: listening on ixl3.15, link-type EN10MB (Ethernet), capture size 262144 bytes
14:14:37.255429 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28
14:14:42.263475 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28
14:15:02.556311 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28
14:15:07.557644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28

The answer cannot be seen on the VLAN interface ):


I hope the list can help me out here, as I am lost.


Thanks & best


Daniel


Best regards,
Zhenlei

--------------1W4CUCK5105E1AOz0qmc0SmV--