Date: Fri, 3 Jan 2025 16:59:42 -0700 From: John Nielsen <lists@jnielsen.net> To: freebsd-net@freebsd.org Cc: Navdeep Parhar <np@freebsd.org> Subject: cxgbe VF vs multicast Message-ID: <4ED3FE1D-776C-4CE2-A676-975CDB757694@jnielsen.net>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_A9F64ED3-7010-43A7-867D-F253F3E3B109 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi- I just moved my router VM from a bridge + virtio setup to using a VF = (virtual function) on my Chelsio T520-LL-CR to improve performance. = It=E2=80=99s working beautifully for the most part, but I have noticed = two things that aren=E2=80=99t, both related to multicast. First, when using dhcpcd to assign IPv6 addresses to my LAN VLAN = interfaces, I get this spammed repeatedly on the console for both VLANs = where I use prefix delegation: kernel: cxlv0.2: a looped back NS message is detected during DAD for = [address]. Another DAD probes are being sent I was able to work around that (I think?) by setting = "net.inet6.ip6.dad_count=3D0=E2=80=9D in /etc/sysctl.conf (as hinted in = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270859 talking about = an mlx5 VF). Second, CARP no longer works. At all. I turned off the peer completely = but the interfaces never came out of the BACKUP state. For now I have = removed CARP from the setup. Both IPv6 address assignment and CARP were working correctly in the = previous setup. Is the VF receiving its own multicast packets back from the PF or = virtual switch in the T5? That would explain both issues. In any case, = is there anything I can do about it? Both host and guest are FreeBSD 14-stable. In addition to the T5 VF the = host passes through an em interface to the guest entirely. Config excerpts from both machines below. Any help/insight appreciated. = Thanks! -John On the host I have: #/boot/loader.conf vmm_load=3D"YES" hw.vmm.amdvi.enable=3D"1" pptdevs=3D"14/0/0=E2=80=9D #em0 hw.cxgbe.tx_vm_wr=3D=E2=80=9C1=E2=80=9D #allow PF to talk to VF #/etc/rc.conf ifconfig_cxl0=3D"up" vlans_cxl0=3D"2 104" ifconfig_cxl0_2=3D"inet ${ADDR2}/24" ifconfig_cxl0_2_ipv6=3D"inet6 accept_rtadv auto_linklocal" ifconfig_cxl0_104=3D"inet ${ADDR104}/24" iovctl_files=3D"/etc/iov/cxl0.conf" #/etc/iov/cxl0.conf=20 PF { device : cxl0 num_vfs : 1 } VF-0 { passthrough : true; } On the guest I have: #/boot/loader.conf if_cxgbev_load=3D"YES" net.inet.ip.fw.default_to_accept=3D=E2=80=9C1=E2=80=9D #allow dhcp6 = before ipfw ruleset is loaded #/etc/rc.conf ifconfig_cxlv0=3D"up" vlans_cxlv0=3D"2 98 103" ifconfig_cxlv0_2=3D"inet ${MY_IP}/24" ifconfig_cxlv0_2_ipv6=3D"inet6 auto_linklocal" # set from dhcpcd via = prefix delegation #ifconfig_cxlv0_2_alias0=3D"inet vhid 1 advskew ${MY_SKEW} pass ${PASS} = alias 192.168.2.1/32" ifconfig_cxlv0_2_alias0=3D"inet alias 192.168.2.1/32" #ifconfig_cxlv0_103=3D"inet vhid 2 advskew ${MY_SKEW} pass ${PASS} = 192.168.103.1/24" ifconfig_cxlv0_103=3D"inet 192.168.103.1/24" #ifconfig_cxlv0_98=3D"inet vhid 3 advskew ${MY_SKEW} pass ${PASS} = 192.168.98.1/24" ifconfig_cxlv0_98=3D"inet 192.168.98.1/24" ifconfig_cxlv0_98_ipv6=3D"inet6 auto_linklocal" rtadvd_interfaces=3D"cxlv0.2 cxlv0.98" dhclient_program=3D"/usr/local/sbin/dhcpcd" #/etc/sysctl.conf net.inet.carp.preempt=3D1 net.inet6.ip6.dad_count=3D0 #/usr/local/etc/dhcpcd.conf duid persistent vendorclassid option classless_static_routes option interface_mtu option rapid_commit require dhcp_server_identifier slaac hwaddr allowinterfaces em0 waitip 4 waitip 6 nodelay timeout 10 interface em0 ia_na ia_pd 1/::/64 cxlv0.2/0/64/1 ia_pd 2/::/64 cxlv0.98/0/64/1 --Apple-Mail=_A9F64ED3-7010-43A7-867D-F253F3E3B109 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: = after-white-space;">Hi-<div><br></div><div>I just moved my router VM = from a bridge + virtio setup to using a VF (virtual function) on my = Chelsio T520-LL-CR to improve performance. It=E2=80=99s working = beautifully for the most part, but I have noticed two things that = aren=E2=80=99t, both related to = multicast.</div><div><br></div><div>First, when using dhcpcd to assign = IPv6 addresses to my LAN VLAN interfaces, I get this spammed repeatedly = on the console for both VLANs where I use prefix = delegation:</div><div>kernel: cxlv0.2: a looped back NS message is = detected during DAD for [address]. Another DAD probes are being = sent</div><div><br></div><div>I was able to work around that (I think?) = by setting "net.inet6.ip6.dad_count=3D0=E2=80=9D in /etc/sysctl.conf (as = hinted in <a = href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270859">https:= //bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270859</a> talking = about an mlx5 VF).</div><div><br></div><div>Second, CARP no longer = works. At all. I turned off the peer completely but the interfaces never = came out of the BACKUP state. For now I have removed CARP from the = setup.</div><div><br></div><div>Both IPv6 address assignment and CARP = were working correctly in the previous = setup.</div><div><br></div><div>Is the VF receiving its own multicast = packets back from the PF or virtual switch in the T5? That would explain = both issues. In any case, is there anything I can do about = it?</div><div><br></div><div>Both host and guest are FreeBSD 14-stable. = In addition to the T5 VF the host passes through an em interface to the = guest entirely.</div><div><br></div><div>Config excerpts from both = machines below. Any help/insight appreciated. = Thanks!</div><div><br></div><div>-John</div><div><br></div><div><br></div>= <div>On the host I = have:</div><div><div><br></div><div>#/boot/loader.conf</div><div>vmm_load=3D= "YES"</div><div>hw.vmm.amdvi.enable=3D"1"</div><div>pptdevs=3D"14/0/0=E2=80= =9D #em0</div><div>hw.cxgbe.tx_vm_wr=3D=E2=80=9C1=E2=80=9D #allow PF to = talk to = VF</div></div><div><br></div><div>#/etc/rc.conf</div><div><div>ifconfig_cx= l0=3D"up"</div><div>vlans_cxl0=3D"2 = 104"</div></div><div><div>ifconfig_cxl0_2=3D"inet = ${ADDR2}/24"</div><div>ifconfig_cxl0_2_ipv6=3D"inet6 accept_rtadv = auto_linklocal"</div><div>ifconfig_cxl0_104=3D"inet = ${ADDR104}/24"</div></div><div><div>iovctl_files=3D"/etc/iov/cxl0.conf"</d= iv></div><div><br></div><div><div>#/etc/iov/cxl0.conf </div><div>PF = {</div><div> device : cxl0</div><div> = num_vfs : 1</div><div>}</div><div>VF-0 = {</div><div> passthrough : = true;</div><div>}</div></div><div><br></div><div><br></div><div>On the = guest I = have:</div><div><br></div><div><div>#/boot/loader.conf</div><div>if_cxgbev= _load=3D"YES"</div><div>net.inet.ip.fw.default_to_accept=3D=E2=80=9C1=E2=80= =9D #allow dhcp6 before ipfw ruleset is = loaded</div></div><div><br></div><div><div>#/etc/rc.conf</div><div>ifconfi= g_cxlv0=3D"up"</div><div>vlans_cxlv0=3D"2 98 = 103"</div><div>ifconfig_cxlv0_2=3D"inet = ${MY_IP}/24"</div><div>ifconfig_cxlv0_2_ipv6=3D"inet6 auto_linklocal" # = set from dhcpcd via prefix = delegation</div><div>#ifconfig_cxlv0_2_alias0=3D"inet vhid 1 advskew = ${MY_SKEW} pass ${PASS} alias = 192.168.2.1/32"</div><div>ifconfig_cxlv0_2_alias0=3D"inet alias = 192.168.2.1/32"</div><div>#ifconfig_cxlv0_103=3D"inet vhid 2 advskew = ${MY_SKEW} pass ${PASS} = 192.168.103.1/24"</div><div>ifconfig_cxlv0_103=3D"inet = 192.168.103.1/24"</div><div>#ifconfig_cxlv0_98=3D"inet vhid 3 advskew = ${MY_SKEW} pass ${PASS} = 192.168.98.1/24"</div><div>ifconfig_cxlv0_98=3D"inet = 192.168.98.1/24"</div><div>ifconfig_cxlv0_98_ipv6=3D"inet6 = auto_linklocal"</div><div>rtadvd_interfaces=3D"cxlv0.2 = cxlv0.98"</div><div><div>dhclient_program=3D"/usr/local/sbin/dhcpcd"</div>= </div></div><div><br></div><div>#/etc/sysctl.conf</div><div><div>net.inet.= carp.preempt=3D1</div><div>net.inet6.ip6.dad_count=3D0</div></div><div><br= ></div><div><div>#/usr/local/etc/dhcpcd.conf</div><div>duid</div><div>pers= istent</div><div>vendorclassid</div><div>option = classless_static_routes</div><div>option interface_mtu</div><div>option = rapid_commit</div><div>require dhcp_server_identifier</div><div>slaac = hwaddr</div><div>allowinterfaces em0</div><div>waitip 4</div><div>waitip = 6</div><div>nodelay</div><div>timeout 10</div><div>interface = em0</div><div> ia_na</div><div> ia_pd 1/::/64 = cxlv0.2/0/64/1</div><div> ia_pd 2/::/64 = cxlv0.98/0/64/1</div></div><div><br></div><div><br></div></body></html>= --Apple-Mail=_A9F64ED3-7010-43A7-867D-F253F3E3B109--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4ED3FE1D-776C-4CE2-A676-975CDB757694>