Date: Wed, 23 Oct 2024 16:43:17 -0700 From: Navdeep Parhar <nparhar@gmail.com> To: Lexi Winter <lexi@le-fay.org> Cc: freebsd-net@freebsd.org Subject: Re: Chelsio / cxlv(4): strange messages, SR-IOV interface does not work Message-ID: <CAPFoGT-Nptqhmb=VY7LGPPgqFqESuUD%2BpoxLMWuxz2xrw9zvGg@mail.gmail.com> In-Reply-To: <f2972877-4ec5-49f5-a335-99c28aab58a8@le-fay.org> References: <f2972877-4ec5-49f5-a335-99c28aab58a8@le-fay.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000e7a81406252d733f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 22, 2024 at 11:21=E2=80=AFPM Lexi Winter <lexi@le-fay.org> wrot= e: > hello, > > i'm trying to configure a cxlv(4) device, which is a VF of a Chelsio > T540-CR on a host running bhyve. > > host: FreeBSD 15.0-CURRENT #3 lf/main-n269068-2cff93ced1d: Wed Oct 23 > 02:48:20 BST 2024 > guest: FreeBSD 15.0-CURRENT #2 lf/main-n269067-56dd459904b: Sat Oct 19 > 18:36:40 BST 2024 > > the VF appears correctly in the VM: > > root@lily:~ # kldload if_cxlv > t5vf0: <Chelsio T540-CR VF> mem > 0xc000e000-0xc000efff,0xc0000000-0xc0007fff,0xc0008000-0xc0009fff at > device 6.0 on pci0 > t5vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq > cxlv0: <port 0> on t5vf0 > cxlv0: 2 txq, 1 rxq (NIC) > > and after bringing the interface 'up' everything seems fine: > > root@lily:~ # ifconfig cxlv0 > cxlv0: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> > metric 0 mtu 1500 > > > options=3D6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HW= CSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWR= XTSTMP,MEXTPG> > ether 06:44:3f:e7:60:30 > media: Ethernet 10Gbase-Twinax <full-duplex> (10Gbase-Twinax > <full-duplex,rxpause,txpause>) > status: active > nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > however, trying to assign an IP address causes immediate problems: > > root@lily:~ # ifconfig cxlv0 inet6 2001:8b0:aab5:7::10/64 > root@lily:~ # Oct 23 06:16:07 lily kernel: cxlv0: a looped back NS > message is detected during DAD for fe80:3::444:3fff:fee7:6030. Another > DAD probes are being sent. > root@lily:~ # dmesg|grep loop > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > > i find this strange because the link local IP address in the kernel > error is not even configured on the interface: > > root@lily:~ # ifconfig cxlv0 > cxlv0: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> > metric 0 mtu 1500 > > > options=3D6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HW= CSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWR= XTSTMP,MEXTPG> > ether 06:44:3f:e7:60:30 > inet6 2001:8b0:aab5:7::10/64 > inet6 fe80::444:3fff:fee7:6030%cxlv0/64 tentative scopeid 0x3 > media: Ethernet 10Gbase-Twinax <full-duplex> (10Gbase-Twinax > <full-duplex,rxpause,txpause>) > status: active > nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> > > am i doing something wrong here? > You can disable IPv6 DAD as a workaround for this issue. The problem is that the VF's multicast tx is getting reflected back to it by the internal switch when it shouldn't. # sysctl net.inet6.ip6.dad_count=3D0 Regards, Navdeep > > thanks, lexi. > > --000000000000e7a81406252d733f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"g= mail_attr">On Tue, Oct 22, 2024 at 11:21=E2=80=AFPM Lexi Winter <<a href= =3D"mailto:lexi@le-fay.org">lexi@le-fay.org</a>> wrote:<br></div><blockq= uote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1p= x solid rgb(204,204,204);padding-left:1ex">hello,<br> <br> i'm trying to configure a cxlv(4) device, which is a VF of a Chelsio <b= r> T540-CR on a host running bhyve.<br> <br> host: FreeBSD 15.0-CURRENT #3 lf/main-n269068-2cff93ced1d: Wed Oct 23 <br> 02:48:20 BST 2024<br> guest: FreeBSD 15.0-CURRENT #2 lf/main-n269067-56dd459904b: Sat Oct 19 <br> 18:36:40 BST 2024<br> <br> the VF appears correctly in the VM:<br> <br> root@lily:~ # kldload if_cxlv<br> t5vf0: <Chelsio T540-CR VF> mem <br> 0xc000e000-0xc000efff,0xc0000000-0xc0007fff,0xc0008000-0xc0009fff at <br> device 6.0 on pci0<br> t5vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq<br> cxlv0: <port 0> on t5vf0<br> cxlv0: 2 txq, 1 rxq (NIC)<br> <br> and after bringing the interface 'up' everything seems fine:<br> <br> root@lily:~ # ifconfig cxlv0<br> cxlv0: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP&g= t; <br> metric 0 mtu 1500<br> <br> options=3D6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_H= WCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HW= RXTSTMP,MEXTPG><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ether 06:44:3f:e7:60:30<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0media: Ethernet 10Gbase-Twinax <full-d= uplex> (10Gbase-Twinax <br> <full-duplex,rxpause,txpause>)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status: active<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nd6 options=3D29<PERFORMNUD,IFDISABLED= ,AUTO_LINKLOCAL><br> <br> however, trying to assign an IP address causes immediate problems:<br> <br> root@lily:~ # ifconfig cxlv0 inet6 2001:8b0:aab5:7::10/64<br> root@lily:~ # Oct 23 06:16:07 lily kernel: cxlv0: a looped back NS <br> message is detected during DAD for fe80:3::444:3fff:fee7:6030.=C2=A0 Anothe= r <br> DAD probes are being sent.<br> root@lily:~ # dmesg|grep loop<br> cxlv0: a looped back NS message is detected during DAD for <br> fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.<br> cxlv0: a looped back NS message is detected during DAD for <br> fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.<br> cxlv0: a looped back NS message is detected during DAD for <br> fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.<br> cxlv0: a looped back NS message is detected during DAD for <br> fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.<br> cxlv0: a looped back NS message is detected during DAD for <br> fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.<br> cxlv0: a looped back NS message is detected during DAD for <br> fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.<br> cxlv0: a looped back NS message is detected during DAD for <br> fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.<br> <br> i find this strange because the link local IP address in the kernel <br> error is not even configured on the interface:<br> <br> root@lily:~ # ifconfig cxlv0<br> cxlv0: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP&g= t; <br> metric 0 mtu 1500<br> <br> options=3D6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_H= WCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HW= RXTSTMP,MEXTPG><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ether 06:44:3f:e7:60:30<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 2001:8b0:aab5:7::10/64<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 fe80::444:3fff:fee7:6030%cxlv0/64 t= entative scopeid 0x3<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0media: Ethernet 10Gbase-Twinax <full-d= uplex> (10Gbase-Twinax <br> <full-duplex,rxpause,txpause>)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status: active<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nd6 options=3D21<PERFORMNUD,AUTO_LINKL= OCAL><br> <br> am i doing something wrong here?<br></blockquote><div><br></div><div>You ca= n disable IPv6 DAD as a=20 workaround for this issue.=C2=A0 The problem is that the VF's multicast= tx is getting reflected back to it by the internal switch when it shouldn't.= <br></div><div><br></div><div># sysctl net.inet6.ip6.dad_count=3D0</div><di= v><br></div><div>Regards,</div><div>Navdeep<br></div><div>=C2=A0</div><bloc= kquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:= 1px solid rgb(204,204,204);padding-left:1ex"> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 thanks, lexi.<br> <br> </blockquote></div></div> --000000000000e7a81406252d733f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPFoGT-Nptqhmb=VY7LGPPgqFqESuUD%2BpoxLMWuxz2xrw9zvGg>