Date: Fri, 11 Oct 2024 17:20:22 -0700 From: Navdeep Parhar <np@freebsd.org> To: John Nielsen <lists@jnielsen.net> Cc: freebsd-net@freebsd.org Subject: Re: Traffic between cxgbe VFs and/or PF on a host Message-ID: <CAPFoGT8zWe0-6C7V5wBmHuBpUHceXVsg%2B3a1r6Yhh3LiNDZxMw@mail.gmail.com> In-Reply-To: <6BC00399-9D4C-4893-9FE4-4F99BD295909@jnielsen.net> References: <6BC00399-9D4C-4893-9FE4-4F99BD295909@jnielsen.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000061e4cb06243c922f
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Fri, Oct 11, 2024 at 3:56=E2=80=AFPM John Nielsen <lists@jnielsen.net> w=
rote:
> Hi-
>
> I=E2=80=99m running a FreeBSD 14-STABLE host with a Chelstio T520. I have=
a bhyve
> VM (also running 14-STABLE) to which I have assigned a VF of the NIC. Tha=
t
> is all working as expected; the host can pass traffic using the PF cxl0 a=
nd
> the guest can pass traffic using the VF cxlv0. However the host cannot
> communicate with the guest. I am looking in to the possibility of enablin=
g
> 802.1qbg / VEPA / reflective relay on the switch port but I=E2=80=99d lik=
e to know
> if the T5 can do that switching itself without sending the packets over t=
he
> wire. The marketing material says the card "integrates a high performance
> packet switch=E2=80=9D but I don=E2=80=99t know how to configure that fun=
ctionality on
> FreeBSD or if this use case is supported. Can anyone shed some light on
> that?
>
The PF driver's tx bypasses the internal switch by default and is not
visible to the VFs because of that. Set this knob to force it go through
the switch.
hw.cxgbe.tx_vm_wr
Setting this to 1 instructs the driver to use VM work requests
to transmit data.
This lets PF interfaces transmit frames to VF interfaces over
the internal switch in
the ASIC. Note that the cxgbev(4) VF driver always uses VM
work requests and is not
affected by this tunable. The default value is 0 and should
be changed only if PF
and VF interfaces need to communicate with each other.
Different interfaces can be
assigned different values using the dev.<port>.X.tx_vm_wr
sysctl when the interface
is administratively down.
Regards,
Navdeep
> The other alternative would be to wire up the second port but if I can ge=
t
> away with not needing to use another SFP+ port on the switch for this tha=
t
> would be ideal.
>
> Thanks!
>
> JN
>
>
>
--00000000000061e4cb06243c922f
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr">On Fri, Oct 11, 2024 at 3:56=E2=80=AFPM J=
ohn Nielsen <<a href=3D"mailto:lists@jnielsen.net">lists@jnielsen.net</a=
>> wrote:</div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quo=
te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204=
);padding-left:1ex">Hi-<br>
<br>
I=E2=80=99m running a FreeBSD 14-STABLE host with a Chelstio T520. I have a=
bhyve VM (also running 14-STABLE) to which I have assigned a VF of the NIC=
. That is all working as expected; the host can pass traffic using the PF c=
xl0 and the guest can pass traffic using the VF cxlv0. However the host can=
not communicate with the guest. I am looking in to the possibility of enabl=
ing 802.1qbg / VEPA / reflective relay on the switch port but I=E2=80=99d l=
ike to know if the T5 can do that switching itself without sending the pack=
ets over the wire. The marketing material says the card "integrates a =
high performance packet switch=E2=80=9D but I don=E2=80=99t know how to con=
figure that functionality on FreeBSD or if this use case is supported. Can =
anyone shed some light on that?<br></blockquote><div><br></div><div>The PF =
driver's tx bypasses the internal switch by default and is not visible =
to the VFs because of that.=C2=A0 Set this knob to force it go through the =
switch.<br></div><div><br></div><div>=C2=A0 =C2=A0=C2=A0 hw.cxgbe.tx_vm_wr<=
br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Setting this to 1 instru=
cts the driver to use VM work requests to transmit data.<br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0This lets PF interfaces transmit frames t=
o VF interfaces over the internal switch in<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0the ASIC.=C2=A0 Note that the cxgbev(4) VF driver alway=
s uses VM work requests and is not<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0affected by this tunable.=C2=A0 The default value is 0 and sho=
uld be changed only if PF<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0and VF interfaces need to communicate with each other.=C2=A0 Different i=
nterfaces can be<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0assigne=
d different values using the dev.<port>.X.tx_vm_wr sysctl when the in=
terface<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0is administrativ=
ely down.<br></div><div><br></div><div>Regards,</div><div>Navdeep <br></div=
><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0=
px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
The other alternative would be to wire up the second port but if I can get =
away with not needing to use another SFP+ port on the switch for this that =
would be ideal.<br>
<br>
Thanks!<br>
<br>
JN<br>
<br>
<br>
</blockquote></div></div>
--00000000000061e4cb06243c922f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPFoGT8zWe0-6C7V5wBmHuBpUHceXVsg%2B3a1r6Yhh3LiNDZxMw>