From nobody Sat Oct 12 21:51:57 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XQxzf44vXz5Z2LN for ; Sat, 12 Oct 2024 21:52:14 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [69.87.218.172]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XQxzd70Nwz4YKG; Sat, 12 Oct 2024 21:52:13 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple ([IPv6:2607:fa18:e2fe:2d6a:940c:f295:967:dc73]) (authenticated bits=0) by webmail5.jnielsen.net (8.17.2/8.17.2) with ESMTPSA id 49CLq9Gs054003 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 12 Oct 2024 15:52:12 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host [IPv6:2607:fa18:e2fe:2d6a:940c:f295:967:dc73] claimed to be smtpclient.apple Content-Type: multipart/alternative; boundary=Apple-Mail-EE54B904-B54D-4006-91E9-414A7E93D260 Content-Transfer-Encoding: 7bit From: John Nielsen List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: Traffic between cxgbe VFs and/or PF on a host Date: Sat, 12 Oct 2024 15:51:57 -0600 Message-Id: <024F07FC-EF22-4259-9678-0F4D4AF2725E@jnielsen.net> References: Cc: freebsd-net@freebsd.org In-Reply-To: To: Navdeep Parhar X-Mailer: iPhone Mail (21G93) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6364, ipnet:69.87.218.0/24, country:US] X-Rspamd-Queue-Id: 4XQxzd70Nwz4YKG X-Spamd-Bar: ---- --Apple-Mail-EE54B904-B54D-4006-91E9-414A7E93D260 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Oct 11, 2024, at 6:21=E2=80=AFPM, Navdeep Parhar &= lt;np@freebsd.org> wrote:
=EF=BB=BF
On Fri, Oct 11,= 2024 at 3:56=E2=80=AFPM John Nielsen <lists@jnielsen.net> wrote:

I=E2=80=99m running a FreeBSD 14-STABLE host with a Chelstio T520. I have a b= hyve VM (also running 14-STABLE) to which I have assigned a VF of the NIC. T= hat is all working as expected; the host can pass traffic using the PF cxl0 a= nd the guest can pass traffic using the VF cxlv0. However the host cannot co= mmunicate with the guest. I am looking in to the possibility of enabling 802= .1qbg / VEPA / reflective relay on the switch port but I=E2=80=99d like to k= now if the T5 can do that switching itself without sending the packets over t= he wire. The marketing material says the card "integrates a high performance= packet switch=E2=80=9D but I don=E2=80=99t know how to configure that funct= ionality on FreeBSD or if this use case is supported. Can anyone shed some l= ight on that?

The PF driver's tx bypasse= s the internal switch by default and is not visible to the VFs because of th= at.  Set this knob to force it go through the switch.
     hw.cxgbe.tx_vm_wr
      &n= bsp;      Setting this to 1 instructs the driver to use VM wo= rk requests to transmit data.
            &= nbsp;This lets PF interfaces transmit frames to VF interfaces over the inter= nal switch in
             the ASIC.&n= bsp; Note that the cxgbev(4) VF driver always uses VM work requests and is n= ot
             affected by this tunab= le.  The default value is 0 and should be changed only if PF
  &= nbsp;          and VF interfaces need to communicat= e with each other.  Different interfaces can be
     =        assigned different values using the dev.<port= >.X.tx_vm_wr sysctl when the interface
        &nb= sp;    is administratively down.

Thank you! That indeed works as described. I did= scan the man page before posting but obviously could have benefited from cl= oser study. I appreciate you pointing me in the right direction.
<= br>
Out of curiosity is there any practical difference (performanc= e; security, etc) between setting this knob and just using an additional VF f= or the host rather than the PF (aside from being one less VF available for u= se elsewhere)?

JN
= --Apple-Mail-EE54B904-B54D-4006-91E9-414A7E93D260--