From nobody Sat Sep 27 16:26:54 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cYtC82jZcz68KmM; Sat, 27 Sep 2025 16:27:16 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [IPv6:2607:f170:34:11::b0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cYtC76K73z4NYt; Sat, 27 Sep 2025 16:27:15 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple ([IPv6:2607:fa18:e2fe:2d6a:c9d8:f1a0:f9f4:9c10]) (authenticated bits=0) by webmail5.jnielsen.net (8.18.1/8.18.1) with ESMTPSA id 58RGR46A069325 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 27 Sep 2025 10:27:07 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host [IPv6:2607:fa18:e2fe:2d6a:c9d8:f1a0:f9f4:9c10] claimed to be smtpclient.apple From: John Nielsen Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: Chelsio (cxgbe) VF on FreeBSD guest under KVM requires -rxcsum Date: Sat, 27 Sep 2025 10:26:54 -0600 In-Reply-To: Cc: net@freebsd.org, virtualization@freebsd.org, np@freebsd.org To: Michael Tuexen References: X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4cYtC76K73z4NYt --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Sep 26, 2025, at 3:44=E2=80=AFPM, Michael Tuexen = wrote: >=20 >> On 26. Sep 2025, at 20:52, John Nielsen wrote: >>=20 >>> On Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen = wrote: >>> =EF=BB=BF >>>>=20 >>>> On 26. Sep 2025, at 02:58, John Nielsen wrote: >>>>=20 >>>> Not sure if this is a known issue or even an issue on the FreeBSD = side but decided to ask here first. I=E2=80=99m happy to put in a bug = report if appropriate. >>>>=20 >>>> I have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected. >>>=20 >>> I don't have access to such a network card. Just to be clear: you = are running the =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D command inside = the guest running FreeBSD, right? >>=20 >> Yes. >>=20 >>> What is the peer, when you mention TCP and UDP do not work? Is it = the host running Linux? Is it another VM? Is it some external host? >>=20 >> My laptop on the same subnet primarily, but. Also tested from another = physical machine running FreeBSD. > OK. That does not seem to be related to what I initially thought. >=20 > Could you run > tcpdump -i outgoing_interface -w laptop.pcap > on your laptop and > tcpdump -i cxlv0 -w vm.pcap > on your vm at the same time and try to do some TCP based = communication. > Maybe two times, one time with ifconfig cxlv0 rxcsum and one time with = ifconfig cxlv0 -rxcsum. >=20 > If you are fine with doing the measurements, you can send the .pcap = files > to tuexen@freebsd.org = . >=20 > At least I would like to understand what is going on. Thank you, I sent you those packet captures under separate cover. =46rom what I can tell so far the VM does receive incoming packets even = when rxcsum is enabled, and the checksums appear correct at least = according to tcpdump. But something prevents the packets from being = processed or sent up the stack. The VM=E2=80=99s sshd never generates a = SYN/ACK in response to a connection attempt. A DNS query from the VM = looks like it gets a response on the wire but the response doesn=E2=80=99t= make it to the process doing the query. -John --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Sep 26, 2025, = at 3:44=E2=80=AFPM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote:

On 26. Sep 2025, at 20:52, John Nielsen = <lists@jnielsen.net> wrote:

On = Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote:
=EF=BB=BF

On 26. Sep 2025, at 02:58, John Nielsen = <lists@jnielsen.net> wrote:

Not sure if this is a known = issue or even an issue on the FreeBSD side but decided to ask here = first. I=E2=80=99m happy to put in a bug report if appropriate.

I = have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected.

I don't have access to such a = network card. Just to be clear: you are running the =E2=80=9Cifconfig = cxlv0 -rxcsum=E2=80=9D command inside the guest running FreeBSD, = right?

Yes.

What is = the peer, when you mention TCP and UDP do not work? Is it the host = running Linux? Is it another VM? Is it some external = host?

My laptop on the same subnet primarily, but. = Also tested from another physical machine running = FreeBSD.
OK. = That does not seem to be related to what I initially thought.

Could you run
tcpdump = -i outgoing_interface -w laptop.pcap
on your = laptop and
tcpdump -i cxlv0 -w = vm.pcap
on your vm at the same = time and try to do some TCP based communication.
Maybe two times, one time with ifconfig = cxlv0 rxcsum and one time with ifconfig cxlv0 -rxcsum.

If you are fine with doing the = measurements, you can send the .pcap files
to tuexen@freebsd.org <mailto:tuexen@freebsd.org>.

At = least I would like to understand what is going on.

Thank you, I sent you those = packet captures under separate cover.

=46rom = what I can tell so far the VM does receive incoming packets even when = rxcsum is enabled, and the checksums appear correct at least according = to tcpdump. But something prevents the packets from being processed or = sent up the stack. The VM=E2=80=99s sshd never generates a SYN/ACK in = response to a connection attempt. A DNS query from the VM looks like it = gets a response on the wire but the response doesn=E2=80=99t make it to = the process doing the = query.

-John

= --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627--