From nobody Tue Sep 30 18:48:47 2025 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cbnCW6lq4z694ZN; Tue, 30 Sep 2025 18:49:11 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [IPv6:2607:f170:34:11::b0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cbnCV0bgCz3QBw; Tue, 30 Sep 2025 18:49:10 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of lists@jnielsen.net designates 2607:f170:34:11::b0 as permitted sender) smtp.mailfrom=lists@jnielsen.net Received: from smtpclient.apple ([50.207.241.62]) (authenticated bits=0) by webmail5.jnielsen.net (8.18.1/8.18.1) with ESMTPSA id 58UImvxn042172 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 30 Sep 2025 12:49:01 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host [50.207.241.62] claimed to be smtpclient.apple Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: Chelsio (cxgbe) VF on FreeBSD guest under KVM requires -rxcsum From: John Nielsen In-Reply-To: <4855A0F6-DD05-42A1-9119-398DC4124FDF@lurchi.franken.de> Date: Tue, 30 Sep 2025 12:48:47 -0600 Cc: net@freebsd.org, virtualization@freebsd.org, np@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <07254AC1-40B9-43B5-A902-08261C4CE2F0@jnielsen.net> References: <4855A0F6-DD05-42A1-9119-398DC4124FDF@lurchi.franken.de> To: Michael Tuexen X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: - X-Spamd-Result: default: False [-1.82 / 15.00]; NEURAL_HAM_LONG(-0.96)[-0.962]; NEURAL_HAM_SHORT(-0.96)[-0.960]; NEURAL_SPAM_MEDIUM(0.40)[0.398]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; APPLE_MAILER_COMMON(0.00)[]; DMARC_NA(0.00)[jnielsen.net]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org,net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4cbnCV0bgCz3QBw On Sep 27, 2025, at 12:37=E2=80=AFPM, Michael Tuexen = wrote: >=20 >> On 27. Sep 2025, at 18:26, John Nielsen wrote: >>=20 >>> On Sep 26, 2025, at 3:44=E2=80=AFPM, Michael Tuexen = wrote: >>>=20 >>>> On 26. Sep 2025, at 20:52, John Nielsen wrote: >>>>=20 >>>>> On Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen = wrote: >>>>> =EF=BB=BF >>>>>>=20 >>>>>> On 26. Sep 2025, at 02:58, John Nielsen = wrote: >>>>>>=20 >>>>>> Not sure if this is a known issue or even an issue on the FreeBSD = side but decided to ask here first. I=E2=80=99m happy to put in a bug = report if appropriate. >>>>>>=20 >>>>>> I have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected. >>>>>=20 >>>>> I don't have access to such a network card. Just to be clear: you = are running the =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D command inside = the guest running FreeBSD, right? >>>>=20 >>>> Yes. >>>>=20 >>>>> What is the peer, when you mention TCP and UDP do not work? Is it = the host running Linux? Is it another VM? Is it some external host? >>>>=20 >>>> My laptop on the same subnet primarily, but. Also tested from = another physical machine running FreeBSD. >>> OK. That does not seem to be related to what I initially thought. >>>=20 >>> Could you run >>> tcpdump -i outgoing_interface -w laptop.pcap >>> on your laptop and >>> tcpdump -i cxlv0 -w vm.pcap >>> on your vm at the same time and try to do some TCP based = communication. >>> Maybe two times, one time with ifconfig cxlv0 rxcsum and one time = with ifconfig cxlv0 -rxcsum. >>>=20 >>> If you are fine with doing the measurements, you can send the .pcap = files >>> to tuexen@freebsd.org . >>>=20 >>> At least I would like to understand what is going on. >>=20 >> Thank you, I sent you those packet captures under separate cover. > Thanks I got the email and sent a response. >>=20 >> =46rom what I can tell so far the VM does receive incoming packets = even when rxcsum is enabled, and the checksums appear correct at least = according to tcpdump. But something prevents the packets from being = processed or sent up the stack. The VM=E2=80=99s sshd never generates a = SYN/ACK in response to a connection attempt. A DNS query from the VM = looks like it gets a response on the wire but the response doesn=E2=80=99t= make it to the process doing the query. > Yes, I agree. For whatever reason the packets received, but are not = delivered to > the transport stack. I would like to figure out where the packets are = dropped or > stored and not processed further. I > sent you some suggestions to try. I hope we can figure out where the = problem is > related. > In addition to what I suggested, you might want to provide the output = of > netstat -s > from before and after running the experiment. Thanks again Michael for your help pinpointing the issue. For the = archives, this is now being tracked under = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D289907 . JN