Date: Sat, 27 Sep 2025 10:26:54 -0600 From: John Nielsen <lists@jnielsen.net> To: Michael Tuexen <michael.tuexen@lurchi.franken.de> Cc: net@freebsd.org, virtualization@freebsd.org, np@freebsd.org Subject: Re: Chelsio (cxgbe) VF on FreeBSD guest under KVM requires -rxcsum Message-ID: <CDFB2AB5-36DD-4B33-9658-9559EEF9BB47@jnielsen.net> In-Reply-To: <FA555678-F299-43BB-BF65-5A429CDF495E@lurchi.franken.de> References: <CF7D677F-2D3A-41B5-A3F1-0FCA844D206F@lurchi.franken.de> <B648CE12-8D42-4978-A50B-DE5E7F39A512@jnielsen.net> <FA555678-F299-43BB-BF65-5A429CDF495E@lurchi.franken.de>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Sep 26, 2025, at 3:44=E2=80=AFPM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote: >=20 >> On 26. Sep 2025, at 20:52, John Nielsen <lists@jnielsen.net> wrote: >>=20 >>> On Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote: >>> =EF=BB=BF >>>>=20 >>>> On 26. Sep 2025, at 02:58, John Nielsen <lists@jnielsen.net> wrote: >>>>=20 >>>> Not sure if this is a known issue or even an issue on the FreeBSD = side but decided to ask here first. I=E2=80=99m happy to put in a bug = report if appropriate. >>>>=20 >>>> I have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected. >>>=20 >>> I don't have access to such a network card. Just to be clear: you = are running the =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D command inside = the guest running FreeBSD, right? >>=20 >> Yes. >>=20 >>> What is the peer, when you mention TCP and UDP do not work? Is it = the host running Linux? Is it another VM? Is it some external host? >>=20 >> My laptop on the same subnet primarily, but. Also tested from another = physical machine running FreeBSD. > OK. That does not seem to be related to what I initially thought. >=20 > Could you run > tcpdump -i outgoing_interface -w laptop.pcap > on your laptop and > tcpdump -i cxlv0 -w vm.pcap > on your vm at the same time and try to do some TCP based = communication. > Maybe two times, one time with ifconfig cxlv0 rxcsum and one time with = ifconfig cxlv0 -rxcsum. >=20 > If you are fine with doing the measurements, you can send the .pcap = files > to tuexen@freebsd.org <mailto:tuexen@freebsd.org> = <mailto:tuexen@freebsd.org>. >=20 > At least I would like to understand what is going on. Thank you, I sent you those packet captures under separate cover. =46rom what I can tell so far the VM does receive incoming packets even = when rxcsum is enabled, and the checksums appear correct at least = according to tcpdump. But something prevents the packets from being = processed or sent up the stack. The VM=E2=80=99s sshd never generates a = SYN/ACK in response to a connection attempt. A DNS query from the VM = looks like it gets a response on the wire but the response doesn=E2=80=99t= make it to the process doing the query. -John --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: = after-white-space;"><div><blockquote type=3D"cite"><div>On Sep 26, 2025, = at 3:44=E2=80=AFPM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote:</div><br = class=3D"Apple-interchange-newline"><div><meta = charset=3D"UTF-8"><blockquote type=3D"cite" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;">On 26. Sep 2025, at 20:52, John Nielsen = <lists@jnielsen.net> wrote:<br><br><blockquote type=3D"cite">On = Sep 26, 2025, at 1:46=E2=80=AFAM, Michael Tuexen = <michael.tuexen@lurchi.franken.de> wrote:<br>=EF=BB=BF<br><blockquot= e type=3D"cite"><br>On 26. Sep 2025, at 02:58, John Nielsen = <lists@jnielsen.net> wrote:<br><br>Not sure if this is a known = issue or even an issue on the FreeBSD side but decided to ask here = first. I=E2=80=99m happy to put in a bug report if appropriate.<br><br>I = have a hypervisor machine running Arch Linux with KVM, Qemu and = libvirtd. The machine has a Chelsio T520-CR adapter. I recently began = passing through virtual functions of the NIC to several of the guests I = run on the hypervisor. One of the guests runs Windows 11, and the change = was seamless. Two of the guests are running FreeBSD (14.3 or so). On = each of them the VFs were readily identified and configured (using DHCP = in one case), and ping and ARP appeared to work fine. However, TCP and = UDP traffic to the guests never received a response. After some = head-scratching and troubleshooting I discovered that running = =E2=80=9Cifconfig cxlv0 -rxcsum=E2=80=9D immediately allowed traffic to = flow as expected.<br></blockquote><br>I don't have access to such a = network card. Just to be clear: you are running the =E2=80=9Cifconfig = cxlv0 -rxcsum=E2=80=9D command inside the guest running FreeBSD, = right?<br></blockquote><br>Yes.<br><br><blockquote type=3D"cite">What is = the peer, when you mention TCP and UDP do not work? Is it the host = running Linux? Is it another VM? Is it some external = host?<br></blockquote><br>My laptop on the same subnet primarily, but. = Also tested from another physical machine running = FreeBSD.<br></blockquote><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">OK. = That does not seem to be related to what I initially thought.</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">Could you run</span><br style=3D"caret-color:= rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">tcpdump = -i outgoing_interface -w laptop.pcap</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">on your = laptop and</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">tcpdump -i cxlv0 -w = vm.pcap</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">on your vm at the same = time and try to do some TCP based communication.</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">Maybe two times, one time with ifconfig = cxlv0 rxcsum and one time with ifconfig cxlv0 -rxcsum.</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">If you are fine with doing the = measurements, you can send the .pcap files</span><br style=3D"caret-color:= rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">to<span = class=3D"Apple-converted-space"> </span></span><a = href=3D"mailto:tuexen@freebsd.org" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;">tuexen@freebsd.org</a><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;"><span = class=3D"Apple-converted-space"> </span><</span><a = href=3D"mailto:tuexen@freebsd.org" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;">mailto:tuexen@freebsd.org</a><span style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline = !important;">>.</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">At = least I would like to understand what is going on.</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: = none;"></div></blockquote></div><br><div>Thank you, I sent you those = packet captures under separate cover.</div><div><br></div><div>=46rom = what I can tell so far the VM does receive incoming packets even when = rxcsum is enabled, and the checksums appear correct at least according = to tcpdump. But something prevents the packets from being processed or = sent up the stack. The VM=E2=80=99s sshd never generates a SYN/ACK in = response to a connection attempt. A DNS query from the VM looks like it = gets a response on the wire but the response doesn=E2=80=99t make it to = the process doing the = query.</div><div><br></div><div>-John</div><div><br></div></body></html>= --Apple-Mail=_A1D93E43-CBCA-46A6-BFD8-7A29BE016627--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CDFB2AB5-36DD-4B33-9658-9559EEF9BB47>