Date: Thu, 3 Aug 2023 03:48:08 -0100 From: skorpio dr <drskorp@gmail.com> To: freebsd-current@freebsd.org Subject: Bhyve NIC Passthru broken Message-ID: <CAEug4ZiSd2U5w4XdMg94A3Efp6TwMUeSq0auPt5hjbydCaq9AQ@mail.gmail.com>
index | next in thread | raw e-mail
[-- Attachment #1 --]
Hello,
Starting a vm (FreeBSD 13.2) with passthru NIC results a constant flood of
error messages on the host (FreeBSD 14-CURRENT).
The NIC refuses to work in the vm (it works on the host if I turn off the
passthru).
This problem is persistent since 13.1-Release.
(Sidenote: the passthru mostly works under the EOL 13.0-p13 with the same
config, only occasionally showing the symptoms after multiple vm reboots,
and a host reboot solves the problem unlike in 13.1-release 13.2-release
and now 14.0-current)
Ruling out NIC/board hardware fault and specific NIC driver problems, I've
tried with the following NICs:
- Intel I350-t4
- Intel 82580
- Broadcom BCM5720
...in different desktop and server boards (with the latest BIOS):
- Asus M5A99X EVO R2.0
- Asus M4A87TD EVO
- Asus KGPE-D16
# freebsd-version -kru
14.0-CURRENT
14.0-CURRENT
14.0-CURRENT
/boot/loader.conf:
vmm_load="YES"
nmdm_load="YES"
hw.vmm.amdvi.enable=1
pptdevs="8/0/0 8/0/1"
# vm passthru|grep ppt0
ppt0 8/0/0 Yes 82580 Gigabit Network Connection
the vm config file for vm-bhyve:
loader="bhyveload"
cpu=2
memory=4G
disk0_type="virtio-blk"
disk0_name="ptproba.img"
# igb0:
passthru0="8/0/0"
uuid="2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77"
vm-bhyve.log:
Aug 03 02:25:16: initialising
Aug 03 02:25:16: [loader: bhyveload]
Aug 03 02:25:16: [cpu: 2]
Aug 03 02:25:16: [memory: 4G]
Aug 03 02:25:16: [hostbridge: standard]
Aug 03 02:25:16: [com ports: com1]
Aug 03 02:25:16: [uuid: 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77]
Aug 03 02:25:16: [debug mode: no]
Aug 03 02:25:16: [primary disk: ptproba.img]
Aug 03 02:25:16: [primary disk dev: file]
Aug 03 02:25:16: booting
Aug 03 02:25:16: bhyveload -c /dev/nmdm-ptproba.1A -S -m 4G -e
autoboot_delay=3 -d /root/vm-bhyve/ptproba/ptproba.img ptproba
Aug 03 02:25:22: [bhyve options: -c 2 -m 4G -AHP -U
2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77 -u -S]
Aug 03 02:25:22: [bhyve devices: -s 0,hostbridge -s 31,lpc -s
4:0,virtio-blk,/root/vm-bhyve/ptproba/ptproba.img -s 5:0,passthru,8/0/0]
Aug 03 02:25:22: [bhyve console: -l com1,/dev/nmdm-ptproba.1A]
Aug 03 02:25:22: starting bhyve (run 1)
The first few dozen messages are like the followings (couple of hundred
lines):
ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10]
[CMD Total 0x24] Tail:0x240, Head:0x240.
ivhd0: [Event0: Head:0x0 Tail:0x10]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20]
[CMD Total 0x26] Tail:0x260, Head:0x260.
ivhd0: [Event0: Head:0x10 Tail:0x20]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30]
[CMD Total 0x26] Tail:0x260, Head:0x260.
ivhd0: [Event0: Head:0x20 Tail:0x30]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
...then the constant flood (about 35 lines per second in /var/log/messages):
ivhd0: [Event63: Head:0x3e0 Tail:0x8d0]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: [Event64: Head:0x3f0 Tail:0x8d0]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
ivhd0: [Event65: Head:0x400 Tail:0x8d0]
[INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]
Any idea?
Best
[-- Attachment #2 --]
<div dir="ltr">Hello,<br> <br>Starting a vm (FreeBSD 13.2) with passthru NIC results a constant flood of error messages on the host (FreeBSD 14-CURRENT).<br>The NIC refuses to work in the vm (it works on the host if I turn off the passthru).<br>This problem is persistent since 13.1-Release.<br>(Sidenote: the passthru mostly works under the EOL 13.0-p13 with the same config, only occasionally showing the symptoms after multiple vm reboots, and a host reboot solves the problem unlike in 13.1-release 13.2-release and now 14.0-current)<br><br>Ruling out NIC/board hardware fault and specific NIC driver problems, I've tried with the following NICs:<br>- Intel I350-t4<br>- Intel 82580<br>- Broadcom BCM5720<br><br>...in different desktop and server boards (with the latest BIOS):<br>- Asus M5A99X EVO R2.0<br>- Asus M4A87TD EVO<br>- Asus KGPE-D16<br><br> # freebsd-version -kru<br>14.0-CURRENT<br>14.0-CURRENT<br>14.0-CURRENT<br><br> /boot/loader.conf:<br>vmm_load="YES"<br>nmdm_load="YES"<br>hw.vmm.amdvi.enable=1<br>pptdevs="8/0/0 8/0/1"<br><br> # vm passthru|grep ppt0<br>ppt0 8/0/0 Yes 82580 Gigabit Network Connection<br><br> the vm config file for vm-bhyve:<br>loader="bhyveload"<br>cpu=2<br>memory=4G<br>disk0_type="virtio-blk"<br>disk0_name="ptproba.img"<br># igb0:<br>passthru0="8/0/0"<br>uuid="2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77"<br><br>vm-bhyve.log:<br><br>Aug 03 02:25:16: initialising<br>Aug 03 02:25:16: [loader: bhyveload]<br>Aug 03 02:25:16: [cpu: 2]<br>Aug 03 02:25:16: [memory: 4G]<br>Aug 03 02:25:16: [hostbridge: standard]<br>Aug 03 02:25:16: [com ports: com1]<br>Aug 03 02:25:16: [uuid: 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77]<br>Aug 03 02:25:16: [debug mode: no]<br>Aug 03 02:25:16: [primary disk: ptproba.img]<br>Aug 03 02:25:16: [primary disk dev: file]<br>Aug 03 02:25:16: booting<br>Aug 03 02:25:16: bhyveload -c /dev/nmdm-ptproba.1A -S -m 4G -e autoboot_delay=3 -d /root/vm-bhyve/ptproba/ptproba.img ptproba<br>Aug 03 02:25:22: [bhyve options: -c 2 -m 4G -AHP -U 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77 -u -S]<br>Aug 03 02:25:22: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/root/vm-bhyve/ptproba/ptproba.img -s 5:0,passthru,8/0/0]<br>Aug 03 02:25:22: [bhyve console: -l com1,/dev/nmdm-ptproba.1A]<br>Aug 03 02:25:22: starting bhyve (run 1)<br><br>The first few dozen messages are like the followings (couple of hundred lines):<br><br>ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10]<br> [CMD Total 0x24] Tail:0x240, Head:0x240.<br>ivhd0: [Event0: Head:0x0 Tail:0x10]<br> [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br>ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20]<br> [CMD Total 0x26] Tail:0x260, Head:0x260.<br>ivhd0: [Event0: Head:0x10 Tail:0x20]<br> [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br>ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30]<br> [CMD Total 0x26] Tail:0x260, Head:0x260.<br>ivhd0: [Event0: Head:0x20 Tail:0x30]<br> [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br><br>...then the constant flood (about 35 lines per second in /var/log/messages):<br><br>ivhd0: [Event63: Head:0x3e0 Tail:0x8d0]<br> [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br>ivhd0: [Event64: Head:0x3f0 Tail:0x8d0]<br> [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br>ivhd0: [Event65: Head:0x400 Tail:0x8d0]<br> [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br><br>Any idea?<br><br>Best<br></div>
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAEug4ZiSd2U5w4XdMg94A3Efp6TwMUeSq0auPt5hjbydCaq9AQ>