Date: Thu, 3 Aug 2023 03:48:08 -0100 From: skorpio dr <drskorp@gmail.com> To: freebsd-current@freebsd.org Subject: Bhyve NIC Passthru broken Message-ID: <CAEug4ZiSd2U5w4XdMg94A3Efp6TwMUeSq0auPt5hjbydCaq9AQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--0000000000003c36770601fd7d3f Content-Type: text/plain; charset="UTF-8" Hello, Starting a vm (FreeBSD 13.2) with passthru NIC results a constant flood of error messages on the host (FreeBSD 14-CURRENT). The NIC refuses to work in the vm (it works on the host if I turn off the passthru). This problem is persistent since 13.1-Release. (Sidenote: the passthru mostly works under the EOL 13.0-p13 with the same config, only occasionally showing the symptoms after multiple vm reboots, and a host reboot solves the problem unlike in 13.1-release 13.2-release and now 14.0-current) Ruling out NIC/board hardware fault and specific NIC driver problems, I've tried with the following NICs: - Intel I350-t4 - Intel 82580 - Broadcom BCM5720 ...in different desktop and server boards (with the latest BIOS): - Asus M5A99X EVO R2.0 - Asus M4A87TD EVO - Asus KGPE-D16 # freebsd-version -kru 14.0-CURRENT 14.0-CURRENT 14.0-CURRENT /boot/loader.conf: vmm_load="YES" nmdm_load="YES" hw.vmm.amdvi.enable=1 pptdevs="8/0/0 8/0/1" # vm passthru|grep ppt0 ppt0 8/0/0 Yes 82580 Gigabit Network Connection the vm config file for vm-bhyve: loader="bhyveload" cpu=2 memory=4G disk0_type="virtio-blk" disk0_name="ptproba.img" # igb0: passthru0="8/0/0" uuid="2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77" vm-bhyve.log: Aug 03 02:25:16: initialising Aug 03 02:25:16: [loader: bhyveload] Aug 03 02:25:16: [cpu: 2] Aug 03 02:25:16: [memory: 4G] Aug 03 02:25:16: [hostbridge: standard] Aug 03 02:25:16: [com ports: com1] Aug 03 02:25:16: [uuid: 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77] Aug 03 02:25:16: [debug mode: no] Aug 03 02:25:16: [primary disk: ptproba.img] Aug 03 02:25:16: [primary disk dev: file] Aug 03 02:25:16: booting Aug 03 02:25:16: bhyveload -c /dev/nmdm-ptproba.1A -S -m 4G -e autoboot_delay=3 -d /root/vm-bhyve/ptproba/ptproba.img ptproba Aug 03 02:25:22: [bhyve options: -c 2 -m 4G -AHP -U 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77 -u -S] Aug 03 02:25:22: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/root/vm-bhyve/ptproba/ptproba.img -s 5:0,passthru,8/0/0] Aug 03 02:25:22: [bhyve console: -l com1,/dev/nmdm-ptproba.1A] Aug 03 02:25:22: starting bhyve (run 1) The first few dozen messages are like the followings (couple of hundred lines): ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10] [CMD Total 0x24] Tail:0x240, Head:0x240. ivhd0: [Event0: Head:0x0 Tail:0x10] [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20] [CMD Total 0x26] Tail:0x260, Head:0x260. ivhd0: [Event0: Head:0x10 Tail:0x20] [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30] [CMD Total 0x26] Tail:0x260, Head:0x260. ivhd0: [Event0: Head:0x20 Tail:0x30] [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] ...then the constant flood (about 35 lines per second in /var/log/messages): ivhd0: [Event63: Head:0x3e0 Tail:0x8d0] [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] ivhd0: [Event64: Head:0x3f0 Tail:0x8d0] [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] ivhd0: [Event65: Head:0x400 Tail:0x8d0] [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] Any idea? Best --0000000000003c36770601fd7d3f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hello,<br>=C2=A0<br>Starting a vm (FreeBSD 13.2) with pass= thru NIC results a constant flood of error messages on the host (FreeBSD 14= -CURRENT).<br>The NIC refuses to work in the vm (it works on the host if I = turn off the passthru).<br>This problem is persistent since 13.1-Release.<b= r>(Sidenote: the passthru mostly works under the EOL 13.0-p13 with the same= config, only occasionally showing the symptoms after multiple vm reboots, = and a host reboot solves the problem unlike in 13.1-release 13.2-release an= d now 14.0-current)<br><br>Ruling out NIC/board hardware fault and specific= NIC driver problems, I've tried with the following NICs:<br>- Intel I3= 50-t4<br>- Intel 82580<br>- Broadcom BCM5720<br><br>...in different desktop= and server boards (with the latest BIOS):<br>- Asus M5A99X EVO R2.0<br>- A= sus M4A87TD EVO<br>- Asus KGPE-D16<br><br>=C2=A0 # freebsd-version -kru<br>= 14.0-CURRENT<br>14.0-CURRENT<br>14.0-CURRENT<br><br>=C2=A0 /boot/loader.con= f:<br>vmm_load=3D"YES"<br>nmdm_load=3D"YES"<br>hw.vmm.a= mdvi.enable=3D1<br>pptdevs=3D"8/0/0 8/0/1"<br><br>=C2=A0 # vm pas= sthru|grep ppt0<br>ppt0 =C2=A0 =C2=A0 =C2=A0 8/0/0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0Yes =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A082580 Gigabit Network Connection<b= r><br>=C2=A0 the vm config file for vm-bhyve:<br>loader=3D"bhyveload&q= uot;<br>cpu=3D2<br>memory=3D4G<br>disk0_type=3D"virtio-blk"<br>di= sk0_name=3D"ptproba.img"<br># igb0:<br>passthru0=3D"8/0/0&qu= ot;<br>uuid=3D"2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77"<br><br>vm-bh= yve.log:<br><br>Aug 03 02:25:16: initialising<br>Aug 03 02:25:16: =C2=A0[lo= ader: bhyveload]<br>Aug 03 02:25:16: =C2=A0[cpu: 2]<br>Aug 03 02:25:16: =C2= =A0[memory: 4G]<br>Aug 03 02:25:16: =C2=A0[hostbridge: standard]<br>Aug 03 = 02:25:16: =C2=A0[com ports: com1]<br>Aug 03 02:25:16: =C2=A0[uuid: 2333ba09= -dfe1-11ed-9ed6-9c5c8e54ad77]<br>Aug 03 02:25:16: =C2=A0[debug mode: no]<br= >Aug 03 02:25:16: =C2=A0[primary disk: ptproba.img]<br>Aug 03 02:25:16: =C2= =A0[primary disk dev: file]<br>Aug 03 02:25:16: booting<br>Aug 03 02:25:16:= bhyveload -c /dev/nmdm-ptproba.1A -S -m 4G -e autoboot_delay=3D3 -d /root/= vm-bhyve/ptproba/ptproba.img ptproba<br>Aug 03 02:25:22: =C2=A0[bhyve optio= ns: -c 2 -m 4G -AHP -U 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77 -u -S]<br>Aug 0= 3 02:25:22: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-b= lk,/root/vm-bhyve/ptproba/ptproba.img -s 5:0,passthru,8/0/0]<br>Aug 03 02:2= 5:22: =C2=A0[bhyve console: -l com1,/dev/nmdm-ptproba.1A]<br>Aug 03 02:25:2= 2: starting bhyve (run 1)<br><br>The first few dozen messages are like the = followings (couple of hundred lines):<br><br>ivhd0: EVT INTR 0 Status:0x1a = EVT Head:0x0 Tail:0x10]<br>=C2=A0 [CMD Total 0x24] Tail:0x240, Head:0x240.<= br>ivhd0: =C2=A0[Event0: Head:0x0 Tail:0x10]<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0= [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br>ivhd0: EVT INTR 1 = Status:0x1a EVT Head:0x10 Tail:0x20]<br>=C2=A0 [CMD Total 0x26] Tail:0x260,= Head:0x260.<br>ivhd0: =C2=A0[Event0: Head:0x10 Tail:0x20]<br>=C2=A0 =C2=A0= =C2=A0 =C2=A0 [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br>ivhd= 0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30]<br>=C2=A0 [CMD Total 0x2= 6] Tail:0x260, Head:0x260.<br>ivhd0: =C2=A0[Event0: Head:0x20 Tail:0x30]<br= >=C2=A0 =C2=A0 =C2=A0 =C2=A0 [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3= tr:0]<br><br>...then the constant flood (about 35 lines per second in /var= /log/messages):<br><br>ivhd0: =C2=A0[Event63: Head:0x3e0 Tail:0x8d0]<br>=C2= =A0 =C2=A0 =C2=A0 =C2=A0 [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:= 0]<br>ivhd0: =C2=A0[Event64: Head:0x3f0 Tail:0x8d0]<br>=C2=A0 =C2=A0 =C2=A0= =C2=A0 [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br>ivhd0: =C2= =A0[Event65: Head:0x400 Tail:0x8d0]<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 [INV_DTE= devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0]<br><br>Any idea?<br><br>Best<b= r></div> --0000000000003c36770601fd7d3f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAEug4ZiSd2U5w4XdMg94A3Efp6TwMUeSq0auPt5hjbydCaq9AQ>