Date: Wed, 04 May 2022 03:38:40 +0000 From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 263768] PCI passthru device not working after guest stop/start Message-ID: <bug-263768-27103@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D263768 Bug ID: 263768 Summary: PCI passthru device not working after guest stop/start Product: Base System Version: 13.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: me@anatoli.ws When bhyve starts a guest for the first time (since the host start) with a physical device via ppt, the device works as expected. After a soft reboot of the guest (`sudo reboot` from within, without exiting the bhyve process) the device works as expected. But after a stop / start of the guest (new bhyve process), the device is detected inside the guest but is not working anymore. My guess is that for a correct initialization of the device it should be in some un-initialized state and after the first bhyve run, the guest OS/bhyve= are not de-initializing it and it stays in some soft-broken state. It seems similar to the bug #205549, but not the same. Like the inverse, i.= e. the guest/bhyve don't do something to clean the device. # On the host $ uname -r 13.0-RELEASE-p11 $ sudo pciconf -r ppt0 0:0x3f 15ff8086 00100006 02000002 00800010 b600000c 00000000 00000000 b781800c 00000000 00000000 00000000 1b7b15d9 b9380000 00000040 00000000 00000144 # Guest started for the 1st time (after host reboot) $ sudo vm start test $ sudo pciconf -r ppt0 0:0x3f 15ff8086 00100406 02000002 00800010 b600000c 00000000 00000000 b781800c 00000000 00000000 00000000 1b7b15d9 00000000 00000040 00000000 0000010b # Inside the guest $ uname -a OpenBSD test 7.0 GENERIC.MP#6 amd64 $ dmesg | grep ixl ixl0 at pci0 dev 8 function 0 "Intel X710 10GBaseT" rev 0x02: port 0, FW 7.2.60285 API 1.9, msix, 4 queues, address 3c:ec:ef:21:3d:02 $ ifconfig ixl0 ixl0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr 3c:ec:ef:21:3d:02 index 2 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex) status: active inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255 $ ping -c 1 172.16.1.1 PING 172.16.1.1 (172.16.1.1): 56 data bytes 64 bytes from 172.16.1.1: icmp_seq=3D0 ttl=3D255 time=3D0.594 ms --- 172.16.1.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev =3D 0.594/0.594/0.594/0.000 ms # After reboot of the guest (from within the guest: `sudo reboot`) $ dmesg | grep ixl ixl0 at pci0 dev 8 function 0 "Intel X710 10GBaseT" rev 0x02: port 0, FW 7.2.60285 API 1.9, msix, 4 queues, address 3c:ec:ef:21:3d:02 $ ifconfig ixl0 ixl0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500 lladdr 3c:ec:ef:21:3d:02 index 2 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex) status: active $ ping -c 1 172.16.1.1 PING 172.16.1.1 (172.16.1.1): 56 data bytes 64 bytes from 172.16.1.1: icmp_seq=3D0 ttl=3D255 time=3D1.674 ms --- 172.16.1.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev =3D 1.674/1.674/1.674/0.000 ms # Guest started for the 2nd time (from within the guest: `shutdown -p now`, then `vm start test`) # On the host (before and after guest start same values) $ sudo pciconf -r ppt0 0:0x3f 15ff8086 00100406 02000002 00800010 b600000c 00000000 00000000 b781800c 00000000 00000000 00000000 1b7b15d9 00000000 00000040 00000000 0000010b # Inside the guest $ dmesg | rg ixl ixl0 at pci0 dev 8 function 0 "Intel X710 10GBaseT" rev 0x02: port 0, FW 0.0.00000 API 0.0, port address is not valid ixl0: no switch config available $ ifconfig ixl0 ixl0: no such interface # The same happens after more guest stops/starts $ sudo vm config test cpu=3D4 memory=3D2G disk0_type=3D"virtio-blk" disk0_name=3D"disk.img" network0_type=3D"virtio-net" network0_switch=3D"public" passthru0=3D"68/0/0=3D8:0" wired_memory=3D"yes" loader=3D"uefi" hostbridge=3D"amd" virt_console0=3D"yes" graphics=3D"no" uuid=3D"98abd580-c123-11ec-bd08-3cecef1c8ff2" network0_mac=3D"58:9c:fc:08:35:3b" $ cat /boot/loader.conf cryptodev_load=3D"YES" zfs_load=3D"YES" vmm_load=3D"YES" nmdm_load=3D"YES" if_tap_load=3D"YES" if_bridge_load=3D"YES" hw.vmm.amdvi.enable=3D"1" pptdevs=3D"68/0/0 68/0/1" $ cat /etc/rc.conf zfs_enable=3D"YES" cloned_interfaces=3D"bridge0 tap0" ifconfig_bridge0=3D"inet 192.168.55.2/24 addm tap0" kld_list=3D"nmdm vmm" vm_enable=3D"YES" vm_dir=3D"zfs:zroot/vm" $ cat vm-bhyve.log --- First start May 03 23:15:06: initialising May 03 23:15:06: [loader: uefi] May 03 23:15:06: [cpu: 4] May 03 23:15:06: [memory: 2G] May 03 23:15:06: [hostbridge: amd] May 03 23:15:06: [com ports: com1] May 03 23:15:06: [uuid: 98abd580-c123-11ec-bd08-3cecef1c8ff2] May 03 23:15:06: [utctime: yes] May 03 23:15:06: [debug mode: no] May 03 23:15:06: [primary disk: disk.img] May 03 23:15:06: [primary disk dev: file] May 03 23:15:06: initialising network device tap1 May 03 23:15:06: adding tap1 -> vm-public (public addm) May 03 23:15:06: bring up tap1 -> vm-public (public addm) May 03 23:15:06: booting May 03 23:15:06: [bhyve options: -c 4 -m 2G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -S -U 98abd580-c123-11ec-bd08-3cecef1c8ff2 -u -S] May 03 23:15:06: [bhyve devices: -s 0,amd_hostbridge -s 31,lpc -s 4:0,virtio-blk,/vm/test/disk.img -s 5:0,virtio-net,tap1,mac=3D58:9c:fc:08:3= 5:3b -s 8:0,passthru,68/0/0 -s 7:0,virtio-console,0=3D/vm/test/vtcon.0] May 03 23:15:06: [bhyve console: -l com1,/dev/nmdm-test.1A] May 03 23:15:06: [bhyve iso device: -s 3:0,ahci-cd,/vm/.config/null.iso] May 03 23:15:06: starting bhyve (run 1) May 03 23:16:50: bhyve exited with status 1 May 03 23:16:50: destroying network device tap1 May 03 23:16:51: stopped --- Second start May 03 23:17:26: initialising May 03 23:17:26: [loader: uefi] May 03 23:17:26: [cpu: 4] May 03 23:17:26: [memory: 2G] May 03 23:17:26: [hostbridge: amd] May 03 23:17:26: [com ports: com1] May 03 23:17:26: [uuid: 98abd580-c123-11ec-bd08-3cecef1c8ff2] May 03 23:17:26: [utctime: yes] May 03 23:17:26: [debug mode: no] May 03 23:17:26: [primary disk: disk.img] May 03 23:17:26: [primary disk dev: file] May 03 23:17:26: initialising network device tap1 May 03 23:17:26: adding tap1 -> vm-public (public addm) May 03 23:17:26: bring up tap1 -> vm-public (public addm) May 03 23:17:26: booting May 03 23:17:26: [bhyve options: -c 4 -m 2G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -S -U 98abd580-c123-11ec-bd08-3cecef1c8ff2 -u -S] May 03 23:17:26: [bhyve devices: -s 0,amd_hostbridge -s 31,lpc -s 4:0,virtio-blk,/vm/test/disk.img -s 5:0,virtio-net,tap1,mac=3D58:9c:fc:08:3= 5:3b -s 8:0,passthru,68/0/0 -s 7:0,virtio-console,0=3D/vm/test/vtcon.0] May 03 23:17:26: [bhyve console: -l com1,/dev/nmdm-test.1A] May 03 23:17:26: [bhyve iso device: -s 3:0,ahci-cd,/vm/.config/null.iso] May 03 23:17:26: starting bhyve (run 1) --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-263768-27103>