From nobody Wed May 4 03:38:40 2022 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 796001AC5420 for ; Wed, 4 May 2022 03:38:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KtMxX1FTgz4ljh for ; Wed, 4 May 2022 03:38:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0ACBE3346 for ; Wed, 4 May 2022 03:38:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2443cd7M005620 for ; Wed, 4 May 2022 03:38:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2443cdOq005619 for virtualization@FreeBSD.org; Wed, 4 May 2022 03:38:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 263768] PCI passthru device not working after guest stop/start Date: Wed, 04 May 2022 03:38:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: me@anatoli.ws X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1651635520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hoSErRdkCxyfGVJL3xvybgwgRuse9Pnx9BPDgDEiIIs=; b=vKHghDGqCHQwZ3VGcJz0P8z2vpQtpIESE8epJ8xzK3tTeb7Qgxz77Rvt3Tu2XlQdNFLpgh 8Bkti+oo4SkiJ3oJAqC404X+XtqqzqW38c7zKSRvvER8DbK7jOmIxbiwZ6CCI+e1N2czAr RJudTeg93f6xQeziviKu9hahzWI7RhH9P8Ycj3KlBjSLHdVOpSsAG4dTxuoFzlmugR5vAH eSzqNGCs0pNHBF+ntC5gt0dfYkCp3lZvVKY6ZDdnbNCjs5PCoccR3zB5AwZmprKWxvJesO 29R1O0wfcylaQJSZPYGVJTnZO7UsDK7WkzabQ9erZdQYupLYQGRk2K2HlLQEGA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1651635520; a=rsa-sha256; cv=none; b=EZucaH+Y4Yp44Fumk7+qAJ2rXWMj36GM5+xSRl53XoMv5JZm2zyQ9Pv8r7957Ysx/EbG5F O8JhA1KzHp4U2znnLyueVWnxYnpfEO5co7wE8yoOME3SzeD1n6RvHxcCqJ3vzxzAb+kK15 w6AL/56SGGfqz4oHvWCHSdCYc/9rfIATGCil/WK+3xKCiOfBQ9+tdPEjBfBPanJd/J/sE9 EMNxgdStCK+gzsHUQl0J3EcrO5xyEoDlW23LUTkpF5+xlIwLpMl+92xv+FEkRtkkt5L8+h 2Gw6gu0TeQzHqhWMF/p7apK/oag5OBu6VmqPU8bZWaKUvV4i8cnoZ+CHY5qg4A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D263768 Bug ID: 263768 Summary: PCI passthru device not working after guest stop/start Product: Base System Version: 13.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: me@anatoli.ws When bhyve starts a guest for the first time (since the host start) with a physical device via ppt, the device works as expected. After a soft reboot of the guest (`sudo reboot` from within, without exiting the bhyve process) the device works as expected. But after a stop / start of the guest (new bhyve process), the device is detected inside the guest but is not working anymore. My guess is that for a correct initialization of the device it should be in some un-initialized state and after the first bhyve run, the guest OS/bhyve= are not de-initializing it and it stays in some soft-broken state. It seems similar to the bug #205549, but not the same. Like the inverse, i.= e. the guest/bhyve don't do something to clean the device. # On the host $ uname -r 13.0-RELEASE-p11 $ sudo pciconf -r ppt0 0:0x3f 15ff8086 00100006 02000002 00800010 b600000c 00000000 00000000 b781800c 00000000 00000000 00000000 1b7b15d9 b9380000 00000040 00000000 00000144 # Guest started for the 1st time (after host reboot) $ sudo vm start test $ sudo pciconf -r ppt0 0:0x3f 15ff8086 00100406 02000002 00800010 b600000c 00000000 00000000 b781800c 00000000 00000000 00000000 1b7b15d9 00000000 00000040 00000000 0000010b # Inside the guest $ uname -a OpenBSD test 7.0 GENERIC.MP#6 amd64 $ dmesg | grep ixl ixl0 at pci0 dev 8 function 0 "Intel X710 10GBaseT" rev 0x02: port 0, FW 7.2.60285 API 1.9, msix, 4 queues, address 3c:ec:ef:21:3d:02 $ ifconfig ixl0 ixl0: flags=3D8843 mtu 1500 lladdr 3c:ec:ef:21:3d:02 index 2 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex) status: active inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255 $ ping -c 1 172.16.1.1 PING 172.16.1.1 (172.16.1.1): 56 data bytes 64 bytes from 172.16.1.1: icmp_seq=3D0 ttl=3D255 time=3D0.594 ms --- 172.16.1.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev =3D 0.594/0.594/0.594/0.000 ms # After reboot of the guest (from within the guest: `sudo reboot`) $ dmesg | grep ixl ixl0 at pci0 dev 8 function 0 "Intel X710 10GBaseT" rev 0x02: port 0, FW 7.2.60285 API 1.9, msix, 4 queues, address 3c:ec:ef:21:3d:02 $ ifconfig ixl0 ixl0: flags=3D8802 mtu 1500 lladdr 3c:ec:ef:21:3d:02 index 2 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex) status: active $ ping -c 1 172.16.1.1 PING 172.16.1.1 (172.16.1.1): 56 data bytes 64 bytes from 172.16.1.1: icmp_seq=3D0 ttl=3D255 time=3D1.674 ms --- 172.16.1.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev =3D 1.674/1.674/1.674/0.000 ms # Guest started for the 2nd time (from within the guest: `shutdown -p now`, then `vm start test`) # On the host (before and after guest start same values) $ sudo pciconf -r ppt0 0:0x3f 15ff8086 00100406 02000002 00800010 b600000c 00000000 00000000 b781800c 00000000 00000000 00000000 1b7b15d9 00000000 00000040 00000000 0000010b # Inside the guest $ dmesg | rg ixl ixl0 at pci0 dev 8 function 0 "Intel X710 10GBaseT" rev 0x02: port 0, FW 0.0.00000 API 0.0, port address is not valid ixl0: no switch config available $ ifconfig ixl0 ixl0: no such interface # The same happens after more guest stops/starts $ sudo vm config test cpu=3D4 memory=3D2G disk0_type=3D"virtio-blk" disk0_name=3D"disk.img" network0_type=3D"virtio-net" network0_switch=3D"public" passthru0=3D"68/0/0=3D8:0" wired_memory=3D"yes" loader=3D"uefi" hostbridge=3D"amd" virt_console0=3D"yes" graphics=3D"no" uuid=3D"98abd580-c123-11ec-bd08-3cecef1c8ff2" network0_mac=3D"58:9c:fc:08:35:3b" $ cat /boot/loader.conf cryptodev_load=3D"YES" zfs_load=3D"YES" vmm_load=3D"YES" nmdm_load=3D"YES" if_tap_load=3D"YES" if_bridge_load=3D"YES" hw.vmm.amdvi.enable=3D"1" pptdevs=3D"68/0/0 68/0/1" $ cat /etc/rc.conf zfs_enable=3D"YES" cloned_interfaces=3D"bridge0 tap0" ifconfig_bridge0=3D"inet 192.168.55.2/24 addm tap0" kld_list=3D"nmdm vmm" vm_enable=3D"YES" vm_dir=3D"zfs:zroot/vm" $ cat vm-bhyve.log --- First start May 03 23:15:06: initialising May 03 23:15:06: [loader: uefi] May 03 23:15:06: [cpu: 4] May 03 23:15:06: [memory: 2G] May 03 23:15:06: [hostbridge: amd] May 03 23:15:06: [com ports: com1] May 03 23:15:06: [uuid: 98abd580-c123-11ec-bd08-3cecef1c8ff2] May 03 23:15:06: [utctime: yes] May 03 23:15:06: [debug mode: no] May 03 23:15:06: [primary disk: disk.img] May 03 23:15:06: [primary disk dev: file] May 03 23:15:06: initialising network device tap1 May 03 23:15:06: adding tap1 -> vm-public (public addm) May 03 23:15:06: bring up tap1 -> vm-public (public addm) May 03 23:15:06: booting May 03 23:15:06: [bhyve options: -c 4 -m 2G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -S -U 98abd580-c123-11ec-bd08-3cecef1c8ff2 -u -S] May 03 23:15:06: [bhyve devices: -s 0,amd_hostbridge -s 31,lpc -s 4:0,virtio-blk,/vm/test/disk.img -s 5:0,virtio-net,tap1,mac=3D58:9c:fc:08:3= 5:3b -s 8:0,passthru,68/0/0 -s 7:0,virtio-console,0=3D/vm/test/vtcon.0] May 03 23:15:06: [bhyve console: -l com1,/dev/nmdm-test.1A] May 03 23:15:06: [bhyve iso device: -s 3:0,ahci-cd,/vm/.config/null.iso] May 03 23:15:06: starting bhyve (run 1) May 03 23:16:50: bhyve exited with status 1 May 03 23:16:50: destroying network device tap1 May 03 23:16:51: stopped --- Second start May 03 23:17:26: initialising May 03 23:17:26: [loader: uefi] May 03 23:17:26: [cpu: 4] May 03 23:17:26: [memory: 2G] May 03 23:17:26: [hostbridge: amd] May 03 23:17:26: [com ports: com1] May 03 23:17:26: [uuid: 98abd580-c123-11ec-bd08-3cecef1c8ff2] May 03 23:17:26: [utctime: yes] May 03 23:17:26: [debug mode: no] May 03 23:17:26: [primary disk: disk.img] May 03 23:17:26: [primary disk dev: file] May 03 23:17:26: initialising network device tap1 May 03 23:17:26: adding tap1 -> vm-public (public addm) May 03 23:17:26: bring up tap1 -> vm-public (public addm) May 03 23:17:26: booting May 03 23:17:26: [bhyve options: -c 4 -m 2G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -S -U 98abd580-c123-11ec-bd08-3cecef1c8ff2 -u -S] May 03 23:17:26: [bhyve devices: -s 0,amd_hostbridge -s 31,lpc -s 4:0,virtio-blk,/vm/test/disk.img -s 5:0,virtio-net,tap1,mac=3D58:9c:fc:08:3= 5:3b -s 8:0,passthru,68/0/0 -s 7:0,virtio-console,0=3D/vm/test/vtcon.0] May 03 23:17:26: [bhyve console: -l com1,/dev/nmdm-test.1A] May 03 23:17:26: [bhyve iso device: -s 3:0,ahci-cd,/vm/.config/null.iso] May 03 23:17:26: starting bhyve (run 1) --=20 You are receiving this mail because: You are the assignee for the bug.=