Date: Thu, 10 Feb 2022 09:39:46 -0500 From: Aryeh Friedman <aryeh.friedman@gmail.com> To: Mario Marietto <marietto2008@gmail.com> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: ERRATA CORRIGE "BUG !!!! X64 Exception Type - 06(#UD - Invalid Opcode) CPU Apic ID - 00000000 !!!! while trying to pass thru my devices with bhyve on FreeBSD 14-CURRENT" Message-ID: <CAGBxaX=XzQQxxwtwc4Ky7-Bhib%2Byof_O5BKu3d%2BufVyM=u4S7A@mail.gmail.com> In-Reply-To: <CA%2B1FSiisvAE2=N-ybp1FWsjr6%2BA=zSnVouX4O=%2BfM-=5hf%2BdfQ@mail.gmail.com> References: <CA%2B1FSiisvAE2=N-ybp1FWsjr6%2BA=zSnVouX4O=%2BfM-=5hf%2BdfQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Deleted all the screen dumps since my question is more general: What happens if you do not turn on Xorg on the host and start the VM from the console (likely should make no difference but worth a try). In your original post you never mentioned it worked on 14-CURRENT but not on 13-RELEASE (w/ patches). If this is the case it is almost certainly a bug, but the general policy of most developers is not to make bug fixes for older versions. This means you are likely stuck with 14-CURRENT. On Thu, Feb 10, 2022 at 9:24 AM Mario Marietto <marietto2008@gmail.com> wrote: > Hello. > > I've bought an additional PCI to USB (Renesas) controller that I've > attached to the PCIe Gen3 x4 (Type 22110) of my MOBO Gigabyte Aorus Pro > with Intel I9 chipset and Nvidia Geforce RTX 2080 ti. On this controller > I've attached two USB disks and my KInect 2. The nVidia geforce RTX 2080 ti > is attached to the PCIe Gen3 x 4 (Type 2280). Check this image : > https://ibb.co/fC0jR1p > > check by yourself the logs showing that my passed through devices are > working great : > > dmesg --> https://pastebin.ubuntu.com/p/cSy7jkHdZz/ > pciconf -vl --> https://pastebin.ubuntu.com/p/xZNggrz6D5/ > > On FreeBSD 14 the bhyve setup below works. I can boot Linux passing > through the devices that you see. But I have used a raw image file (not a > physical installation on a sata or nvme disk,because in this case it wont > work and for this I've opened another bug). With this I want to demonstrate > that there aren't any problems with my bios settings and with my passed > thru devices. > > bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H \ > -s 0,hostbridge \ > -s 1,virtio-blk,/mnt/da1p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \ > -s 3:0,passthru,2/0/0 \ > -s 3:1,passthru,2/0/1 \ > -s 3:2,passthru,2/0/2 \ > -s 3:3,passthru,2/0/3 \ > -s 4:0,passthru,1/0/0 \ > -s 9,virtio-net,tap0 \ > -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > -l com1,stdio \ > vm0 > > My host graphic card is not an Nvidia card. It's the graphic chipset > integrated on the motherboard (CoffeeLake-S GT2 [UHD Graphics 630); to use > it I've installed the xf86-video-intel driver with this xorg.conf file : > > Section "Device" > Identifier "Card0" > Driver "intel" > BusID "PCI:0:2:0" > EndSection > > so,now,let's to straight to this problem : > > I've created a new VM with the goal to passthru my graphic card and my usb > controller from the host os (freebsd 14-CURRENT) to the guest os (freebsd > 13R p7) and I've used the parameters below : > > bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \ > -s 0,hostbridge \ > -s 1,ahci-hd,/mnt/da2p2/bhyve/os/BSD/freebsd13vm.img \ > -s 2,passthru,1/0/0 \ > -s 3:0,passthru,2/0/0 \ > -s 3:1,passthru,2/0/1 \ > -s 3:2,passthru,2/0/2 \ > -s 3:3,passthru,2/0/3 \ > -s 8,virtio-net,tap1 \ > -s 9,virtio-9p,sharename=/ \ > -s 29,fbuf,tcp=0.0.0.0:5901,w=1440,h=900,wait \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > -l com1,stdio \ > vm1 > > inside /etc/rc.conf I've added the following parameters : > > linux_enable="YES" > linux_mounts_enable="YES" > vm_enable="YES" > kdm5_enable="YES" > dbus_enable="YES" > sddm_enable="YES" > > on /boot/loader.conf : > > nvidia_load="YES" > vmm_load="YES" > nmdm_load="YES" > if_tap_load="YES" > if_bridge_load="YES" > bridgestp_load="YES" > linprocfs_load="YES" > linsysfs_load="YES" > tmpfs_load="YES" > verbose_loading="YES" > hw.vmm.topology.cores_per_package=8 > kern.racct.enable=1 > snd_hda_load="YES" > > on /etc/sysctl.conf > > net.link.tap.up_on_open=1 > net.inet.ip.forwarding=1 > net.inet.ip.random_id=1 > kern.evdev.rcpt_mask=6 > kern.init_shutdown_timeout="900" > kern.capmode_coredump=1 > > and with this xorg.conf file : > > Section "ServerLayout" > Identifier "X.org Configured" > Screen 0 "Screen0" 0 0 > #Screen 1 "Screen1" RightOf "Screen0" > InputDevice "Mouse0" "CorePointer" > InputDevice "Keyboard0" "CoreKeyboard" > EndSection > > Section "Module" > Load "glx" > EndSection > > Section "Files" > ModulePath "/usr/local/lib/xorg/modules" > FontPath "/usr/local/share/fonts/misc/" > FontPath "/usr/local/share/fonts/TTF/" > FontPath "/usr/local/share/fonts/OTF/" > FontPath "/usr/local/share/fonts/Type1/" > FontPath "/usr/local/share/fonts/100dpi/" > FontPath "/usr/local/share/fonts/75dpi/" > FontPath "catalogue:/usr/local/etc/X11/fontpath.d" > EndSection > > Section "InputDevice" > Identifier "Mouse0" > Driver "mouse" > Option "Protocol" "auto" > Option "Device" "/dev/sysmouse" > Option "ZAxisMapping" "4 5 6 7" > EndSection > > Section "Monitor" > Identifier "Monitor0" > VendorName "Monitor Vendor" > ModelName "Monitor Model" > EndSection > > Section "Device" > ### Available Driver options are:- > ### Values: <i>: integer, <f>: float, <bool>: "True"/"False", > ### <string>: "String", <freq>: "<f> Hz/kHz/MHz", > ### <percent>: "<f>%" > ### [arg]: arg optional > #Option "SWcursor" # [<bool>] > #Option "kmsdev" # <str> > #Option "ShadowFB" # [<bool>] > #Option "AccelMethod" # <str> > #Option "PageFlip" # [<bool>] > #Option "ZaphodHeads" # <str> > #Option "DoubleShadow" # [<bool>] > #Option "Atomic" # [<bool>] > Identifier "Card0" > Driver "nvidia" > BusID "PCI:0:4:0" > EndSection > > this is what happens when I boot it : > > Loading kernel... > /boot/kernel/kernel text=0x17b9e0 text=0xdd6d50 text=0x65ba0c data=0x140 > data=0x1b9348+0x445cb8 syms=[0x8+0x178ed8+0x8+0x1990a2] > Loading configured modules... > snd_hda...if_bridge.../boot/kernel/if_bridge.ko size 0xfc48 at 0x2112000 > loading required module 'bridgestp' > /boot/kernel/bridgestp.ko size 0x8178 at 0x2122000 > tmpfs.../boot/kernel/tmpfs.ko size 0x14820 at 0x212b000 > /etc/hostid.../etc/hostid size=0x25 > /boot/entropy.../boot/entropy size=0x1000 > if_tap...linprocfs.../boot/kernel/linprocfs.ko size 0x10e50 at 0x2141000 > loading required module 'linux_common' > /boot/kernel/linux_common.ko size 0x2c650 at 0x2152000 > nvidia.../boot/modules/nvidia.ko size 0x28c8640 at 0x217f000 > loading required module 'linux' > /boot/kernel/linux.ko size 0x9e070 at 0x4a48000 > nmdm.../boot/kernel/nmdm.ko size 0x4110 at 0x4ae7000 > bridgestp...linsysfs.../boot/kernel/linsysfs.ko size 0x79b8 at 0x4aec000 > vmm.../boot/kernel/vmm.ko size 0x5748f0 at 0x4af4000 > Start @ 0xffffffff8037c000 ... > efi_check_space: Unable to expand staging area > efi_check_space: Unable to expand staging area > EFI framebuffer information: > addr, size 0xc3000000, 0x1000000 > dimensions 1440 x 900 > stride 1440 > masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000 > !!!! X64 Exception Type - 06(#UD - Invalid Opcode) CPU Apic ID - 00000000 > !!!! > RIP - 00000000000C000E, CS - 0000000000000038, RFLAGS - 0000000000010086 > RAX - 0000000054A218D8, RCX - 0000000000000002, RDX - 0000000000000002 > RBX - 0000000005069001, RSP - 00000000BFF58098, RBP - 00000000BFF58100 > RSI - 0000000000001000, RDI - 0000000040001000 > R8 - 0000000000000001, R9 - 00000000BFF580D8, R10 - 00000000B902D6E0 > R11 - 0000000000000D80, R12 - 00000000BE819050, R13 - 0000000005069000 > R14 - 00000000BE819058, R15 - 00000000BE819048 > DS - 0000000000000030, ES - 0000000000000030, FS - 0000000000000030 > GS - 0000000000000030, SS - 0000000000000030 > CR0 - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BFC01000 > CR4 - 0000000000000668, CR8 - 0000000000000000 > DR0 - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000 > DR3 - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400 > GDTR - 00000000BFBEEA98 0000000000000047, LDTR - 0000000000000000 > IDTR - 00000000BF832018 0000000000000FFF, TR - 0000000000000000 > FXSAVE_STATE - 00000000BFF57CF0 > !!!! Can't find image information. !!!! > > it freezes here and I can't do anything anymore. > > -- > Mario. > -- Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org [-- Attachment #2 --] <div dir="ltr"><div>Deleted all the screen dumps since my question is more general: <br></div><div><br></div><div>What happens if you do not turn on Xorg on the host and start the VM from the console (likely should make no difference but worth a try).</div><div><br></div><div>In your original post you never mentioned it worked on 14-CURRENT but not on 13-RELEASE (w/ patches). If this is the case it is almost certainly a bug, but the general policy of most developers is not to make bug fixes for older versions. This means you are likely stuck with 14-CURRENT.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 10, 2022 at 9:24 AM Mario Marietto <<a href="mailto:marietto2008@gmail.com">marietto2008@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">Hello.<br><br><div>I've bought an additional PCI to USB (Renesas) controller that I've attached to the PCIe Gen3 x4 (Type 22110) of my MOBO Gigabyte Aorus Pro with Intel I9 chipset and Nvidia Geforce RTX 2080 ti. On this controller I've attached two USB disks and my KInect 2. The nVidia geforce RTX 2080 ti is attached to the PCIe Gen3 x 4 (Type 2280). Check this image : <a href="https://ibb.co/fC0jR1p" target="_blank">https://ibb.co/fC0jR1p</a></div><br>check by yourself the logs showing that my passed through devices are working great :<br><br>dmesg --> <a href="https://pastebin.ubuntu.com/p/cSy7jkHdZz/" target="_blank">https://pastebin.ubuntu.com/p/cSy7jkHdZz/</a><br>pciconf -vl --> <a href="https://pastebin.ubuntu.com/p/xZNggrz6D5/" target="_blank">https://pastebin.ubuntu.com/p/xZNggrz6D5/</a><br><br>On FreeBSD 14 the bhyve setup below works. I can boot Linux passing through the devices that you see. But I have used a raw image file (not a physical installation on a sata or nvme disk,because in this case it wont work and for this I've opened another bug). With this I want to demonstrate that there aren't any problems with my bios settings and with my passed thru devices.<br><br>bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H \<br>-s 0,hostbridge \<br>-s 1,virtio-blk,/mnt/da1p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \<br>-s 3:0,passthru,2/0/0 \<br>-s 3:1,passthru,2/0/1 \<br>-s 3:2,passthru,2/0/2 \<br>-s 3:3,passthru,2/0/3 \<br>-s 4:0,passthru,1/0/0 \<br>-s 9,virtio-net,tap0 \<br>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5900" target="_blank">0.0.0.0:5900</a>,w=1440,h=900 \<br>-s 30,xhci,tablet \<br>-s 31,lpc \<br>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \<br>-l com1,stdio \<br>vm0<br> <br>My host graphic card is not an Nvidia card. It's the graphic chipset integrated on the motherboard (CoffeeLake-S GT2 [UHD Graphics 630); to use it I've installed the xf86-video-intel driver with this xorg.conf file :<br><br>Section "Device"<br>Identifier "Card0"<br>Driver "intel"<br>BusID "PCI:0:2:0"<br>EndSection<br><br>so,now,let's to straight to this problem :<br><br>I've created a new VM with the goal to passthru my graphic card and my usb controller from the host os (freebsd 14-CURRENT) to the guest os (freebsd 13R p7) and I've used the parameters below :<br><br>bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \<br>-s 0,hostbridge \<br>-s 1,ahci-hd,/mnt/da2p2/bhyve/os/BSD/freebsd13vm.img \<br>-s 2,passthru,1/0/0 \<br>-s 3:0,passthru,2/0/0 \ <br>-s 3:1,passthru,2/0/1 \<br>-s 3:2,passthru,2/0/2 \<br>-s 3:3,passthru,2/0/3 \<br>-s 8,virtio-net,tap1 \<br>-s 9,virtio-9p,sharename=/ \<br>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5901" target="_blank">0.0.0.0:5901</a>,w=1440,h=900,wait \<br>-s 30,xhci,tablet \<br>-s 31,lpc \<br>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \<br>-l com1,stdio \<br>vm1<br><br>inside /etc/rc.conf I've added the following parameters :<br><br>linux_enable="YES"<br>linux_mounts_enable="YES"<br>vm_enable="YES"<br>kdm5_enable="YES"<br>dbus_enable="YES"<br>sddm_enable="YES"<br><br>on /boot/loader.conf :<br><br>nvidia_load="YES"<br>vmm_load="YES"<br>nmdm_load="YES"<br>if_tap_load="YES"<br>if_bridge_load="YES"<br>bridgestp_load="YES"<br>linprocfs_load="YES"<br>linsysfs_load="YES"<br>tmpfs_load="YES"<br>verbose_loading="YES"<br>hw.vmm.topology.cores_per_package=8<br>kern.racct.enable=1<br>snd_hda_load="YES"<br><br>on /etc/sysctl.conf<br><br>net.link.tap.up_on_open=1<br>net.inet.ip.forwarding=1<br>net.inet.ip.random_id=1<br>kern.evdev.rcpt_mask=6<br>kern.init_shutdown_timeout="900"<br>kern.capmode_coredump=1<br><br>and with this xorg.conf file :<br><br>Section "ServerLayout"<br> Identifier "X.org Configured"<br> Screen 0 "Screen0" 0 0<br> #Screen 1 "Screen1" RightOf "Screen0"<br> InputDevice "Mouse0" "CorePointer"<br> InputDevice "Keyboard0" "CoreKeyboard"<br>EndSection<br><br>Section "Module"<br> Load "glx"<br>EndSection<br><br>Section "Files"<br> ModulePath "/usr/local/lib/xorg/modules"<br> FontPath "/usr/local/share/fonts/misc/"<br> FontPath "/usr/local/share/fonts/TTF/"<br> FontPath "/usr/local/share/fonts/OTF/"<br> FontPath "/usr/local/share/fonts/Type1/"<br> FontPath "/usr/local/share/fonts/100dpi/"<br> FontPath "/usr/local/share/fonts/75dpi/"<br> FontPath "catalogue:/usr/local/etc/X11/fontpath.d"<br>EndSection<br><br>Section "InputDevice"<br> Identifier "Mouse0"<br> Driver "mouse"<br> Option "Protocol" "auto"<br> Option "Device" "/dev/sysmouse"<br> Option "ZAxisMapping" "4 5 6 7"<br>EndSection<br><br>Section "Monitor"<br> Identifier "Monitor0"<br> VendorName "Monitor Vendor"<br> ModelName "Monitor Model"<br>EndSection<br><br>Section "Device"<br> ### Available Driver options are:-<br> ### Values: <i>: integer, <f>: float, <bool>: "True"/"False",<br> ### <string>: "String", <freq>: "<f> Hz/kHz/MHz",<br> ### <percent>: "<f>%"<br> ### [arg]: arg optional<br> #Option "SWcursor" # [<bool>]<br> #Option "kmsdev" # <str><br> #Option "ShadowFB" # [<bool>]<br> #Option "AccelMethod" # <str><br> #Option "PageFlip" # [<bool>]<br> #Option "ZaphodHeads" # <str><br> #Option "DoubleShadow" # [<bool>]<br> #Option "Atomic" # [<bool>]<br> Identifier "Card0"<br> Driver "nvidia"<br> BusID "PCI:0:4:0"<br>EndSection<br><br>this is what happens when I boot it :<br><br>Loading kernel...<br>/boot/kernel/kernel text=0x17b9e0 text=0xdd6d50 text=0x65ba0c data=0x140 data=0x1b9348+0x445cb8 syms=[0x8+0x178ed8+0x8+0x1990a2]<br>Loading configured modules...<br>snd_hda...if_bridge.../boot/kernel/if_bridge.ko size 0xfc48 at 0x2112000<br>loading required module 'bridgestp'<br>/boot/kernel/bridgestp.ko size 0x8178 at 0x2122000<br>tmpfs.../boot/kernel/tmpfs.ko size 0x14820 at 0x212b000<br>/etc/hostid.../etc/hostid size=0x25<br>/boot/entropy.../boot/entropy size=0x1000<br>if_tap...linprocfs.../boot/kernel/linprocfs.ko size 0x10e50 at 0x2141000<br>loading required module 'linux_common'<br>/boot/kernel/linux_common.ko size 0x2c650 at 0x2152000<br>nvidia.../boot/modules/nvidia.ko size 0x28c8640 at 0x217f000<br>loading required module 'linux'<br>/boot/kernel/linux.ko size 0x9e070 at 0x4a48000<br>nmdm.../boot/kernel/nmdm.ko size 0x4110 at 0x4ae7000<br>bridgestp...linsysfs.../boot/kernel/linsysfs.ko size 0x79b8 at 0x4aec000<br>vmm.../boot/kernel/vmm.ko size 0x5748f0 at 0x4af4000<br>Start @ 0xffffffff8037c000 ...<br>efi_check_space: Unable to expand staging area<br>efi_check_space: Unable to expand staging area<br>EFI framebuffer information:<br>addr, size 0xc3000000, 0x1000000<br>dimensions 1440 x 900<br>stride 1440<br>masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000<br>!!!! X64 Exception Type - 06(#UD - Invalid Opcode) CPU Apic ID - 00000000 !!!!<br>RIP - 00000000000C000E, CS - 0000000000000038, RFLAGS - 0000000000010086<br>RAX - 0000000054A218D8, RCX - 0000000000000002, RDX - 0000000000000002<br>RBX - 0000000005069001, RSP - 00000000BFF58098, RBP - 00000000BFF58100<br>RSI - 0000000000001000, RDI - 0000000040001000<br>R8 - 0000000000000001, R9 - 00000000BFF580D8, R10 - 00000000B902D6E0<br>R11 - 0000000000000D80, R12 - 00000000BE819050, R13 - 0000000005069000<br>R14 - 00000000BE819058, R15 - 00000000BE819048<br>DS - 0000000000000030, ES - 0000000000000030, FS - 0000000000000030<br>GS - 0000000000000030, SS - 0000000000000030<br>CR0 - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BFC01000<br>CR4 - 0000000000000668, CR8 - 0000000000000000<br>DR0 - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000<br>DR3 - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400<br>GDTR - 00000000BFBEEA98 0000000000000047, LDTR - 0000000000000000<br>IDTR - 00000000BF832018 0000000000000FFF, TR - 0000000000000000<br>FXSAVE_STATE - 00000000BFF57CF0<br>!!!! Can't find image information. !!!!<br><br>it freezes here and I can't do anything anymore.<br clear="all"><br clear="all"></div></div>-- <br><div dir="ltr">Mario.<br></div></div> </blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Aryeh M. Friedman, Lead Developer, <a href="http://www.PetiteCloud.org" target="_blank">http://www.PetiteCloud.org</a><br></div></div>;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGBxaX=XzQQxxwtwc4Ky7-Bhib%2Byof_O5BKu3d%2BufVyM=u4S7A>