Date: Sun, 12 Jan 2020 02:22:38 +0000 From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 241774] FreeBSD 11.3 & 12.0 has broken SCSI & Networking on KVM/QEMU Q35 with OVMF Message-ID: <bug-241774-27103-2ls8RnuHto@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-241774-27103@https.bugs.freebsd.org/bugzilla/> References: <bug-241774-27103@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241774 --- Comment #29 from Tommy P <tommyhp2@gmail.com> --- John & Matt, What is/are the CPU/s for your host OS? Did you verify that the relevant virtual technology features are enabled on the host: IOMMU for AMD and VT/D= (?) for Intel. IIRC, some older versions of Linux (CentOS, Ubuntu, and Fedora) requires additional manual entry in the grub's boot loader. The modern Lin= ux _shouldn't_ but doesn't hurt to confirm: dmesg|grep -i ommu [ 0.000000] AGP: Please enable the IOMMU option in the BIOS setup [ 3.434650] iommu: Adding device 0000:00:00.0 to group 0 <snip/> [ 3.536829] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 3.536834] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40 grep -e '^#' -v /etc/default/grub GRUB_DEFAULT=3D0 GRUB_TIMEOUT=3D10 GRUB_DISTRIBUTOR=3D`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT=3D"" GRUB_CMDLINE_LINUX=3D"" GRUB_DISABLE_SUBMENU=3Dy How are you creating/managing your VMs on the host? I use virt-manager (to reduce typos, misconfiguration, or unsupported configuation errors) on Ubun= tu to manage. Did you customize any of the configurations for the VM in the s= hell script (if used to maintain the VMs) or XML backend? Here's a screen shot for the VM's config: https://imgur.com/a/1WbzvhQ Below is the command line startup for that VM without any manual customizat= ion in the XML configuration file: ps -ef|grep 'fbsd11_ovmf' root 8914 20540 0 15:19 pts/7 00:00:00 grep --color=3Dauto fbsd11_= ovmf libvirt+ 25561 1 40 Jan09 ? 17:21:30 qemu-system-x86_64 -enable-= kvm -name guest=3Dz_fbsd12_ovmf,debug-threads=3Don -S -object secret,id=3DmasterKey0,format=3Draw,file=3D/var/lib/libvirt/qemu/domain-35-= z_fbsd12_ovmf/master-key.aes -machine pc-q35-2.11,accel=3Dkvm,usb=3Doff,vmport=3Doff,dump-guest-core=3Do= ff -cpu Opteron_G5,vme=3Don,ht=3Don,monitor=3Don,osxsave=3Don,bmi1=3Don,mmxext=3Don= ,fxsr_opt=3Don,cmp_legacy=3Don,extapic=3Don,cr8legacy=3Don,osvw=3Don,ibs=3D= on,skinit=3Don,wdt=3Don,tce=3Don,nodeid_msr=3Don,topoext=3Don,perfctr_core= =3Don,perfctr_nb=3Don,invtsc=3Don,ibpb=3Don -drive file=3D/usr/share/OVMF/OVMF_CODE.fd,if=3Dpflash,format=3Draw,unit=3D0,reado= nly=3Don -drive file=3D/var/lib/libvirt/qemu/nvram/z_fbsd12_ovmf_VARS.fd,if=3Dpflash,format= =3Draw,unit=3D1 <...snip...> -no-user-config -nodefaults <...snip...> -rtc base=3Dutc,driftfix=3Dslew -global kvm-pit.lost_tick_policy=3Ddelay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=3D1 -global ICH9-LPC.disable_s4=3D1 -boot menu=3Don,strict=3Don -device i82801b11-bridge,id=3Dpci.1,bus=3Dpcie.0,addr=3D0x1e -device pci-bridge,chassis_nr=3D2,id=3Dpci.2,bus=3Dpci.1,addr=3D0x0 (<<< t= his should be em0 NIC) -device pcie-root-port,port=3D0x10,chassis=3D3,id=3Dpci.3,bus=3Dpcie.0,multifunctio= n=3Don,addr=3D0x2 (<<< this should be the VirtIO NIC) -device pcie-root-port,port=3D0x11,chassis=3D4,id=3Dpci.4,bus=3Dpcie.0,addr= =3D0x2.0x1=20 (<<< sym0 LSI/Symbios SCSI) -device pcie-root-port,port=3D0x12,chassis=3D5,id=3Dpci.5,bus=3Dpcie.0,addr= =3D0x2.0x2=20 (<<< VirtIO SCSI) -device pcie-root-port,port=3D0x13,chassis=3D6,id=3Dpci.6,bus=3Dpcie.0,addr= =3D0x2.0x3=20 (<<< VirtIO Serial) -device pcie-root-port,port=3D0x14,chassis=3D7,id=3Dpci.7,bus=3Dpcie.0,addr= =3D0x2.0x4=20 (<<< VirtIO Memory Balloon) -device pcie-root-port,port=3D0x15,chassis=3D8,id=3Dpci.8,bus=3Dpcie.0,addr= =3D0x2.0x5=20 (<<< 'Controller PCIe 0' seen in the screen shot ?) -device ich9-usb-ehci1,id=3Dusb,bus=3Dpcie.0,addr=3D0x1d.0x7 -device ich9-usb-uhci1,masterbus=3Dusb.0,firstport=3D0,bus=3Dpcie.0,multifunction= =3Don,addr=3D0x1d -device ich9-usb-uhci2,masterbus=3Dusb.0,firstport=3D2,bus=3Dpcie.0,addr=3D= 0x1d.0x1 -device ich9-usb-uhci3,masterbus=3Dusb.0,firstport=3D4,bus=3Dpcie.0,addr=3D= 0x1d.0x2 -device lsi,id=3Dscsi0,bus=3Dpci.4,addr=3D0x0 (<<< sym0 LSI/Symbios = SCSI) -device virtio-scsi-pci,id=3Dscsi1,bus=3Dpci.5,addr=3D0x0 -device virtio-serial-pci,id=3Dvirtio-serial0,bus=3Dpci.6,addr=3D0x0 <...snip...> -netdev tap,fd=3D39,id=3Dhostnet0 -device e1000,netdev=3Dhostnet0,id=3Dnet0,mac=3D52:54:00:4e:d4:8f,bus=3Dpci= .2,addr=3D0x1 -netdev tap,fd=3D41,id=3Dhostnet1,vhost=3Don,vhostfd=3D42 -device virtio-net-pci,netdev=3Dhostnet1,id=3Dnet1,mac=3D52:54:00:a9:08:fb,bus=3Dpc= i.3,addr=3D0x0 <...snip...> -device virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dch= annel0,name=3Dcom.redhat.spice.0 <...snip...> -device qxl-vga,id=3Dvideo0,ram_size=3D67108864,vram_size=3D67108864,vram64_size_mb= =3D0,vgamem_mb=3D16,max_outputs=3D1,bus=3Dpcie.0,addr=3D0x1 <...snip...> -device virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.7,addr=3D0x0 -msg timestamp=3Don --------------------------------------------------------------- 11.3-RELEASE-p5 r356613 boot log messages for em0 NIC: root@fbsd11-ovmf:~ # egrep 'em0' /var/log/messages Jan 10 00:29:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at de= vice 1.0 on pci2 Jan 10 00:29:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 10 00:29:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 10 00:29:26 fbsd11-ovmf kernel: em0: link state changed to UP Jan 10 01:08:26 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at de= vice 1.0 on pci2 Jan 10 01:08:26 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 10 01:08:26 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 10 01:08:26 fbsd11-ovmf kernel: em0: link state changed to UP Jan 10 01:26:48 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at de= vice 1.0 on pci2 Jan 10 01:26:48 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 10 01:26:48 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 10 01:26:48 fbsd11-ovmf kernel: em0: link state changed to UP Jan 11 16:00:03 fbsd11-ovmf kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0x6000-0x603f mem 0x98000000-0x9801ffff irq 21 at de= vice 1.0 on pci2 Jan 11 16:00:03 fbsd11-ovmf kernel: em0: Ethernet address: 52:54:00:4e:d4:8f Jan 11 16:00:03 fbsd11-ovmf kernel: em0: netmap queues/slots: TX 1/256, RX 1/256 Jan 11 16:00:03 fbsd11-ovmf kernel: em0: link state changed to UP --------------------------------------------------------------- Snippets of my custom kernel since 11.2-RELEASE-p15 and upgraded to 11.3-RELEASE-p5 r356613: root@fbsd11-ovmf:~ # grep -e '^#' -v /usr/src/sys/amd64/conf/Custom|grep -e '^$' -v cpu HAMMER ident GENERIC options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption <snip> device acpi options ACPI_DMAR device pci options PCI_HP # PCI-Express native HotPlug options PCI_IOV # PCI SR-IOV support device ahci # AHCI-compatible SATA controllers device ata # Legacy ATA/SATA controllers <snip> device em # Intel PRO/1000 Gigabit Ethernet Family device miibus # MII bus support device re # RealTek 8139C+/8169/8169S/8110S <snip> device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device xhci # XHCI PCI->USB interface (USB 3.0) device usb # USB Bus (required) device ukbd # Keyboard device umass # Disks/Mass storage - Requires scb= us and da device virtio # Generic VirtIO bus (required) device virtio_pci # VirtIO PCI device device vtnet # VirtIO Ethernet device device virtio_blk # VirtIO Block device device virtio_scsi # VirtIO SCSI device device virtio_balloon # VirtIO Memory Balloon device device netmap # netmap(4) support device crypto # Required by IPSEC --------------------------------------------------------------- dmesg of 11.2 & upgraded to 11.3 CPU detection which matches my Opteron 637= 0p: FreeBSD 11.2-RELEASE-p15 #0 r356556: Thu Jan 9 19:50:57 PST 2020 root@fbsd11-ovmf:/usr/obj/usr/src11.2/sys/Custom amd64 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) VT(efifb): resolution 800x600 CPU: AMD Opteron 63xx class CPU (2003.51-MHz K8-class CPU) Origin=3D"AuthenticAMD" Id=3D0x600f20 Family=3D0x15 Model=3D0x2 Stepp= ing=3D0 =20 Features=3D0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,= MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> =20 Features2=3D0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,P= OPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV> AMD Features=3D0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM> AMD Features2=3D0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,= TBM> Structured Extended Features=3D0x8<BMI1> AMD Extended Feature Extensions ID EBX=3D0x1000 SVM: NAsids=3D16 TSC: P-state invariant Hypervisor: Origin =3D "KVMKVMKVM" FreeBSD 11.3-RELEASE-p5 #0 r356613: Sat Jan 11 15:56:34 PST 2020 root@fbsd11-ovmf:/usr/obj/usr/src11.3/sys/Custom amd64 FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0) VT(efifb): resolution 800x600 CPU: AMD Opteron 63xx class CPU (2004.09-MHz K8-class CPU) Origin=3D"AuthenticAMD" Id=3D0x600f20 Family=3D0x15 Model=3D0x2 Stepp= ing=3D0 =20 Features=3D0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,= MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> =20 Features2=3D0xbeb83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,P= OPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV> AMD Features=3D0x26500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,LM> AMD Features2=3D0x210bf7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,XOP,FMA4,= TBM> Structured Extended Features=3D0x8<BMI1> AMD Extended Feature Extensions ID EBX=3D0x1000 SVM: NAsids=3D16 TSC: P-state invariant Hypervisor: Origin =3D "KVMKVMKVM" cat /proc/cpuinfo (on host Ubuntu 18.0.4.3 LTS) processor : 0 vendor_id : AuthenticAMD cpu family : 21 model : 2 model name : AMD Opteron(tm) Processor 6370P stepping : 0 <snip> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca = cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xs= ave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilt= er pfthreshold bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass <snip> --------------------------------------------------------------- This VM was initially created to ensure that my proposed VirtIO patches wou= ld also work in Q35 + OMVF for FreeBSD 12 (hence it's name: z_fbsd12_ovmf). I= 've since used this configuration to install FreeBSD 11.2 to test viability of backporting my proposed patches. The order of the devices seen in the comm= and line startup matches the order of customizations I've made/added in the GUI virt-manager during the creation of the VM. Did you have a chane to test my proposed VirtIO patches for FreeBSD 12.x within your environment? The Virt= IO devices should work along with SATA storage devices and em NICs. I'm unsure about the VirtIO + netmap. Regards, Tommy --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-241774-27103-2ls8RnuHto>