Date: Mon, 27 Jan 2020 01:43:16 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 243640] QEMU / KVM Q35 V4.X PCIe Virtual and Physical (Passthrough) Devices not detected Message-ID: <bug-243640-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243640 Bug ID: 243640 Summary: QEMU / KVM Q35 V4.X PCIe Virtual and Physical (Passthrough) Devices not detected Product: Base System Version: 12.1-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: drum@graphica.com.au Bug / Defect: PCIe attached devices are not detected on when running FreeBSD 12.1 on QEMU= Q35 V 4.x Virtual Machines. This bug affects the following PCIe based devices: VirtIO - All em - When using e1000e QEMU emulator (PCIe attached Intel 1GbE NIC ix - PCI Passthrough Intel X550 10GbE NIC Likely all other PCIe devices whether via emulation or PCI Passthrough Issues was discovered while testing: Q35 with VirtIO: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D236922 Q35 / OVMF with SCSI and Network Devices: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241774 Also appears to be root cause of defect raised with Q35 V4 and PCI Passthro= ugh: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241581 Observed Behaviour: Running QEMU/KVM VM - Q35 V4.x / OVMF / SATA / VirtIO / e1000e / e1000 / vmxnet3 / PCI Passthrough to Intel X550 10GbE / FreeBSD 12.1 with recompiled kernel to disable netmap ("dev netmap") sys/amd64/conf/GENERIC to address f= ound bug with Q35 + netmap devices (see above reported and resolved bugs). SATA - disk found VirtIO - disk not found e1000e - PCIe NIC not found e1000 - Legacy PCI NIC found as em0 vmxnet3 - PCI connected NIC found as vmx0 X550 10GbE - not found. Get the following: dmesg errors <<DMESG>> ... pcib2: <PCI-PCI bridge> mem 0xc8b87000-0xc8b87fff irq 22 at device 2.1 on p= ci0 pcib2: Failed to allocate interrupt for PCI-e events pcib3: <PCI-PCI bridge> mem 0xc8b86000-0xc8b86fff irq 22 at device 2.2 on p= ci0 pcib3: Failed to allocate interrupt for PCI-e events pcib4: <PCI-PCI bridge> mem 0xc8b85000-0xc8b85fff irq 22 at device 2.3 on p= ci0 pcib4: Failed to allocate interrupt for PCI-e events pcib5: <PCI-PCI bridge> mem 0xc8b84000-0xc8b84fff irq 22 at device 2.4 on p= ci0 pcib5: Failed to allocate interrupt for PCI-e events ... <<END DMESG>> ifconfig <<IFCONFIG>> # ifconfig -a lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> vmx0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3De403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:53:01:17:15:aa inet XX.XXX.XXX.53 netmask 0xffffff80 broadcast 203.XXX.XXX.127 media: Ethernet autoselect status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3D81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGI= C,VLAN_HWFILTER> ether 52:54:00:a4:13:df media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> <<END IFCONFIG>> Expected Behaviour: Should get all NIC visible:=20 m0 (e1000), em1 (e1000e), ix0 (X550 10GbE PCI Passthrough), vmx0 (vmxnet3) Should get: -- SATA - /dev/adaNpN storage devices -- SCSI - /dev/daNpN VirtIO SCSI devices I then retested with except with QEMU Q35 V3.1: Storage" SATA - OK VirtIO SCSI - only ok when built with new VirtIO sub-system as per https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D236922 Networking: e1000 - ok comes up as em0 e1000e - ok comes up as em1 vmxnet3 - ok comes up as vmx0 X550 10GbE - ok comes up as ix0 ifconfig <<IFCONFIG>> # ifconfig -a vmx0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3De403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 52:53:01:17:15:aa inet XXX.XXX.XXX.53 netmask 0xffffff80 broadcast XXX.XXX.XXX.127 media: Ethernet autoselect status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3D81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGI= C,VLAN_HWFILTER> ether 52:54:00:a4:13:df media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ix0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3De53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCS= UM_IPV6,TXCSUM_IPV6> ether b4:96:91:21:4a:ce media: Ethernet autoselect status: no carrier nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> em1: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3D81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_= MAGIC,VLAN_HWFILTER> ether 52:54:00:f8:3b:94 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> <<END IFCONFIG>> Here is QEMU Q35 V3.1 with PCI passthrough XML snippet: <<LIBVIRT XML>> virsh dumpxml test-freebsd-12.1=20 <domain type=3D'kvm' id=3D'5'> <name>test-freebsd-12.1</name> <uuid>a50005d7-7425-435f-82e9-e76f18784693</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo=3D"http://libosinfo.org/xmlns/libvirt/domain/1.0">; <libosinfo:os id=3D"http://freebsd.org/freebsd/12.0"/>; </libosinfo:libosinfo> </metadata> <memory unit=3D'KiB'>4194304</memory> <currentMemory unit=3D'KiB'>4194304</currentMemory> <vcpu placement=3D'static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch=3D'x86_64' machine=3D'pc-q35-3.1'>hvm</type> <loader readonly=3D'yes' type=3D'pflash'>/usr/share/OVMF/OVMF_CODE.fd</= loader> <nvram>/home/XXX/DIR/OVMF_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <vmport state=3D'off'/> </features> <cpu mode=3D'custom' match=3D'exact' check=3D'full'> <model fallback=3D'forbid'>Broadwell-IBRS</model> <vendor>Intel</vendor> <feature policy=3D'require' name=3D'vme'/> <feature policy=3D'require' name=3D'ss'/> <feature policy=3D'require' name=3D'vmx'/> <feature policy=3D'require' name=3D'f16c'/> <feature policy=3D'require' name=3D'rdrand'/> <feature policy=3D'require' name=3D'hypervisor'/> <feature policy=3D'require' name=3D'arat'/> <feature policy=3D'require' name=3D'tsc_adjust'/> <feature policy=3D'require' name=3D'umip'/> <feature policy=3D'require' name=3D'md-clear'/> <feature policy=3D'require' name=3D'stibp'/> <feature policy=3D'require' name=3D'arch-capabilities'/> <feature policy=3D'require' name=3D'ssbd'/> <feature policy=3D'require' name=3D'xsaveopt'/> <feature policy=3D'require' name=3D'pdpe1gb'/> <feature policy=3D'require' name=3D'abm'/> <feature policy=3D'disable' name=3D'skip-l1dfl-vmentry'/> </cpu> <clock offset=3D'utc'> <timer name=3D'rtc' tickpolicy=3D'catchup'/> <timer name=3D'pit' tickpolicy=3D'delay'/> <timer name=3D'hpet' present=3D'no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled=3D'no'/> <suspend-to-disk enabled=3D'no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type=3D'file' device=3D'disk'> <driver name=3D'qemu' type=3D'qcow2'/> <source file=3D'/home/XXX/DIR/test-hd1-01.qcow2'/> <backingStore/> <target dev=3D'sda' bus=3D'sata'/> <boot order=3D'1'/> <alias name=3D'sata0-0-0'/> <address type=3D'drive' controller=3D'0' bus=3D'0' target=3D'0' unit= =3D'0'/> </disk> <disk type=3D'file' device=3D'cdrom'> <driver name=3D'qemu' type=3D'raw'/> <source file=3D'/home/XXX/DIR/FreeBSD-12.1-RELEASE-amd64-dvd1.iso'/> <backingStore/> <target dev=3D'sdb' bus=3D'sata'/> <readonly/> <boot order=3D'2'/> <alias name=3D'sata0-0-1'/> <address type=3D'drive' controller=3D'0' bus=3D'0' target=3D'0' unit= =3D'1'/> </disk> ... ... <controller type=3D'sata' index=3D'0'> <alias name=3D'ide'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x1f' function=3D'0x2'/> </controller> <controller type=3D'pci' index=3D'0' model=3D'pcie-root'> <alias name=3D'pcie.0'/> </controller> <controller type=3D'pci' index=3D'1' model=3D'pcie-root-port'> <model name=3D'pcie-root-port'/> <target chassis=3D'1' port=3D'0x10'/> <alias name=3D'pci.1'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02' fu= nction=3D'0x0' multifunction=3D'on'/> </controller> <controller type=3D'pci' index=3D'2' model=3D'pcie-to-pci-bridge'> <model name=3D'pcie-pci-bridge'/> <alias name=3D'pci.2'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x01' slot=3D'0x00' function=3D'0x0'/> </controller> <controller type=3D'pci' index=3D'3' model=3D'pcie-root-port'> <model name=3D'pcie-root-port'/> <target chassis=3D'3' port=3D'0x11'/> <alias name=3D'pci.3'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02' function=3D'0x1'/> </controller> <controller type=3D'pci' index=3D'4' model=3D'pcie-root-port'> <model name=3D'pcie-root-port'/> <target chassis=3D'4' port=3D'0x12'/> <alias name=3D'pci.4'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02' function=3D'0x2'/> </controller> <controller type=3D'pci' index=3D'5' model=3D'pcie-root-port'> <model name=3D'pcie-root-port'/> <target chassis=3D'5' port=3D'0x13'/> <alias name=3D'pci.5'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02' function=3D'0x3'/> </controller> <controller type=3D'pci' index=3D'6' model=3D'pcie-root-port'> <model name=3D'pcie-root-port'/> <target chassis=3D'6' port=3D'0x14'/> <alias name=3D'pci.6'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02' function=3D'0x4'/> </controller> <controller type=3D'virtio-serial' index=3D'0'> <alias name=3D'virtio-serial0'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x03' slot=3D'0x00' function=3D'0x0'/> </controller> <interface type=3D'bridge'> <mac address=3D'52:54:00:18:15:aa'/> <source bridge=3D'br20'/> <target dev=3D'vnet0'/> <model type=3D'vmxnet3'/> <alias name=3D'net0'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x02' slot=3D'0x01' function=3D'0x0'/> </interface> <interface type=3D'bridge'> <mac address=3D'52:54:00:a4:13:df'/> <source bridge=3D'br20'/> <target dev=3D'vnet1'/> <model type=3D'e1000'/> <alias name=3D'net1'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x02' slot=3D'0x02' function=3D'0x0'/> </interface> <interface type=3D'bridge'> <mac address=3D'52:54:00:f8:3b:94'/> <source bridge=3D'br20'/> <target dev=3D'vnet2'/> <model type=3D'e1000e'/> <alias name=3D'net2'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x06' slot=3D'0x00' function=3D'0x0'/> </interface> ... ... <hostdev mode=3D'subsystem' type=3D'pci' managed=3D'yes'> <driver name=3D'vfio'/> <source> <address domain=3D'0x0000' bus=3D'0x06' slot=3D'0x00' function=3D'0= x1'/> </source> <alias name=3D'hostdev0'/> <address type=3D'pci' domain=3D'0x0000' bus=3D'0x05' slot=3D'0x00' function=3D'0x0'/> </hostdev> ... ... </domain> <<END LIBVIRT XML>> Diagnosis: Considerable testing and diagnosis as been done as part of find and resolvi= ng VirtIO and netmap issues. This has found how to replicate and work around the issues Q35 V4.X (Issue)= and Q35 V31 (work around). So problem appears to be within PCI device driver and relates to Q35 V4.x u= sing PCIe GEN4 specification aligned emulation. Tommy P has done considerable work to pin point likely source of issues: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241774#c74 I quote: "Starting with the Q35 QEMU 4.0 machine type, generic pcie-root-po= rt will default to the maximum PCIe link speed (16GT/s) and width (x32) provid= ed by the PCIe 4.0 specification." This now needs to be confirmed by FreeBSD PCI core team for resolution. Please advise if you need additional testing or diagnostic information. Cheers, John Hartley. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-243640-227>