From owner-freebsd-bugs@freebsd.org  Mon Jan 27 01:43:16 2020
Return-Path: <owner-freebsd-bugs@freebsd.org>
Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id BC0E722B999
 for <freebsd-bugs@mailman.nyi.freebsd.org>;
 Mon, 27 Jan 2020 01:43:16 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::50:13])
 by mx1.freebsd.org (Postfix) with ESMTP id 485XZX5PD3z3FwD
 for <freebsd-bugs@freebsd.org>; Mon, 27 Jan 2020 01:43:16 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: by mailman.nyi.freebsd.org (Postfix)
 id B8D7422B998; Mon, 27 Jan 2020 01:43:16 +0000 (UTC)
Delivered-To: bugs@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id B89B422B997
 for <bugs@mailman.nyi.freebsd.org>; Mon, 27 Jan 2020 01:43:16 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 485XZX5Fbpz3FwC
 for <bugs@FreeBSD.org>; Mon, 27 Jan 2020 01:43:16 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org (kenobi.freebsd.org
 [IPv6:2610:1c1:1:606c::50:1d])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AEE5A267A
 for <bugs@FreeBSD.org>; Mon, 27 Jan 2020 01:43:16 +0000 (UTC)
 (envelope-from bugzilla-noreply@freebsd.org)
Received: from kenobi.freebsd.org ([127.0.1.5])
 by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 00R1hGrM053742
 for <bugs@FreeBSD.org>; Mon, 27 Jan 2020 01:43:16 GMT
 (envelope-from bugzilla-noreply@freebsd.org)
Received: (from www@localhost)
 by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 00R1hGL4053741
 for bugs@FreeBSD.org; Mon, 27 Jan 2020 01:43:16 GMT
 (envelope-from bugzilla-noreply@freebsd.org)
X-Authentication-Warning: kenobi.freebsd.org: www set sender to
 bugzilla-noreply@freebsd.org using -f
From: bugzilla-noreply@freebsd.org
To: bugs@FreeBSD.org
Subject: [Bug 243640] QEMU / KVM Q35 V4.X PCIe Virtual and Physical
 (Passthrough) Devices not detected
Date: Mon, 27 Jan 2020 01:43:16 +0000
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Base System
X-Bugzilla-Component: kern
X-Bugzilla-Version: 12.1-STABLE
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Affects Only Me
X-Bugzilla-Who: drum@graphica.com.au
X-Bugzilla-Status: New
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: ---
X-Bugzilla-Assigned-To: bugs@FreeBSD.org
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform
 op_sys bug_status bug_severity priority component assigned_to reporter
Message-ID: <bug-243640-227@https.bugs.freebsd.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-BeenThere: freebsd-bugs@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Bug reports <freebsd-bugs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-bugs>,
 <mailto:freebsd-bugs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-bugs/>
List-Post: <mailto:freebsd-bugs@freebsd.org>
List-Help: <mailto:freebsd-bugs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-bugs>,
 <mailto:freebsd-bugs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 27 Jan 2020 01:43:16 -0000

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D243640

            Bug ID: 243640
           Summary: QEMU / KVM Q35 V4.X PCIe Virtual and Physical
                    (Passthrough) Devices not detected
           Product: Base System
           Version: 12.1-STABLE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: drum@graphica.com.au

Bug / Defect:

PCIe attached devices are not detected on when running FreeBSD 12.1 on QEMU=
 Q35
V 4.x Virtual Machines.

This bug affects the following PCIe based devices:

VirtIO - All
em - When using e1000e QEMU emulator (PCIe attached Intel 1GbE NIC
ix - PCI Passthrough Intel X550 10GbE NIC

Likely all other PCIe devices whether via emulation or PCI Passthrough

Issues was discovered while testing:

Q35 with VirtIO:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D236922

Q35 / OVMF with SCSI and Network Devices:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241774

Also appears to be root cause of defect raised with Q35 V4 and PCI Passthro=
ugh:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241581


Observed Behaviour:

Running QEMU/KVM VM - Q35 V4.x / OVMF / SATA / VirtIO / e1000e / e1000 /
vmxnet3 / PCI Passthrough to Intel X550 10GbE / FreeBSD 12.1 with recompiled
kernel to disable netmap ("dev netmap") sys/amd64/conf/GENERIC to address f=
ound
bug with Q35 + netmap devices (see above reported and resolved bugs).

SATA - disk found
VirtIO - disk not found
e1000e - PCIe NIC not found
e1000 - Legacy PCI NIC found as em0
vmxnet3 - PCI connected NIC found as vmx0
X550 10GbE - not found.

Get the following:

dmesg errors

<<DMESG>>
...
pcib2: <PCI-PCI bridge> mem 0xc8b87000-0xc8b87fff irq 22 at device 2.1 on p=
ci0
pcib2: Failed to allocate interrupt for PCI-e events
pcib3: <PCI-PCI bridge> mem 0xc8b86000-0xc8b86fff irq 22 at device 2.2 on p=
ci0
pcib3: Failed to allocate interrupt for PCI-e events
pcib4: <PCI-PCI bridge> mem 0xc8b85000-0xc8b85fff irq 22 at device 2.3 on p=
ci0
pcib4: Failed to allocate interrupt for PCI-e events
pcib5: <PCI-PCI bridge> mem 0xc8b84000-0xc8b84fff irq 22 at device 2.4 on p=
ci0
pcib5: Failed to allocate interrupt for PCI-e events
...
<<END DMESG>>

ifconfig

<<IFCONFIG>>
# ifconfig -a
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
vmx0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
=20=20=20=20=20=20=20
options=3De403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU=
M,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 52:53:01:17:15:aa
        inet XX.XXX.XXX.53 netmask 0xffffff80 broadcast 203.XXX.XXX.127
        media: Ethernet autoselect
        status: active
        nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
=20=20=20=20=20=20=20
options=3D81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGI=
C,VLAN_HWFILTER>
        ether 52:54:00:a4:13:df
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<<END IFCONFIG>>


Expected Behaviour:

Should get all NIC visible:=20
m0 (e1000), em1 (e1000e), ix0 (X550 10GbE PCI Passthrough), vmx0 (vmxnet3)

Should get:
-- SATA - /dev/adaNpN storage devices
-- SCSI - /dev/daNpN VirtIO SCSI devices


I then retested with except with QEMU Q35 V3.1:

Storage"
SATA - OK
VirtIO SCSI - only ok when built with new VirtIO sub-system as per
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D236922

Networking:
e1000 - ok comes up as em0
e1000e - ok comes up as em1
vmxnet3 - ok comes up as vmx0
X550 10GbE - ok comes up as ix0

ifconfig

<<IFCONFIG>>
# ifconfig -a
vmx0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
=20=20=20=20=20=20=20
options=3De403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU=
M,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 52:53:01:17:15:aa
        inet XXX.XXX.XXX.53 netmask 0xffffff80 broadcast XXX.XXX.XXX.127
        media: Ethernet autoselect
        status: active
        nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
=20=20=20=20=20=20=20
options=3D81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGI=
C,VLAN_HWFILTER>
        ether 52:54:00:a4:13:df
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
=20=20=20=20=20=20=20
options=3De53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU=
M,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCS=
UM_IPV6,TXCSUM_IPV6>
        ether b4:96:91:21:4a:ce
        media: Ethernet autoselect
        status: no carrier
        nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
em1: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
=20=20=20=20=20=20=20
options=3D81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_=
MAGIC,VLAN_HWFILTER>
        ether 52:54:00:f8:3b:94
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL>
<<END IFCONFIG>>

Here is QEMU Q35 V3.1 with PCI passthrough XML snippet:

<<LIBVIRT XML>>
virsh dumpxml test-freebsd-12.1=20
<domain type=3D'kvm' id=3D'5'>
  <name>test-freebsd-12.1</name>
  <uuid>a50005d7-7425-435f-82e9-e76f18784693</uuid>
  <metadata>
    <libosinfo:libosinfo
xmlns:libosinfo=3D"http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id=3D"http://freebsd.org/freebsd/12.0"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit=3D'KiB'>4194304</memory>
  <currentMemory unit=3D'KiB'>4194304</currentMemory>
  <vcpu placement=3D'static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch=3D'x86_64' machine=3D'pc-q35-3.1'>hvm</type>
    <loader readonly=3D'yes' type=3D'pflash'>/usr/share/OVMF/OVMF_CODE.fd</=
loader>
    <nvram>/home/XXX/DIR/OVMF_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state=3D'off'/>
  </features>
  <cpu mode=3D'custom' match=3D'exact' check=3D'full'>
    <model fallback=3D'forbid'>Broadwell-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy=3D'require' name=3D'vme'/>
    <feature policy=3D'require' name=3D'ss'/>
    <feature policy=3D'require' name=3D'vmx'/>
    <feature policy=3D'require' name=3D'f16c'/>
    <feature policy=3D'require' name=3D'rdrand'/>
    <feature policy=3D'require' name=3D'hypervisor'/>
    <feature policy=3D'require' name=3D'arat'/>
    <feature policy=3D'require' name=3D'tsc_adjust'/>
    <feature policy=3D'require' name=3D'umip'/>
    <feature policy=3D'require' name=3D'md-clear'/>
    <feature policy=3D'require' name=3D'stibp'/>
    <feature policy=3D'require' name=3D'arch-capabilities'/>
    <feature policy=3D'require' name=3D'ssbd'/>
    <feature policy=3D'require' name=3D'xsaveopt'/>
    <feature policy=3D'require' name=3D'pdpe1gb'/>
    <feature policy=3D'require' name=3D'abm'/>
    <feature policy=3D'disable' name=3D'skip-l1dfl-vmentry'/>
  </cpu>
  <clock offset=3D'utc'>
    <timer name=3D'rtc' tickpolicy=3D'catchup'/>
    <timer name=3D'pit' tickpolicy=3D'delay'/>
    <timer name=3D'hpet' present=3D'no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled=3D'no'/>
    <suspend-to-disk enabled=3D'no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type=3D'file' device=3D'disk'>
      <driver name=3D'qemu' type=3D'qcow2'/>
      <source file=3D'/home/XXX/DIR/test-hd1-01.qcow2'/>
      <backingStore/>
      <target dev=3D'sda' bus=3D'sata'/>
      <boot order=3D'1'/>
      <alias name=3D'sata0-0-0'/>
      <address type=3D'drive' controller=3D'0' bus=3D'0' target=3D'0' unit=
=3D'0'/>
    </disk>
    <disk type=3D'file' device=3D'cdrom'>
      <driver name=3D'qemu' type=3D'raw'/>
      <source file=3D'/home/XXX/DIR/FreeBSD-12.1-RELEASE-amd64-dvd1.iso'/>
      <backingStore/>
      <target dev=3D'sdb' bus=3D'sata'/>
      <readonly/>
      <boot order=3D'2'/>
      <alias name=3D'sata0-0-1'/>
      <address type=3D'drive' controller=3D'0' bus=3D'0' target=3D'0' unit=
=3D'1'/>
    </disk>
...
...
    <controller type=3D'sata' index=3D'0'>
      <alias name=3D'ide'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x1f'
function=3D'0x2'/>
    </controller>
    <controller type=3D'pci' index=3D'0' model=3D'pcie-root'>
      <alias name=3D'pcie.0'/>
    </controller>
    <controller type=3D'pci' index=3D'1' model=3D'pcie-root-port'>
      <model name=3D'pcie-root-port'/>
      <target chassis=3D'1' port=3D'0x10'/>
      <alias name=3D'pci.1'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02' fu=
nction=3D'0x0'
multifunction=3D'on'/>
    </controller>
    <controller type=3D'pci' index=3D'2' model=3D'pcie-to-pci-bridge'>
      <model name=3D'pcie-pci-bridge'/>
      <alias name=3D'pci.2'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x01' slot=3D'0x00'
function=3D'0x0'/>
    </controller>
    <controller type=3D'pci' index=3D'3' model=3D'pcie-root-port'>
      <model name=3D'pcie-root-port'/>
      <target chassis=3D'3' port=3D'0x11'/>
      <alias name=3D'pci.3'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02'
function=3D'0x1'/>
    </controller>
    <controller type=3D'pci' index=3D'4' model=3D'pcie-root-port'>
      <model name=3D'pcie-root-port'/>
      <target chassis=3D'4' port=3D'0x12'/>
      <alias name=3D'pci.4'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02'
function=3D'0x2'/>
    </controller>
    <controller type=3D'pci' index=3D'5' model=3D'pcie-root-port'>
      <model name=3D'pcie-root-port'/>
      <target chassis=3D'5' port=3D'0x13'/>
      <alias name=3D'pci.5'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02'
function=3D'0x3'/>
    </controller>
    <controller type=3D'pci' index=3D'6' model=3D'pcie-root-port'>
      <model name=3D'pcie-root-port'/>
      <target chassis=3D'6' port=3D'0x14'/>
      <alias name=3D'pci.6'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x02'
function=3D'0x4'/>
    </controller>
    <controller type=3D'virtio-serial' index=3D'0'>
      <alias name=3D'virtio-serial0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x03' slot=3D'0x00'
function=3D'0x0'/>
    </controller>
    <interface type=3D'bridge'>
      <mac address=3D'52:54:00:18:15:aa'/>
      <source bridge=3D'br20'/>
      <target dev=3D'vnet0'/>
      <model type=3D'vmxnet3'/>
      <alias name=3D'net0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x02' slot=3D'0x01'
function=3D'0x0'/>
    </interface>
    <interface type=3D'bridge'>
      <mac address=3D'52:54:00:a4:13:df'/>
      <source bridge=3D'br20'/>
      <target dev=3D'vnet1'/>
      <model type=3D'e1000'/>
      <alias name=3D'net1'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x02' slot=3D'0x02'
function=3D'0x0'/>
    </interface>
    <interface type=3D'bridge'>
      <mac address=3D'52:54:00:f8:3b:94'/>
      <source bridge=3D'br20'/>
      <target dev=3D'vnet2'/>
      <model type=3D'e1000e'/>
      <alias name=3D'net2'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x06' slot=3D'0x00'
function=3D'0x0'/>
    </interface>
...
...
    <hostdev mode=3D'subsystem' type=3D'pci' managed=3D'yes'>
      <driver name=3D'vfio'/>
      <source>
        <address domain=3D'0x0000' bus=3D'0x06' slot=3D'0x00' function=3D'0=
x1'/>
      </source>
      <alias name=3D'hostdev0'/>
      <address type=3D'pci' domain=3D'0x0000' bus=3D'0x05' slot=3D'0x00'
function=3D'0x0'/>
    </hostdev>
...
...

</domain>

<<END LIBVIRT XML>>

Diagnosis:

Considerable testing and diagnosis as been done as part of find and resolvi=
ng
VirtIO and netmap issues.

This has found how to replicate and work around the issues Q35 V4.X (Issue)=
 and
Q35 V31 (work around).

So problem appears to be within PCI device driver and relates to Q35 V4.x u=
sing
PCIe GEN4 specification aligned emulation.

Tommy P has done considerable work to pin point likely source of issues:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241774#c74

I quote: "Starting with the Q35 QEMU 4.0 machine type, generic pcie-root-po=
rt
will default to the maximum PCIe link speed (16GT/s) and width (x32) provid=
ed
by the PCIe 4.0 specification."

This now needs to be confirmed by FreeBSD PCI core team for resolution.

Please advise if you need additional testing or diagnostic information.

Cheers,


John Hartley.

--=20
You are receiving this mail because:
You are the assignee for the bug.=