Date: Thu, 28 Feb 2019 11:58:16 -0600 From: Kevin Day <toasty@dragondata.com> To: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve PCIe passthrough on AMD Epyc Message-ID: <98792A76-1723-458D-A7B4-BB3F74D9F26C@dragondata.com> In-Reply-To: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net> References: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes =
<freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote:
>=20
>>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes =
<freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote:
>>>=20
>>>> I'm trying to get PCIe device (video card) passthrough working to a =
Windows bhyve VM, should this work now?
>>>=20
>>> No, this should not work now.
>>=20
>> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right =
now?
> I do not know what would be special about Epyc so can not say
> for certain.
I'm just pointing out that it's an AMD Epyc because I know the AMD =
IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is =
maybe less tested.
>> Just trying the same thing with a simple PCIe USB card gives me:
>>=20
>> CPU0: local APIC error 0x40
>> CPU0: local APIC error 0x40
>> ivhd0: Error: completion failed tail:0xda0, head:0x0.
>> ivhd0: Dump all the commands:
>> [CMD0, off:0xff0] opcode=3D 0x0 0x0 0x0 0x0
>> [CMD1, off:0x0] opcode=3D 0x3 0x0 0x0 0x7ffffffffffff003
>> [CMD2, off:0x10] opcode=3D 0x0 0x0 0x0 0x0
>> [CMD3, off:0x20] opcode=3D 0x2 0x0 0x0 0x0
>> [CMD4, off:0x30] opcode=3D 0x2 0x2 0x0 0x0
>> [...snip... lots of similar stuff]
>>=20
>> Then I get a bunch more APIC errors, and the system freezes for about =
60 seconds before recovering.
>=20
> You do not get any host dmesg errors about unaligned BAR
> ranges or sizes?
>=20
Correct. I did with the video card, but not with the xhci USB card. I =
get a ton of dmesg errors, and the host system repeatedly freezes until =
I stop the VM. (Console is non-responsive, no ICMP replies, etc)
I didn't actually look at this until now, but Windows actually does see =
the USB controller, but it complains that "port reset failed" on any =
device I attach to it.
> Can you get the pciconf -l -b info for the device you passed, =
something like:
> # pciconf -l -b uhci0
> uhci0@pci0:0:26:0: class=3D0x0c0300 card=3D0x20f017aa =
chip=3D0x29378086 rev=3D0x03 hdr=3D0x00
> bar [20] =3D type I/O Port, range 32, base 0x1840, size 32, =
enabled
xhci3@pci0:97:0:0: class=3D0x0c0330 card=3D0xffffffff =
chip=3D0x00151912 rev=3D0x02 hdr=3D0x00
bar [10] =3D type Memory, range 64, base 0xe1d00000, size 8192, =
enabled
cap 01[50] =3D powerspec 3 supports D0 D3 current D0
cap 05[70] =3D MSI supports 8 messages, 64 bit enabled with 1 =
message
cap 11[90] =3D MSI-X supports 8 messages
Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
cap 10[a0] =3D PCI-Express 2 endpoint max data 128(128) NS
link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1)
ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0018[150] =3D LTR 1
Also just in case it's relevant, "acpidump" on this system hangs after =
allocating ~3GB of ram and just spins at 100% CPU. I am seeing that =
AMD-Vi is enabled in the boot dmesg though:
AMD-Vi: IVRS Info VAsize =3D 64 PAsize =3D 48 GVAsize =3D 2 flags:0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?98792A76-1723-458D-A7B4-BB3F74D9F26C>