Date: Thu, 28 Feb 2019 10:28:30 -0800 From: Anish <akgupt3@gmail.com> To: Kevin Day <toasty@dragondata.com> Cc: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.cn85.dnsmgr.net>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: bhyve PCIe passthrough on AMD Epyc Message-ID: <CALnRwMT=VNYL8A8=rs3Spbiw7jvKyKxwZAv%2BrwTPmmO8gXaSFw@mail.gmail.com> In-Reply-To: <98792A76-1723-458D-A7B4-BB3F74D9F26C@dragondata.com> References: <201902280534.x1S5YMQS054621@pdx.rh.CN85.dnsmgr.net> <98792A76-1723-458D-A7B4-BB3F74D9F26C@dragondata.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Kevin, >I'm just pointing out that it's an AMD Epyc because I know the AMD IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is maybe less tested. I tested on Ryzen-I, not sure if anyone tried it on newer Ryzen/Epyc systems. >CPU0: local APIC error 0x40 This is concerning, as per AMD docs, bit6 is for illegal vector which indicate CPU is getting invalid interrupts. Can you share hw.vmm sysctl output? -Anish On Thu, Feb 28, 2019 at 9:59 AM Kevin Day <toasty@dragondata.com> wrote: > > > > On Feb 27, 2019, at 11:34 PM, Rodney W. Grimes < > freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote: > > > >>> On Feb 27, 2019, at 12:32 AM, Rodney W. Grimes < > freebsd-rwg@pdx.rh.CN85.dnsmgr.net> wrote: > >>> > >>>> I'm trying to get PCIe device (video card) passthrough working to a > Windows bhyve VM, should this work now? > >>> > >>> No, this should not work now. > >> > >> Ok, easy enough. Should *any* PCIe passthrough work on Epyc right now? > > I do not know what would be special about Epyc so can not say > > for certain. > > I'm just pointing out that it's an AMD Epyc because I know the AMD > IOMMU/AMD-Vi code was added somewhat more recently than Intel's and is > maybe less tested. > > >> Just trying the same thing with a simple PCIe USB card gives me: > >> > >> CPU0: local APIC error 0x40 > >> CPU0: local APIC error 0x40 > >> ivhd0: Error: completion failed tail:0xda0, head:0x0. > >> ivhd0: Dump all the commands: > >> [CMD0, off:0xff0] opcode= 0x0 0x0 0x0 0x0 > >> [CMD1, off:0x0] opcode= 0x3 0x0 0x0 0x7ffffffffffff003 > >> [CMD2, off:0x10] opcode= 0x0 0x0 0x0 0x0 > >> [CMD3, off:0x20] opcode= 0x2 0x0 0x0 0x0 > >> [CMD4, off:0x30] opcode= 0x2 0x2 0x0 0x0 > >> [...snip... lots of similar stuff] > >> > >> Then I get a bunch more APIC errors, and the system freezes for about > 60 seconds before recovering. > > > > You do not get any host dmesg errors about unaligned BAR > > ranges or sizes? > > > > Correct. I did with the video card, but not with the xhci USB card. I get > a ton of dmesg errors, and the host system repeatedly freezes until I stop > the VM. (Console is non-responsive, no ICMP replies, etc) > > I didn't actually look at this until now, but Windows actually does see > the USB controller, but it complains that "port reset failed" on any device > I attach to it. > > > Can you get the pciconf -l -b info for the device you passed, something > like: > > # pciconf -l -b uhci0 > > uhci0@pci0:0:26:0: class=0x0c0300 card=0x20f017aa chip=0x29378086 > rev=0x03 hdr=0x00 > > bar [20] = type I/O Port, range 32, base 0x1840, size 32, enabled > > xhci3@pci0:97:0:0: class=0x0c0330 card=0xffffffff chip=0x00151912 > rev=0x02 hdr=0x00 > bar [10] = type Memory, range 64, base 0xe1d00000, size 8192, enabled > cap 01[50] = powerspec 3 supports D0 D3 current D0 > cap 05[70] = MSI supports 8 messages, 64 bit enabled with 1 message > cap 11[90] = MSI-X supports 8 messages > Table in map 0x10[0x1000], PBA in map 0x10[0x1080] > cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS > link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1) > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0018[150] = LTR 1 > > Also just in case it's relevant, "acpidump" on this system hangs after > allocating ~3GB of ram and just spins at 100% CPU. I am seeing that AMD-Vi > is enabled in the boot dmesg though: > > AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0 > > > > > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALnRwMT=VNYL8A8=rs3Spbiw7jvKyKxwZAv%2BrwTPmmO8gXaSFw>