Date: Fri, 13 Jan 2017 22:54:01 -0800 From: Peter Grehan <grehan@freebsd.org> To: soralx@cydem.org Cc: freebsd-virtualization@freebsd.org Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <cf1b8234-81ae-7741-d25e-30854690f0fc@freebsd.org> In-Reply-To: <20170113001737.5fe3001b@mscad14> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> <7cfaedbd-df48-53a8-2510-5f180ce1f2f6@freebsd.org> <20170113001737.5fe3001b@mscad14>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, >> That is extremely likely. bhyve itself doesn't have a BIOS, though >> bhyve/UEFI could be modified to handle options ROMs (see >> http://awilliam.github.io/presentations/KVM-Forum-2014/#/) > > Hm, interesting. I wonder if a card that's not designed for use > with UEFI is destined not to work well/at_all with bhyve... > I'll read the presentation later. I think in general almost all cards have UEFI ROM support these days since it has been mandated by Microsoft. However, as Rod mentioned, the bhyve UEFI implementation does not run PCI device option ROMs. (see http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html) >>> - GPU UUID : >>> GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 >>> + GPU UUID : Unknown Error >> >> That implies some type of h/w access isn't working, either MMIO >> registers or response from a DMA command. > > I have a feeling it's something to do with DMA that's > not getting configured correctly for data transfers, > and returns wrong data (or good data to wrong location). Yes. >> A general issue with PCI passthrough is that often MMIO from the >> guest works, since that is just VT-x remapping, but DMA doesn't work >> due to issues with IOMMU programming (or incorrect mappings being >> used). This gives a device that partially works in that registers can >> be read, but data transfer doesn't work. > > Didn't we verify that the BARs are programmed correctly? The BAR values you see are fictional and are created by bhyve. The actual physical BAR values are those set up by the host BIOS. bhyve uses EPT mappings to translate between the 'fake' value and the real value. > So you're saying that bhyve has a bug in that it doesn't > program the IOMMU right to match guest's memory-mapped > address regions to host's addresses? There isn't a known bug, but the 64-bit BAR region hasn't been tested for a long time so it's possible there is an issue with it. >>> BTW, is it [generally] safe to decrease the BAR base address further? >>> My workstation has a CPU with just 36 address bits... >> Yes. The only potential conflict is with the top of guest RAM, and 36 >> bits is a lot of RAM :) > > 64G of RAM isn't that much these days, how incredible is that :) > But you're saying there's nothing else inbetween the top of > guest's RAM and the BAR base? In that case it's nothing to > worry about at all, as a guest will always have less RAM that > the host's CPU can address. Right - the 64-bit PCI decode region would be set dynamically based on the phys address bits, rather than being a hard-coded value. later, Peter.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?cf1b8234-81ae-7741-d25e-30854690f0fc>