Date: Sun, 5 Aug 2018 13:35:00 +0300 From: Konstantin Belousov <kib@freebsd.org> To: Johannes Lundberg <johalun0@gmail.com> Cc: freebsd-current <freebsd-current@freebsd.org> Subject: Re: IOMMU for GPUs Message-ID: <20180805103500.GW6049@kib.kiev.ua> In-Reply-To: <CAECmPwsdeG74_o25FXiKek=6yKK-tkRxeQ0rXOt3dN=oGAOh1w@mail.gmail.com> References: <CAECmPwsdeG74_o25FXiKek=6yKK-tkRxeQ0rXOt3dN=oGAOh1w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Aug 05, 2018 at 10:34:56AM +0100, Johannes Lundberg wrote: > Hi > > First I have to say I don't know much when it comes to virtual GPUs and > IOMMU. I'm trying to figure out what we have and what is missing in regards > to sharing the GPU to virtual guests on Intel and AMD and things like the > amdkfd driver (for Radeon open compute). > > Looking at the state of IOMMU in FreeBSD there seem to be (what I guess is) > a general driver for Intel at /usr/src/sys/x86/iommu/. This is indeed a generic driver, which also provides busdma implementation using DMAR. Its main use right now is to debug other drivers. It is also helpful if some device has very restrictive DMA alignment, contiguous or location requirements which cannot be satisfied by VM subsystem easily, esp. under the load. In this case its behaviour might be better than the usual bounce busdma. Also the Intel driver knows about significant set of the DMAR erratas in the existing chipsets, although I did not updated the list recently. There is no similar driver for AMD IOMMU. > > Then there's the support for AMD's IOMMU in bhyve at > /usr/src/sys/amd64/vmm/io/iommu.c. Bhyve has both Intel DMAR and AMD IOMMU simple drivers, only providing for the PCI pass-through. Intention is to switch to x86/iommu DMAR driver eventually, at least on Intel. I did not found time to work on this still. > > Without looking too much into the details my guess is we have an iommu > driver for Intel that Intel's i915/gvt can use but for AMD there's only the > specific implementation for bhyve and no general driver that other clients > can use... Or? In-tree code in dev/drm2 does not work with GPU IOMMU turned on. GTT code for Intel/GEM assumes that physical == GTT address. I strongly suspect that dev/drm2/ati does the same for TTM buffers. > > If anyone wants to work on this, it's up for grabs :) > We'll probably have to add some glue in linuxkpi as well. As far as I know (would be glad to appear wrong) linuxkpi does not wrap Linux DMA KPI into our busdma, if such wrapping is ever possible. More probably, DMA interfaces should be implemented from scratch and perhaps use existing Intel DMAR driver as one of the substrates.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180805103500.GW6049>