From owner-freebsd-current@freebsd.org Sun Aug 5 10:35:12 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 551FB10720F4 for ; Sun, 5 Aug 2018 10:35:12 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A92BD8ACF0 for ; Sun, 5 Aug 2018 10:35:11 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w75AZ0uY024176 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 5 Aug 2018 13:35:03 +0300 (EEST) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w75AZ0uY024176 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w75AZ0pU024175; Sun, 5 Aug 2018 13:35:00 +0300 (EEST) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Sun, 5 Aug 2018 13:35:00 +0300 From: Konstantin Belousov To: Johannes Lundberg Cc: freebsd-current Subject: Re: IOMMU for GPUs Message-ID: <20180805103500.GW6049@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Aug 2018 10:35:12 -0000 On Sun, Aug 05, 2018 at 10:34:56AM +0100, Johannes Lundberg wrote: > Hi > > First I have to say I don't know much when it comes to virtual GPUs and > IOMMU. I'm trying to figure out what we have and what is missing in regards > to sharing the GPU to virtual guests on Intel and AMD and things like the > amdkfd driver (for Radeon open compute). > > Looking at the state of IOMMU in FreeBSD there seem to be (what I guess is) > a general driver for Intel at /usr/src/sys/x86/iommu/. This is indeed a generic driver, which also provides busdma implementation using DMAR. Its main use right now is to debug other drivers. It is also helpful if some device has very restrictive DMA alignment, contiguous or location requirements which cannot be satisfied by VM subsystem easily, esp. under the load. In this case its behaviour might be better than the usual bounce busdma. Also the Intel driver knows about significant set of the DMAR erratas in the existing chipsets, although I did not updated the list recently. There is no similar driver for AMD IOMMU. > > Then there's the support for AMD's IOMMU in bhyve at > /usr/src/sys/amd64/vmm/io/iommu.c. Bhyve has both Intel DMAR and AMD IOMMU simple drivers, only providing for the PCI pass-through. Intention is to switch to x86/iommu DMAR driver eventually, at least on Intel. I did not found time to work on this still. > > Without looking too much into the details my guess is we have an iommu > driver for Intel that Intel's i915/gvt can use but for AMD there's only the > specific implementation for bhyve and no general driver that other clients > can use... Or? In-tree code in dev/drm2 does not work with GPU IOMMU turned on. GTT code for Intel/GEM assumes that physical == GTT address. I strongly suspect that dev/drm2/ati does the same for TTM buffers. > > If anyone wants to work on this, it's up for grabs :) > We'll probably have to add some glue in linuxkpi as well. As far as I know (would be glad to appear wrong) linuxkpi does not wrap Linux DMA KPI into our busdma, if such wrapping is ever possible. More probably, DMA interfaces should be implemented from scratch and perhaps use existing Intel DMAR driver as one of the substrates.