From owner-freebsd-virtualization@freebsd.org Sat Jan 14 06:54:10 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26682CAFFC2 for ; Sat, 14 Jan 2017 06:54:10 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id AF8DB1EF7 for ; Sat, 14 Jan 2017 06:54:09 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id C44A520B4B42 for ; Sat, 14 Jan 2017 16:53:47 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id BCD492809CC for ; Sat, 14 Jan 2017 16:53:47 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id nsesBwNRLpKR for ; Sat, 14 Jan 2017 16:53:47 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 98E4C280996; Sat, 14 Jan 2017 16:53:45 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> <7cfaedbd-df48-53a8-2510-5f180ce1f2f6@freebsd.org> <20170113001737.5fe3001b@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: Date: Fri, 13 Jan 2017 22:54:01 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170113001737.5fe3001b@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=okpfQa3PAAAA:8 a=T6pNkRvUAAAA:8 a=HsyY3lUf_bTxMnHSjM4A:9 a=TgtqQ1FgE2jwKowV:21 a=SnqjUP613dYyXqCm:21 a=pILNOxqGKmIA:10 a=kPJpQtqUOtsA:10 a=gA6IeH5FQcgA:10 a=NWVoK91CQyQA:10 a=ajisY8KShb3P6wK432_8:22 a=hgMNUkviuzi0Hbumswsj:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jan 2017 06:54:10 -0000 Hi, >> That is extremely likely. bhyve itself doesn't have a BIOS, though >> bhyve/UEFI could be modified to handle options ROMs (see >> http://awilliam.github.io/presentations/KVM-Forum-2014/#/) > > Hm, interesting. I wonder if a card that's not designed for use > with UEFI is destined not to work well/at_all with bhyve... > I'll read the presentation later. I think in general almost all cards have UEFI ROM support these days since it has been mandated by Microsoft. However, as Rod mentioned, the bhyve UEFI implementation does not run PCI device option ROMs. (see http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html) >>> - GPU UUID : >>> GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 >>> + GPU UUID : Unknown Error >> >> That implies some type of h/w access isn't working, either MMIO >> registers or response from a DMA command. > > I have a feeling it's something to do with DMA that's > not getting configured correctly for data transfers, > and returns wrong data (or good data to wrong location). Yes. >> A general issue with PCI passthrough is that often MMIO from the >> guest works, since that is just VT-x remapping, but DMA doesn't work >> due to issues with IOMMU programming (or incorrect mappings being >> used). This gives a device that partially works in that registers can >> be read, but data transfer doesn't work. > > Didn't we verify that the BARs are programmed correctly? The BAR values you see are fictional and are created by bhyve. The actual physical BAR values are those set up by the host BIOS. bhyve uses EPT mappings to translate between the 'fake' value and the real value. > So you're saying that bhyve has a bug in that it doesn't > program the IOMMU right to match guest's memory-mapped > address regions to host's addresses? There isn't a known bug, but the 64-bit BAR region hasn't been tested for a long time so it's possible there is an issue with it. >>> BTW, is it [generally] safe to decrease the BAR base address further? >>> My workstation has a CPU with just 36 address bits... >> Yes. The only potential conflict is with the top of guest RAM, and 36 >> bits is a lot of RAM :) > > 64G of RAM isn't that much these days, how incredible is that :) > But you're saying there's nothing else inbetween the top of > guest's RAM and the BAR base? In that case it's nothing to > worry about at all, as a guest will always have less RAM that > the host's CPU can address. Right - the 64-bit PCI decode region would be set dynamically based on the phys address bits, rather than being a hard-coded value. later, Peter.