From owner-freebsd-virtualization@freebsd.org Fri Jan 13 02:42:51 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1B59CACCE6 for ; Fri, 13 Jan 2017 02:42:51 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 642CE1EFA for ; Fri, 13 Jan 2017 02:42:50 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 1081A2052DED for ; Fri, 13 Jan 2017 12:42:29 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 096822809D3 for ; Fri, 13 Jan 2017 12:42:29 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id yH_-fsWcsHvf for ; Fri, 13 Jan 2017 12:42:28 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id E03CA280992; Fri, 13 Jan 2017 12:42:26 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> From: Peter Grehan Cc: freebsd-virtualization@freebsd.org Message-ID: <7cfaedbd-df48-53a8-2510-5f180ce1f2f6@freebsd.org> Date: Thu, 12 Jan 2017 18:42:44 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170111195402.785f27c6@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=okpfQa3PAAAA:8 a=aU_aj2REWSNhthCTgJkA:9 a=pILNOxqGKmIA:10 a=ajisY8KShb3P6wK432_8:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 02:42:51 -0000 Hi, > First, `nvidia-smi -q` output diff [0] is interesting. It suggests that > the card may be in some incompletely initialized state: notice the > "Unknown Error" instead of real UUID, and the P8 power state. Could it > be that the driver doesn't put the card's BIOS in the right state? That is extremely likely. bhyve itself doesn't have a BIOS, though bhyve/UEFI could be modified to handle options ROMs (see http://awilliam.github.io/presentations/KVM-Forum-2014/#/) > The command was run in both host and guest without Xorg loaded. Thanks for the diff; this is very useful. > - GPU UUID : GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 > + GPU UUID : Unknown Error That implies some type of h/w access isn't working, either MMIO registers or response from a DMA command. > - Board ID : 0x100 > + Board ID : 0x4 The same ? > PCIe Generation > Max : 2 > - Current : 2 > + Current : 1 bhyve's emulated PCI hostbridge only advertises gen-1 - that could be easily changed to gen2. That could make a difference for some of the clock issues below (source is pci_emul.c:pci_emul_add_pciecap()) > Link Width > Max : 16x > Current : 16x That's a bit unexpected since the hostbridge only advertises 1x, but the driver is probably exporting the host value here. > - Performance State : P0 > + Performance State : P8 Note sure what's happening here. > Clocks > - Graphics : 625 MHz > - SM : 1251 MHz > - Memory : 1304 MHz > - Video : 540 MHz > + Graphics : 405 MHz > + SM : 810 MHz > + Memory : 324 MHz > + Video : 405 MHz This may be related to the gen1 vs gen2 issue above. > When rebooting, I get this: > nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000857d:0:0:0x00000040 This may be DMA not working. A general issue with PCI passthrough is that often MMIO from the guest works, since that is just VT-x remapping, but DMA doesn't work due to issues with IOMMU programming (or incorrect mappings being used). This gives a device that partially works in that registers can be read, but data transfer doesn't work. > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine core DMA push buffer Not sure what's happening with those. Would it be possible to try the nouveau driver ? At least the source is available, so it may be easier to determine what is broken. later, Peter.