From owner-freebsd-virtualization@freebsd.org Wed Jan 11 10:08:27 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B7F3CAA808 for ; Wed, 11 Jan 2017 10:08:27 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 2CFC91388 for ; Wed, 11 Jan 2017 10:08:26 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id D7DEC2041641 for ; Wed, 11 Jan 2017 20:07:59 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id D2E69280991 for ; Wed, 11 Jan 2017 20:07:59 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ncQi3bRcgIgB for ; Wed, 11 Jan 2017 20:07:59 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id B8513280500; Wed, 11 Jan 2017 20:07:57 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> Date: Wed, 11 Jan 2017 02:08:12 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170111014544.70670784@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=FO24ivJdlaGtZuhHtzoA:9 a=_0mYaXJXlwf1-dBP:21 a=SLvJyNt8ObrNJ6nn:21 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 10:08:27 -0000 Hi, >> The problem appears to be in the area of assigning memory-mapped >> I/O ranges by bhyve for the VGA card to a region outside of the >> CPU's addressable space; i.e., bhyve does not check CPUID's >> 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while >> bhyve assigns 0xd000000000 & above for the large Prefetch Memory >> chunks, which requires 40 address bits). That's correct - it's a bug in bhyve. > To test this, I tried writing to PCI BARs in FreeBSD guest using > `pciconf -w`. Not much use that was: I could read back the values > written to the registers (e.g., `pciconf -r pci0:0:4:0 0x14:48`), > but `pciconf -lvb` still showed the same huge base addresses -- > they did not want to change. PCI passthru doesn't allow the BAR values to be modified (this could be changed, but it's a lot of work for little gain). > OK, I had enough of that. So I went to dig in the source, and > changed the "#define PCI_EMUL_MEMBASE64" from '0xD000000000UL' > to '0x3400000000UL' in src/usr.sbin/bhyve/pci_emul.c. Yep, that's a good way to test. > But: > # ./nvidia-smi > No devices were found > dmesg: > [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) > [ 173.499115] NVRM: rm_init_adapter failed for device bearing minor number 0 Looks like you're getting close :) later, Peter.