From owner-freebsd-virtualization@freebsd.org Thu Jan 12 03:09:27 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0E85CA9734 for ; Thu, 12 Jan 2017 03:09:27 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 82F1F1D97 for ; Thu, 12 Jan 2017 03:09:27 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 6A44C20195A1 for ; Thu, 12 Jan 2017 13:09:06 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 6463A2809D3 for ; Thu, 12 Jan 2017 13:09:06 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id levYwLj62SOw for ; Thu, 12 Jan 2017 13:09:06 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 485DB280992; Thu, 12 Jan 2017 13:09:04 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111184810.5e9fb07f@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <839b4e54-fd7a-20fd-630f-f8aaee0c79ee@freebsd.org> Date: Wed, 11 Jan 2017 19:09:21 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170111184810.5e9fb07f@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=8SLKNYVD9Kc_1ussAjsA:9 a=iZbocsCRyau62-kp:21 a=WgxIE6v9D2JIhToZ:21 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 03:09:28 -0000 Hi, >> That's correct - it's a bug in bhyve. > > Baking a proper fix will be more complicated by the fact that PCIe > cards themselves may have limitations. For example, most nVidia GPUs > have 40 bits DMA addressing capability, some 39, an a few (still > modern) ones -- just 37 [ref. nVidia "README" in the driver package]. That's a different issue - it's unlikely, if not impossible, to configure bhyve with enough RAM to hit 37 bits worth where that would become a problem. No need to worry about that. >> PCI passthru doesn't allow the BAR values to be modified (this could >> be changed, but it's a lot of work for little gain). > > Removing another signature of detecting virtualization and increasing > compatibility would be negligible gain? Just asking... There are lots of BIOS/UEFI implementations out there that have the same restriction. In general, there should be no need for a guest to reprogram device BARs. After changing the 64-bit BAR base address, did you still need the pci=nocrs option for Linux ? I'd hope this would be no longer necessary. >>> But: >>> # ./nvidia-smi >>> No devices were found >>> dmesg: >>> [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) >>> [ 173.499115] NVRM: rm_init_adapter failed for device bearing >>> minor number 0 >> >> Looks like you're getting close :) > > Hmm, I'm not seeing myself getting much closer here. Do you know > something I don't? ;) I really hope bhyve developers can spare a > bit of time on getting GPU passthrough to work... I know nothing > about these things, and where I waste half a day messing around, > the problem could be fixed in half an hour by someone who knows. The problem is the knowledge set of graphics/GPU knowhow and equipment access, and bhyve/PCI programming, are disjoint. The time I've spent on it has been the inverse, where I feel that I've spent a half-day doing things that anyone who knew about graphics could get done in a half-hour :) For these type of issues, joint work is best to leverage the knowledge of both sides. From my point-of-view, the work you've done has been very helpful. later, Peter.