From owner-freebsd-virtualization@freebsd.org Thu Jan 12 02:48:12 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49134CABDDD for ; Thu, 12 Jan 2017 02:48:12 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 36DED107B; Thu, 12 Jan 2017 02:48:11 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 951ACF805; Wed, 11 Jan 2017 18:48:11 -0800 (PST) Date: Wed, 11 Jan 2017 18:48:10 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111184810.5e9fb07f@mscad14> In-Reply-To: <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 02:48:12 -0000 > Hi, > > >> The problem appears to be in the area of assigning memory-mapped > >> I/O ranges by bhyve for the VGA card to a region outside of the > >> CPU's addressable space; i.e., bhyve does not check CPUID's > >> 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while > >> bhyve assigns 0xd000000000 & above for the large Prefetch Memory > >> chunks, which requires 40 address bits). > > That's correct - it's a bug in bhyve. Baking a proper fix will be more complicated by the fact that PCIe cards themselves may have limitations. For example, most nVidia GPUs have 40 bits DMA addressing capability, some 39, an a few (still modern) ones -- just 37 [ref. nVidia "README" in the driver package]. > PCI passthru doesn't allow the BAR values to be modified (this could > be changed, but it's a lot of work for little gain). Removing another signature of detecting virtualization and increasing compatibility would be negligible gain? Just asking... > > But: > > # ./nvidia-smi > > No devices were found > > dmesg: > > [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) > > [ 173.499115] NVRM: rm_init_adapter failed for device bearing > > minor number 0 > > Looks like you're getting close :) Hmm, I'm not seeing myself getting much closer here. Do you know something I don't? ;) I really hope bhyve developers can spare a bit of time on getting GPU passthrough to work... I know nothing about these things, and where I waste half a day messing around, the problem could be fixed in half an hour by someone who knows. > later, > Peter. -- [SorAlx] ridin' VN2000 Classic LT