From owner-freebsd-virtualization@freebsd.org Wed Sep 7 07:02:39 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 193AFAC4E95 for ; Wed, 7 Sep 2016 07:02:39 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id D0533A70 for ; Wed, 7 Sep 2016 07:02:38 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 7C5A620A40CD for ; Wed, 7 Sep 2016 17:02:25 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 75FFF280A36 for ; Wed, 7 Sep 2016 17:02:25 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id PO1pO-GM6yFn for ; Wed, 7 Sep 2016 17:02:25 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 12058280991; Wed, 7 Sep 2016 17:02:22 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: Dom References: <727839e2-4be1-5661-8c19-6d13f73ad36f@talk2dom.com> From: Peter Grehan Cc: freebsd-virtualization@freebsd.org Message-ID: <62b2a72a-3662-5b3e-5029-f284a625d05b@freebsd.org> Date: Wed, 7 Sep 2016 00:02:31 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <727839e2-4be1-5661-8c19-6d13f73ad36f@talk2dom.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YvxNfMQX c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=GW1xBdLrtEIA:10 a=9zVSu4PRvGFPjgcPEUoA:9 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2016 07:02:39 -0000 Hi Dom, > Bhyve's ACPI table produces this error linux-side regardless of > "pci=" setting: > > [ 0.163085] acpi PNP0A03:00: host bridge window > [0xd000000000-0xd0100fffff] (ignored, not CPU addressable) That one is most likely a bug in bhyve, where the space used for 64-bit BAR placement isn't tested against the max physaddr width of the host CPU. To confirm, would you be able to report on this value on your system ? # sudo pkg install cpuid # cpuid | grep ^80000008 80000008 00003027 00000000 00000000 00000000 ^^ This is the output from a Xeon E3-1220 v3: 0x27 == 39 bits of phys address (0x8000000000 max) 0xd000000000 requires >= 40 bits. > 2. "can't derive routing for PCI INT" > > Linux-side dmesg related output: > > [ 1.677168] nvidia 0000:00:06.0: can't derive routing for PCI INT > A [ 1.677600] nvidia 0000:00:06.0: PCI INT A: no GSI ... > Host-side info (when GTX960 is NOT configured as a pass-thru dev): ... > cap 05[68] = MSI supports 1 message, 64 bit enabled with 1 message The device has an MSI capability, but the nvidia driver may not use it. bhyve PCI passthrough requires the use of MSI/MSI-x interrupts, and doesn't support using legacy interrupts. This could be confirmed from the output of /proc/interrupts when booting Linux on the system. later, Peter.