From owner-freebsd-virtualization@freebsd.org Thu Mar 12 00:31:47 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A05326F469 for ; Thu, 12 Mar 2020 00:31:47 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48d8sH0CdMz4BbX for ; Thu, 12 Mar 2020 00:31:46 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito.onthenet.com.au (Postfix) with ESMTPS id AF5DF20CA9C7 for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 95DCE20B49B0 for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 8FA4920E467D for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Dp1MNvduvkgR for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from MacBook-Air-4.local (otn-120-29-62-248.broadband.onthenet.net [120.29.62.248]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id C328720E467A; Thu, 12 Mar 2020 10:31:40 +1000 (AEST) Subject: Re: [GPU pass-through] no compatible bridge window for claimed BAR To: Alex Troussov References: From: Peter Grehan Cc: freebsd-virtualization@freebsd.org Message-ID: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> Date: Thu, 12 Mar 2020 10:31:39 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=UIBp4xXy c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=c3PU01mjbo3xNLXCTk0A:9 a=D_dgQrR0NpD0Az2v:21 a=854ja8IJPeEXZL8-:21 a=QEXdDO2ut3YA:10 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=d81BNirE c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=c3PU01mjbo3xNLXCTk0A:9 a=D_dgQrR0NpD0Az2v:21 a=854ja8IJPeEXZL8-:21 a=QEXdDO2ut3YA:10 X-Rspamd-Queue-Id: 48d8sH0CdMz4BbX X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(0.99)[0.987,0]; ASN(0.00)[asn:9313, ipnet:203.22.124.0/23, country:AU]; NEURAL_HAM_LONG(-0.99)[-0.986,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 00:31:47 -0000 Hi Alex, >> dmesg | grep "no compatible bridge window" > pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff 64bit > pref]: no compatible bridge window ...> From what I can read from all the info above, > somehow the requested memory space for BAR1 for that device is > 0xd000000000-0xd00fffffff which is out of addressable space on the system. Yep, that's the issue, and it's a bhyve bug - there is no check to see if the 64-bit window is within the addressable range of the processor. A quick fix is to change the constant for that range in pci_emul.c #define PCI_EMUL_MEMBASE64 0xD000000000UL #define PCI_EMUL_MEMLIMIT64 0xFD00000000UL .. to a value that is within the address bits of the CPU, but also above guest DRAM. later, Peter.