From nobody Wed Jun 23 09:00:33 2021 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 15A1F11D33F2 for ; Wed, 23 Jun 2021 09:00:48 +0000 (UTC) (envelope-from SRS0=iE+8=LR=freebsd.org=grehan@iredmail.onthenet.com.au) Received: from azure.onthenet.com.au (azure2.OntheNet.com.au [203.13.68.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4G8y0Y0cHhz3rtL for ; Wed, 23 Jun 2021 09:00:44 +0000 (UTC) (envelope-from SRS0=iE+8=LR=freebsd.org=grehan@iredmail.onthenet.com.au) Received: from alto.onthenet.com.au (alto3.onthenet.com.au [203.13.68.39]) by azure.onthenet.com.au (Postfix) with ESMTPS id E1E5420B49AA for ; Wed, 23 Jun 2021 19:00:33 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id C86F2209C044 for ; Wed, 23 Jun 2021 19:00:33 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id C1A332104D28 for ; Wed, 23 Jun 2021 19:00:33 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id m0-Hvb1meFzo for ; Wed, 23 Jun 2021 19:00:33 +1000 (AEST) Received: from MacBook-Air-4.local (unknown [120.29.45.45]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 991C42104D27; Wed, 23 Jun 2021 19:00:33 +1000 (AEST) Subject: Re: [PATCH 0/4] Prepare bhyve's OVMF for GPU-Passthrough To: =?UTF-8?Q?Corvin_K=c3=b6hne?= Cc: =?UTF-8?Q?Patrick_Br=c3=bcnn?= , "freebsd-virtualization@freebsd.org" References: <20210615083000.1286-1-c.koehne@beckhoff.com> From: Peter Grehan Message-ID: <443c1663-979d-bd79-7bde-d308ce71d861@freebsd.org> Date: Wed, 23 Jun 2021 19:00:33 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=BZdDLol2 c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=s0GJETs4rT0hs0moWij6rw==:17 a=IkcTkHD0fZMA:10 a=r6YtysWOX24A:10 a=6I5d2MoRAAAA:8 a=xdQocayyOkz9LgrFYKAA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=HqHt53bS c=1 sm=1 tr=0 a=/VfLw706i2ychnmP9KhCsg==:117 a=s0GJETs4rT0hs0moWij6rw==:17 a=IkcTkHD0fZMA:10 a=r6YtysWOX24A:10 a=6I5d2MoRAAAA:8 a=xdQocayyOkz9LgrFYKAA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-Rspamd-Queue-Id: 4G8y0Y0cHhz3rtL X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=iE@iredmail.onthenet.com.au has no SPF policy when checking 203.13.68.15) smtp.mailfrom=SRS0=iE@iredmail.onthenet.com.au X-Spamd-Result: default: False [-1.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[grehan@freebsd.org,SRS0=iE@iredmail.onthenet.com.au]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[203.13.68.15:from]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/23, country:AU]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[8=LR=freebsd.org=grehan]; FROM_NEQ_ENVFROM(0.00)[grehan@freebsd.org,SRS0=iE@iredmail.onthenet.com.au]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; AUTH_NA(1.00)[]; DMARC_NA(0.00)[freebsd.org]; SPAMHAUS_ZRD(0.00)[203.13.68.15:from:127.0.2.255]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-ThisMailContainsUnwantedMimeParts: N Hi Corvin, > As far as I know, QEMU uses OVMF with bus enumeration enabled. How > does QEMU solve such issues? Full-time corp Linux staff working on Qemu, KVM and EFI. >> - there is no need for EFI to perform a slow scan via PCI bus >> operations, resulting in VM-exits, where bhyve can perform all this >> in memory, which can result in faster boot. > > I didn't measured boot time yet. But I didn't noticed any difference > in boot time. Ok, thanks. >> Is it possible to fix this in bhyve ? Can pass-thru ROMs be mapped >> just like mmio BARs are ? > > I'm mapping passthru ROMs like MMIO BARs (see my patch for AMD GPUs: > https://reviews.freebsd.org/D27456). There are two issues for GPU > Passthrough to work properly: > > 1. Linux amdgpu driver needs a ROM for AMD GPUs. Linux assumes that > the ROM is shadowed because it's the primary video card. Shadowing is > normally done by EFI. I don't know if it's possible that bhyve > shadows the ROM. I don't think so. EFI thinks it has ownership of all RAM except for the < 1MB reserved area, so bhyve-relocated ROMs could be overwritten by EFI. While it's possible for PCI ROM regions to be actual executable RAM in a hypervisor, I suspect that o/s's wouldn't allow this. The only other possibility would be for EFI to relocate ROMs even if BAR allocation was done externally. I'm not sure how complex that would be. 2. It's necessary that the ROM is executed by EFI. > Otherwise, it's impossible to get a display output when no OS driver > is loaded. This means, that you don't have a display output while > inside EFI, a bootloader menu (like grub menu) or while installing an > OS. Understood. later, Peter.