From owner-freebsd-virtualization@freebsd.org Sun Mar 15 14:20:59 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 374C8266D61 for ; Sun, 15 Mar 2020 14:20:59 +0000 (UTC) (envelope-from crowston@protonmail.com) Received: from mail-40131.protonmail.ch (mail-40131.protonmail.ch [185.70.40.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "protonmail.com", Issuer "SwissSign Server Gold CA 2014 - G22" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48gM6d0F8Lz41pv; Sun, 15 Mar 2020 14:20:56 +0000 (UTC) (envelope-from crowston@protonmail.com) Date: Sun, 15 Mar 2020 14:20:51 +0000 To: Alex Erley From: Robert Crowston Cc: Peter Grehan , "freebsd-virtualization@freebsd.org" Reply-To: Robert Crowston Subject: Re: [GPU pass-through] no compatible bridge window for claimed BAR Message-ID: In-Reply-To: References: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> Feedback-ID: 2OVbcR1yHYpdkD8cgQllkFwcuMVZg_LiVMMPvptooFDfHD_03MuQO4ZaF626jWHZYFEhNR2cmIbZ53j4QGWMBQ==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM shortcircuit=no autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mail.protonmail.ch X-Rspamd-Queue-Id: 48gM6d0F8Lz41pv X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.09 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_REPLYTO(0.00)[crowston@protonmail.com]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[protonmail.com]; R_SPF_ALLOW(-0.20)[+ip4:185.70.40.0/24:c]; DKIM_TRACE(0.00)[protonmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[protonmail.com,quarantine]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.80), ipnet: 185.70.40.0/24(-4.89), asn: 62371(-3.91), country: CH(0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[protonmail.com]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[131.40.70.185.list.dnswl.org : 127.0.5.1]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.988,0]; R_DKIM_ALLOW(-0.20)[protonmail.com:s=default]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[protonmail.com]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Mar 2020 14:20:59 -0000 I suggest you map the BAR into the 32 bit address space, unless you have so= many PCI devices that this is not feasible. Just raise the limit of the sp= ecial 64 bit handling to 1 GB or something big. - Many/most(?) consumer BIOS/UEFIs map 64 bit bars into the 32 bit address = space by default, so this configuration is much more tested for device driv= ers and guest operating systems. - Passthrough doesn't work for me at all above the 4GB memory window on my = recent AMD system. It exits back to the hypervisor, which then crashes with= a failed assert because it doesn't expect to handle this. With this tweak it is possible to use the open source nVidia driver on Linu= x. However, your next problem---if you want to use proprietary drivers---is th= at access to the ROM BAR is not supported or emulated. If that could be fix= ed, it would be a big step forwards. =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Saturday, 14 March 2020 12:50, Alex Erley wrote: > Hello, > > Some new findings to share. > > 1. Changing PCI_EMUL_MEMBASE64 from 0xD000000000 to any value > below 0x0440000000 makes bhyve fail when starting VM with message: > bhyve: failed to initialize BARs for PCI 1/0/0 > device emulation initialization error: Cannot allocate memory > > 2. Having PCI_EMUL_MEMBASE64 set to 0x0440000000 (or above) guest VM > can not configure BARs of pass-through device properly. > > =3D=3D (a) =3D=3D > On BHyve host ppt device is: > > > > devinfo -rv > > ... > pci0 > hostb0 at slot=3D0 function=3D0 dbsf=3Dpci0:0:0:0 > pcib1 at slot=3D1 function=3D0 dbsf=3Dpci0:0:1:0 handle=3D\SB.PCI0.P0P2 > I/O ports: 0xe000-0xefff > I/O memory addresses: > 0x00c0000000-0x00d30fffff <-- covers all child mem windows > pci1 > ppt0 at slot=3D0 function=3D0 dbsf=3Dpci0:1:0:0 > pcib1 I/O port window: 0xe000-0xe07f > pcib1 memory window: > 0x00c0000000-0x00cfffffff <-- 256M > 0x00d0000000-0x00d1ffffff <-- 32M > 0x00d2000000-0x00d2ffffff <-- 16M > ppt1 at slot=3D0 function=3D1 dbsf=3Dpci0:1:0:1 > pcib1 memory window: > 0xd3080000-0xd3083fff <-- 16K > ... > and there is no other device attached to pci1. > > =3D=3D (b) =3D=3D > On guest VM dmesg shows (timestamps are removed): > ... > BIOS-provided physical RAM map: > BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable > BIOS-e820: [mem 0x0000000000100000-0x00000000bea95fff] usable > BIOS-e820: [mem 0x00000000bea96000-0x00000000bea97fff] reserved > BIOS-e820: [mem 0x00000000bea98000-0x00000000bea99fff] ACPI data > BIOS-e820: [mem 0x00000000bea9a000-0x00000000beaa8fff] reserved > BIOS-e820: [mem 0x00000000beaa9000-0x00000000bfb28fff] usable > BIOS-e820: [mem 0x00000000bfb29000-0x00000000bfb58fff] type 20 > BIOS-e820: [mem 0x00000000bfb59000-0x00000000bfb7cfff] reserved > BIOS-e820: [mem 0x00000000bfb7d000-0x00000000bfb81fff] usable > BIOS-e820: [mem 0x00000000bfb82000-0x00000000bfb88fff] ACPI data > BIOS-e820: [mem 0x00000000bfb89000-0x00000000bfb8cfff] ACPI NVS > BIOS-e820: [mem 0x00000000bfb8d000-0x00000000bffcffff] usable > BIOS-e820: [mem 0x00000000bffd0000-0x00000000bffeffff] reserved > BIOS-e820: [mem 0x00000000bfff0000-0x00000000bfffffff] usable > BIOS-e820: [mem 0x0000000100000000-0x000000043fffffff] usable > ^^^-upper limit for adressable memory > ... > PM: Registered nosave memory: [mem 0xc0000000-0xffffffff] > [mem 0xc0000000-0xffffffff] available for PCI devices > ... > pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] > pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] > pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] > ^-- 128K > pci_bus 0000:00: root bus resource [mem 0xc0000000-0xdfffffff window] > ^-- 512M > pci_bus 0000:00: root bus resource [mem 0xf0000000-0xf07fffff window] > ^-- 8M > pci_bus 0000:00: root bus resource [bus 00-ff] > > =3D=3D (c) =3D=3D > Until now all runs OK. > > Guest Linux then allocates memory regions for devices. > Allocation is done from lower reg (0x10) to higher (0x30) > for each device (i.e. from 00.0 to 1f.0) on PCI bus. > > Here I reordered dmesg output to groups to show continuous RAM regions: > (pass-through device is marked with ) > pci 0000:00:01.0: reg 0x24: [io 0x2000-0x207f] > pci 0000:00:02.0: reg 0x10: [io 0x2080-0x209f] > pci 0000:00:03.0: reg 0x10: [io 0x20c0-0x20ff] > ... > pci 0000:00:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref] > pci 0000:00:02.0: reg 0x30: [mem 0x00000000-0x000007ff pref] > pci 0000:00:03.0: reg 0x30: [mem 0x00000000-0x000007ff pref] > pci 0000:00:1d.0: reg 0x30: [mem 0x00000000-0x000007ff pref] > pci 0000:00:1e.0: reg 0x30: [mem 0x00000000-0x000007ff pref] > pci 0000:00:1f.0: reg 0x30: [mem 0x00000000-0x000007ff pref] > ... > pci 0000:00:01.0: reg 0x10:[mem 0xc0000000-0xc0ffffff] 16M... 0xc1000000-= 0xc1ffffff 16M gap > pci 0000:00:01.0: reg 0x1c:[mem 0xc2000000-0xc3ffffff 64bit pref] 32M > pci 0000:00:01.1: reg 0x10:[mem 0xc4000000-0xc4003fff]pci 0000:00:02.0: r= eg 0x14: [mem 0xc4004000-0xc4005fff] > pci 0000:00:03.0: reg 0x14: [mem 0xc4006000-0xc4007fff] > pci 0000:00:1d.0: reg 0x10: [mem 0xc4008000-0xc400807f] > ... 0xc4008080-0xc4ffffff <16M gap > pci 0000:00:1d.0: reg 0x14: [mem 0xc5000000-0xc5ffffff] 16M > pci 0000:00:1e.0: reg 0x10: [mem 0xc6000000-0xc6000fff] > ... 0xc6001000-0xd2ffffff <208M gap > pci 0000:00:01.0: reg 0x30:[mem 0xd3000000-0xd307ffff pref] 512K > 0xd3080000-0xdfffffff <208M gap > pci 0000:00:01.0: reg0x14:[mem 0x440000000-0x44fffffff 64bit pref] 256M^^= ^- this value is outside allowed range > > =3D=3D (d) =3D=3D > So, there is no window for 256M BAR, although there are 2 big gapes > of 208M in 512M space provided for BAR allocation by PCI bus. > > So, BAR reg 0x14 of size 256M for device 01.0 must be inside provisioned > 512M region 0xc0000000-0xdfffffff. > But refering to (1) above, setting base address to any value below > 0x440000000 breaks bhyve on start. > According to (b), this value corresponds to upper addressable memory > limit in guest VM. > > So I'm blocked here at the moment: > > - Guest VM requires a value which BHyve doesn't like. > - Guest VM allocates BARs with huge gapes. > > I have little knowledge about PCI bus internals, although I already r= ead > some articles on internet. > Could it be some ACPI trick to do? > I'd be happy to hear any ideas... > > PS > I suspect that if I take other OS as a guest VM or other pass-through > GPU model, it would probably allocate BARs properly. > But this is not what I want for this config. > There should be a way to allocate 256M BAR in guest Linux. > > Have a nice day, > Alex > > > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org"