From owner-freebsd-virtualization@freebsd.org Thu Mar 12 14:25:17 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 05A9D26224F for ; Thu, 12 Mar 2020 14:25:17 +0000 (UTC) (envelope-from erleya@gmail.com) Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48dWM05c2Dz4gJ8; Thu, 12 Mar 2020 14:25:16 +0000 (UTC) (envelope-from erleya@gmail.com) Received: by mail-wr1-x436.google.com with SMTP id m9so7699302wro.12; Thu, 12 Mar 2020 07:25:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=VBcLRiPH2QBZYvCoGmfvom4ON2Uv+P3pLfdBwBTlp8A=; b=A4Jqv5f4GTiGK+IgXCJhQNq6BMq0JPYBSckGF5u3SROvE3J/Tb3PfyRPOvjJQwyrtP L8VMSiPXEp46qoaPValHVdt10Mo9cf6x9zalmoMp8guJu/u1wUlAAuzVtITyAfsD5wwM xLK/pLlGxJ/PpbKreussK10S/63hQjV9Jf1WEA50cA7SO3ZvWmQ09lXFA/9qxI1ktCHB 68qCHhcYGiicJ1e19wDZ/BMV1mVa5bk3fwOtob0ww02JXoeG/MHEar/McohT4oENNLYc 0a69naZt7MZeeizQMM/4RlRfFQQ/Ou/CcqCBSvPK+Kh7ck05EWia7Tk7tFFj3mqmgFQb DZug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=VBcLRiPH2QBZYvCoGmfvom4ON2Uv+P3pLfdBwBTlp8A=; b=CJZHKQOEOL9+kVHbdhNFzejYGdFIAqtaT/TY2fJxopmKB+tOOjP/pQODgfbWHGzoK4 CZVVSmLGCCgC0bJDI4T+3tK0eiY47w++I/4jUNyw7B7CRx5rk8FGYR43Nr1xj/OXPcFQ H7c0OwTPj1vsV1YX2TwTTqCSiA8CFty3GFEcPvXenlp8RvrP3qXN11ptXnQ2cDgB6EIP lg06ZkTVDdWU24miz3becRATy5FOhxS2CI1NCxADv6pnyoByIQaZmJXPUDnlhRfEjNk9 NbybEQf9ST9moUalRmGiM3zdx2W/OB6AMO206WDEE4rf1CRHfBN5t1j/2oFKG86HY/Cr PwBg== X-Gm-Message-State: ANhLgQ13Ae2PXoYMW2YOlWvYBuNBloBQ54cX+VQUq2HgwomYa2fkMyc0 bdCoHOmVIstxPWFHDMdOGUSruNt8sz4= X-Google-Smtp-Source: ADFU+vvOZeJDk+h1o0qA9qs93L5Q7kH8f4gsz9DWA8CDVZ7I71XoRO550Ap2h/xi5bG+zew5GOSEyg== X-Received: by 2002:a5d:4711:: with SMTP id y17mr11767554wrq.358.1584023114950; Thu, 12 Mar 2020 07:25:14 -0700 (PDT) Received: from erley.ru (erley.ru. [83.153.157.67]) by smtp.gmail.com with ESMTPSA id i1sm57975796wrs.18.2020.03.12.07.25.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 12 Mar 2020 07:25:14 -0700 (PDT) Received: by erley.ru (OpenSMTPD) with ESMTPSA id 2df0d96a (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Thu, 12 Mar 2020 15:25:13 +0100 (CET) Subject: Re: [GPU pass-through] no compatible bridge window for claimed BAR To: Peter Grehan Cc: freebsd-virtualization@freebsd.org References: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> From: Alex Erley Message-ID: <9b26dee7-8656-b5ba-5d72-9a01638ee438@gmail.com> Date: Thu, 12 Mar 2020 15:25:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48dWM05c2Dz4gJ8 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.74 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; REPLY(-4.00)[]; NEURAL_SPAM_MEDIUM(0.26)[0.257,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 14:25:17 -0000 Hello Peter, Many thanks for your insight! Changing PCI_EMUL_MEMBASE64 from 0xD000000000 to 0x0440000000 (16Gb guest memory limit) and PCI_EMUL_MEMLIMIT64 from 0xFD00000000 to 0x07ffffffff (32Gb host memory limit) still fails. From guest dmesg: ... [mem 0xc0000000-0xffffffff] available for PCI devices ... pci 0000:00:01.0: [10de:1c03] type 00 class 0x030000 pci 0000:00:01.0: reg 0x10: [mem 0xc0000000-0xc0ffffff] pci 0000:00:01.0: reg 0x14: [mem 0x440000000-0x44fffffff 64bit pref] pci 0000:00:01.0: reg 0x1c: [mem 0xc2000000-0xc3ffffff 64bit pref] pci 0000:00:01.0: reg 0x24: [io 0x2000-0x207f] pci 0000:00:01.0: reg 0x30: [mem 0xd3000000-0xd307ffff pref] pci 0000:00:01.1: [10de:10f1] type 00 class 0x040300 pci 0000:00:01.1: reg 0x10: [mem 0xc4000000-0xc4003fff] ... pci 0000:00:01.0: can't claim BAR 1 [mem 0x440000000-0x44fffffff 64bit pref]: no compatible bridge window ... pci 0000:00:01.0: BAR 1: no space for [mem size 0x10000000 64bit pref] pci 0000:00:01.0: BAR 1: trying firmware assignment [mem 0x440000000-0x44fffffff 64bit pref] pci 0000:00:01.0: BAR 1: assigned [mem 0x440000000-0x44fffffff 64bit pref] It seems guest requires BAR to be within its addressable space. Changing PCI_EMUL_MEMBASE64 to 0x00c0000000 (as guest suggests in its dmesg) didn't work either: bhyve: failed to initialize BARs for PCI 1/0/0 device emulation initialization error: Cannot allocate memory From the BHyve sources it becomes evident that when claimed BAR size is <= 32Mb, it is mandatory allocated as 32-bit BAR. The only case BAR goes for 64-bit allocation is when its size > 32Mb. In my case it is 256 Mb and it fails only for this BAR. I think the 64-bit BAR allocation still has to be fixed in BHyve code. Anyway, we are getting closer :) Any ideas are really appreciated. Have a nice day, Alex On 3/12/20 1:31 AM, Peter Grehan wrote: > Hi Alex, > >>> dmesg | grep "no compatible bridge window" >> pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff >> 64bit pref]: no compatible bridge window > ...> From what I can read from all the info above, >> somehow the requested memory space for BAR1 for that device is >> 0xd000000000-0xd00fffffff which is out of addressable space on the >> system. > > Yep, that's the issue, and it's a bhyve bug - there is no check to > see if the 64-bit window is within the addressable range of the > processor. > > A quick fix is to change the constant for that range in pci_emul.c > > #define PCI_EMUL_MEMBASE64 0xD000000000UL #define > PCI_EMUL_MEMLIMIT64 0xFD00000000UL > > .. to a value that is within the address bits of the CPU, but also > above guest DRAM. > > later, > > Peter.