From owner-freebsd-virtualization@freebsd.org Thu Mar 12 00:18:06 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1656226ED47 for ; Thu, 12 Mar 2020 00:18:06 +0000 (UTC) (envelope-from erleya@gmail.com) Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48d8YS4shBz3KJJ for ; Thu, 12 Mar 2020 00:18:03 +0000 (UTC) (envelope-from erleya@gmail.com) Received: by mail-wr1-x443.google.com with SMTP id v9so5112445wrf.10 for ; Wed, 11 Mar 2020 17:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=iG1i2MHmP7F+ydPtabDxjHouoMtbmtZjo5EZu0SZ+1I=; b=uq5hfvbH0HWgYrpu/4zcvo5TbFk4WVgNNAQH5m/zAA3mGPHMTkIYt8FRzgvRrgIzoB 4yX/z/NusGyODeZcRKx4pSqfAUvrBF6tfCpaYf/TQz7by44l3YAMwOHk8lx9CNTBaYKR f9Wdo4KtKG6ERznO0T8ndjPD+5pwOIWnN+wOOK51Kc16A7ZmTX51jMdmM1WsZ8ab5dHb X25ndhnT7zH3q4cKEzPm45pmniVk/jz0kaFttF2sB8womLPItej3lzyC0WM+N9OtW4Li 2jjFFABrQopC0H1EtjmCZ69V3LlXVMzitFr4n3Yztt8mOipIr1Z1qda5J5cPfKBUn/S7 pjNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=iG1i2MHmP7F+ydPtabDxjHouoMtbmtZjo5EZu0SZ+1I=; b=fItldMqPms1Gd4QM9lTIx8NUPWJMuZE6gNrimnq7XKcSpDh00HXlNH1jhju/CmwTWx 8pQcbnGzfHzofteHM/y12z3nAhRWto0E59QkDPUzHmnYnypx0OqIoAv4llnIHRIVW3Bq Y+8P8IwAjT6JErdyz88bFlbBwsAXhxBWPuLyGVx2s3sA6rYBMAKlUIYwgQQjzqhc7+1d GMyKhAZ8vZaQZVWKfE/y3JdZV6Rp7pW+lweHgZgVsr6zZp7hdBb3N5KEOJKyXG+r8XLx hcr+3fXU3J3KNzKBKk7HCXYLsfJ0unf1EhDQdjP1i37ZIwYxeyYNoXCTPgQwL76WXc5j L0sg== X-Gm-Message-State: ANhLgQ2AVwUQfLzNs+5CgsDyMx/EtOLrLjlK5PE3SOZ7V99He4CO67Je mFqaSfQi1JzA8DCtr3YVh3IXivR2OBAuh/mKSQKRo18/ X-Google-Smtp-Source: ADFU+vvgtscBgELtAum6CLtaeTmiwEqjQl79LtOe24YU6CT6yXQ55ykBRU1OgxNk6s0xeszsAdin8LOvpmSH36lgs/k= X-Received: by 2002:adf:9071:: with SMTP id h104mr7169540wrh.359.1583972280954; Wed, 11 Mar 2020 17:18:00 -0700 (PDT) MIME-Version: 1.0 From: Alex Troussov Date: Thu, 12 Mar 2020 01:18:18 +0100 Message-ID: Subject: [GPU pass-through] no compatible bridge window for claimed BAR To: freebsd-virtualization@freebsd.org X-Rspamd-Queue-Id: 48d8YS4shBz3KJJ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=uq5hfvbH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of erleya@gmail.com designates 2a00:1450:4864:20::443 as permitted sender) smtp.mailfrom=erleya@gmail.com X-Spamd-Result: default: False [-1.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (2.39), ipnet: 2a00:1450::/32(-2.40), asn: 15169(-1.65), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[3.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; HTTP_TO_IP(1.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 00:18:06 -0000 Hello all, While trying to pass-through GPU card from Bhyve host (FreeBSD 12.1) to Debian Buster VM, I get error during VM boot like: > dmesg | grep "no compatible bridge window" pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff 64bit pref]: no compatible bridge window but GPU appears in VM: > lspci -v 00:01.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] Flags: fast devsel, IRQ 17 Memory at c1000000 (32-bit, non-prefetchable) [disabled] [size=16M] Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at c2000000 (64-bit, prefetchable) [disabled] [size=32M] I/O ports at 2080 [disabled] [size=128] [virtual] Expansion ROM at 000c0000 [disabled] [size=128K] Kernel driver in use: nvidia Kernel modules: nvidia 00:01.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) Subsystem: NVIDIA Corporation GP106 High Definition Audio Controller Flags: fast devsel, IRQ 18 Memory at c4000000 (32-bit, non-prefetchable) [disabled] [size=16K] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel and its nvidia driver is partially loaded ("nvidia-smi -L" doesn't detect it). On VM complete device tree is: > lspci -tv -[0000:00]-+-00.0 Network Appliance Corporation Device 1275 +-01.0 NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] +-01.1 NVIDIA Corporation GP106 High Definition Audio Controller +-02.0 Red Hat, Inc Virtio network device +-03.0 Red Hat, Inc Virtio block device +-1d.0 Device fb5d:40fb +-1e.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller \-1f.0 Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] And my BHyve command-line parameters are: > bhyve -AHPS -c 4 -m 16G -u \ -s 0,hostbridge \ -s 1:0,passthru,1/0/0 -s 1:1,passthru,1/0/1 \ -s 2,virtio-net,tap0,mac=00:a0:98:fa:5c:8e \ -s 3,virtio-blk,/dev/zvol/attic/tuxdisk0 \ -s 29,fbuf,tcp=192.168.0.2:5900,w=640,h=480,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -l com1,stdio \ tux I tried to remove/rescan GPU card on VM with no success. On host system card is properly masked: > pciconv -lv ppt0@pci0:1:0:0: class=0x030000 card=0x11d710de chip=0x1c0310de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GP106 [GeForce GTX 1060 6GB]' class = display subclass = VGA ppt1@pci0:1:0:1: class=0x040300 card=0x11d710de chip=0x10f110de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GP106 High Definition Audio Controller' class = multimedia subclass = HDA What would be a good strategy to find a solution? >From what I can read from all the info above, somehow the requested memory space for BAR1 for that device is 0xd000000000-0xd00fffffff which is out of addressable space on the system. In the same time device reports prefetchable memory slot of 256 Mb starting from address 0xd0000000. Why it is different? Bhyve or UEFI bug? VM Linux config? Please help... -- Regards, Alex