From nobody Tue Sep 16 11:12:39 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cQzlF47mKz68nRr for ; Tue, 16 Sep 2025 11:12:41 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cQzlF3XpZz453p; Tue, 16 Sep 2025 11:12:41 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758021161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BIlTslhkh+8YEgcQ7CScMYc8JtsUFSvnLDnOvTP67Ic=; b=lbteu2bVFMw6iIXbsapsTt0nXn5LRK4ca5mUGYGEvHDyk36g2zWp7G8x8KPkA7GxJ9SnF8 OE4zhLu2fhGseS7cFvxiiMzpEx+T1IfTgC1htzYX6e/ymqJ2aVz2fLNlQh3la+fV5df0x9 9IST3Gv8DgDtx2x3qhmZigp3PiEOTyJaa/31EDx0idm5zPQ/mxw6eLxkL6QM2I+yafRbAX 24knTGU8wf5Wq2RT2nAUEzF9oYV/WPaM5ZCoQkOEIXy1/SOhh5uOr39etTdyMdQ4n8p55n zzyoFyCYmlnxxhi3wg+55nappR5J9ihfJjnyNz+zHVLc4QxW0UQ7A4W9RHxWTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758021161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BIlTslhkh+8YEgcQ7CScMYc8JtsUFSvnLDnOvTP67Ic=; b=F3anEjlBf8sRs8ob9zvVu9JLCHTPdTCvaTe7OTuR6FiODxNWguNbJ405S/f0oEze7wdc96 FmsSP4yl3Zn6Lv6nB+42Hfsa3BZt4KR0gu1fyWZkajb8P/1kF1cwAWxZc1+3Pp8l7c5B/v +xGU2r44dr40Oun1lTrcd1bzFsb/nx8wqQVcHWPHtrupII3Y+XQ0DHLiCu/3/fx86VR5GJ fXs0RhfBejJyBCj3P58m8EEsIBFcxhC+kWIGJ17xTWETVRzP2NjDI/KISRGtrH16BVa0dG oK9evboFHlIQ+UC5t/yhUCbb/tT1IAgyILX2kEW5SLL08A5iVEHiqN7qfpxXtg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758021161; a=rsa-sha256; cv=none; b=blOXMqEexhDGxToPzyXVDZl2OmO+Excd7yN8/WsNx+G3p4T/bbO/GPvXfnslwleKRyXQ/g ZBR2jHk4vrMEp98DTTY7RTUESPgz9NzAK14VZogF39oZg2vjIDRdmDqXVrLTyDwBS8PAvI nXOo2ZhBSKzchNr15mbkq5PJf+Ifzh7QhU4ANviwWqnkKCPSeNW+QUpYw/4C8CuL8lnQiO avQDvRYFgetxlRIjHPfMTzeXsvVuFB0+c+RudGfafDrI2Cz0UevE/NtpqMT4/8a5b1Sqw2 tGVS0QGOVzIuXAjma6CBgqVOYdbsCyaGiNLVN1lw2khiiCk5q9BB2aukKmhoGw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [10.253.7.34] (nat-184-125.net.cam.ac.uk [131.111.184.125]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 4cQzlF0vMlzLZl; Tue, 16 Sep 2025 11:12:40 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: <83666dad-078b-4634-b66c-3153be957567@FreeBSD.org> Date: Tue, 16 Sep 2025 12:12:39 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Max BAR size for passthru? Content-Language: en-US To: Chuck Tuffli , freebsd-virtualization@freebsd.org Cc: Mark Johnston References: From: John Baldwin In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 9/15/25 14:58, Chuck Tuffli wrote: > I'm trying to get PCI passthru of an AMD GPU working, and am seeing > what I think is a problem mapping the device's BAR into the guest. The > host shows: > > # pciconf -lbevV pci0:3:0:0 > vgapci0@pci0:3:0:0: class=0x030000 rev=0xc0 hdr=0x00 vendor=0x1002 > device=0x7590 subvendor=0x148c subdevice=0x2437 > vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]' > device = 'Navi 44 [Radeon RX 9060 XT]' > class = display > subclass = VGA > bar [10] = type Prefetchable Memory, range 64, base > 0x4000000000, size 268435456, enabled > bar [18] = type Prefetchable Memory, range 64, base > 0x4010000000, size 2097152, enabled > bar [20] = type I/O Port, range 32, base 0x6000, size 256, enabled > bar [24] = type Memory, range 32, base 0x81100000, size 524288, enabled > PCI-e errors = Correctable Error Detected > Unsupported Request Detected > Non-fatal = Unsupported Request > Corrected = Advisory Non-Fatal Error > > (yes, a 256 MiB BAR). After starting the Linux guest, bhyve spits out the error: > > bhyve: pci_passthru: map_pptdev_mmio failed. I've traced this a bit > and see that vm_map_find returns 3 (KERN_NO_SPACE) with arguments > off=0, addr=0x800000000, len=2097152, max_addr=0 > > In the guest, lspci reports > # lspci -s 00:07.0 -v > 00:07.0 VGA compatible controller: Advanced Micro Devices, Inc. > [AMD/ATI] Device 7590 (rev c0) (prog-if 00 [VGA controller]) > Subsystem: Tul Corporation / PowerColor Device 2437 > Flags: bus master, fast devsel, latency 0 > Memory at 800000000 (64-bit, prefetchable) [size=256M] > Memory at 810000000 (64-bit, prefetchable) [size=2M] > I/O ports at 2000 [size=256] > Memory at c1000000 (32-bit, non-prefetchable) [size=512K] > Expansion ROM at 000c0000 [virtual] [disabled] [size=128K] > > Has anyone run into something like this before? Is this a limit I'm > bumping into? Suggestions for what to poke next, etc.? TIA! Hmmm, I think it would be useful if you could dump the effective memory mappings for the guest PAs (the vm_map passed to vm_map_find) to see why it is failing. Note that the request that is failing is for 2M, not 256M, so I'm curious if the address is wrong somehow (why isn't it 0x810000000?) Might be good to confirm which BAR it is trying to allocate GPA space for. -- John Baldwin