From nobody Tue Dec 9 19:49:18 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dQqDh2cwBz6Jhlq for ; Tue, 09 Dec 2025 19:49:24 +0000 (UTC) (envelope-from taro@desudesu.org) Received: from desudesu.org (desudesu.org [5.9.28.141]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4dQqDg6WXRz3rXv for ; Tue, 09 Dec 2025 19:49:23 +0000 (UTC) (envelope-from taro@desudesu.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of taro@desudesu.org designates 5.9.28.141 as permitted sender) smtp.mailfrom=taro@desudesu.org Date: Tue, 9 Dec 2025 11:49:18 -0800 From: Taro To: freebsd-virtualization@freebsd.org Subject: [vmm] patch for nvidia pci passthru on releng/15 Message-ID: <20251209114918.2212e7b4@blade> Content-Type: multipart/mixed; boundary="MP_/Hn1kj2w2sXOdhLOwYgW0Cmd" X-Spamd-Bar: / X-Spamd-Result: default: False [-0.68 / 15.00]; MISSING_MIME_VERSION(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.98)[-0.984]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+mx:c]; ONCE_RECEIVED(0.10)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain,text/x-patch]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:24940, ipnet:5.9.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+,1:+,2:+]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[desudesu.org]; HAS_ATTACHMENT(0.00)[] X-Rspamd-Queue-Id: 4dQqDg6WXRz3rXv List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org --MP_/Hn1kj2w2sXOdhLOwYgW0Cmd Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello! When I updated from 14.3 to 15.0, my nvidia 5090 passthru to an Ubuntu guest broke (nvidia-smi on the guest returns "no devices"). Emitted to the host's dmesg: ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10] [CMD Total 0x42] Tail:0x420, Head:0x420. ivhd0: [Event0: Head:0x0 Tail:0x10] [IO_PAGE_FAULT EVT: devId:0x100 DomId:0x1 Addr:0x1271c0000 0x0] (The Ubuntu guest is configured with 32GB of RAM. I later noted that it works correctly when the value is set less than VM_LOWMEM_LIMIT.) git bisect indicated the first bad commit was 08c7dd2f After some trial and error, I added printfs to vm_mmap_memseg to see if there was any difference in the working/bad versions, and there was: when splitting a segment mapping, the previous working version would first map the higher, then the lower. The broken version swapped the ordering: ## WORKING ## vmm: mmap{gpa=4294967296, segid=0, segoff=4294967296, len=31138512896} vmm: mmap{gpa=0, segid=0, segoff=0, len=3221225472} vmm: mmap{gpa=4291313664, segid=8, segoff=13123584, len=3653632} vmm: mmap{gpa=4278190080, segid=8, segoff=0, len=4096} vmm: mmap{gpa=3288334336, segid=9, segoff=0, len=33554432} ## BROKEN ## vmm: mmap{gpa=0, segid=0, segoff=0, len=3221225472} vmm: mmap{gpa=4294967296, segid=0, segoff=3221225472, len=31138512896} vmm: mmap{gpa=4291313664, segid=8, segoff=13123584, len=3653632} vmm: mmap{gpa=4278190080, segid=8, segoff=0, len=4096} vmm: mmap{gpa=3288334336, segid=9, segoff=0, len=33554432} Putting the ordering back (patch attached) fixes the problem for me. I don't know why. I do note that the segoff is also different between working/broken, but am trying not to think about it. I am a humble web programmer, lost and afraid. Thanks! --MP_/Hn1kj2w2sXOdhLOwYgW0Cmd Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=nvidia-passthru-fix.patch diff --git a/lib/libvmmapi/vmmapi.c b/lib/libvmmapi/vmmapi.c index 77f0f8f5c58..a86b444b642 100644 --- a/lib/libvmmapi/vmmapi.c +++ b/lib/libvmmapi/vmmapi.c @@ -562,18 +562,20 @@ vm_setup_memory_domains(struct vmctx *ctx, enum vm_mmap_style vms, if (gpa <= VM_LOWMEM_LIMIT && gpa + dom->size > VM_LOWMEM_LIMIT) { low_len = VM_LOWMEM_LIMIT - gpa; + /* Map remainder first */ + error = map_memory_segment(ctx, segid, VM_HIGHMEM_BASE, + dom->size - low_len, low_len, baseaddr); + if (error) + return (error); + /* Then map lower segment */ error = map_memory_segment(ctx, segid, gpa, low_len, 0, baseaddr); if (error) return (error); + /* Update vars to post-remainder setup */ ctx->lowmem_size = VM_LOWMEM_LIMIT; - /* Map the remainder. */ gpa = VM_HIGHMEM_BASE; len = dom->size - low_len; - error = map_memory_segment(ctx, segid, gpa, len, - low_len, baseaddr); - if (error) - return (error); } else { len = dom->size; error = map_memory_segment(ctx, segid, gpa, len, 0, --MP_/Hn1kj2w2sXOdhLOwYgW0Cmd--