From nobody Mon Mar 24 15:38:52 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLy0Q6F13z5s3yt for ; Mon, 24 Mar 2025 15:39:34 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLy0P3dDHz3mF2 for ; Mon, 24 Mar 2025 15:39:33 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=XcYAzo0k; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::102e as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x102e.google.com with SMTP id 98e67ed59e1d1-2ff6a98c638so9554063a91.0 for ; Mon, 24 Mar 2025 08:39:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742830771; x=1743435571; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=+MCLZMN2fg5vUJVZKn41goHH+CnaYCjuVW1V5rWEFzQ=; b=XcYAzo0k+b6AJpOI27maG9LVYDhzYzllxJ7SgVIR7xSaHuWXDvXGVlCGsvoA6iJWI9 bsBO0MKJ7uA3vu0wVxdaBPD8JzS4GF4Wd6eMAFQgquNg8cuqPOiNtriRklR37d/UMX7N IRn/IzKkVbR5/JaZJfdDZQCGFvWedFBk3sBHXspuPOzPghE+2wtmdFuK9ztXs/gLLMyo WMIs0SFCT7VSuAQAgpySywJJfsDV4n+CKuyBSuVOuDNf1wV52k2930uThps7zUrHwLXQ v0TQMTIcoGMetBi+AKRl7o+bXBdvROXJsBToex7Udn2RC/VRIqZUDi123EuSneRFJVZl jAJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742830771; x=1743435571; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+MCLZMN2fg5vUJVZKn41goHH+CnaYCjuVW1V5rWEFzQ=; b=V2/dplHA0foPhmLOByI+EVGZomZx7aIevD7/SZAumY47yat/9r1qntKttnfoBE8A94 oVuA2zXyuNZlKofU7RfB7uMaiU6JyFp9lP5A+yHvUz783FEqOMgLssO7DeSxkxYprpak WQRWr7f04wtU4AfBGt3JEbdSGAalPNlnD02gwxsXaurRKDJzG3M0pWOJCA2Q2cFmfIwV bpYGyG4uOasVnD7eaHLiSuxgH91aG7A1BF+ABBs00PGTeXjfhNOrr/wXPpVybWeQOXBX jbxTDhrOmW0dAVcpXNh53yGasU9kV9QsUeHoL7cWad6gSVNgOrUwFk/FqkKYU3V1KDg1 fiyQ== X-Gm-Message-State: AOJu0YzQ85zj44FUp/hNu45I4ilW6Vdmo3XshZfYJQ0B7oNYHwBE6PrG LAtRfO8WzDc4VjgeoZ5sYncASY8hbV+DG4fwL+tSnYMROeUQ2PcQYjz0WSpRx58zqW9PErAT2X4 gMFBZCQLtGsBDwsqgGDTdWDSIjm0thzKr X-Gm-Gg: ASbGnctlW5WBDmzFEPJ2nAIYWaNeDFeKz4hFiv6Hl0P1iHL/1SxtaKt70lOJrIYWua+ WxonEExb4efGNJev0UOUsp4pkiG1g4YSLVPCStoONNV9JUHWSmFGvJ/0K/Ne8MVVrLdDwgQXnoN 7rHpYhCQX+c70tFMsBWUZU4cBd4Fs= X-Google-Smtp-Source: AGHT+IEVuGD6fKZZT69OzSv/dZZ6+G+xr1/nh2s9oTuIIg2MfaGmzSVCcoxNzlzdnwUeqXi2zp15M7DIKfpHwyYTmH4= X-Received: by 2002:a17:90b:53d0:b0:2ee:df70:1ff3 with SMTP id 98e67ed59e1d1-3030fd36f80mr26545538a91.0.1742830770504; Mon, 24 Mar 2025 08:39:30 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 16:38:52 +0100 X-Gm-Features: AQ5f1JpWesxAEyR4YhAda09II1-TG04rYs29DDE-0Cu-0LSoOkuYlEhYpimN32s Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000facef9063118686b" X-Spamd-Result: default: False [1.04 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; NEURAL_HAM_SHORT(-0.96)[-0.959]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102e:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLy0P3dDHz3mF2 X-Spamd-Bar: + --000000000000facef9063118686b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Shamim. You don't have any bhyve-win. You can use bhyve. I use bhyve-win to pass through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are two different executables,because for some reason,my bhyve-lin executable is not able to pass my GPU in a Windows vm. On Mon, Mar 24, 2025 at 4:29=E2=80=AFPM Shamim Shahriar wrote: > Hi Mario > > I will try the options you have used later and let you know how it goes. > Not sure if this is intentional or typo, but I don't recall any bhyve-win > binary in my system. > > Best regards > SS > > On Mon, 24 Mar 2025 at 14:59, Mario Marietto > wrote: > >> Oh sorry. I forgot to add the following bhyve parameters necessary to >> passthru my GPU :D >> >> -s 8:0,passthru,2/0/0 \ >> -s 8:1,passthru,2/0/1 \ >> -s 8:2,passthru,2/0/2 \ >> -s 8:3,passthru,2/0/3 \ >> >> On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto >> wrote: >> >>> Usually I use this kind of script to launch a bhyve vm : >>> >>> #!/bin/sh >>> >>> setxkbmap it >>> kldload vmm.ko >>> vms=3D"$(ls /dev/vmm/*)" >>> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >>> kldload vmm.ko >>> >>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >>> kldload nvidia-modeset >>> devctl clear driver -f pci0:2:0:0 >>> devctl set driver -f pci0:2:0:0 ppt >>> else >>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:1 >>> devctl set driver -f pci0:2:0:1 ppt >>> else >>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:2 >>> devctl set driver -f pci0:2:0:2 ppt >>> else >>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >>> fi >>> >>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:2:0:3 >>> devctl set driver -f pci0:2:0:3 ppt >>> else >>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >>> fi >>> >>> echo "rtx 2080ti is fully attached to ppt" >>> >>> for vm in $vms; do >>> session=3D"${vm##*/}" >>> echo "bhyve session =3D $session" >>> echo "vnc session =3D $vncs" >>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; >>> then >>> printf 'VNC session not found,destroying ghost vms\n' >>> bhyvectl --vm=3D$session --destroy >>> else >>> printf 'Found VNC session %s\n' >>> "${session},no ghost vms found,not destroying them" >>> fi >>> done >>> >>> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (AM13N4CZ)/ && d{print d}'` >>> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >>> >>> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2015040204055E)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >>> >>> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2027285F1175)/ && d{print d}'` >>> echo "CT1000P1SSD8 ; $vmdisk2" >>> >>> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (20130508005976F)/ && d{print d}'` >>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >>> >>> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (BE0191510218)/ && d{print d}'` >>> echo "G-DRIVE USB ; $vmdisk4" >>> >>> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (38234B4237354B45)/ && d{print d}'` >>> echo "Elements 25A3 ; $vmdisk5" >>> >>> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (WD-WCAV2X797309)/ && d{print d}'` >>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >>> >>> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (20140108006C)/ && d{print d}'` >>> echo "Corsair Force 3 SSD ; $vmdisk7" >>> >>> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (1924E50B2AE5)/ && d{print d}'` >>> echo "CT500MX500SSD4 ; $vmdisk8" >>> >>> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (0774921DDC4200A6)/ && d{print d}'` >>> echo "SanDisk Cruzer-15GB ; $vmdisk9" >>> >>> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (60A44D4138D8F311190A0149)/ && d{print d}'` >>> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >>> >>> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (575845483038524844323238)/ && d{print d}'` >>> echo "WD 2500BMV External ; $vmdisk11" >>> >>> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (57442E575845323039544433303334)/ && d{print d}'` >>> echo "WD 3200BMV External ; $vmdisk12" >>> >>> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>> (2414E989076B)/ && d{print d}'` >>> echo "CT500BX500SSD1 ; $vmdisk13" >>> >>> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w = -H -A \ >>> -s 0,hostbridge \ >>> -s 1,ahci-hd,/dev/$vmdisk13 \ >>> -s 10,virtio-net,tap6 \ >>> -s 11,virtio-9p,sharename=3D/ \ >>> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ >>> -s 31,lpc \ >>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >>> >>> As you can see,each vm has a unique ID (connected with the vncviewer >>> instance) and the vncviewer window is launched as soon as the bhyve vm = is >>> called...and my Geforce RTX 2080 ti is passed through inside the vm. >>> >>> >>> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto >>> wrote: >>> >>>> First of all,you could use a more flexible technique than using the >>>> pptdevs on /boot/loader.conf. I would use a script like this : >>>> >>>> 1) >>>> >>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>>> echo "18/0/0 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:18:0:0 >>>> devctl set driver -f pci0:18:0:0 ppt >>>> else >>>> echo "18/0/0 already attached to ppt" >>>> fi >>>> >>>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>>> >>>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >>>> shamim.shahriar@gmail.com> wrote: >>>> >>>>> Good afternoon everyone. >>>>> >>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have che= cked >>>>> the device IDs and put pptdevs in place >>>>> >>>>> # cat /boot/loader.conf >>>>> pptdevs=3D"18/0/0 19/0/0" >>>>> pptdevs2=3D"72/0/0 73/0/0" >>>>> >>>>> this is showing the GPUs as pptdev on the pciconf >>>>> >>>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>>> vendor =3D 'NVIDIA Corporation' >>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>> class =3D display >>>>> subclass =3D 3D >>>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>>> vendor =3D 'NVIDIA Corporation' >>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>> class =3D display >>>>> subclass =3D 3D >>>>> >>>>> as I am using vm-bhyve, I have put the configuration as below >>>>> >>>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>>> loader=3D"uefi" >>>>> cpu=3D16 >>>>> memory=3D128G >>>>> xhci_mouse=3D"yes" >>>>> debug=3D"true" >>>>> >>>>> graphics=3D"yes" >>>>> graphics_listen=3D"127.0.0.1" >>>>> graphics_port=3D"5920" >>>>> graphics_res=3D"1024x768" >>>>> graphics_wait=3D"no" >>>>> #graphics_vga=3D"io" >>>>> >>>>> network0_type=3D"virtio-net" >>>>> network0_switch=3D"swUNI" >>>>> network0_mac=3D"58:9c:fc:06:3f:af" >>>>> >>>>> disk0_type=3D"nvme" >>>>> disk0_name=3D"jagadish-disk0.img" >>>>> #disk0_size=3D"128G" >>>>> >>>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>>>> >>>>> passthru0=3D"19/0/0=3D6:0" >>>>> >>>>> bhyve_options=3D"-A -H -P" >>>>> #END >>>>> >>>>> however, when I start the vm (for OS installation to start with), it >>>>> shows it is running but I am unable to access the VNC for a while, th= en >>>>> when I finally manage to connect via VNC, there is nothing on the scr= een, >>>>> just blank dark screen >>>>> >>>>> the vm-bhyve.log shows >>>>> >>>>> Mar 24 13:59:26: initialising >>>>> Mar 24 13:59:26: [loader: uefi] >>>>> Mar 24 13:59:26: [cpu: 16] >>>>> Mar 24 13:59:26: [memory: 128G] >>>>> Mar 24 13:59:26: [hostbridge: standard] >>>>> Mar 24 13:59:26: [com ports: com1] >>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>>> Mar 24 13:59:26: [debug mode: true] >>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>>> Mar 24 13:59:26: [primary disk dev: file] >>>>> Mar 24 13:59:26: initialising network device tap0 >>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>>> Mar 24 13:59:26: booting >>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>>> Mar 24 13:59:26: [bhyve iso device: -s >>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>>> Mar 24 13:59:26: starting bhyve (run 1) >>>>> >>>>> based on what I can see, and a little out of desparation, I decided t= o >>>>> run the installer in the foreground, and below is what I got >>>>> >>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>>> >>>>> it stays there for as long as I wait until I poweroff the vm. >>>>> >>>>> tried with Debian installer, >>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>>> Starting jagadish >>>>> * found guest in /mnt/VMs/jagadish >>>>> * booting... >>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>>> >>>>> >>>>> ideally I need to install Debian for my user base on these GPU based >>>>> systems. But even that is proving to be impossible since the installe= r does >>>>> not move any further from where it is. >>>>> >>>>> Any thoughts/ideas/suggestions what I else I can try to make this >>>>> work? Anything that you have tried that worked? have I missed somethi= ng? >>>>> >>>>> Would appreciate any and all thoughts/suggestions >>>>> >>>>> best regards >>>>> SS >>>>> >>>> >>>> >>>> -- >>>> Mario. >>>> >>> >>> >>> -- >>> Mario. >>> >> >> >> -- >> Mario. >> > --=20 Mario. --000000000000facef9063118686b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Shamim.

You don't = have any bhyve-win. You can use bhyve. I use bhyve-win to pass through my G= PU in a Windows vm,bhyve-lin for a Linux vm. They are two different executa= bles,because for some reason,my bhyve-lin executable is not able to pass my= GPU in a Windows vm.=C2=A0

On Mon, Mar 24, 2025= at 4:29=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Hi Mario

=
I will try the options you have used later and let you know how it goe= s. Not sure if this is intentional or typo, but I don't recall any bhyv= e-win binary in my system.

Best regards
= SS

On Mon, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com> wr= ote:
Oh sorry. I forgot to add the following bhyve parameters necessa= ry to passthru my GPU :D

-s 8:0,passthru,2/0/0 \-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2= /0/3 \

On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
Usually I use this kind of script to launch a bhyve vm :=

#!/bin/sh

setxkbmap it
kldload vmm.ko
vms= =3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | awk '/vncview= er [0]/{print $6}')"
kldload vmm.ko

if ! pciconf -l pci0= :2:0:0 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/0= is not attached to ppt,attaching..."
kldload nvidia-modeset
dev= ctl clear driver -f pci0:2:0:0
devctl set driver -f pci0:2:0:0 ppt
el= se
echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
= fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
ec= ho "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."devctl clear driver -f pci0:2:0:1
devctl set driver -f pci0:2:0:1 pptelse
echo "rtx 2080ti slot 2/0/1 is already attached to ppt"=
fi

if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; thenecho "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."= ;
devctl clear driver -f pci0:2:0:2
devctl set driver -f pci0:2:0:2 p= pt
else
echo "rtx 2080ti slot 2/0/2 is already attached to ppt&q= uot;
fi

if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; th= en
echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching...&= quot;
devctl clear driver -f pci0:2:0:3
devctl set driver -f pci0:2:0= :3 ppt
else
echo "rtx 2080ti slot 2/0/3 is already attached to p= pt"
fi

echo "rtx 2080ti is fully attached to ppt"<= br>
for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve session =3D $session&qu= ot;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "v= nc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 if ! printf '%s\n' "${vncs}" | grep "${se= ssion#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf '= VNC session not found,destroying ghost vms\n'
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 printf 'Found VNC session %s\n' "${session},no ghos= t vms found,not destroying them"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geom disk list | awk = 9;/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ && d{print d}'= `
echo "Seagate M3 Portable 1.8T ; $vmdisk0"

vmdisk1=3D= `geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (201504020405= 5E)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 1.8= T ; $vmdisk1"

vmdisk2=3D`geom disk list | awk '/^Geom name:= /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{print d}'`
echo = "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`geom disk list | awk = '/^Geom name: /{d=3D$NF} /^ *ident: (20130508005976F)/ && d{pri= nt d}'`
echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"<= br>
vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ide= nt: (BE0191510218)/ && d{print d}'`
echo "G-DRIVE USB ;= $vmdisk4"

vmdisk5=3D`geom disk list | awk '/^Geom name: /{= d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{print d}'`
echo= "Elements 25A3 ; $vmdisk5"

vmdisk6=3D`geom disk list | aw= k '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X797309)/ && d{p= rint d}'`
echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"=

vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *id= ent: (20140108006C)/ && d{print d}'`
echo "Corsair Forc= e 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk list | awk '/^Geom= name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'`echo "CT500MX500SSD4 ; $vmdisk8"

vmdisk9=3D`geom disk lis= t | awk '/^Geom name: /{d=3D$NF} /^ *ident: (0774921DDC4200A6)/ &&a= mp; d{print d}'`
echo "SanDisk Cruzer-15GB ; $vmdisk9"
=
vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident= : (60A44D4138D8F311190A0149)/ && d{print d}'`
echo "Kin= gston DataTraveler 2.0 ; $vmdisk10"

vmdisk11=3D`geom disk list = | awk '/^Geom name: /{d=3D$NF} /^ *ident: (575845483038524844323238)/ &= amp;& d{print d}'`
echo "WD 2500BMV External ; $vmdisk11&qu= ot;

vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^= *ident: (57442E575845323039544433303334)/ && d{print d}'`
e= cho "WD 3200BMV External ; $vmdisk12"

vmdisk13=3D`geom dis= k list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2414E989076B)/ &&= amp; d{print d}'`
echo "CT500BX500SSD1 ; $vmdisk13"
/usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 = -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/$vmdisk13 \
-= s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp= =3D0.0.0.0:5906,w=3D1= 600,h=3D950 \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/B= HYVE_BHF_CODE.fd \
vm0:6 < /dev/null & sleep 5 && vncview= er 0:6 &

As you can see,each vm has a unique I= D (connected with the vncviewer instance) and the vncviewer window is launc= hed as soon as the bhyve vm is called...and my Geforce RTX 2080 ti is passe= d through inside the vm.


On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM M= ario Marietto <marietto2008@gmail.com> wrote:
First of all,you could use a= more flexible technique than using the pptdevs on /boot/loader.conf. I wou= ld use a script like this :

1)

if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
ech= o "18/0/0 is not attached to ppt,attaching..."
devctl clear dr= iver -f pci0:18:0:0
devctl set driver -f pci0:18:0:0 ppt
else
echo= "18/0/0 already attached to ppt"
fi

2)= I would not like to use vm-bhyve. It adds only some confusion...

On M= on, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com&g= t; wrote:
Good afternoon everyone.

I am trying to have VMs with GPU= passthrough. The setup is a Dell server with NVidia Quadro RTX 6000/8000 i= nstalled already. I have checked the device IDs and put pptdevs in place
# cat /boot/loader.conf
pptdevs=3D"18/0/0 19/0/0"
pptd= evs2=3D"72/0/0 73/0/0"

this is showing the GPUs as pptdev = on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x0302= 00 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de= subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA C= orporation'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Qua= dro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D disp= lay
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt1@pci0:19:0:0: =C2=A0 =C2= =A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D= 0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0= =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2= =A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
as I am using vm-bhyve, I have put the configuration as below

# cat= /mnt/VMs/jagadish/jagadish.conf
loader=3D"uefi"
cpu=3D16memory=3D128G
xhci_mouse=3D"yes"
debug=3D"true"=

graphics=3D"yes"
graphics_listen=3D"127.0.0.1&quo= t;
graphics_port=3D"5920"
graphics_res=3D"1024x768&quo= t;
graphics_wait=3D"no"
#graphics_vga=3D"io"
<= br>network0_type=3D"virtio-net"
network0_switch=3D"swUNI&= quot;
network0_mac=3D"58:9c:fc:06:3f:af"

disk0_type=3D&= quot;nvme"
disk0_name=3D"jagadish-disk0.img"
#disk0_si= ze=3D"128G"

uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe3= 4de"

passthru0=3D"19/0/0=3D6:0"

bhyve_options= =3D"-A -H -P"
#END

however, when I start the vm (for OS= installation to start with), it shows it is running but I am unable to acc= ess the VNC for a while, then when I finally manage to connect via VNC, the= re is nothing on the screen, just blank dark screen

the vm-bhyve.log= shows

Mar 24 13:59:26: initialising
Mar 24 13:59:26: =C2=A0[load= er: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]
Mar 24 13:59:26: =C2=A0[me= mory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridge: standard]
Mar 24 13:5= 9:26: =C2=A0[com ports: com1]
Mar 24 13:59:26: =C2=A0[uuid: 966e909b-129= 3-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: =C2=A0[debug mode: true]
M= ar 24 13:59:26: =C2=A0[primary disk: jagadish-disk0.img]
Mar 24 13:59:26= : =C2=A0[primary disk dev: file]
Mar 24 13:59:26: initialising network d= evice tap0
Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
M= ar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:2= 6: booting
Mar 24 13:59:26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l= bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-= 1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 13:59:26: =C2=A0[bhyve devices= : -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.im= g -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s = 7:0,fbuf,tcp=3D127.0.0.= 1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[= bhyve console: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bh= yve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.= iso,ro]
Mar 24 13:59:26: starting bhyve (run 1)

based on what I c= an see, and a little out of desparation, I decided to run the installer in = the foreground, and below is what I got

# vm install -f jagadish Fre= eBSD-14.2-RELEASE-amd64-disc1.iso
Starting jagadish
=C2=A0 * found gu= est in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: = 0x112245400000 [sz 16777216]

it stays there for as long as I wait un= til I poweroff the vm.

tried with Debian installer,
# vm install= -f jagadish debian-12.5.0-amd64-netinst.iso
Starting jagadish
=C2=A0= * found guest in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame bu= ffer base: 0x2747e2400000 [sz 16777216]


ideally I need to instal= l Debian for my user base on these GPU based systems. But even that is prov= ing to be impossible since the installer does not move any further from whe= re it is.

Any thoughts/ideas/suggestions what I else I can try to ma= ke this work? Anything that you have tried that worked? have I missed somet= hing?

Would appreciate any and all thoughts/suggestions

best = regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000facef9063118686b--