From nobody Mon Mar 24 15:29:45 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLxnL4ywhz5s3Xk for ; Mon, 24 Mar 2025 15:29:58 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com [IPv6:2607:f8b0:4864:20::112c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLxnL2tcbz3hs2 for ; Mon, 24 Mar 2025 15:29:58 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-700b17551cdso41146627b3.0 for ; Mon, 24 Mar 2025 08:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742830197; x=1743434997; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=b+nrJ8y5G8HwazmiK6gZkUF6xv2OuMkMw1IxKbfvbTI=; b=Wu0WKVB81X7EB40cXdudUFTS8f0RLLZyvd6+jImYt2/ATdlhxSk3or+PWuYEpXY9fx VRer0PByIMyUEfRh7tYb82IQB8LlZepggT4UvREehwuV+WUvnUpvCrTOb2eInxkIlYCZ A3E9BCWovIgt+Xce3/6zsEEuB7yrDfOpUXoEwG7F9BHetugLR/kF+Bxy1IkhAtdHh5Hu Tpg+VGOeHWrY9psCUIqJgIq1B64a9zP4fM80Hd/89ocDt/VKEulxfdPxm+sN4ilwphLl DvqRL5FNh/iLWAchov94gzaZUT+YRvMYVUkc12GS8S2E1OM/tvM22XvdXMN56Bzdm5KD kEeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742830197; x=1743434997; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=b+nrJ8y5G8HwazmiK6gZkUF6xv2OuMkMw1IxKbfvbTI=; b=jRoJ84N1P9qplS12FwNU4UcQ0Kz8BufMhJ0DxSeJ00+6BgDTylYspVvrLGxACbxNUe 24byj2upoTFhAoL6bzPkYK8U5aelb8fPrGgY5fhnBNERKotZuCSE9oMXhHGLOQhNgQ8G SJkbEs44/O8pO0tDQvIUNH3qrMLizhehbz8SZCRhMmlacTJu7+0jIk2oMcehZbX4RMXd sE2JRGIJkMqGzdnqeRBGF04B2xjqwAZSFLpui8IcXUiEU/r7o6hiswGKGhJWFg1S+4R0 lAUYvfToPp/7uEZ21LA9Odggw0/I87PLmLuKga1WobzCFQ1TloT2rXvlZzBCHn4SYdi0 9UYQ== X-Gm-Message-State: AOJu0Yz2Q6fOqVG7IapL53kw0zxpTFOnOkg5pJDj3V3YH42qMFj67gdP 6H5EcImWfZagExYnAjQM+Qci1R9F6hE3CiyyYZ0wVTkPxTOxbxCgVwSYlLEh+bLholY9zwJH6Ke ozWn+8vAIYUmwvhmPu+guj47HBSf2 X-Gm-Gg: ASbGncurr0DkffFYPAb74tFBrzLxjzIgmqn6sLou1xV1C6M/OkXSaF1sKvM1dVtHkr+ 7d4ShPDQwDrsCEjBZ919LT/b1uT9BZetyxhLaaFjHkontO3zFJxVRKaqUf4xGHfVjOWjwUiyfKR 4PvGvXjnzswz1SD3lYQow31Ck= X-Google-Smtp-Source: AGHT+IH5dOdXOtyElKV1Ck9vZy/oyI7b9aQf3+8uawzR3bMeMVMLVTZ72gG4IJekbbwxy6wArIqYy7CE4jrviSQJchg= X-Received: by 2002:a05:690c:4444:b0:6fe:e79f:bd8f with SMTP id 00721157ae682-700bacd58eemr193187517b3.26.1742830197228; Mon, 24 Mar 2025 08:29:57 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Shamim Shahriar Date: Mon, 24 Mar 2025 15:29:45 +0000 X-Gm-Features: AQ5f1JrgPqhMVT4Ec2V0-vi-yijJ0Ma3B5TrzkMCb3bbxqMi9ooaaETdvjhTXc8 Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Mario Marietto Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000cf4c600631184611" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4ZLxnL2tcbz3hs2 X-Spamd-Bar: ---- --000000000000cf4c600631184611 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mario I will try the options you have used later and let you know how it goes. Not sure if this is intentional or typo, but I don't recall any bhyve-win binary in my system. Best regards SS On Mon, 24 Mar 2025 at 14:59, Mario Marietto wrote= : > Oh sorry. I forgot to add the following bhyve parameters necessary to > passthru my GPU :D > > -s 8:0,passthru,2/0/0 \ > -s 8:1,passthru,2/0/1 \ > -s 8:2,passthru,2/0/2 \ > -s 8:3,passthru,2/0/3 \ > > On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto > wrote: > >> Usually I use this kind of script to launch a bhyve vm : >> >> #!/bin/sh >> >> setxkbmap it >> kldload vmm.ko >> vms=3D"$(ls /dev/vmm/*)" >> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >> kldload vmm.ko >> >> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >> kldload nvidia-modeset >> devctl clear driver -f pci0:2:0:0 >> devctl set driver -f pci0:2:0:0 ppt >> else >> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >> fi >> >> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:2:0:1 >> devctl set driver -f pci0:2:0:1 ppt >> else >> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >> fi >> >> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:2:0:2 >> devctl set driver -f pci0:2:0:2 ppt >> else >> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >> fi >> >> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:2:0:3 >> devctl set driver -f pci0:2:0:3 ppt >> else >> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >> fi >> >> echo "rtx 2080ti is fully attached to ppt" >> >> for vm in $vms; do >> session=3D"${vm##*/}" >> echo "bhyve session =3D $session" >> echo "vnc session =3D $vncs" >> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; the= n >> >> printf 'VNC session not found,destroying ghost vms\n' >> bhyvectl --vm=3D$session --destroy >> else >> printf 'Found VNC session %s\n' >> "${session},no ghost vms found,not destroying them" >> fi >> done >> >> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (AM13N4CZ)/ && d{print d}'` >> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >> >> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (2015040204055E)/ && d{print d}'` >> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >> >> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (2027285F1175)/ && d{print d}'` >> echo "CT1000P1SSD8 ; $vmdisk2" >> >> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (20130508005976F)/ && d{print d}'` >> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >> >> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (BE0191510218)/ && d{print d}'` >> echo "G-DRIVE USB ; $vmdisk4" >> >> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (38234B4237354B45)/ && d{print d}'` >> echo "Elements 25A3 ; $vmdisk5" >> >> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (WD-WCAV2X797309)/ && d{print d}'` >> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >> >> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (20140108006C)/ && d{print d}'` >> echo "Corsair Force 3 SSD ; $vmdisk7" >> >> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (1924E50B2AE5)/ && d{print d}'` >> echo "CT500MX500SSD4 ; $vmdisk8" >> >> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (0774921DDC4200A6)/ && d{print d}'` >> echo "SanDisk Cruzer-15GB ; $vmdisk9" >> >> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (60A44D4138D8F311190A0149)/ && d{print d}'` >> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >> >> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (575845483038524844323238)/ && d{print d}'` >> echo "WD 2500BMV External ; $vmdisk11" >> >> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (57442E575845323039544433303334)/ && d{print d}'` >> echo "WD 3200BMV External ; $vmdisk12" >> >> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >> (2414E989076B)/ && d{print d}'` >> echo "CT500BX500SSD1 ; $vmdisk13" >> >> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -= H -A \ >> -s 0,hostbridge \ >> -s 1,ahci-hd,/dev/$vmdisk13 \ >> -s 10,virtio-net,tap6 \ >> -s 11,virtio-9p,sharename=3D/ \ >> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ >> -s 31,lpc \ >> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >> >> As you can see,each vm has a unique ID (connected with the vncviewer >> instance) and the vncviewer window is launched as soon as the bhyve vm i= s >> called...and my Geforce RTX 2080 ti is passed through inside the vm. >> >> >> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto >> wrote: >> >>> First of all,you could use a more flexible technique than using the >>> pptdevs on /boot/loader.conf. I would use a script like this : >>> >>> 1) >>> >>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>> echo "18/0/0 is not attached to ppt,attaching..." >>> devctl clear driver -f pci0:18:0:0 >>> devctl set driver -f pci0:18:0:0 ppt >>> else >>> echo "18/0/0 already attached to ppt" >>> fi >>> >>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>> >>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >>> shamim.shahriar@gmail.com> wrote: >>> >>>> Good afternoon everyone. >>>> >>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>> server with NVidia Quadro RTX 6000/8000 installed already. I have chec= ked >>>> the device IDs and put pptdevs in place >>>> >>>> # cat /boot/loader.conf >>>> pptdevs=3D"18/0/0 19/0/0" >>>> pptdevs2=3D"72/0/0 73/0/0" >>>> >>>> this is showing the GPUs as pptdev on the pciconf >>>> >>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>> vendor =3D 'NVIDIA Corporation' >>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>> class =3D display >>>> subclass =3D 3D >>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>>> vendor =3D 'NVIDIA Corporation' >>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>> class =3D display >>>> subclass =3D 3D >>>> >>>> as I am using vm-bhyve, I have put the configuration as below >>>> >>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>> loader=3D"uefi" >>>> cpu=3D16 >>>> memory=3D128G >>>> xhci_mouse=3D"yes" >>>> debug=3D"true" >>>> >>>> graphics=3D"yes" >>>> graphics_listen=3D"127.0.0.1" >>>> graphics_port=3D"5920" >>>> graphics_res=3D"1024x768" >>>> graphics_wait=3D"no" >>>> #graphics_vga=3D"io" >>>> >>>> network0_type=3D"virtio-net" >>>> network0_switch=3D"swUNI" >>>> network0_mac=3D"58:9c:fc:06:3f:af" >>>> >>>> disk0_type=3D"nvme" >>>> disk0_name=3D"jagadish-disk0.img" >>>> #disk0_size=3D"128G" >>>> >>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>>> >>>> passthru0=3D"19/0/0=3D6:0" >>>> >>>> bhyve_options=3D"-A -H -P" >>>> #END >>>> >>>> however, when I start the vm (for OS installation to start with), it >>>> shows it is running but I am unable to access the VNC for a while, the= n >>>> when I finally manage to connect via VNC, there is nothing on the scre= en, >>>> just blank dark screen >>>> >>>> the vm-bhyve.log shows >>>> >>>> Mar 24 13:59:26: initialising >>>> Mar 24 13:59:26: [loader: uefi] >>>> Mar 24 13:59:26: [cpu: 16] >>>> Mar 24 13:59:26: [memory: 128G] >>>> Mar 24 13:59:26: [hostbridge: standard] >>>> Mar 24 13:59:26: [com ports: com1] >>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>> Mar 24 13:59:26: [debug mode: true] >>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>> Mar 24 13:59:26: [primary disk dev: file] >>>> Mar 24 13:59:26: initialising network device tap0 >>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>> Mar 24 13:59:26: booting >>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>> Mar 24 13:59:26: [bhyve iso device: -s >>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>> Mar 24 13:59:26: starting bhyve (run 1) >>>> >>>> based on what I can see, and a little out of desparation, I decided to >>>> run the installer in the foreground, and below is what I got >>>> >>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>> Starting jagadish >>>> * found guest in /mnt/VMs/jagadish >>>> * booting... >>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>> >>>> it stays there for as long as I wait until I poweroff the vm. >>>> >>>> tried with Debian installer, >>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>> Starting jagadish >>>> * found guest in /mnt/VMs/jagadish >>>> * booting... >>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>> >>>> >>>> ideally I need to install Debian for my user base on these GPU based >>>> systems. But even that is proving to be impossible since the installer= does >>>> not move any further from where it is. >>>> >>>> Any thoughts/ideas/suggestions what I else I can try to make this work= ? >>>> Anything that you have tried that worked? have I missed something? >>>> >>>> Would appreciate any and all thoughts/suggestions >>>> >>>> best regards >>>> SS >>>> >>> >>> >>> -- >>> Mario. >>> >> >> >> -- >> Mario. >> > > > -- > Mario. > --000000000000cf4c600631184611 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mario

I will try the options you hav= e used later and let you know how it goes. Not sure if this is intentional = or typo, but I don't recall any bhyve-win binary in my system.

Best regards
SS

On Mo= n, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com> wrote:
Oh sorry. I forgot t= o add the following bhyve parameters necessary to passthru my GPU :D
<= div>
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
= -s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \

On Mon, Mar 24, 2025 at 3= :57=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
Usually I us= e this kind of script to launch a bhyve vm :

#!/bin/sh<= br>
setxkbmap it
kldload vmm.ko
vms=3D"$(ls /dev/vmm/*)"=
vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')"kldload vmm.ko

if ! pciconf -l pci0:2:0:0 | grep -q "^ppt&quo= t;; then
echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attachi= ng..."
kldload nvidia-modeset
devctl clear driver -f pci0:2:0:0<= br>devctl set driver -f pci0:2:0:0 ppt
else
echo "rtx 2080ti slo= t 2/0/0 is already attached to ppt"
fi

if ! pciconf -l pci0:= 2:0:1 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/1 = is not attached to ppt,attaching..."
devctl clear driver -f pci0:2:= 0:1
devctl set driver -f pci0:2:0:1 ppt
else
echo "rtx 2080ti= slot 2/0/1 is already attached to ppt"
fi

if ! pciconf -l p= ci0:2:0:2 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/= 0/2 is not attached to ppt,attaching..."
devctl clear driver -f pci= 0:2:0:2
devctl set driver -f pci0:2:0:2 ppt
else
echo "rtx 20= 80ti slot 2/0/2 is already attached to ppt"
fi

if ! pciconf = -l pci0:2:0:3 | grep -q "^ppt"; then
echo "rtx 2080ti slo= t 2/0/3 is not attached to ppt,attaching..."
devctl clear driver -f= pci0:2:0:3
devctl set driver -f pci0:2:0:3 ppt
else
echo "rt= x 2080ti slot 2/0/3 is already attached to ppt"
fi

echo &quo= t;rtx 2080ti is fully attached to ppt"

for vm in $vms; do
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${v= m##*/}" =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 echo "bhyve session =3D $session"
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "vnc session =3D $vncs"=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '= %s\n' "${vncs}" | grep "${session#vm}"; then =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'VNC session not found,destro= ying ghost vms\n'
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --v= m=3D$session --destroy
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC = session %s\n' "${session},no ghost vms found,not destroying them&q= uot;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
done<= br>
vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ide= nt: (AM13N4CZ)/ && d{print d}'`
echo "Seagate M3 Portab= le 1.8T ; $vmdisk0"

vmdisk1=3D`geom disk list | awk '/^Geom= name: /{d=3D$NF} /^ *ident: (2015040204055E)/ && d{print d}'`<= br>echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"

vmdisk2= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1= 175)/ && d{print d}'`
echo "CT1000P1SSD8 ; $vmdisk2&quo= t;

vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *= ident: (20130508005976F)/ && d{print d}'`
echo "TOSHIBA= External USB 3.0 932 GB ; $vmdisk3"

vmdisk4=3D`geom disk list = | awk '/^Geom name: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{= print d}'`
echo "G-DRIVE USB ; $vmdisk4"

vmdisk5=3D= `geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (38234B423735= 4B45)/ && d{print d}'`
echo "Elements 25A3 ; $vmdisk5&q= uot;

vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^= *ident: (WD-WCAV2X797309)/ && d{print d}'`
echo "WDC W= D3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"

vmdisk7=3D`geom disk list= | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d= {print d}'`
echo "Corsair Force 3 SSD ; $vmdisk7"

v= mdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (192= 4E50B2AE5)/ && d{print d}'`
echo "CT500MX500SSD4 ; $vmd= isk8"

vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$= NF} /^ *ident: (0774921DDC4200A6)/ && d{print d}'`
echo &quo= t;SanDisk Cruzer-15GB ; $vmdisk9"

vmdisk10=3D`geom disk list | = awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ &am= p;& d{print d}'`
echo "Kingston DataTraveler 2.0 ; $vmdisk1= 0"

vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF= } /^ *ident: (575845483038524844323238)/ && d{print d}'`
ech= o "WD 2500BMV External ; $vmdisk11"

vmdisk12=3D`geom disk = list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57442E57584532303954443= 3303334)/ && d{print d}'`
echo "WD 3200BMV External ; $= vmdisk12"

vmdisk13=3D`geom disk list | awk '/^Geom name: /{= d=3D$NF} /^ *ident: (2414E989076B)/ && d{print d}'`
echo &qu= ot;CT500BX500SSD1 ; $vmdisk13"

/usr/sbin/./bhyve-wi= n -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H -A \
-s 0,hostbrid= ge \
-s 1,ahci-hd,/dev/$vmdisk13 \
-s 10,virtio-net,tap6 \
-s 11,v= irtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
-s 31,lpc \
-= l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:6 < = /dev/null & sleep 5 && vncviewer 0:6 &

=
As you can see,each vm has a unique ID (connected with the vncviewer i= nstance) and the vncviewer window is launched as soon as the bhyve vm is ca= lled...and my Geforce RTX 2080 ti is passed through inside the vm.

On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gmail.com>= wrote:
First of all,you could use a more flexible technique than usi= ng the pptdevs on /boot/loader.conf. I would use a script like this :
=

1)

if ! pciconf -l pci0:18:0:0= | grep -q "^ppt"; then
echo "18/0/0 is not attached to= ppt,attaching..."
devctl clear driver -f pci0:18:0:0
devctl set= driver -f pci0:18:0:0 ppt
else
echo "18/0/0 already attached to= ppt"
fi

2) I would not like to use vm-bhyve= . It adds only some confusion...

=
On Mon, Mar 24, 2025 at 3:22=E2=80=AF= PM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Good afternoon everyo= ne.

I am trying to have VMs with GPU passthrough. The setup is a Del= l server with NVidia Quadro RTX 6000/8000 installed already. I have checked= the device IDs and put pptdevs in place

# cat /boot/loader.conf
= pptdevs=3D"18/0/0 19/0/0"
pptdevs2=3D"72/0/0 73/0/0"=

this is showing the GPUs as pptdev on the pciconf

ppt0@pci0:= 18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 = =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2= =A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
= =C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subcla= ss =C2=A0 =3D 3D
ppt1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200= rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de s= ubdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Cor= poration'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadr= o RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D displa= y
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D

as I am using vm-bhyve, I = have put the configuration as below

# cat /mnt/VMs/jagadish/jagadish= .conf
loader=3D"uefi"
cpu=3D16
memory=3D128G
xhci_mou= se=3D"yes"
debug=3D"true"

graphics=3D"ye= s"
graphics_listen=3D"127.0.0.1"
graphics_port=3D"= ;5920"
graphics_res=3D"1024x768"
graphics_wait=3D"= ;no"
#graphics_vga=3D"io"

network0_type=3D"vi= rtio-net"
network0_switch=3D"swUNI"
network0_mac=3D&qu= ot;58:9c:fc:06:3f:af"

disk0_type=3D"nvme"
disk0_na= me=3D"jagadish-disk0.img"
#disk0_size=3D"128G"
uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de"

passthru0= =3D"19/0/0=3D6:0"

bhyve_options=3D"-A -H -P"
= #END

however, when I start the vm (for OS installation to start with= ), it shows it is running but I am unable to access the VNC for a while, th= en when I finally manage to connect via VNC, there is nothing on the screen= , just blank dark screen

the vm-bhyve.log shows

Mar 24 13:59:= 26: initialising
Mar 24 13:59:26: =C2=A0[loader: uefi]
Mar 24 13:59:2= 6: =C2=A0[cpu: 16]
Mar 24 13:59:26: =C2=A0[memory: 128G]
Mar 24 13:59= :26: =C2=A0[hostbridge: standard]
Mar 24 13:59:26: =C2=A0[com ports: com= 1]
Mar 24 13:59:26: =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]Mar 24 13:59:26: =C2=A0[debug mode: true]
Mar 24 13:59:26: =C2=A0[prim= ary disk: jagadish-disk0.img]
Mar 24 13:59:26: =C2=A0[primary disk dev: = file]
Mar 24 13:59:26: initialising network device tap0
Mar 24 13:59:= 26: adding tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: bring up ta= p0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: booting
Mar 24 13:59:= 26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/u= efi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de= -u -S]
Mar 24 13:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc= -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,ma= c=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768= -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev= /nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci= -cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
Mar 24 13:59:26:= starting bhyve (run 1)

based on what I can see, and a little out of= desparation, I decided to run the installer in the foreground, and below i= s what I got

# vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-dis= c1.iso
Starting jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish=C2=A0 * booting...
fbuf frame buffer base: 0x112245400000 [sz 16777216= ]

it stays there for as long as I wait until I poweroff the vm.
<= br>tried with Debian installer,
# vm install -f jagadish debian-12.5.0-= amd64-netinst.iso
Starting jagadish
=C2=A0 * found guest in /mnt/VMs/= jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0x2747e2400000 [= sz 16777216]


ideally I need to install Debian for my user base o= n these GPU based systems. But even that is proving to be impossible since = the installer does not move any further from where it is.

Any though= ts/ideas/suggestions what I else I can try to make this work? Anything that= you have tried that worked? have I missed something?

Would apprecia= te any and all thoughts/suggestions

best regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000cf4c600631184611--