Date: Mon, 24 Mar 2025 15:58:49 +0100 From: Mario Marietto <marietto2008@gmail.com> To: Shamim Shahriar <shamim.shahriar@gmail.com> Cc: freebsd-virtualization@freebsd.org Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 Message-ID: <CA%2B1FSigdkJRRD42TrD0wR0_gspFkL2neqvNF2wY0fDTE%2BPpP-g@mail.gmail.com> In-Reply-To: <CA%2B1FSiiu0cL2XNQk7oVByqt=BWn8yd-9h42wuDP11eq=cd1ndQ@mail.gmail.com> References: <CAOyJeZTtM80ceqdZffNXef4LVoEA3GBYRrn508pS0y12yT8dOQ@mail.gmail.com> <CA%2B1FSigyrtAiM%2By6p%2Bqc2JDYa=eRRZ814aHwfHV-kjepMNYNNg@mail.gmail.com> <CA%2B1FSiiu0cL2XNQk7oVByqt=BWn8yd-9h42wuDP11eq=cd1ndQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000c66db6063117d908
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Oh sorry. I forgot to add the following bhyve parameters necessary to
passthru my GPU :D
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto <marietto2008@gmail.=
com>
wrote:
> Usually I use this kind of script to launch a bhyve vm :
>
> #!/bin/sh
>
> setxkbmap it
> kldload vmm.ko
> vms=3D"$(ls /dev/vmm/*)"
> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')"
> kldload vmm.ko
>
> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."
> kldload nvidia-modeset
> devctl clear driver -f pci0:2:0:0
> devctl set driver -f pci0:2:0:0 ppt
> else
> echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
> fi
>
> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."
> devctl clear driver -f pci0:2:0:1
> devctl set driver -f pci0:2:0:1 ppt
> else
> echo "rtx 2080ti slot 2/0/1 is already attached to ppt"
> fi
>
> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
> devctl clear driver -f pci0:2:0:2
> devctl set driver -f pci0:2:0:2 ppt
> else
> echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
> fi
>
> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
> devctl clear driver -f pci0:2:0:3
> devctl set driver -f pci0:2:0:3 ppt
> else
> echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
> fi
>
> echo "rtx 2080ti is fully attached to ppt"
>
> for vm in $vms; do
> session=3D"${vm##*/}"
> echo "bhyve session =3D $session"
> echo "vnc session =3D $vncs"
> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then
>
> printf 'VNC session not found,destroying ghost vms\n'
> bhyvectl --vm=3D$session --destroy
> else
> printf 'Found VNC session %s\n'
> "${session},no ghost vms found,not destroying them"
> fi
> done
>
> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (AM13N4CZ)/ && d{print d}'`
> echo "Seagate M3 Portable 1.8T ; $vmdisk0"
>
> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (2015040204055E)/ && d{print d}'`
> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"
>
> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (2027285F1175)/ && d{print d}'`
> echo "CT1000P1SSD8 ; $vmdisk2"
>
> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (20130508005976F)/ && d{print d}'`
> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"
>
> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (BE0191510218)/ && d{print d}'`
> echo "G-DRIVE USB ; $vmdisk4"
>
> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (38234B4237354B45)/ && d{print d}'`
> echo "Elements 25A3 ; $vmdisk5"
>
> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (WD-WCAV2X797309)/ && d{print d}'`
> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"
>
> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (20140108006C)/ && d{print d}'`
> echo "Corsair Force 3 SSD ; $vmdisk7"
>
> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (1924E50B2AE5)/ && d{print d}'`
> echo "CT500MX500SSD4 ; $vmdisk8"
>
> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (0774921DDC4200A6)/ && d{print d}'`
> echo "SanDisk Cruzer-15GB ; $vmdisk9"
>
> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (60A44D4138D8F311190A0149)/ && d{print d}'`
> echo "Kingston DataTraveler 2.0 ; $vmdisk10"
>
> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (575845483038524844323238)/ && d{print d}'`
> echo "WD 2500BMV External ; $vmdisk11"
>
> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (57442E575845323039544433303334)/ && d{print d}'`
> echo "WD 3200BMV External ; $vmdisk12"
>
> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
> (2414E989076B)/ && d{print d}'`
> echo "CT500BX500SSD1 ; $vmdisk13"
>
> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H=
-A \
> -s 0,hostbridge \
> -s 1,ahci-hd,/dev/$vmdisk13 \
> -s 10,virtio-net,tap6 \
> -s 11,virtio-9p,sharename=3D/ \
> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &
>
> As you can see,each vm has a unique ID (connected with the vncviewer
> instance) and the vncviewer window is launched as soon as the bhyve vm is
> called...and my Geforce RTX 2080 ti is passed through inside the vm.
>
>
> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gmai=
l.com>
> wrote:
>
>> First of all,you could use a more flexible technique than using the
>> pptdevs on /boot/loader.conf. I would use a script like this :
>>
>> 1)
>>
>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
>> echo "18/0/0 is not attached to ppt,attaching..."
>> devctl clear driver -f pci0:18:0:0
>> devctl set driver -f pci0:18:0:0 ppt
>> else
>> echo "18/0/0 already attached to ppt"
>> fi
>>
>> 2) I would not like to use vm-bhyve. It adds only some confusion...
>>
>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar <
>> shamim.shahriar@gmail.com> wrote:
>>
>>> Good afternoon everyone.
>>>
>>> I am trying to have VMs with GPU passthrough. The setup is a Dell serve=
r
>>> with NVidia Quadro RTX 6000/8000 installed already. I have checked the
>>> device IDs and put pptdevs in place
>>>
>>> # cat /boot/loader.conf
>>> pptdevs=3D"18/0/0 19/0/0"
>>> pptdevs2=3D"72/0/0 73/0/0"
>>>
>>> this is showing the GPUs as pptdev on the pciconf
>>>
>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=
=3D0x10de
>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
>>> vendor =3D 'NVIDIA Corporation'
>>> device =3D 'TU102GL [Quadro RTX 6000/8000]'
>>> class =3D display
>>> subclass =3D 3D
>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=
=3D0x10de
>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
>>> vendor =3D 'NVIDIA Corporation'
>>> device =3D 'TU102GL [Quadro RTX 6000/8000]'
>>> class =3D display
>>> subclass =3D 3D
>>>
>>> as I am using vm-bhyve, I have put the configuration as below
>>>
>>> # cat /mnt/VMs/jagadish/jagadish.conf
>>> loader=3D"uefi"
>>> cpu=3D16
>>> memory=3D128G
>>> xhci_mouse=3D"yes"
>>> debug=3D"true"
>>>
>>> graphics=3D"yes"
>>> graphics_listen=3D"127.0.0.1"
>>> graphics_port=3D"5920"
>>> graphics_res=3D"1024x768"
>>> graphics_wait=3D"no"
>>> #graphics_vga=3D"io"
>>>
>>> network0_type=3D"virtio-net"
>>> network0_switch=3D"swUNI"
>>> network0_mac=3D"58:9c:fc:06:3f:af"
>>>
>>> disk0_type=3D"nvme"
>>> disk0_name=3D"jagadish-disk0.img"
>>> #disk0_size=3D"128G"
>>>
>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de"
>>>
>>> passthru0=3D"19/0/0=3D6:0"
>>>
>>> bhyve_options=3D"-A -H -P"
>>> #END
>>>
>>> however, when I start the vm (for OS installation to start with), it
>>> shows it is running but I am unable to access the VNC for a while, then
>>> when I finally manage to connect via VNC, there is nothing on the scree=
n,
>>> just blank dark screen
>>>
>>> the vm-bhyve.log shows
>>>
>>> Mar 24 13:59:26: initialising
>>> Mar 24 13:59:26: [loader: uefi]
>>> Mar 24 13:59:26: [cpu: 16]
>>> Mar 24 13:59:26: [memory: 128G]
>>> Mar 24 13:59:26: [hostbridge: standard]
>>> Mar 24 13:59:26: [com ports: com1]
>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
>>> Mar 24 13:59:26: [debug mode: true]
>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img]
>>> Mar 24 13:59:26: [primary disk dev: file]
>>> Mar 24 13:59:26: initialising network device tap0
>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
>>> Mar 24 13:59:26: booting
>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l
>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U
>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s
>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s
>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s
>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet]
>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A]
>>> Mar 24 13:59:26: [bhyve iso device: -s
>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
>>> Mar 24 13:59:26: starting bhyve (run 1)
>>>
>>> based on what I can see, and a little out of desparation, I decided to
>>> run the installer in the foreground, and below is what I got
>>>
>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
>>> Starting jagadish
>>> * found guest in /mnt/VMs/jagadish
>>> * booting...
>>> fbuf frame buffer base: 0x112245400000 [sz 16777216]
>>>
>>> it stays there for as long as I wait until I poweroff the vm.
>>>
>>> tried with Debian installer,
>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso
>>> Starting jagadish
>>> * found guest in /mnt/VMs/jagadish
>>> * booting...
>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216]
>>>
>>>
>>> ideally I need to install Debian for my user base on these GPU based
>>> systems. But even that is proving to be impossible since the installer =
does
>>> not move any further from where it is.
>>>
>>> Any thoughts/ideas/suggestions what I else I can try to make this work?
>>> Anything that you have tried that worked? have I missed something?
>>>
>>> Would appreciate any and all thoughts/suggestions
>>>
>>> best regards
>>> SS
>>>
>>
>>
>> --
>> Mario.
>>
>
>
> --
> Mario.
>
--=20
Mario.
--000000000000c66db6063117d908
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Oh sorry. I forgot to add the following bhyve paramet=
ers necessary to passthru my GPU :D</div><div><br></div><div>-s 8:0,passthr=
u,2/0/0 \</div>-s 8:1,passthru,2/0/1 \<br>-s 8:2,passthru,2/0/2 \<br>-s 8:3=
,passthru,2/0/3 \</div><br><div class=3D"gmail_quote gmail_quote_container"=
><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Mar 24, 2025 at 3:57=E2=80=
=AFPM Mario Marietto <<a href=3D"mailto:marietto2008@gmail.com">marietto=
2008@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd=
ing-left:1ex"><div dir=3D"ltr"><div>Usually I use this kind of script to la=
unch a bhyve vm :</div><div><br></div>#!/bin/sh<br><br>setxkbmap it<br>kldl=
oad vmm.ko<br>vms=3D"$(ls /dev/vmm/*)"<br>vncs=3D"$(ps ax | =
awk '/vncviewer [0]/{print $6}')"<br>kldload vmm.ko<br><br>if =
! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then<br>echo "rtx =
2080ti slot 2/0/0 is not attached to ppt,attaching..."<br>kldload nvid=
ia-modeset<br>devctl clear driver -f pci0:2:0:0<br>devctl set driver -f pci=
0:2:0:0 ppt<br>else<br>echo "rtx 2080ti slot 2/0/0 is already attached=
to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:1 | grep -q "^ppt&=
quot;; then<br>echo "rtx 2080ti slot 2/0/1 is not attached to ppt,atta=
ching..."<br>devctl clear driver -f pci0:2:0:1<br>devctl set driver -f=
pci0:2:0:1 ppt<br>else<br>echo "rtx 2080ti slot 2/0/1 is already atta=
ched to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:2 | grep -q "^=
ppt"; then<br>echo "rtx 2080ti slot 2/0/2 is not attached to ppt,=
attaching..."<br>devctl clear driver -f pci0:2:0:2<br>devctl set drive=
r -f pci0:2:0:2 ppt<br>else<br>echo "rtx 2080ti slot 2/0/2 is already =
attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:3 | grep -q &qu=
ot;^ppt"; then<br>echo "rtx 2080ti slot 2/0/3 is not attached to =
ppt,attaching..."<br>devctl clear driver -f pci0:2:0:3<br>devctl set d=
river -f pci0:2:0:3 ppt<br>else<br>echo "rtx 2080ti slot 2/0/3 is alre=
ady attached to ppt"<br>fi<br><br>echo "rtx 2080ti is fully attac=
hed to ppt"<br><br>for vm in $vms; do<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${vm##*/}" =C2=A0 <br>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve sessi=
on =3D $session"<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 echo "vnc session =3D $vncs"<br>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '%s\n' "${vncs}&qu=
ot; | grep "${session#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 printf 'VNC session not found,destroying ghost vms\n'<br=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy<b=
r>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else<br>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC session %s\n' &qu=
ot;${session},no ghost vms found,not destroying them"<br>=C2=A0 =C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi<br>done<br><br>vmdisk0=3D`geo=
m disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ &=
& d{print d}'`<br>echo "Seagate M3 Portable 1.8T ; $vmdisk0&qu=
ot;<br><br>vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ =
*ident: (2015040204055E)/ && d{print d}'`<br>echo "TOSHIBA=
External USB 3.0 1.8T ; $vmdisk1"<br><br>vmdisk2=3D`geom disk list | =
awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{pr=
int d}'`<br>echo "CT1000P1SSD8 ; $vmdisk2"<br><br>vmdisk3=3D`=
geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2013050800597=
6F)/ && d{print d}'`<br>echo "TOSHIBA External USB 3.0 932=
GB ; $vmdisk3"<br><br>vmdisk4=3D`geom disk list | awk '/^Geom nam=
e: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{print d}'`<br>ech=
o "G-DRIVE USB ; $vmdisk4"<br><br>vmdisk5=3D`geom disk list | awk=
'/^Geom name: /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{p=
rint d}'`<br>echo "Elements 25A3 ; $vmdisk5"<br><br>vmdisk6=
=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X=
797309)/ && d{print d}'`<br>echo "WDC WD3200AAJS-00L7A0 ; =
298 GB ; $vmdisk6"<br><br>vmdisk7=3D`geom disk list | awk '/^Geom =
name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d{print d}'`<br>=
echo "Corsair Force 3 SSD ; $vmdisk7"<br><br>vmdisk8=3D`geom disk=
list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ &&a=
mp; d{print d}'`<br>echo "CT500MX500SSD4 ; $vmdisk8"<br><br>v=
mdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (077=
4921DDC4200A6)/ && d{print d}'`<br>echo "SanDisk Cruzer-15=
GB ; $vmdisk9"<br><br>vmdisk10=3D`geom disk list | awk '/^Geom nam=
e: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}&=
#39;`<br>echo "Kingston DataTraveler 2.0 ; $vmdisk10"<br><br>vmdi=
sk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57584=
5483038524844323238)/ && d{print d}'`<br>echo "WD 2500BMV =
External ; $vmdisk11"<br><br>vmdisk12=3D`geom disk list | awk '/^G=
eom name: /{d=3D$NF} /^ *ident: (57442E575845323039544433303334)/ &&=
; d{print d}'`<br>echo "WD 3200BMV External ; $vmdisk12"<br><=
br>vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:=
(2414E989076B)/ && d{print d}'`<br>echo "CT500BX500SSD1 ;=
$vmdisk13"<div><br></div><div>/usr/sbin/./bhyve-win -S -c sockets=3D4=
,cores=3D1,threads=3D1 -m 4G -w -H -A \<br>-s 0,hostbridge \<br>-s 1,ahci-h=
d,/dev/$vmdisk13 \<br>-s 10,virtio-net,tap6 \<br>-s 11,virtio-9p,sharename=
=3D/ \<br>-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5906" target=3D"_blank=
">0.0.0.0:5906</a>,w=3D1600,h=3D950 \<br>-s 31,lpc \<br>-l bootrom,/usr/loc=
al/share/uefi-firmware/BHYVE_BHF_CODE.fd \<br>vm0:6 < /dev/null & sl=
eep 5 && vncviewer 0:6 &</div><div><br></div><div>As you can se=
e,each vm has a unique ID (connected with the vncviewer instance) and the v=
ncviewer window is launched as soon as the bhyve vm is called...and my Gefo=
rce RTX 2080 ti is passed through inside the vm.</div><br></div><br><div cl=
ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Mar 24, 2=
025 at 3:47=E2=80=AFPM Mario Marietto <<a href=3D"mailto:marietto2008@gm=
ail.com" target=3D"_blank">marietto2008@gmail.com</a>> wrote:<br></div><=
blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l=
eft:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Firs=
t of all,you could use a more flexible technique than using the pptdevs on =
/boot/loader.conf. I would use a script like this :</div><div><br></div><di=
v>1)</div><div><br></div><div>if ! pciconf -l pci0:18:0:0 | grep -q "^=
ppt"; then</div>echo "18/0/0 is not attached to ppt,attaching...&=
quot;<br>devctl clear driver -f pci0:18:0:0<br>devctl set driver -f pci0:18=
:0:0 ppt<br>else<br>echo "18/0/0 already attached to ppt"<br>fi<b=
r><div><br></div><div>2) I would not like to use vm-bhyve. It adds only som=
e confusion...</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" c=
lass=3D"gmail_attr">On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar=
<<a href=3D"mailto:shamim.shahriar@gmail.com" target=3D"_blank">shamim.=
shahriar@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote=
" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);=
padding-left:1ex"><div dir=3D"ltr">Good afternoon everyone.<br><br>I am try=
ing to have VMs with GPU passthrough. The setup is a Dell server with NVidi=
a Quadro RTX 6000/8000 installed already. I have checked the device IDs and=
put pptdevs in place<br><br># cat /boot/loader.conf<br>pptdevs=3D"18/=
0/0 19/0/0"<br>pptdevs2=3D"72/0/0 73/0/0"<br><br>this is sho=
wing the GPUs as pptdev on the pciconf<br><br>ppt0@pci0:18:0:0: =C2=A0 =C2=
=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D=
0x1e78 subvendor=3D0x10de subdevice=3D0x13d8<br>=C2=A0 =C2=A0 vendor =C2=A0=
=C2=A0 =3D 'NVIDIA Corporation'<br>=C2=A0 =C2=A0 device =C2=A0 =C2=
=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'<br>=C2=A0 =C2=A0 class =C2=
=A0 =C2=A0 =C2=A0=3D display<br>=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D<br>ppt=
1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 =
vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8<br>=
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'<br>=C2=
=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'=
;<br>=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display<br>=C2=A0 =C2=A0 s=
ubclass =C2=A0 =3D 3D<br><br>as I am using vm-bhyve, I have put the configu=
ration as below<br><br># cat /mnt/VMs/jagadish/jagadish.conf<br>loader=3D&q=
uot;uefi"<br>cpu=3D16<br>memory=3D128G<br>xhci_mouse=3D"yes"=
<br>debug=3D"true"<br><br>graphics=3D"yes"<br>graphics_=
listen=3D"127.0.0.1"<br>graphics_port=3D"5920"<br>graph=
ics_res=3D"1024x768"<br>graphics_wait=3D"no"<br>#graphi=
cs_vga=3D"io"<br><br>network0_type=3D"virtio-net"<br>ne=
twork0_switch=3D"swUNI"<br>network0_mac=3D"58:9c:fc:06:3f:af=
"<br><br>disk0_type=3D"nvme"<br>disk0_name=3D"jagadish-=
disk0.img"<br>#disk0_size=3D"128G"<br><br>uuid=3D"966e9=
09b-1293-11ef-a9a4-e4434bfe34de"<br><br>passthru0=3D"19/0/0=3D6:0=
"<br><br>bhyve_options=3D"-A -H -P"<br>#END<br><br>however, =
when I start the vm (for OS installation to start with), it shows it is run=
ning but I am unable to access the VNC for a while, then when I finally man=
age to connect via VNC, there is nothing on the screen, just blank dark scr=
een<br><br>the vm-bhyve.log shows<br><br>Mar 24 13:59:26: initialising<br>M=
ar 24 13:59:26: =C2=A0[loader: uefi]<br>Mar 24 13:59:26: =C2=A0[cpu: 16]<br=
>Mar 24 13:59:26: =C2=A0[memory: 128G]<br>Mar 24 13:59:26: =C2=A0[hostbridg=
e: standard]<br>Mar 24 13:59:26: =C2=A0[com ports: com1]<br>Mar 24 13:59:26=
: =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]<br>Mar 24 13:59:26: =
=C2=A0[debug mode: true]<br>Mar 24 13:59:26: =C2=A0[primary disk: jagadish-=
disk0.img]<br>Mar 24 13:59:26: =C2=A0[primary disk dev: file]<br>Mar 24 13:=
59:26: initialising network device tap0<br>Mar 24 13:59:26: adding tap0 -&g=
t; vm-swUNI (swUNI addm)<br>Mar 24 13:59:26: bring up tap0 -> vm-swUNI (=
swUNI addm)<br>Mar 24 13:59:26: booting<br>Mar 24 13:59:26: =C2=A0[bhyve op=
tions: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_=
UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]<br>Mar 24 1=
3:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/V=
Ms/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:=
af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D<a href=3D"http://127.0.0.1:592=
0" target=3D"_blank">127.0.0.1:5920</a>,w=3D1024,h=3D768 -s 8:0,xhci,tablet=
]<br>Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev/nmdm-jagadish.1A]<=
br>Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/l=
ubuntu-24.04.1-desktop-amd64.iso,ro]<br>Mar 24 13:59:26: starting bhyve (ru=
n 1)<br><br>based on what I can see, and a little out of desparation, I dec=
ided to run the installer in the foreground, and below is what I got<br><br=
># vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso<br>Starting =
jagadish<br>=C2=A0 * found guest in /mnt/VMs/jagadish<br>=C2=A0 * booting..=
.<br>fbuf frame buffer base: 0x112245400000 [sz 16777216]<br><br>it stays t=
here for as long as I wait until I poweroff the vm.<br><br>tried with Debia=
n installer, <br># vm install -f jagadish debian-12.5.0-amd64-netinst.iso<b=
r>Starting jagadish<br>=C2=A0 * found guest in /mnt/VMs/jagadish<br>=C2=A0 =
* booting...<br>fbuf frame buffer base: 0x2747e2400000 [sz 16777216]<br><br=
><br>ideally I need to install Debian for my user base on these GPU based s=
ystems. But even that is proving to be impossible since the installer does =
not move any further from where it is.<br><br>Any thoughts/ideas/suggestion=
s what I else I can try to make this work? Anything that you have tried tha=
t worked? have I missed something?<br><br>Would appreciate any and all thou=
ghts/suggestions<br><br>best regards<br><div>SS</div></div>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma=
rio.<br></div>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma=
rio.<br></div>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma=
rio.<br></div>
--000000000000c66db6063117d908--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSigdkJRRD42TrD0wR0_gspFkL2neqvNF2wY0fDTE%2BPpP-g>