Date: Mon, 24 Mar 2025 15:29:45 +0000 From: Shamim Shahriar <shamim.shahriar@gmail.com> To: Mario Marietto <marietto2008@gmail.com> Cc: freebsd-virtualization@freebsd.org Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 Message-ID: <CAOyJeZRRB7nuSgzVL9J-OwamuLmBi3_a20BRaJhb5PWkpdUwKQ@mail.gmail.com> In-Reply-To: <CA%2B1FSigdkJRRD42TrD0wR0_gspFkL2neqvNF2wY0fDTE%2BPpP-g@mail.gmail.com> References: <CAOyJeZTtM80ceqdZffNXef4LVoEA3GBYRrn508pS0y12yT8dOQ@mail.gmail.com> <CA%2B1FSigyrtAiM%2By6p%2Bqc2JDYa=eRRZ814aHwfHV-kjepMNYNNg@mail.gmail.com> <CA%2B1FSiiu0cL2XNQk7oVByqt=BWn8yd-9h42wuDP11eq=cd1ndQ@mail.gmail.com> <CA%2B1FSigdkJRRD42TrD0wR0_gspFkL2neqvNF2wY0fDTE%2BPpP-g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000cf4c600631184611
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi Mario
I will try the options you have used later and let you know how it goes.
Not sure if this is intentional or typo, but I don't recall any bhyve-win
binary in my system.
Best regards
SS
On Mon, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com> wrote=
:
> Oh sorry. I forgot to add the following bhyve parameters necessary to
> passthru my GPU :D
>
> -s 8:0,passthru,2/0/0 \
> -s 8:1,passthru,2/0/1 \
> -s 8:2,passthru,2/0/2 \
> -s 8:3,passthru,2/0/3 \
>
> On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto <marietto2008@gmai=
l.com>
> wrote:
>
>> Usually I use this kind of script to launch a bhyve vm :
>>
>> #!/bin/sh
>>
>> setxkbmap it
>> kldload vmm.ko
>> vms=3D"$(ls /dev/vmm/*)"
>> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')"
>> kldload vmm.ko
>>
>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."
>> kldload nvidia-modeset
>> devctl clear driver -f pci0:2:0:0
>> devctl set driver -f pci0:2:0:0 ppt
>> else
>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
>> fi
>>
>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."
>> devctl clear driver -f pci0:2:0:1
>> devctl set driver -f pci0:2:0:1 ppt
>> else
>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt"
>> fi
>>
>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
>> devctl clear driver -f pci0:2:0:2
>> devctl set driver -f pci0:2:0:2 ppt
>> else
>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
>> fi
>>
>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
>> devctl clear driver -f pci0:2:0:3
>> devctl set driver -f pci0:2:0:3 ppt
>> else
>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
>> fi
>>
>> echo "rtx 2080ti is fully attached to ppt"
>>
>> for vm in $vms; do
>> session=3D"${vm##*/}"
>> echo "bhyve session =3D $session"
>> echo "vnc session =3D $vncs"
>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; the=
n
>>
>> printf 'VNC session not found,destroying ghost vms\n'
>> bhyvectl --vm=3D$session --destroy
>> else
>> printf 'Found VNC session %s\n'
>> "${session},no ghost vms found,not destroying them"
>> fi
>> done
>>
>> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (AM13N4CZ)/ && d{print d}'`
>> echo "Seagate M3 Portable 1.8T ; $vmdisk0"
>>
>> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (2015040204055E)/ && d{print d}'`
>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"
>>
>> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (2027285F1175)/ && d{print d}'`
>> echo "CT1000P1SSD8 ; $vmdisk2"
>>
>> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (20130508005976F)/ && d{print d}'`
>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"
>>
>> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (BE0191510218)/ && d{print d}'`
>> echo "G-DRIVE USB ; $vmdisk4"
>>
>> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (38234B4237354B45)/ && d{print d}'`
>> echo "Elements 25A3 ; $vmdisk5"
>>
>> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (WD-WCAV2X797309)/ && d{print d}'`
>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"
>>
>> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (20140108006C)/ && d{print d}'`
>> echo "Corsair Force 3 SSD ; $vmdisk7"
>>
>> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (1924E50B2AE5)/ && d{print d}'`
>> echo "CT500MX500SSD4 ; $vmdisk8"
>>
>> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (0774921DDC4200A6)/ && d{print d}'`
>> echo "SanDisk Cruzer-15GB ; $vmdisk9"
>>
>> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (60A44D4138D8F311190A0149)/ && d{print d}'`
>> echo "Kingston DataTraveler 2.0 ; $vmdisk10"
>>
>> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (575845483038524844323238)/ && d{print d}'`
>> echo "WD 2500BMV External ; $vmdisk11"
>>
>> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (57442E575845323039544433303334)/ && d{print d}'`
>> echo "WD 3200BMV External ; $vmdisk12"
>>
>> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
>> (2414E989076B)/ && d{print d}'`
>> echo "CT500BX500SSD1 ; $vmdisk13"
>>
>> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -=
H -A \
>> -s 0,hostbridge \
>> -s 1,ahci-hd,/dev/$vmdisk13 \
>> -s 10,virtio-net,tap6 \
>> -s 11,virtio-9p,sharename=3D/ \
>> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
>> -s 31,lpc \
>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &
>>
>> As you can see,each vm has a unique ID (connected with the vncviewer
>> instance) and the vncviewer window is launched as soon as the bhyve vm i=
s
>> called...and my Geforce RTX 2080 ti is passed through inside the vm.
>>
>>
>> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gma=
il.com>
>> wrote:
>>
>>> First of all,you could use a more flexible technique than using the
>>> pptdevs on /boot/loader.conf. I would use a script like this :
>>>
>>> 1)
>>>
>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
>>> echo "18/0/0 is not attached to ppt,attaching..."
>>> devctl clear driver -f pci0:18:0:0
>>> devctl set driver -f pci0:18:0:0 ppt
>>> else
>>> echo "18/0/0 already attached to ppt"
>>> fi
>>>
>>> 2) I would not like to use vm-bhyve. It adds only some confusion...
>>>
>>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar <
>>> shamim.shahriar@gmail.com> wrote:
>>>
>>>> Good afternoon everyone.
>>>>
>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell
>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have chec=
ked
>>>> the device IDs and put pptdevs in place
>>>>
>>>> # cat /boot/loader.conf
>>>> pptdevs=3D"18/0/0 19/0/0"
>>>> pptdevs2=3D"72/0/0 73/0/0"
>>>>
>>>> this is showing the GPUs as pptdev on the pciconf
>>>>
>>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=
=3D0x10de
>>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
>>>> vendor =3D 'NVIDIA Corporation'
>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]'
>>>> class =3D display
>>>> subclass =3D 3D
>>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=
=3D0x10de
>>>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
>>>> vendor =3D 'NVIDIA Corporation'
>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]'
>>>> class =3D display
>>>> subclass =3D 3D
>>>>
>>>> as I am using vm-bhyve, I have put the configuration as below
>>>>
>>>> # cat /mnt/VMs/jagadish/jagadish.conf
>>>> loader=3D"uefi"
>>>> cpu=3D16
>>>> memory=3D128G
>>>> xhci_mouse=3D"yes"
>>>> debug=3D"true"
>>>>
>>>> graphics=3D"yes"
>>>> graphics_listen=3D"127.0.0.1"
>>>> graphics_port=3D"5920"
>>>> graphics_res=3D"1024x768"
>>>> graphics_wait=3D"no"
>>>> #graphics_vga=3D"io"
>>>>
>>>> network0_type=3D"virtio-net"
>>>> network0_switch=3D"swUNI"
>>>> network0_mac=3D"58:9c:fc:06:3f:af"
>>>>
>>>> disk0_type=3D"nvme"
>>>> disk0_name=3D"jagadish-disk0.img"
>>>> #disk0_size=3D"128G"
>>>>
>>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de"
>>>>
>>>> passthru0=3D"19/0/0=3D6:0"
>>>>
>>>> bhyve_options=3D"-A -H -P"
>>>> #END
>>>>
>>>> however, when I start the vm (for OS installation to start with), it
>>>> shows it is running but I am unable to access the VNC for a while, the=
n
>>>> when I finally manage to connect via VNC, there is nothing on the scre=
en,
>>>> just blank dark screen
>>>>
>>>> the vm-bhyve.log shows
>>>>
>>>> Mar 24 13:59:26: initialising
>>>> Mar 24 13:59:26: [loader: uefi]
>>>> Mar 24 13:59:26: [cpu: 16]
>>>> Mar 24 13:59:26: [memory: 128G]
>>>> Mar 24 13:59:26: [hostbridge: standard]
>>>> Mar 24 13:59:26: [com ports: com1]
>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
>>>> Mar 24 13:59:26: [debug mode: true]
>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img]
>>>> Mar 24 13:59:26: [primary disk dev: file]
>>>> Mar 24 13:59:26: initialising network device tap0
>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
>>>> Mar 24 13:59:26: booting
>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l
>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U
>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s
>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s
>>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s
>>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet]
>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A]
>>>> Mar 24 13:59:26: [bhyve iso device: -s
>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
>>>> Mar 24 13:59:26: starting bhyve (run 1)
>>>>
>>>> based on what I can see, and a little out of desparation, I decided to
>>>> run the installer in the foreground, and below is what I got
>>>>
>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
>>>> Starting jagadish
>>>> * found guest in /mnt/VMs/jagadish
>>>> * booting...
>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216]
>>>>
>>>> it stays there for as long as I wait until I poweroff the vm.
>>>>
>>>> tried with Debian installer,
>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso
>>>> Starting jagadish
>>>> * found guest in /mnt/VMs/jagadish
>>>> * booting...
>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216]
>>>>
>>>>
>>>> ideally I need to install Debian for my user base on these GPU based
>>>> systems. But even that is proving to be impossible since the installer=
does
>>>> not move any further from where it is.
>>>>
>>>> Any thoughts/ideas/suggestions what I else I can try to make this work=
?
>>>> Anything that you have tried that worked? have I missed something?
>>>>
>>>> Would appreciate any and all thoughts/suggestions
>>>>
>>>> best regards
>>>> SS
>>>>
>>>
>>>
>>> --
>>> Mario.
>>>
>>
>>
>> --
>> Mario.
>>
>
>
> --
> Mario.
>
--000000000000cf4c600631184611
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi Mario<div><br></div><div>I will try the options you hav=
e used later and let you know how it goes. Not sure if this is intentional =
or typo, but I don't recall any bhyve-win binary in my system.</div><di=
v><br></div><div>Best regards</div><div>SS</div></div><br><div class=3D"gma=
il_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Mo=
n, 24 Mar 2025 at 14:59, Mario Marietto <<a href=3D"mailto:marietto2008@=
gmail.com">marietto2008@gmail.com</a>> wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Oh sorry. I forgot t=
o add the following bhyve parameters necessary to passthru my GPU :D</div><=
div><br></div><div>-s 8:0,passthru,2/0/0 \</div>-s 8:1,passthru,2/0/1 \<br>=
-s 8:2,passthru,2/0/2 \<br>-s 8:3,passthru,2/0/3 \</div><br><div class=3D"g=
mail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Mar 24, 2025 at 3=
:57=E2=80=AFPM Mario Marietto <<a href=3D"mailto:marietto2008@gmail.com"=
target=3D"_blank">marietto2008@gmail.com</a>> wrote:<br></div><blockquo=
te class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px =
solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Usually I us=
e this kind of script to launch a bhyve vm :</div><div><br></div>#!/bin/sh<=
br><br>setxkbmap it<br>kldload vmm.ko<br>vms=3D"$(ls /dev/vmm/*)"=
<br>vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')"<b=
r>kldload vmm.ko<br><br>if ! pciconf -l pci0:2:0:0 | grep -q "^ppt&quo=
t;; then<br>echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attachi=
ng..."<br>kldload nvidia-modeset<br>devctl clear driver -f pci0:2:0:0<=
br>devctl set driver -f pci0:2:0:0 ppt<br>else<br>echo "rtx 2080ti slo=
t 2/0/0 is already attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:=
2:0:1 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/1 =
is not attached to ppt,attaching..."<br>devctl clear driver -f pci0:2:=
0:1<br>devctl set driver -f pci0:2:0:1 ppt<br>else<br>echo "rtx 2080ti=
slot 2/0/1 is already attached to ppt"<br>fi<br><br>if ! pciconf -l p=
ci0:2:0:2 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/=
0/2 is not attached to ppt,attaching..."<br>devctl clear driver -f pci=
0:2:0:2<br>devctl set driver -f pci0:2:0:2 ppt<br>else<br>echo "rtx 20=
80ti slot 2/0/2 is already attached to ppt"<br>fi<br><br>if ! pciconf =
-l pci0:2:0:3 | grep -q "^ppt"; then<br>echo "rtx 2080ti slo=
t 2/0/3 is not attached to ppt,attaching..."<br>devctl clear driver -f=
pci0:2:0:3<br>devctl set driver -f pci0:2:0:3 ppt<br>else<br>echo "rt=
x 2080ti slot 2/0/3 is already attached to ppt"<br>fi<br><br>echo &quo=
t;rtx 2080ti is fully attached to ppt"<br><br>for vm in $vms; do<br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${v=
m##*/}" =C2=A0 <br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 echo "bhyve session =3D $session"<br>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "vnc session =3D $vncs"<b=
r>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '=
%s\n' "${vncs}" | grep "${session#vm}"; then =C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<br>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'VNC session not found,destro=
ying ghost vms\n'<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --v=
m=3D$session --destroy<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 else<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC =
session %s\n' "${session},no ghost vms found,not destroying them&q=
uot;<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi<br>done<=
br><br>vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ide=
nt: (AM13N4CZ)/ && d{print d}'`<br>echo "Seagate M3 Portab=
le 1.8T ; $vmdisk0"<br><br>vmdisk1=3D`geom disk list | awk '/^Geom=
name: /{d=3D$NF} /^ *ident: (2015040204055E)/ && d{print d}'`<=
br>echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"<br><br>vmdisk2=
=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1=
175)/ && d{print d}'`<br>echo "CT1000P1SSD8 ; $vmdisk2&quo=
t;<br><br>vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *=
ident: (20130508005976F)/ && d{print d}'`<br>echo "TOSHIBA=
External USB 3.0 932 GB ; $vmdisk3"<br><br>vmdisk4=3D`geom disk list =
| awk '/^Geom name: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{=
print d}'`<br>echo "G-DRIVE USB ; $vmdisk4"<br><br>vmdisk5=3D=
`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (38234B423735=
4B45)/ && d{print d}'`<br>echo "Elements 25A3 ; $vmdisk5&q=
uot;<br><br>vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^=
*ident: (WD-WCAV2X797309)/ && d{print d}'`<br>echo "WDC W=
D3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"<br><br>vmdisk7=3D`geom disk list=
| awk '/^Geom name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d=
{print d}'`<br>echo "Corsair Force 3 SSD ; $vmdisk7"<br><br>v=
mdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (192=
4E50B2AE5)/ && d{print d}'`<br>echo "CT500MX500SSD4 ; $vmd=
isk8"<br><br>vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$=
NF} /^ *ident: (0774921DDC4200A6)/ && d{print d}'`<br>echo &quo=
t;SanDisk Cruzer-15GB ; $vmdisk9"<br><br>vmdisk10=3D`geom disk list | =
awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ &am=
p;& d{print d}'`<br>echo "Kingston DataTraveler 2.0 ; $vmdisk1=
0"<br><br>vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF=
} /^ *ident: (575845483038524844323238)/ && d{print d}'`<br>ech=
o "WD 2500BMV External ; $vmdisk11"<br><br>vmdisk12=3D`geom disk =
list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57442E57584532303954443=
3303334)/ && d{print d}'`<br>echo "WD 3200BMV External ; $=
vmdisk12"<br><br>vmdisk13=3D`geom disk list | awk '/^Geom name: /{=
d=3D$NF} /^ *ident: (2414E989076B)/ && d{print d}'`<br>echo &qu=
ot;CT500BX500SSD1 ; $vmdisk13"<div><br></div><div>/usr/sbin/./bhyve-wi=
n -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H -A \<br>-s 0,hostbrid=
ge \<br>-s 1,ahci-hd,/dev/$vmdisk13 \<br>-s 10,virtio-net,tap6 \<br>-s 11,v=
irtio-9p,sharename=3D/ \<br>-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5906=
" target=3D"_blank">0.0.0.0:5906</a>,w=3D1600,h=3D950 \<br>-s 31,lpc \<br>-=
l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \<br>vm0:6 < =
/dev/null & sleep 5 && vncviewer 0:6 &</div><div><br></div>=
<div>As you can see,each vm has a unique ID (connected with the vncviewer i=
nstance) and the vncviewer window is launched as soon as the bhyve vm is ca=
lled...and my Geforce RTX 2080 ti is passed through inside the vm.</div><br=
></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr"=
>On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto <<a href=3D"mail=
to:marietto2008@gmail.com" target=3D"_blank">marietto2008@gmail.com</a>>=
wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px =
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=
=3D"ltr"><div>First of all,you could use a more flexible technique than usi=
ng the pptdevs on /boot/loader.conf. I would use a script like this :</div>=
<div><br></div><div>1)</div><div><br></div><div>if ! pciconf -l pci0:18:0:0=
| grep -q "^ppt"; then</div>echo "18/0/0 is not attached to=
ppt,attaching..."<br>devctl clear driver -f pci0:18:0:0<br>devctl set=
driver -f pci0:18:0:0 ppt<br>else<br>echo "18/0/0 already attached to=
ppt"<br>fi<br><div><br></div><div>2) I would not like to use vm-bhyve=
. It adds only some confusion...</div></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Mon, Mar 24, 2025 at 3:22=E2=80=AF=
PM Shamim Shahriar <<a href=3D"mailto:shamim.shahriar@gmail.com" target=
=3D"_blank">shamim.shahriar@gmail.com</a>> wrote:<br></div><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Good afternoon everyo=
ne.<br><br>I am trying to have VMs with GPU passthrough. The setup is a Del=
l server with NVidia Quadro RTX 6000/8000 installed already. I have checked=
the device IDs and put pptdevs in place<br><br># cat /boot/loader.conf<br>=
pptdevs=3D"18/0/0 19/0/0"<br>pptdevs2=3D"72/0/0 73/0/0"=
<br><br>this is showing the GPUs as pptdev on the pciconf<br><br>ppt0@pci0:=
18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=
=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8<br>=C2=A0 =
=C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'<br>=C2=A0 =C2=
=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'<br>=
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display<br>=C2=A0 =C2=A0 subcla=
ss =C2=A0 =3D 3D<br>ppt1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200=
rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de s=
ubdevice=3D0x13d8<br>=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Cor=
poration'<br>=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadr=
o RTX 6000/8000]'<br>=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D displa=
y<br>=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D<br><br>as I am using vm-bhyve, I =
have put the configuration as below<br><br># cat /mnt/VMs/jagadish/jagadish=
.conf<br>loader=3D"uefi"<br>cpu=3D16<br>memory=3D128G<br>xhci_mou=
se=3D"yes"<br>debug=3D"true"<br><br>graphics=3D"ye=
s"<br>graphics_listen=3D"127.0.0.1"<br>graphics_port=3D"=
;5920"<br>graphics_res=3D"1024x768"<br>graphics_wait=3D"=
;no"<br>#graphics_vga=3D"io"<br><br>network0_type=3D"vi=
rtio-net"<br>network0_switch=3D"swUNI"<br>network0_mac=3D&qu=
ot;58:9c:fc:06:3f:af"<br><br>disk0_type=3D"nvme"<br>disk0_na=
me=3D"jagadish-disk0.img"<br>#disk0_size=3D"128G"<br><b=
r>uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de"<br><br>passthru0=
=3D"19/0/0=3D6:0"<br><br>bhyve_options=3D"-A -H -P"<br>=
#END<br><br>however, when I start the vm (for OS installation to start with=
), it shows it is running but I am unable to access the VNC for a while, th=
en when I finally manage to connect via VNC, there is nothing on the screen=
, just blank dark screen<br><br>the vm-bhyve.log shows<br><br>Mar 24 13:59:=
26: initialising<br>Mar 24 13:59:26: =C2=A0[loader: uefi]<br>Mar 24 13:59:2=
6: =C2=A0[cpu: 16]<br>Mar 24 13:59:26: =C2=A0[memory: 128G]<br>Mar 24 13:59=
:26: =C2=A0[hostbridge: standard]<br>Mar 24 13:59:26: =C2=A0[com ports: com=
1]<br>Mar 24 13:59:26: =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]<b=
r>Mar 24 13:59:26: =C2=A0[debug mode: true]<br>Mar 24 13:59:26: =C2=A0[prim=
ary disk: jagadish-disk0.img]<br>Mar 24 13:59:26: =C2=A0[primary disk dev: =
file]<br>Mar 24 13:59:26: initialising network device tap0<br>Mar 24 13:59:=
26: adding tap0 -> vm-swUNI (swUNI addm)<br>Mar 24 13:59:26: bring up ta=
p0 -> vm-swUNI (swUNI addm)<br>Mar 24 13:59:26: booting<br>Mar 24 13:59:=
26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/u=
efi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de=
-u -S]<br>Mar 24 13:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc=
-s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,ma=
c=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D<a href=3D"h=
ttp://127.0.0.1:5920" target=3D"_blank">127.0.0.1:5920</a>,w=3D1024,h=3D768=
-s 8:0,xhci,tablet]<br>Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev=
/nmdm-jagadish.1A]<br>Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci=
-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]<br>Mar 24 13:59:26:=
starting bhyve (run 1)<br><br>based on what I can see, and a little out of=
desparation, I decided to run the installer in the foreground, and below i=
s what I got<br><br># vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-dis=
c1.iso<br>Starting jagadish<br>=C2=A0 * found guest in /mnt/VMs/jagadish<br=
>=C2=A0 * booting...<br>fbuf frame buffer base: 0x112245400000 [sz 16777216=
]<br><br>it stays there for as long as I wait until I poweroff the vm.<br><=
br>tried with Debian installer, <br># vm install -f jagadish debian-12.5.0-=
amd64-netinst.iso<br>Starting jagadish<br>=C2=A0 * found guest in /mnt/VMs/=
jagadish<br>=C2=A0 * booting...<br>fbuf frame buffer base: 0x2747e2400000 [=
sz 16777216]<br><br><br>ideally I need to install Debian for my user base o=
n these GPU based systems. But even that is proving to be impossible since =
the installer does not move any further from where it is.<br><br>Any though=
ts/ideas/suggestions what I else I can try to make this work? Anything that=
you have tried that worked? have I missed something?<br><br>Would apprecia=
te any and all thoughts/suggestions<br><br>best regards<br><div>SS</div></d=
iv>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma=
rio.<br></div>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma=
rio.<br></div>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma=
rio.<br></div>
</blockquote></div>
--000000000000cf4c600631184611--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOyJeZRRB7nuSgzVL9J-OwamuLmBi3_a20BRaJhb5PWkpdUwKQ>