Date: Mon, 24 Mar 2025 16:48:35 +0100 From: Mario Marietto <marietto2008@gmail.com> To: Shamim Shahriar <shamim.shahriar@gmail.com> Cc: freebsd-virtualization@freebsd.org Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 Message-ID: <CA%2B1FSig6v3p9%2BC5AJCJ26mbo-psy9KVrWyJMVGw0a=Ov1opzRA@mail.gmail.com> In-Reply-To: <CA%2B1FSigKfpzaCmuLEtzgy2Co5=aSuABeo_m284aq54ZadS7eKg@mail.gmail.com> References: <CAOyJeZTtM80ceqdZffNXef4LVoEA3GBYRrn508pS0y12yT8dOQ@mail.gmail.com> <CA%2B1FSigyrtAiM%2By6p%2Bqc2JDYa=eRRZ814aHwfHV-kjepMNYNNg@mail.gmail.com> <CA%2B1FSiiu0cL2XNQk7oVByqt=BWn8yd-9h42wuDP11eq=cd1ndQ@mail.gmail.com> <CA%2B1FSigdkJRRD42TrD0wR0_gspFkL2neqvNF2wY0fDTE%2BPpP-g@mail.gmail.com> <CAOyJeZRRB7nuSgzVL9J-OwamuLmBi3_a20BRaJhb5PWkpdUwKQ@mail.gmail.com> <CA%2B1FSigKfpzaCmuLEtzgy2Co5=aSuABeo_m284aq54ZadS7eKg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Ohhh man.
I forgot to say that you need to apply the Corvin's patches if you want to
pass your gpu to a LINUX vm. And you should apply another patch if you want
to pass it to a Windows VM. It seems that the necessary patches have never
been uploaded mainstream.
On Mon, Mar 24, 2025 at 4:38 PM Mario Marietto <marietto2008@gmail.com>
wrote:
> Hello Shamim.
>
> You don't have any bhyve-win. You can use bhyve. I use bhyve-win to pass
> through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are two
> different executables,because for some reason,my bhyve-lin executable is
> not able to pass my GPU in a Windows vm.
>
> On Mon, Mar 24, 2025 at 4:29 PM Shamim Shahriar <shamim.shahriar@gmail.com>
> wrote:
>
>> Hi Mario
>>
>> I will try the options you have used later and let you know how it goes.
>> Not sure if this is intentional or typo, but I don't recall any bhyve-win
>> binary in my system.
>>
>> Best regards
>> SS
>>
>> On Mon, 24 Mar 2025 at 14:59, Mario Marietto <marietto2008@gmail.com>
>> wrote:
>>
>>> Oh sorry. I forgot to add the following bhyve parameters necessary to
>>> passthru my GPU :D
>>>
>>> -s 8:0,passthru,2/0/0 \
>>> -s 8:1,passthru,2/0/1 \
>>> -s 8:2,passthru,2/0/2 \
>>> -s 8:3,passthru,2/0/3 \
>>>
>>> On Mon, Mar 24, 2025 at 3:57 PM Mario Marietto <marietto2008@gmail.com>
>>> wrote:
>>>
>>>> Usually I use this kind of script to launch a bhyve vm :
>>>>
>>>> #!/bin/sh
>>>>
>>>> setxkbmap it
>>>> kldload vmm.ko
>>>> vms="$(ls /dev/vmm/*)"
>>>> vncs="$(ps ax | awk '/vncviewer [0]/{print $6}')"
>>>> kldload vmm.ko
>>>>
>>>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
>>>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."
>>>> kldload nvidia-modeset
>>>> devctl clear driver -f pci0:2:0:0
>>>> devctl set driver -f pci0:2:0:0 ppt
>>>> else
>>>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
>>>> fi
>>>>
>>>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
>>>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."
>>>> devctl clear driver -f pci0:2:0:1
>>>> devctl set driver -f pci0:2:0:1 ppt
>>>> else
>>>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt"
>>>> fi
>>>>
>>>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
>>>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
>>>> devctl clear driver -f pci0:2:0:2
>>>> devctl set driver -f pci0:2:0:2 ppt
>>>> else
>>>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
>>>> fi
>>>>
>>>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
>>>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
>>>> devctl clear driver -f pci0:2:0:3
>>>> devctl set driver -f pci0:2:0:3 ppt
>>>> else
>>>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
>>>> fi
>>>>
>>>> echo "rtx 2080ti is fully attached to ppt"
>>>>
>>>> for vm in $vms; do
>>>> session="${vm##*/}"
>>>> echo "bhyve session = $session"
>>>> echo "vnc session = $vncs"
>>>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}";
>>>> then
>>>> printf 'VNC session not found,destroying ghost vms\n'
>>>> bhyvectl --vm=$session --destroy
>>>> else
>>>> printf 'Found VNC session %s\n'
>>>> "${session},no ghost vms found,not destroying them"
>>>> fi
>>>> done
>>>>
>>>> vmdisk0=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (AM13N4CZ)/ && d{print d}'`
>>>> echo "Seagate M3 Portable 1.8T ; $vmdisk0"
>>>>
>>>> vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (2015040204055E)/ && d{print d}'`
>>>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"
>>>>
>>>> vmdisk2=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (2027285F1175)/ && d{print d}'`
>>>> echo "CT1000P1SSD8 ; $vmdisk2"
>>>>
>>>> vmdisk3=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (20130508005976F)/ && d{print d}'`
>>>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"
>>>>
>>>> vmdisk4=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (BE0191510218)/ && d{print d}'`
>>>> echo "G-DRIVE USB ; $vmdisk4"
>>>>
>>>> vmdisk5=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (38234B4237354B45)/ && d{print d}'`
>>>> echo "Elements 25A3 ; $vmdisk5"
>>>>
>>>> vmdisk6=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (WD-WCAV2X797309)/ && d{print d}'`
>>>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"
>>>>
>>>> vmdisk7=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (20140108006C)/ && d{print d}'`
>>>> echo "Corsair Force 3 SSD ; $vmdisk7"
>>>>
>>>> vmdisk8=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (1924E50B2AE5)/ && d{print d}'`
>>>> echo "CT500MX500SSD4 ; $vmdisk8"
>>>>
>>>> vmdisk9=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (0774921DDC4200A6)/ && d{print d}'`
>>>> echo "SanDisk Cruzer-15GB ; $vmdisk9"
>>>>
>>>> vmdisk10=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (60A44D4138D8F311190A0149)/ && d{print d}'`
>>>> echo "Kingston DataTraveler 2.0 ; $vmdisk10"
>>>>
>>>> vmdisk11=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (575845483038524844323238)/ && d{print d}'`
>>>> echo "WD 2500BMV External ; $vmdisk11"
>>>>
>>>> vmdisk12=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (57442E575845323039544433303334)/ && d{print d}'`
>>>> echo "WD 3200BMV External ; $vmdisk12"
>>>>
>>>> vmdisk13=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
>>>> (2414E989076B)/ && d{print d}'`
>>>> echo "CT500BX500SSD1 ; $vmdisk13"
>>>>
>>>> /usr/sbin/./bhyve-win -S -c sockets=4,cores=1,threads=1 -m 4G -w -H -A \
>>>> -s 0,hostbridge \
>>>> -s 1,ahci-hd,/dev/$vmdisk13 \
>>>> -s 10,virtio-net,tap6 \
>>>> -s 11,virtio-9p,sharename=/ \
>>>> -s 29,fbuf,tcp=0.0.0.0:5906,w=1600,h=950 \
>>>> -s 31,lpc \
>>>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
>>>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &
>>>>
>>>> As you can see,each vm has a unique ID (connected with the vncviewer
>>>> instance) and the vncviewer window is launched as soon as the bhyve vm is
>>>> called...and my Geforce RTX 2080 ti is passed through inside the vm.
>>>>
>>>>
>>>> On Mon, Mar 24, 2025 at 3:47 PM Mario Marietto <marietto2008@gmail.com>
>>>> wrote:
>>>>
>>>>> First of all,you could use a more flexible technique than using the
>>>>> pptdevs on /boot/loader.conf. I would use a script like this :
>>>>>
>>>>> 1)
>>>>>
>>>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then
>>>>> echo "18/0/0 is not attached to ppt,attaching..."
>>>>> devctl clear driver -f pci0:18:0:0
>>>>> devctl set driver -f pci0:18:0:0 ppt
>>>>> else
>>>>> echo "18/0/0 already attached to ppt"
>>>>> fi
>>>>>
>>>>> 2) I would not like to use vm-bhyve. It adds only some confusion...
>>>>>
>>>>> On Mon, Mar 24, 2025 at 3:22 PM Shamim Shahriar <
>>>>> shamim.shahriar@gmail.com> wrote:
>>>>>
>>>>>> Good afternoon everyone.
>>>>>>
>>>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell
>>>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have checked
>>>>>> the device IDs and put pptdevs in place
>>>>>>
>>>>>> # cat /boot/loader.conf
>>>>>> pptdevs="18/0/0 19/0/0"
>>>>>> pptdevs2="72/0/0 73/0/0"
>>>>>>
>>>>>> this is showing the GPUs as pptdev on the pciconf
>>>>>>
>>>>>> ppt0@pci0:18:0:0: class=0x030200 rev=0xa1 hdr=0x00
>>>>>> vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8
>>>>>> vendor = 'NVIDIA Corporation'
>>>>>> device = 'TU102GL [Quadro RTX 6000/8000]'
>>>>>> class = display
>>>>>> subclass = 3D
>>>>>> ppt1@pci0:19:0:0: class=0x030200 rev=0xa1 hdr=0x00
>>>>>> vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8
>>>>>> vendor = 'NVIDIA Corporation'
>>>>>> device = 'TU102GL [Quadro RTX 6000/8000]'
>>>>>> class = display
>>>>>> subclass = 3D
>>>>>>
>>>>>> as I am using vm-bhyve, I have put the configuration as below
>>>>>>
>>>>>> # cat /mnt/VMs/jagadish/jagadish.conf
>>>>>> loader="uefi"
>>>>>> cpu=16
>>>>>> memory=128G
>>>>>> xhci_mouse="yes"
>>>>>> debug="true"
>>>>>>
>>>>>> graphics="yes"
>>>>>> graphics_listen="127.0.0.1"
>>>>>> graphics_port="5920"
>>>>>> graphics_res="1024x768"
>>>>>> graphics_wait="no"
>>>>>> #graphics_vga="io"
>>>>>>
>>>>>> network0_type="virtio-net"
>>>>>> network0_switch="swUNI"
>>>>>> network0_mac="58:9c:fc:06:3f:af"
>>>>>>
>>>>>> disk0_type="nvme"
>>>>>> disk0_name="jagadish-disk0.img"
>>>>>> #disk0_size="128G"
>>>>>>
>>>>>> uuid="966e909b-1293-11ef-a9a4-e4434bfe34de"
>>>>>>
>>>>>> passthru0="19/0/0=6:0"
>>>>>>
>>>>>> bhyve_options="-A -H -P"
>>>>>> #END
>>>>>>
>>>>>> however, when I start the vm (for OS installation to start with), it
>>>>>> shows it is running but I am unable to access the VNC for a while, then
>>>>>> when I finally manage to connect via VNC, there is nothing on the screen,
>>>>>> just blank dark screen
>>>>>>
>>>>>> the vm-bhyve.log shows
>>>>>>
>>>>>> Mar 24 13:59:26: initialising
>>>>>> Mar 24 13:59:26: [loader: uefi]
>>>>>> Mar 24 13:59:26: [cpu: 16]
>>>>>> Mar 24 13:59:26: [memory: 128G]
>>>>>> Mar 24 13:59:26: [hostbridge: standard]
>>>>>> Mar 24 13:59:26: [com ports: com1]
>>>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
>>>>>> Mar 24 13:59:26: [debug mode: true]
>>>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img]
>>>>>> Mar 24 13:59:26: [primary disk dev: file]
>>>>>> Mar 24 13:59:26: initialising network device tap0
>>>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
>>>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
>>>>>> Mar 24 13:59:26: booting
>>>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l
>>>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U
>>>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
>>>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s
>>>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s
>>>>>> 5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s
>>>>>> 7:0,fbuf,tcp=127.0.0.1:5920,w=1024,h=768 -s 8:0,xhci,tablet]
>>>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A]
>>>>>> Mar 24 13:59:26: [bhyve iso device: -s
>>>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
>>>>>> Mar 24 13:59:26: starting bhyve (run 1)
>>>>>>
>>>>>> based on what I can see, and a little out of desparation, I decided
>>>>>> to run the installer in the foreground, and below is what I got
>>>>>>
>>>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
>>>>>> Starting jagadish
>>>>>> * found guest in /mnt/VMs/jagadish
>>>>>> * booting...
>>>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216]
>>>>>>
>>>>>> it stays there for as long as I wait until I poweroff the vm.
>>>>>>
>>>>>> tried with Debian installer,
>>>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso
>>>>>> Starting jagadish
>>>>>> * found guest in /mnt/VMs/jagadish
>>>>>> * booting...
>>>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216]
>>>>>>
>>>>>>
>>>>>> ideally I need to install Debian for my user base on these GPU based
>>>>>> systems. But even that is proving to be impossible since the installer does
>>>>>> not move any further from where it is.
>>>>>>
>>>>>> Any thoughts/ideas/suggestions what I else I can try to make this
>>>>>> work? Anything that you have tried that worked? have I missed something?
>>>>>>
>>>>>> Would appreciate any and all thoughts/suggestions
>>>>>>
>>>>>> best regards
>>>>>> SS
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mario.
>>>>>
>>>>
>>>>
>>>> --
>>>> Mario.
>>>>
>>>
>>>
>>> --
>>> Mario.
>>>
>>
>
> --
> Mario.
>
--
Mario.
[-- Attachment #2 --]
<div dir="ltr"><div>Ohhh man. <br></div><div><br></div><div>I forgot to say that you need to apply the Corvin's patches if you want to pass your gpu to a LINUX vm. And you should apply another patch if you want to pass it to a Windows VM. It seems that the necessary patches have never been uploaded mainstream. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 24, 2025 at 4:38 PM Mario Marietto <<a href="mailto:marietto2008@gmail.com" target="_blank">marietto2008@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello Shamim.</div><div><br></div><div>You don't have any bhyve-win. You can use bhyve. I use bhyve-win to pass through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are two different executables,because for some reason,my bhyve-lin executable is not able to pass my GPU in a Windows vm. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 24, 2025 at 4:29 PM Shamim Shahriar <<a href="mailto:shamim.shahriar@gmail.com" target="_blank">shamim.shahriar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Mario<div><br></div><div>I will try the options you have used later and let you know how it goes. Not sure if this is intentional or typo, but I don't recall any bhyve-win binary in my system.</div><div><br></div><div>Best regards</div><div>SS</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 24 Mar 2025 at 14:59, Mario Marietto <<a href="mailto:marietto2008@gmail.com" target="_blank">marietto2008@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Oh sorry. I forgot to add the following bhyve parameters necessary to passthru my GPU :D</div><div><br></div><div>-s 8:0,passthru,2/0/0 \</div>-s 8:1,passthru,2/0/1 \<br>-s 8:2,passthru,2/0/2 \<br>-s 8:3,passthru,2/0/3 \</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 24, 2025 at 3:57 PM Mario Marietto <<a href="mailto:marietto2008@gmail.com" target="_blank">marietto2008@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Usually I use this kind of script to launch a bhyve vm :</div><div><br></div>#!/bin/sh<br><br>setxkbmap it<br>kldload vmm.ko<br>vms="$(ls /dev/vmm/*)"<br>vncs="$(ps ax | awk '/vncviewer [0]/{print $6}')"<br>kldload vmm.ko<br><br>if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."<br>kldload nvidia-modeset<br>devctl clear driver -f pci0:2:0:0<br>devctl set driver -f pci0:2:0:0 ppt<br>else<br>echo "rtx 2080ti slot 2/0/0 is already attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."<br>devctl clear driver -f pci0:2:0:1<br>devctl set driver -f pci0:2:0:1 ppt<br>else<br>echo "rtx 2080ti slot 2/0/1 is already attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."<br>devctl clear driver -f pci0:2:0:2<br>devctl set driver -f pci0:2:0:2 ppt<br>else<br>echo "rtx 2080ti slot 2/0/2 is already attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."<br>devctl clear driver -f pci0:2:0:3<br>devctl set driver -f pci0:2:0:3 ppt<br>else<br>echo "rtx 2080ti slot 2/0/3 is already attached to ppt"<br>fi<br><br>echo "rtx 2080ti is fully attached to ppt"<br><br>for vm in $vms; do<br> session="${vm##*/}" <br> echo "bhyve session = $session"<br> echo "vnc session = $vncs"<br> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then <br> printf 'VNC session not found,destroying ghost vms\n'<br> bhyvectl --vm=$session --destroy<br> else<br> printf 'Found VNC session %s\n' "${session},no ghost vms found,not destroying them"<br> fi<br>done<br><br>vmdisk0=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (AM13N4CZ)/ && d{print d}'`<br>echo "Seagate M3 Portable 1.8T ; $vmdisk0"<br><br>vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (2015040204055E)/ && d{print d}'`<br>echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1"<br><br>vmdisk2=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (2027285F1175)/ && d{print d}'`<br>echo "CT1000P1SSD8 ; $vmdisk2"<br><br>vmdisk3=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (20130508005976F)/ && d{print d}'`<br>echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3"<br><br>vmdisk4=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (BE0191510218)/ && d{print d}'`<br>echo "G-DRIVE USB ; $vmdisk4"<br><br>vmdisk5=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (38234B4237354B45)/ && d{print d}'`<br>echo "Elements 25A3 ; $vmdisk5"<br><br>vmdisk6=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (WD-WCAV2X797309)/ && d{print d}'`<br>echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6"<br><br>vmdisk7=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (20140108006C)/ && d{print d}'`<br>echo "Corsair Force 3 SSD ; $vmdisk7"<br><br>vmdisk8=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'`<br>echo "CT500MX500SSD4 ; $vmdisk8"<br><br>vmdisk9=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (0774921DDC4200A6)/ && d{print d}'`<br>echo "SanDisk Cruzer-15GB ; $vmdisk9"<br><br>vmdisk10=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}'`<br>echo "Kingston DataTraveler 2.0 ; $vmdisk10"<br><br>vmdisk11=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (575845483038524844323238)/ && d{print d}'`<br>echo "WD 2500BMV External ; $vmdisk11"<br><br>vmdisk12=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (57442E575845323039544433303334)/ && d{print d}'`<br>echo "WD 3200BMV External ; $vmdisk12"<br><br>vmdisk13=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (2414E989076B)/ && d{print d}'`<br>echo "CT500BX500SSD1 ; $vmdisk13"<div><br></div><div>/usr/sbin/./bhyve-win -S -c sockets=4,cores=1,threads=1 -m 4G -w -H -A \<br>-s 0,hostbridge \<br>-s 1,ahci-hd,/dev/$vmdisk13 \<br>-s 10,virtio-net,tap6 \<br>-s 11,virtio-9p,sharename=/ \<br>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5906" target="_blank">0.0.0.0:5906</a>,w=1600,h=950 \<br>-s 31,lpc \<br>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \<br>vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &</div><div><br></div><div>As you can see,each vm has a unique ID (connected with the vncviewer instance) and the vncviewer window is launched as soon as the bhyve vm is called...and my Geforce RTX 2080 ti is passed through inside the vm.</div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 24, 2025 at 3:47 PM Mario Marietto <<a href="mailto:marietto2008@gmail.com" target="_blank">marietto2008@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>First of all,you could use a more flexible technique than using the pptdevs on /boot/loader.conf. I would use a script like this :</div><div><br></div><div>1)</div><div><br></div><div>if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then</div>echo "18/0/0 is not attached to ppt,attaching..."<br>devctl clear driver -f pci0:18:0:0<br>devctl set driver -f pci0:18:0:0 ppt<br>else<br>echo "18/0/0 already attached to ppt"<br>fi<br><div><br></div><div>2) I would not like to use vm-bhyve. It adds only some confusion...</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 24, 2025 at 3:22 PM Shamim Shahriar <<a href="mailto:shamim.shahriar@gmail.com" target="_blank">shamim.shahriar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Good afternoon everyone.<br><br>I am trying to have VMs with GPU passthrough. The setup is a Dell server with NVidia Quadro RTX 6000/8000 installed already. I have checked the device IDs and put pptdevs in place<br><br># cat /boot/loader.conf<br>pptdevs="18/0/0 19/0/0"<br>pptdevs2="72/0/0 73/0/0"<br><br>this is showing the GPUs as pptdev on the pciconf<br><br>ppt0@pci0:18:0:0: class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8<br> vendor = 'NVIDIA Corporation'<br> device = 'TU102GL [Quadro RTX 6000/8000]'<br> class = display<br> subclass = 3D<br>ppt1@pci0:19:0:0: class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8<br> vendor = 'NVIDIA Corporation'<br> device = 'TU102GL [Quadro RTX 6000/8000]'<br> class = display<br> subclass = 3D<br><br>as I am using vm-bhyve, I have put the configuration as below<br><br># cat /mnt/VMs/jagadish/jagadish.conf<br>loader="uefi"<br>cpu=16<br>memory=128G<br>xhci_mouse="yes"<br>debug="true"<br><br>graphics="yes"<br>graphics_listen="127.0.0.1"<br>graphics_port="5920"<br>graphics_res="1024x768"<br>graphics_wait="no"<br>#graphics_vga="io"<br><br>network0_type="virtio-net"<br>network0_switch="swUNI"<br>network0_mac="58:9c:fc:06:3f:af"<br><br>disk0_type="nvme"<br>disk0_name="jagadish-disk0.img"<br>#disk0_size="128G"<br><br>uuid="966e909b-1293-11ef-a9a4-e4434bfe34de"<br><br>passthru0="19/0/0=6:0"<br><br>bhyve_options="-A -H -P"<br>#END<br><br>however, when I start the vm (for OS installation to start with), it shows it is running but I am unable to access the VNC for a while, then when I finally manage to connect via VNC, there is nothing on the screen, just blank dark screen<br><br>the vm-bhyve.log shows<br><br>Mar 24 13:59:26: initialising<br>Mar 24 13:59:26: [loader: uefi]<br>Mar 24 13:59:26: [cpu: 16]<br>Mar 24 13:59:26: [memory: 128G]<br>Mar 24 13:59:26: [hostbridge: standard]<br>Mar 24 13:59:26: [com ports: com1]<br>Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]<br>Mar 24 13:59:26: [debug mode: true]<br>Mar 24 13:59:26: [primary disk: jagadish-disk0.img]<br>Mar 24 13:59:26: [primary disk dev: file]<br>Mar 24 13:59:26: initialising network device tap0<br>Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)<br>Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)<br>Mar 24 13:59:26: booting<br>Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]<br>Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=<a href="http://127.0.0.1:5920" target="_blank">127.0.0.1:5920</a>,w=1024,h=768 -s 8:0,xhci,tablet]<br>Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A]<br>Mar 24 13:59:26: [bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]<br>Mar 24 13:59:26: starting bhyve (run 1)<br><br>based on what I can see, and a little out of desparation, I decided to run the installer in the foreground, and below is what I got<br><br># vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso<br>Starting jagadish<br> * found guest in /mnt/VMs/jagadish<br> * booting...<br>fbuf frame buffer base: 0x112245400000 [sz 16777216]<br><br>it stays there for as long as I wait until I poweroff the vm.<br><br>tried with Debian installer, <br># vm install -f jagadish debian-12.5.0-amd64-netinst.iso<br>Starting jagadish<br> * found guest in /mnt/VMs/jagadish<br> * booting...<br>fbuf frame buffer base: 0x2747e2400000 [sz 16777216]<br><br><br>ideally I need to install Debian for my user base on these GPU based systems. But even that is proving to be impossible since the installer does not move any further from where it is.<br><br>Any thoughts/ideas/suggestions what I else I can try to make this work? Anything that you have tried that worked? have I missed something?<br><br>Would appreciate any and all thoughts/suggestions<br><br>best regards<br><div>SS</div></div>
</blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>
</blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>
</blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>
</blockquote></div>
</blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>
</blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSig6v3p9%2BC5AJCJ26mbo-psy9KVrWyJMVGw0a=Ov1opzRA>