From nobody Mon Mar 24 14:58:49 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLx6B6gZwz5s1v5 for ; Mon, 24 Mar 2025 14:59:30 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLx696b0yz3XDs for ; Mon, 24 Mar 2025 14:59:29 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=h3+KqXWc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::630 as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-227d6b530d8so12390655ad.3 for ; Mon, 24 Mar 2025 07:59:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742828368; x=1743433168; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/3vwxiqm/CU6etvYPXXXelYSt+UGZmTt+cQ2Bq6SpGY=; b=h3+KqXWcLM6zLXtVfGvZQlODa3PsGe5LPAppHDojf3vBUJraS8rikwBHagNPR1jdKW gE7ET+LhCmO47CIYNc7NH6tBsgJhrHT5Eb1fI1eqICjoQh6a4pbdFILfqobSdOF3YzAY 5K/ghnrJc5vmmwgASKAg8bSpwv2YE8+7PQu2UddoNzZQZLVdC4TNMIafeKS4dLLY9Ll/ AzE3PuFr/F2YFYFX7mu5lZ6SDu2wcCAm11n7o4MTd5Nz4igl4k4sAPuTMXR20TcPysPF JtIXCN6nvSF2ToU5SkQcoxQLhRM6rHPuTovHPSEhDhUpG6HgX4ZDthaBGGUfb8PAG7QC DuFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742828368; x=1743433168; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/3vwxiqm/CU6etvYPXXXelYSt+UGZmTt+cQ2Bq6SpGY=; b=Jjx5+MYtJhPxaHCd/WyfW7RcZl8TO2XhytXPjrzyIfKEco6PHGfj8QMhi5Tv9sRVMw jXuLhO+tshu1b+/x1RIv5GAEg7s9d6ClMVdYx7EG9cHSfDIU8IHuKyS272B6ttSkzTFz ahl4298bPZsbQu1AyiouCmC2q94VsBCvUfOilYmYhXHg77NTU/ZjICOflY6uYJzA81U1 vvozHwhi/djbgC9FPpV61cGJ0Qr8bUSB7Jsx/vzyDiI5fxoaUNNm8r8X4hrC8w6r6wz1 CecWJxJrqJQ9Rbk0WMRgujwFrGzUehjoOBx08YrJtrXGF4noVXITnUCKmunzrUDwlvwb RDrA== X-Gm-Message-State: AOJu0YxvmeL0nS05ynxr6q7gg/ENG6LiLreXCiSOvwflN83SAYLVz/XG SIPjb+4vdwwNjdsrvE+HNxxUctFDzhmhTmAFkqwBW8Sga7nmX4ex9H0fIZuRlw9pCXeCLBCvgdj +DR8DLWiEUWIwup1Xh62QJHw+Ou0= X-Gm-Gg: ASbGnctXPCFXN3bwKjq+twV04tEcoRAieViSppABIrsJT05kPM5V2HKaDN/wNKCoLKd lCZ7TFkLZfGlMF3Lb0Du+ceMQIDbrbLDJMADZeTPMkI//97EVFL3UOIThRWYgDI5qAXTciVh+up QOa551AGmB6BaLW80U3asX7fNhuYA= X-Google-Smtp-Source: AGHT+IH6LmVZ/UIqhoW+JXPfDlCqoYDGERM1xumeLdO/YkEr3TQQB6voRLcVyWySjiOqfZlcQBs0VWjcbi35b+p2X3A= X-Received: by 2002:a05:6a00:4648:b0:730:927c:d451 with SMTP id d2e1a72fcca58-73905a234f3mr22338241b3a.20.1742828367930; Mon, 24 Mar 2025 07:59:27 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 15:58:49 +0100 X-Gm-Features: AQ5f1JonC5p5Mrok0qCItX9CeQnUEBYMdpwoCFwYfanmI-TTlFPxAGbK2qYYatY Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000c66db6063117d908" X-Spamd-Result: default: False [2.15 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; NEURAL_SPAM_SHORT(0.15)[0.150]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::630:from]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLx696b0yz3XDs X-Spamd-Bar: ++ --000000000000c66db6063117d908 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Oh sorry. I forgot to add the following bhyve parameters necessary to passthru my GPU :D -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto wrote: > Usually I use this kind of script to launch a bhyve vm : > > #!/bin/sh > > setxkbmap it > kldload vmm.ko > vms=3D"$(ls /dev/vmm/*)" > vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" > kldload vmm.ko > > if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." > kldload nvidia-modeset > devctl clear driver -f pci0:2:0:0 > devctl set driver -f pci0:2:0:0 ppt > else > echo "rtx 2080ti slot 2/0/0 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." > devctl clear driver -f pci0:2:0:1 > devctl set driver -f pci0:2:0:1 ppt > else > echo "rtx 2080ti slot 2/0/1 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." > devctl clear driver -f pci0:2:0:2 > devctl set driver -f pci0:2:0:2 ppt > else > echo "rtx 2080ti slot 2/0/2 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." > devctl clear driver -f pci0:2:0:3 > devctl set driver -f pci0:2:0:3 ppt > else > echo "rtx 2080ti slot 2/0/3 is already attached to ppt" > fi > > echo "rtx 2080ti is fully attached to ppt" > > for vm in $vms; do > session=3D"${vm##*/}" > echo "bhyve session =3D $session" > echo "vnc session =3D $vncs" > if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then > > printf 'VNC session not found,destroying ghost vms\n' > bhyvectl --vm=3D$session --destroy > else > printf 'Found VNC session %s\n' > "${session},no ghost vms found,not destroying them" > fi > done > > vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (AM13N4CZ)/ && d{print d}'` > echo "Seagate M3 Portable 1.8T ; $vmdisk0" > > vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2015040204055E)/ && d{print d}'` > echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" > > vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2027285F1175)/ && d{print d}'` > echo "CT1000P1SSD8 ; $vmdisk2" > > vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (20130508005976F)/ && d{print d}'` > echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" > > vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (BE0191510218)/ && d{print d}'` > echo "G-DRIVE USB ; $vmdisk4" > > vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (38234B4237354B45)/ && d{print d}'` > echo "Elements 25A3 ; $vmdisk5" > > vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (WD-WCAV2X797309)/ && d{print d}'` > echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" > > vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (20140108006C)/ && d{print d}'` > echo "Corsair Force 3 SSD ; $vmdisk7" > > vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (1924E50B2AE5)/ && d{print d}'` > echo "CT500MX500SSD4 ; $vmdisk8" > > vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (0774921DDC4200A6)/ && d{print d}'` > echo "SanDisk Cruzer-15GB ; $vmdisk9" > > vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (60A44D4138D8F311190A0149)/ && d{print d}'` > echo "Kingston DataTraveler 2.0 ; $vmdisk10" > > vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (575845483038524844323238)/ && d{print d}'` > echo "WD 2500BMV External ; $vmdisk11" > > vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (57442E575845323039544433303334)/ && d{print d}'` > echo "WD 3200BMV External ; $vmdisk12" > > vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2414E989076B)/ && d{print d}'` > echo "CT500BX500SSD1 ; $vmdisk13" > > /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H= -A \ > -s 0,hostbridge \ > -s 1,ahci-hd,/dev/$vmdisk13 \ > -s 10,virtio-net,tap6 \ > -s 11,virtio-9p,sharename=3D/ \ > -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ > vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & > > As you can see,each vm has a unique ID (connected with the vncviewer > instance) and the vncviewer window is launched as soon as the bhyve vm is > called...and my Geforce RTX 2080 ti is passed through inside the vm. > > > On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto > wrote: > >> First of all,you could use a more flexible technique than using the >> pptdevs on /boot/loader.conf. I would use a script like this : >> >> 1) >> >> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >> echo "18/0/0 is not attached to ppt,attaching..." >> devctl clear driver -f pci0:18:0:0 >> devctl set driver -f pci0:18:0:0 ppt >> else >> echo "18/0/0 already attached to ppt" >> fi >> >> 2) I would not like to use vm-bhyve. It adds only some confusion... >> >> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >> shamim.shahriar@gmail.com> wrote: >> >>> Good afternoon everyone. >>> >>> I am trying to have VMs with GPU passthrough. The setup is a Dell serve= r >>> with NVidia Quadro RTX 6000/8000 installed already. I have checked the >>> device IDs and put pptdevs in place >>> >>> # cat /boot/loader.conf >>> pptdevs=3D"18/0/0 19/0/0" >>> pptdevs2=3D"72/0/0 73/0/0" >>> >>> this is showing the GPUs as pptdev on the pciconf >>> >>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>> vendor =3D 'NVIDIA Corporation' >>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>> class =3D display >>> subclass =3D 3D >>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >>> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >>> vendor =3D 'NVIDIA Corporation' >>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>> class =3D display >>> subclass =3D 3D >>> >>> as I am using vm-bhyve, I have put the configuration as below >>> >>> # cat /mnt/VMs/jagadish/jagadish.conf >>> loader=3D"uefi" >>> cpu=3D16 >>> memory=3D128G >>> xhci_mouse=3D"yes" >>> debug=3D"true" >>> >>> graphics=3D"yes" >>> graphics_listen=3D"127.0.0.1" >>> graphics_port=3D"5920" >>> graphics_res=3D"1024x768" >>> graphics_wait=3D"no" >>> #graphics_vga=3D"io" >>> >>> network0_type=3D"virtio-net" >>> network0_switch=3D"swUNI" >>> network0_mac=3D"58:9c:fc:06:3f:af" >>> >>> disk0_type=3D"nvme" >>> disk0_name=3D"jagadish-disk0.img" >>> #disk0_size=3D"128G" >>> >>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>> >>> passthru0=3D"19/0/0=3D6:0" >>> >>> bhyve_options=3D"-A -H -P" >>> #END >>> >>> however, when I start the vm (for OS installation to start with), it >>> shows it is running but I am unable to access the VNC for a while, then >>> when I finally manage to connect via VNC, there is nothing on the scree= n, >>> just blank dark screen >>> >>> the vm-bhyve.log shows >>> >>> Mar 24 13:59:26: initialising >>> Mar 24 13:59:26: [loader: uefi] >>> Mar 24 13:59:26: [cpu: 16] >>> Mar 24 13:59:26: [memory: 128G] >>> Mar 24 13:59:26: [hostbridge: standard] >>> Mar 24 13:59:26: [com ports: com1] >>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>> Mar 24 13:59:26: [debug mode: true] >>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>> Mar 24 13:59:26: [primary disk dev: file] >>> Mar 24 13:59:26: initialising network device tap0 >>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>> Mar 24 13:59:26: booting >>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>> Mar 24 13:59:26: [bhyve iso device: -s >>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>> Mar 24 13:59:26: starting bhyve (run 1) >>> >>> based on what I can see, and a little out of desparation, I decided to >>> run the installer in the foreground, and below is what I got >>> >>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>> Starting jagadish >>> * found guest in /mnt/VMs/jagadish >>> * booting... >>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>> >>> it stays there for as long as I wait until I poweroff the vm. >>> >>> tried with Debian installer, >>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>> Starting jagadish >>> * found guest in /mnt/VMs/jagadish >>> * booting... >>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>> >>> >>> ideally I need to install Debian for my user base on these GPU based >>> systems. But even that is proving to be impossible since the installer = does >>> not move any further from where it is. >>> >>> Any thoughts/ideas/suggestions what I else I can try to make this work? >>> Anything that you have tried that worked? have I missed something? >>> >>> Would appreciate any and all thoughts/suggestions >>> >>> best regards >>> SS >>> >> >> >> -- >> Mario. >> > > > -- > Mario. > --=20 Mario. --000000000000c66db6063117d908 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Oh sorry. I forgot to add the following bhyve paramet= ers necessary to passthru my GPU :D

-s 8:0,passthr= u,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3= ,passthru,2/0/3 \

On Mon, Mar 24, 2025 at 3:57=E2=80= =AFPM Mario Marietto <marietto= 2008@gmail.com> wrote:
Usually I use this kind of script to la= unch a bhyve vm :

#!/bin/sh

setxkbmap it
kldl= oad vmm.ko
vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | = awk '/vncviewer [0]/{print $6}')"
kldload vmm.ko

if = ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
echo "rtx = 2080ti slot 2/0/0 is not attached to ppt,attaching..."
kldload nvid= ia-modeset
devctl clear driver -f pci0:2:0:0
devctl set driver -f pci= 0:2:0:0 ppt
else
echo "rtx 2080ti slot 2/0/0 is already attached= to ppt"
fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt&= quot;; then
echo "rtx 2080ti slot 2/0/1 is not attached to ppt,atta= ching..."
devctl clear driver -f pci0:2:0:1
devctl set driver -f= pci0:2:0:1 ppt
else
echo "rtx 2080ti slot 2/0/1 is already atta= ched to ppt"
fi

if ! pciconf -l pci0:2:0:2 | grep -q "^= ppt"; then
echo "rtx 2080ti slot 2/0/2 is not attached to ppt,= attaching..."
devctl clear driver -f pci0:2:0:2
devctl set drive= r -f pci0:2:0:2 ppt
else
echo "rtx 2080ti slot 2/0/2 is already = attached to ppt"
fi

if ! pciconf -l pci0:2:0:3 | grep -q &qu= ot;^ppt"; then
echo "rtx 2080ti slot 2/0/3 is not attached to = ppt,attaching..."
devctl clear driver -f pci0:2:0:3
devctl set d= river -f pci0:2:0:3 ppt
else
echo "rtx 2080ti slot 2/0/3 is alre= ady attached to ppt"
fi

echo "rtx 2080ti is fully attac= hed to ppt"

for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve sessi= on =3D $session"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 echo "vnc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '%s\n' "${vncs}&qu= ot; | grep "${session#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 printf 'VNC session not found,destroying ghost vms\n'=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC session %s\n' &qu= ot;${session},no ghost vms found,not destroying them"
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geo= m disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ &= & d{print d}'`
echo "Seagate M3 Portable 1.8T ; $vmdisk0&qu= ot;

vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ = *ident: (2015040204055E)/ && d{print d}'`
echo "TOSHIBA= External USB 3.0 1.8T ; $vmdisk1"

vmdisk2=3D`geom disk list | = awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{pr= int d}'`
echo "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`= geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2013050800597= 6F)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 932= GB ; $vmdisk3"

vmdisk4=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{print d}'`
ech= o "G-DRIVE USB ; $vmdisk4"

vmdisk5=3D`geom disk list | awk= '/^Geom name: /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{p= rint d}'`
echo "Elements 25A3 ; $vmdisk5"

vmdisk6= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X= 797309)/ && d{print d}'`
echo "WDC WD3200AAJS-00L7A0 ; = 298 GB ; $vmdisk6"

vmdisk7=3D`geom disk list | awk '/^Geom = name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d{print d}'`
= echo "Corsair Force 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk= list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ &&a= mp; d{print d}'`
echo "CT500MX500SSD4 ; $vmdisk8"

v= mdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (077= 4921DDC4200A6)/ && d{print d}'`
echo "SanDisk Cruzer-15= GB ; $vmdisk9"

vmdisk10=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}&= #39;`
echo "Kingston DataTraveler 2.0 ; $vmdisk10"

vmdi= sk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57584= 5483038524844323238)/ && d{print d}'`
echo "WD 2500BMV = External ; $vmdisk11"

vmdisk12=3D`geom disk list | awk '/^G= eom name: /{d=3D$NF} /^ *ident: (57442E575845323039544433303334)/ &&= ; d{print d}'`
echo "WD 3200BMV External ; $vmdisk12"
<= br>vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:= (2414E989076B)/ && d{print d}'`
echo "CT500BX500SSD1 ;= $vmdisk13"

/usr/sbin/./bhyve-win -S -c sockets=3D4= ,cores=3D1,threads=3D1 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-h= d,/dev/$vmdisk13 \
-s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename= =3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
-s 31,lpc \
-l bootrom,/usr/loc= al/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:6 < /dev/null & sl= eep 5 && vncviewer 0:6 &

As you can se= e,each vm has a unique ID (connected with the vncviewer instance) and the v= ncviewer window is launched as soon as the bhyve vm is called...and my Gefo= rce RTX 2080 ti is passed through inside the vm.


On Mon, Mar 24, 2= 025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">
Firs= t of all,you could use a more flexible technique than using the pptdevs on = /boot/loader.conf. I would use a script like this :

1)

if ! pciconf -l pci0:18:0:0 | grep -q "^= ppt"; then
echo "18/0/0 is not attached to ppt,attaching...&= quot;
devctl clear driver -f pci0:18:0:0
devctl set driver -f pci0:18= :0:0 ppt
else
echo "18/0/0 already attached to ppt"
fi

2) I would not like to use vm-bhyve. It adds only som= e confusion...

On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar= <shamim.= shahriar@gmail.com> wrote:
Good afternoon everyone.

I am try= ing to have VMs with GPU passthrough. The setup is a Dell server with NVidi= a Quadro RTX 6000/8000 installed already. I have checked the device IDs and= put pptdevs in place

# cat /boot/loader.conf
pptdevs=3D"18/= 0/0 19/0/0"
pptdevs2=3D"72/0/0 73/0/0"

this is sho= wing the GPUs as pptdev on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2= =A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D= 0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0= =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2= =A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt= 1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 = vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
= =C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2= =A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'= ;
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 s= ubclass =C2=A0 =3D 3D

as I am using vm-bhyve, I have put the configu= ration as below

# cat /mnt/VMs/jagadish/jagadish.conf
loader=3D&q= uot;uefi"
cpu=3D16
memory=3D128G
xhci_mouse=3D"yes"=
debug=3D"true"

graphics=3D"yes"
graphics_= listen=3D"127.0.0.1"
graphics_port=3D"5920"
graph= ics_res=3D"1024x768"
graphics_wait=3D"no"
#graphi= cs_vga=3D"io"

network0_type=3D"virtio-net"
ne= twork0_switch=3D"swUNI"
network0_mac=3D"58:9c:fc:06:3f:af= "

disk0_type=3D"nvme"
disk0_name=3D"jagadish-= disk0.img"
#disk0_size=3D"128G"

uuid=3D"966e9= 09b-1293-11ef-a9a4-e4434bfe34de"

passthru0=3D"19/0/0=3D6:0= "

bhyve_options=3D"-A -H -P"
#END

however, = when I start the vm (for OS installation to start with), it shows it is run= ning but I am unable to access the VNC for a while, then when I finally man= age to connect via VNC, there is nothing on the screen, just blank dark scr= een

the vm-bhyve.log shows

Mar 24 13:59:26: initialising
M= ar 24 13:59:26: =C2=A0[loader: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]Mar 24 13:59:26: =C2=A0[memory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridg= e: standard]
Mar 24 13:59:26: =C2=A0[com ports: com1]
Mar 24 13:59:26= : =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: = =C2=A0[debug mode: true]
Mar 24 13:59:26: =C2=A0[primary disk: jagadish-= disk0.img]
Mar 24 13:59:26: =C2=A0[primary disk dev: file]
Mar 24 13:= 59:26: initialising network device tap0
Mar 24 13:59:26: adding tap0 -&g= t; vm-swUNI (swUNI addm)
Mar 24 13:59:26: bring up tap0 -> vm-swUNI (= swUNI addm)
Mar 24 13:59:26: booting
Mar 24 13:59:26: =C2=A0[bhyve op= tions: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_= UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 1= 3:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/V= Ms/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:= af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet= ]
Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev/nmdm-jagadish.1A]<= br>Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/l= ubuntu-24.04.1-desktop-amd64.iso,ro]
Mar 24 13:59:26: starting bhyve (ru= n 1)

based on what I can see, and a little out of desparation, I dec= ided to run the installer in the foreground, and below is what I got
# vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
Starting = jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 * booting..= .
fbuf frame buffer base: 0x112245400000 [sz 16777216]

it stays t= here for as long as I wait until I poweroff the vm.

tried with Debia= n installer,
# vm install -f jagadish debian-12.5.0-amd64-netinst.isoStarting jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 = * booting...
fbuf frame buffer base: 0x2747e2400000 [sz 16777216]

ideally I need to install Debian for my user base on these GPU based s= ystems. But even that is proving to be impossible since the installer does = not move any further from where it is.

Any thoughts/ideas/suggestion= s what I else I can try to make this work? Anything that you have tried tha= t worked? have I missed something?

Would appreciate any and all thou= ghts/suggestions

best regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000c66db6063117d908--