From nobody Mon Mar 24 14:57:19 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLx4R3Zksz5s1mT for ; Mon, 24 Mar 2025 14:57:59 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLx4Q40Fnz3WDc for ; Mon, 24 Mar 2025 14:57:58 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hQabPplC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::1030 as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-3018e2d042bso5507652a91.2 for ; Mon, 24 Mar 2025 07:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742828277; x=1743433077; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=b3ubtgJDbIbo+oY3JwTrrB3ewyWFAY81utGdzTIBNZM=; b=hQabPplCu/LPOPd7uBOMwgHZ+IlthgaQUqmwTahGc2BpXVm/H1kkXy2P7AVzNlmNBc zaSfEDMZrSkNzxm8+bt8y0YoV3uPDonxg6jYGS6EDCH1Nr4zosPHhdIlw9/2r3DtI15o XXYKWThl9dUfdkKf3cyjvhjYMNi+6DXkkG6J5tTR51N7jCz8HOapbF/n3S+HweyxakPP a9OKOiQrRKKmDli970jMjTaYBjja+RG+6jEXWMtnowEMN4PtViExBm4Iu4dlL3pnGFKb vjZnqFNLErXIThUqGLqPx7KEafWA6guCH/6YZeco6+8gyukPJIOCF9NubAkHWEX/iOXv PW1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742828277; x=1743433077; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=b3ubtgJDbIbo+oY3JwTrrB3ewyWFAY81utGdzTIBNZM=; b=QxqVW1Rv8I7YVd9sQReyde8+Uc6EPxytpBv9o4q9z4Kv1CAJg4+qoRQFdkptrdN57M U39aeIVdr62N3+WsoQVZ86Ch47jnEOX107wbn6T9rHjVQxOhpIyGl/Mf++hKLbQE8gw5 iUFU9FP9Mn0bSRpFl29hT6zb9Rr9oWItRaRLiQFBICvtbVsMwxwheDNG9n+7cDAAcHuu j9wlfydKxU/haNScHwVSerUrO9roWSsKYb1bZD/AZliJxisDM/Lb3jxJFe0dxOlZCTMM fXXosdv9ITTp3xW89hKMPIk41zMsAGMsqxbGcgVoM0Qz7mm9N3LY45gmsOaITFd4u4G9 oBtw== X-Gm-Message-State: AOJu0YyDefkp87Wth+YyuSRGVkeBL/QgckuJgiu5A3hFR0OytQqrwQ5G MJ7GmznEqK7UGCthtOb2csxqcNAhj6uMIWT7kI/B/ykZu+uXsNfKJjsQWF10HMUDO14bH4lQ/11 I5QOb6IzI6Smx1FclgPWtPLo0py5nlKU09ZE= X-Gm-Gg: ASbGncsT8glmyMw2X1TKDQAe1ZrP+PpygzeEyIEu1OXRrt8jme1tJmJ+4gD6LNiQlBc FI2n+bK/TiSbUCo5S/pmSoOrqtmbVUdmcm7VkuoDqL8FIhT8VyjFimxfkAF8nDFSLUR+IIHs0xV gkeKBVqH0UyOty65LRy3Jdf5Y2AM4= X-Google-Smtp-Source: AGHT+IFXW45p4sSR+V7+AqZUhAs/BUkd7/F9CTQPRwul0O9/I9bCBLkHyqGYGSyXAqNcBElskvLxdFGueOQLHx9FkZI= X-Received: by 2002:a17:90a:da85:b0:2fe:8217:2da6 with SMTP id 98e67ed59e1d1-3030fef09a8mr17614005a91.22.1742828276764; Mon, 24 Mar 2025 07:57:56 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 15:57:19 +0100 X-Gm-Features: AQ5f1Jq4gUcTKx1JLpUqj6hBNOkOUoHxZz2lhf_cxbaH4u_Evys23nhoftqwBJ0 Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000575a23063117d498" X-Spamd-Result: default: False [2.15 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; NEURAL_SPAM_SHORT(0.15)[0.148]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1030:from]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLx4Q40Fnz3WDc X-Spamd-Bar: ++ --000000000000575a23063117d498 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Usually I use this kind of script to launch a bhyve vm : #!/bin/sh setxkbmap it kldload vmm.ko vms=3D"$(ls /dev/vmm/*)" vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" kldload vmm.ko if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." kldload nvidia-modeset devctl clear driver -f pci0:2:0:0 devctl set driver -f pci0:2:0:0 ppt else echo "rtx 2080ti slot 2/0/0 is already attached to ppt" fi if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." devctl clear driver -f pci0:2:0:1 devctl set driver -f pci0:2:0:1 ppt else echo "rtx 2080ti slot 2/0/1 is already attached to ppt" fi if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." devctl clear driver -f pci0:2:0:2 devctl set driver -f pci0:2:0:2 ppt else echo "rtx 2080ti slot 2/0/2 is already attached to ppt" fi if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." devctl clear driver -f pci0:2:0:3 devctl set driver -f pci0:2:0:3 ppt else echo "rtx 2080ti slot 2/0/3 is already attached to ppt" fi echo "rtx 2080ti is fully attached to ppt" for vm in $vms; do session=3D"${vm##*/}" echo "bhyve session =3D $session" echo "vnc session =3D $vncs" if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then printf 'VNC session not found,destroying ghost vms\n' bhyvectl --vm=3D$session --destroy else printf 'Found VNC session %s\n' "${session},no ghost vms found,not destroying them" fi done vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4= CZ)/ && d{print d}'` echo "Seagate M3 Portable 1.8T ; $vmdisk0" vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2015040204055E)/ && d{print d}'` echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{print d}'` echo "CT1000P1SSD8 ; $vmdisk2" vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20130508005976F)/ && d{print d}'` echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{print d}'` echo "G-DRIVE USB ; $vmdisk4" vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{print d}'` echo "Elements 25A3 ; $vmdisk5" vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X797309)/ && d{print d}'` echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d{print d}'` echo "Corsair Force 3 SSD ; $vmdisk7" vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'` echo "CT500MX500SSD4 ; $vmdisk8" vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (0774921DDC4200A6)/ && d{print d}'` echo "SanDisk Cruzer-15GB ; $vmdisk9" vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}'` echo "Kingston DataTraveler 2.0 ; $vmdisk10" vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (575845483038524844323238)/ && d{print d}'` echo "WD 2500BMV External ; $vmdisk11" vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57442E575845323039544433303334)/ && d{print d}'` echo "WD 3200BMV External ; $vmdisk12" vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2414E989076B)/ && d{print d}'` echo "CT500BX500SSD1 ; $vmdisk13" /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w -H -= A \ -s 0,hostbridge \ -s 1,ahci-hd,/dev/$vmdisk13 \ -s 10,virtio-net,tap6 \ -s 11,virtio-9p,sharename=3D/ \ -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & As you can see,each vm has a unique ID (connected with the vncviewer instance) and the vncviewer window is launched as soon as the bhyve vm is called...and my Geforce RTX 2080 ti is passed through inside the vm. On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto wrote: > First of all,you could use a more flexible technique than using the > pptdevs on /boot/loader.conf. I would use a script like this : > > 1) > > if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then > echo "18/0/0 is not attached to ppt,attaching..." > devctl clear driver -f pci0:18:0:0 > devctl set driver -f pci0:18:0:0 ppt > else > echo "18/0/0 already attached to ppt" > fi > > 2) I would not like to use vm-bhyve. It adds only some confusion... > > On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar > wrote: > >> Good afternoon everyone. >> >> I am trying to have VMs with GPU passthrough. The setup is a Dell server >> with NVidia Quadro RTX 6000/8000 installed already. I have checked the >> device IDs and put pptdevs in place >> >> # cat /boot/loader.conf >> pptdevs=3D"18/0/0 19/0/0" >> pptdevs2=3D"72/0/0 73/0/0" >> >> this is showing the GPUs as pptdev on the pciconf >> >> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D= 0x10de >> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >> vendor =3D 'NVIDIA Corporation' >> device =3D 'TU102GL [Quadro RTX 6000/8000]' >> class =3D display >> subclass =3D 3D >> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D= 0x10de >> device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8 >> vendor =3D 'NVIDIA Corporation' >> device =3D 'TU102GL [Quadro RTX 6000/8000]' >> class =3D display >> subclass =3D 3D >> >> as I am using vm-bhyve, I have put the configuration as below >> >> # cat /mnt/VMs/jagadish/jagadish.conf >> loader=3D"uefi" >> cpu=3D16 >> memory=3D128G >> xhci_mouse=3D"yes" >> debug=3D"true" >> >> graphics=3D"yes" >> graphics_listen=3D"127.0.0.1" >> graphics_port=3D"5920" >> graphics_res=3D"1024x768" >> graphics_wait=3D"no" >> #graphics_vga=3D"io" >> >> network0_type=3D"virtio-net" >> network0_switch=3D"swUNI" >> network0_mac=3D"58:9c:fc:06:3f:af" >> >> disk0_type=3D"nvme" >> disk0_name=3D"jagadish-disk0.img" >> #disk0_size=3D"128G" >> >> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >> >> passthru0=3D"19/0/0=3D6:0" >> >> bhyve_options=3D"-A -H -P" >> #END >> >> however, when I start the vm (for OS installation to start with), it >> shows it is running but I am unable to access the VNC for a while, then >> when I finally manage to connect via VNC, there is nothing on the screen= , >> just blank dark screen >> >> the vm-bhyve.log shows >> >> Mar 24 13:59:26: initialising >> Mar 24 13:59:26: [loader: uefi] >> Mar 24 13:59:26: [cpu: 16] >> Mar 24 13:59:26: [memory: 128G] >> Mar 24 13:59:26: [hostbridge: standard] >> Mar 24 13:59:26: [com ports: com1] >> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >> Mar 24 13:59:26: [debug mode: true] >> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >> Mar 24 13:59:26: [primary disk dev: file] >> Mar 24 13:59:26: initialising network device tap0 >> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >> Mar 24 13:59:26: booting >> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s >> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >> Mar 24 13:59:26: [bhyve iso device: -s >> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >> Mar 24 13:59:26: starting bhyve (run 1) >> >> based on what I can see, and a little out of desparation, I decided to >> run the installer in the foreground, and below is what I got >> >> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >> Starting jagadish >> * found guest in /mnt/VMs/jagadish >> * booting... >> fbuf frame buffer base: 0x112245400000 [sz 16777216] >> >> it stays there for as long as I wait until I poweroff the vm. >> >> tried with Debian installer, >> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >> Starting jagadish >> * found guest in /mnt/VMs/jagadish >> * booting... >> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >> >> >> ideally I need to install Debian for my user base on these GPU based >> systems. But even that is proving to be impossible since the installer d= oes >> not move any further from where it is. >> >> Any thoughts/ideas/suggestions what I else I can try to make this work? >> Anything that you have tried that worked? have I missed something? >> >> Would appreciate any and all thoughts/suggestions >> >> best regards >> SS >> > > > -- > Mario. > --=20 Mario. --000000000000575a23063117d498 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Usually I use this kind of script to launch a bhyve v= m :

#!/bin/sh

setxkbmap it
kldload vmm.ko
= vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | awk '/vncv= iewer [0]/{print $6}')"
kldload vmm.ko

if ! pciconf -l p= ci0:2:0:0 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/= 0/0 is not attached to ppt,attaching..."
kldload nvidia-modeset
= devctl clear driver -f pci0:2:0:0
devctl set driver -f pci0:2:0:0 pptelse
echo "rtx 2080ti slot 2/0/0 is already attached to ppt"<= br>fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; thenecho "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."=
devctl clear driver -f pci0:2:0:1
devctl set driver -f pci0:2:0:1 pp= t
else
echo "rtx 2080ti slot 2/0/1 is already attached to ppt&qu= ot;
fi

if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; the= n
echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching...&q= uot;
devctl clear driver -f pci0:2:0:2
devctl set driver -f pci0:2:0:= 2 ppt
else
echo "rtx 2080ti slot 2/0/2 is already attached to pp= t"
fi

if ! pciconf -l pci0:2:0:3 | grep -q "^ppt";= then
echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching.= .."
devctl clear driver -f pci0:2:0:3
devctl set driver -f pci0:= 2:0:3 ppt
else
echo "rtx 2080ti slot 2/0/3 is already attached t= o ppt"
fi

echo "rtx 2080ti is fully attached to ppt&quo= t;

for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve session =3D $sessio= n"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo &qu= ot;vnc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 if ! printf '%s\n' "${vncs}" | grep "= ${session#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf &#= 39;VNC session not found,destroying ghost vms\n'
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 printf 'Found VNC session %s\n' "${session},no g= host vms found,not destroying them"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geom disk list | awk = '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ && d{print d}&#= 39;`
echo "Seagate M3 Portable 1.8T ; $vmdisk0"

vmdisk1= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (201504020= 4055E)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 = 1.8T ; $vmdisk1"

vmdisk2=3D`geom disk list | awk '/^Geom na= me: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{print d}'`
ec= ho "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`geom disk list | a= wk '/^Geom name: /{d=3D$NF} /^ *ident: (20130508005976F)/ && d{= print d}'`
echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3&quo= t;

vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *= ident: (BE0191510218)/ && d{print d}'`
echo "G-DRIVE US= B ; $vmdisk4"

vmdisk5=3D`geom disk list | awk '/^Geom name:= /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{print d}'`
e= cho "Elements 25A3 ; $vmdisk5"

vmdisk6=3D`geom disk list |= awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X797309)/ && = d{print d}'`
echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6&qu= ot;

vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ = *ident: (20140108006C)/ && d{print d}'`
echo "Corsair F= orce 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk list | awk '/^G= eom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ && d{print d}'`=
echo "CT500MX500SSD4 ; $vmdisk8"

vmdisk9=3D`geom disk = list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (0774921DDC4200A6)/ &= ;& d{print d}'`
echo "SanDisk Cruzer-15GB ; $vmdisk9"<= br>
vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *id= ent: (60A44D4138D8F311190A0149)/ && d{print d}'`
echo "= Kingston DataTraveler 2.0 ; $vmdisk10"

vmdisk11=3D`geom disk li= st | awk '/^Geom name: /{d=3D$NF} /^ *ident: (575845483038524844323238)= / && d{print d}'`
echo "WD 2500BMV External ; $vmdisk11= "

vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF}= /^ *ident: (57442E575845323039544433303334)/ && d{print d}'`echo "WD 3200BMV External ; $vmdisk12"

vmdisk13=3D`geom = disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2414E989076B)/ &am= p;& d{print d}'`
echo "CT500BX500SSD1 ; $vmdisk13"
/usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads= =3D1 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/$vmdisk13 \=
-s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbu= f,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \=
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE= .fd \
vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 &<= /div>

As you can see,each vm has a unique ID (connected = with the vncviewer instance) and the vncviewer window is launched as soon a= s the bhyve vm is called...and my Geforce RTX 2080 ti is passed through ins= ide the vm.


On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto= <marietto20= 08@gmail.com> wrote:
First of all,you could use a more flexibl= e technique than using the pptdevs on /boot/loader.conf. I would use a scri= pt like this :

1)

if ! pc= iconf -l pci0:18:0:0 | grep -q "^ppt"; then
echo "18/0/= 0 is not attached to ppt,attaching..."
devctl clear driver -f pci0:= 18:0:0
devctl set driver -f pci0:18:0:0 ppt
else
echo "18/0/0= already attached to ppt"
fi

2) I would not = like to use vm-bhyve. It adds only some confusion...

On Mon, Mar 24, 2= 025 at 3:22=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
=
Go= od afternoon everyone.

I am trying to have VMs with GPU passthrough.= The setup is a Dell server with NVidia Quadro RTX 6000/8000 installed alre= ady. I have checked the device IDs and put pptdevs in place

# cat /b= oot/loader.conf
pptdevs=3D"18/0/0 19/0/0"
pptdevs2=3D"= 72/0/0 73/0/0"

this is showing the GPUs as pptdev on the pcicon= f

ppt0@pci0:18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1= hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D= 0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation= 9;
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/= 8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0= =C2=A0 subclass =C2=A0 =3D 3D
ppt1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 cl= ass=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subven= dor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D &= #39;NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D '= TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 = =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D

as I am us= ing vm-bhyve, I have put the configuration as below

# cat /mnt/VMs/j= agadish/jagadish.conf
loader=3D"uefi"
cpu=3D16
memory=3D= 128G
xhci_mouse=3D"yes"
debug=3D"true"

gra= phics=3D"yes"
graphics_listen=3D"127.0.0.1"
graph= ics_port=3D"5920"
graphics_res=3D"1024x768"
graph= ics_wait=3D"no"
#graphics_vga=3D"io"

network0= _type=3D"virtio-net"
network0_switch=3D"swUNI"
ne= twork0_mac=3D"58:9c:fc:06:3f:af"

disk0_type=3D"nvme&q= uot;
disk0_name=3D"jagadish-disk0.img"
#disk0_size=3D"= 128G"

uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de"
passthru0=3D"19/0/0=3D6:0"

bhyve_options=3D"-A = -H -P"
#END

however, when I start the vm (for OS installatio= n to start with), it shows it is running but I am unable to access the VNC = for a while, then when I finally manage to connect via VNC, there is nothin= g on the screen, just blank dark screen

the vm-bhyve.log shows
Mar 24 13:59:26: initialising
Mar 24 13:59:26: =C2=A0[loader: uefi]Mar 24 13:59:26: =C2=A0[cpu: 16]
Mar 24 13:59:26: =C2=A0[memory: 128G]<= br>Mar 24 13:59:26: =C2=A0[hostbridge: standard]
Mar 24 13:59:26: =C2=A0= [com ports: com1]
Mar 24 13:59:26: =C2=A0[uuid: 966e909b-1293-11ef-a9a4-= e4434bfe34de]
Mar 24 13:59:26: =C2=A0[debug mode: true]
Mar 24 13:59:= 26: =C2=A0[primary disk: jagadish-disk0.img]
Mar 24 13:59:26: =C2=A0[pri= mary disk dev: file]
Mar 24 13:59:26: initialising network device tap0Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:= 26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar 24 13:59:26: bootingMar 24 13:59:26: =C2=A0[bhyve options: -c 16 -m 128G -AHPw -l bootrom,/us= r/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9= a4-e4434bfe34de -u -S]
Mar 24 13:59:26: =C2=A0[bhyve devices: -s 0,hostb= ridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,vir= tio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp= =3D127.0.0.1:5920,w= =3D1024,h=3D768 -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[bhyve consol= e: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bhyve iso devi= ce: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]
M= ar 24 13:59:26: starting bhyve (run 1)

based on what I can see, and = a little out of desparation, I decided to run the installer in the foregrou= nd, and below is what I got

# vm install -f jagadish FreeBSD-14.2-RE= LEASE-amd64-disc1.iso
Starting jagadish
=C2=A0 * found guest in /mnt/= VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0x1122454000= 00 [sz 16777216]

it stays there for as long as I wait until I powero= ff the vm.

tried with Debian installer,
# vm install -f jagadish= debian-12.5.0-amd64-netinst.iso
Starting jagadish
=C2=A0 * found gue= st in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0= x2747e2400000 [sz 16777216]


ideally I need to install Debian for= my user base on these GPU based systems. But even that is proving to be im= possible since the installer does not move any further from where it is.
Any thoughts/ideas/suggestions what I else I can try to make this work= ? Anything that you have tried that worked? have I missed something?
Would appreciate any and all thoughts/suggestions

best regards
<= div>SS


--
Ma= rio.


--
Ma= rio.
--000000000000575a23063117d498--