From nobody Mon Mar 24 15:48:35 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLyCc1cXHz5s4RZ for ; Mon, 24 Mar 2025 15:49:16 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLyCb1pvJz3q5L for ; Mon, 24 Mar 2025 15:49:15 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=TzNVRRj0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::102f as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-2ff799d99dcso7571170a91.1 for ; Mon, 24 Mar 2025 08:49:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742831353; x=1743436153; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=UQxtYh2Btv7uI0Q0CCjPEiMu/aYXx1wCdyierlZ5PMg=; b=TzNVRRj0bK1/s1aEV3IEZKdosFI/h+9fQY1T427S6j6BwQk26f+BpKfTb6CvnnlcDA 6wY3x3R/Rz45eRalYai6UtJjok2tkhzAiPMDso1F7af7YBh/djROwvusxkx5RISjj5sW r8DZZcvwRs0i3a6nZhUo6/u8ECC0QBKVFgBW6F/1UJuBxmcptzPbKj65Q8V/M8HztWLj 5sn/zzMq9ih5D6JyiER68Q2D1Rl8tLeayPPVmcr9v9sSoNIhOD8ixilqS82JLsj42bKV oT8/T6RQbH6VgDS51alKJ3P7qy8jPn5jTZnrCPUdWbQ9tiAr+5xyOJZugipHKRERdTmo /EuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742831353; x=1743436153; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UQxtYh2Btv7uI0Q0CCjPEiMu/aYXx1wCdyierlZ5PMg=; b=k8GURFMdyqlSY9QTdLFBqsXSNTspC6/F0b1jbFJwKFu9XazE14KMTyxz7ZhvSWxNJR u9GYzmYnzulxwS1nm2WEzsVq/xDmchc1xtsW1ZmnLXSxwM9XN4tCtqO7b9sv8JdvIMJ4 u6UYkg/hKX1sVw4feJSEMzXVkwfsygZksAcmIwyoWDLQgtwrPPkSHlSDb69Tamb2T69V i1a49F3j3iZjU43+LPVeH5im4QjmB+3KqpKabnDvkmZHaW+cp4gSQTM/cMSo8+2qXKe2 mdYU5uL5OFrrIArQ8iam/DCZlAn6PJr0GM7E+YYOpoMv+Iqzw1Q9SgwZLsYzJ2wie8pS Ps5A== X-Gm-Message-State: AOJu0YyD3Yz83/bTVQ4ar3ghgid6/rASeVUi9Cxu6uOXEOVoakJhbYZF RJ9uObfyQmk7y1HpKp6EdP27BLS6islKuCVnOtwj3g7XaBBnauzsrgq/jNlmt7OET3Bq1liTZDf pUh0Q+hZKNSjhEBjmnEty7bjvWrLwTjb3 X-Gm-Gg: ASbGncvW7Afo1nvw7wH+BJIUKT24vpGOvFR38VSRB4KgyMdFxouEbMezXOgTSBUMIYQ zlIYsDTZGKZkdAmOnBJJKbhm3fb8FSal6aC1Idgk98flWeNvi86xyJtTlphcji+t9wULKaFRYzT +cwo7YQstp+IPag8hYt4xze03lnJ4= X-Google-Smtp-Source: AGHT+IHRKMNSrYRqmMXa7nuV7zCxdnwkJHYIjGx/RaMu0I0JcfxWOyxR0xdnYC5A2YHHGKzo/TxiRSQDFZ9yhSBL7T0= X-Received: by 2002:a17:90a:d008:b0:2ff:693a:7590 with SMTP id 98e67ed59e1d1-3030ff08de9mr26351330a91.33.1742831353190; Mon, 24 Mar 2025 08:49:13 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 24 Mar 2025 16:48:35 +0100 X-Gm-Features: AQ5f1Jp4wslYlF0ooDdNFX4E0rqtDEM0PIxsiR-u9Wqwr7xagDmZQ1CCMTRmZg8 Message-ID: Subject: Re: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: Shamim Shahriar Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000b5df190631188bc5" X-Spamd-Result: default: False [2.99 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[0.998]; NEURAL_SPAM_SHORT(0.99)[0.993]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102f:from]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLyCb1pvJz3q5L X-Spamd-Bar: ++ --000000000000b5df190631188bc5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ohhh man. I forgot to say that you need to apply the Corvin's patches if you want to pass your gpu to a LINUX vm. And you should apply another patch if you want to pass it to a Windows VM. It seems that the necessary patches have never been uploaded mainstream. On Mon, Mar 24, 2025 at 4:38=E2=80=AFPM Mario Marietto wrote: > Hello Shamim. > > You don't have any bhyve-win. You can use bhyve. I use bhyve-win to pass > through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are two > different executables,because for some reason,my bhyve-lin executable is > not able to pass my GPU in a Windows vm. > > On Mon, Mar 24, 2025 at 4:29=E2=80=AFPM Shamim Shahriar > wrote: > >> Hi Mario >> >> I will try the options you have used later and let you know how it goes. >> Not sure if this is intentional or typo, but I don't recall any bhyve-wi= n >> binary in my system. >> >> Best regards >> SS >> >> On Mon, 24 Mar 2025 at 14:59, Mario Marietto >> wrote: >> >>> Oh sorry. I forgot to add the following bhyve parameters necessary to >>> passthru my GPU :D >>> >>> -s 8:0,passthru,2/0/0 \ >>> -s 8:1,passthru,2/0/1 \ >>> -s 8:2,passthru,2/0/2 \ >>> -s 8:3,passthru,2/0/3 \ >>> >>> On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marietto >>> wrote: >>> >>>> Usually I use this kind of script to launch a bhyve vm : >>>> >>>> #!/bin/sh >>>> >>>> setxkbmap it >>>> kldload vmm.ko >>>> vms=3D"$(ls /dev/vmm/*)" >>>> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >>>> kldload vmm.ko >>>> >>>> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." >>>> kldload nvidia-modeset >>>> devctl clear driver -f pci0:2:0:0 >>>> devctl set driver -f pci0:2:0:0 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/0 is already attached to ppt" >>>> fi >>>> >>>> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:2:0:1 >>>> devctl set driver -f pci0:2:0:1 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/1 is already attached to ppt" >>>> fi >>>> >>>> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:2:0:2 >>>> devctl set driver -f pci0:2:0:2 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/2 is already attached to ppt" >>>> fi >>>> >>>> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then >>>> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." >>>> devctl clear driver -f pci0:2:0:3 >>>> devctl set driver -f pci0:2:0:3 ppt >>>> else >>>> echo "rtx 2080ti slot 2/0/3 is already attached to ppt" >>>> fi >>>> >>>> echo "rtx 2080ti is fully attached to ppt" >>>> >>>> for vm in $vms; do >>>> session=3D"${vm##*/}" >>>> echo "bhyve session =3D $session" >>>> echo "vnc session =3D $vncs" >>>> if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; >>>> then >>>> printf 'VNC session not found,destroying ghost vms\n' >>>> bhyvectl --vm=3D$session --destroy >>>> else >>>> printf 'Found VNC session %s\n' >>>> "${session},no ghost vms found,not destroying them" >>>> fi >>>> done >>>> >>>> vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (AM13N4CZ)/ && d{print d}'` >>>> echo "Seagate M3 Portable 1.8T ; $vmdisk0" >>>> >>>> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (2015040204055E)/ && d{print d}'` >>>> echo "TOSHIBA External USB 3.0 1.8T ; $vmdisk1" >>>> >>>> vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (2027285F1175)/ && d{print d}'` >>>> echo "CT1000P1SSD8 ; $vmdisk2" >>>> >>>> vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (20130508005976F)/ && d{print d}'` >>>> echo "TOSHIBA External USB 3.0 932 GB ; $vmdisk3" >>>> >>>> vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (BE0191510218)/ && d{print d}'` >>>> echo "G-DRIVE USB ; $vmdisk4" >>>> >>>> vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (38234B4237354B45)/ && d{print d}'` >>>> echo "Elements 25A3 ; $vmdisk5" >>>> >>>> vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (WD-WCAV2X797309)/ && d{print d}'` >>>> echo "WDC WD3200AAJS-00L7A0 ; 298 GB ; $vmdisk6" >>>> >>>> vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (20140108006C)/ && d{print d}'` >>>> echo "Corsair Force 3 SSD ; $vmdisk7" >>>> >>>> vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (1924E50B2AE5)/ && d{print d}'` >>>> echo "CT500MX500SSD4 ; $vmdisk8" >>>> >>>> vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (0774921DDC4200A6)/ && d{print d}'` >>>> echo "SanDisk Cruzer-15GB ; $vmdisk9" >>>> >>>> vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (60A44D4138D8F311190A0149)/ && d{print d}'` >>>> echo "Kingston DataTraveler 2.0 ; $vmdisk10" >>>> >>>> vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (575845483038524844323238)/ && d{print d}'` >>>> echo "WD 2500BMV External ; $vmdisk11" >>>> >>>> vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (57442E575845323039544433303334)/ && d{print d}'` >>>> echo "WD 3200BMV External ; $vmdisk12" >>>> >>>> vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: >>>> (2414E989076B)/ && d{print d}'` >>>> echo "CT500BX500SSD1 ; $vmdisk13" >>>> >>>> /usr/sbin/./bhyve-win -S -c sockets=3D4,cores=3D1,threads=3D1 -m 4G -w= -H -A \ >>>> -s 0,hostbridge \ >>>> -s 1,ahci-hd,/dev/$vmdisk13 \ >>>> -s 10,virtio-net,tap6 \ >>>> -s 11,virtio-9p,sharename=3D/ \ >>>> -s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \ >>>> -s 31,lpc \ >>>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ >>>> vm0:6 < /dev/null & sleep 5 && vncviewer 0:6 & >>>> >>>> As you can see,each vm has a unique ID (connected with the vncviewer >>>> instance) and the vncviewer window is launched as soon as the bhyve vm= is >>>> called...and my Geforce RTX 2080 ti is passed through inside the vm. >>>> >>>> >>>> On Mon, Mar 24, 2025 at 3:47=E2=80=AFPM Mario Marietto >>>> wrote: >>>> >>>>> First of all,you could use a more flexible technique than using the >>>>> pptdevs on /boot/loader.conf. I would use a script like this : >>>>> >>>>> 1) >>>>> >>>>> if ! pciconf -l pci0:18:0:0 | grep -q "^ppt"; then >>>>> echo "18/0/0 is not attached to ppt,attaching..." >>>>> devctl clear driver -f pci0:18:0:0 >>>>> devctl set driver -f pci0:18:0:0 ppt >>>>> else >>>>> echo "18/0/0 already attached to ppt" >>>>> fi >>>>> >>>>> 2) I would not like to use vm-bhyve. It adds only some confusion... >>>>> >>>>> On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar < >>>>> shamim.shahriar@gmail.com> wrote: >>>>> >>>>>> Good afternoon everyone. >>>>>> >>>>>> I am trying to have VMs with GPU passthrough. The setup is a Dell >>>>>> server with NVidia Quadro RTX 6000/8000 installed already. I have ch= ecked >>>>>> the device IDs and put pptdevs in place >>>>>> >>>>>> # cat /boot/loader.conf >>>>>> pptdevs=3D"18/0/0 19/0/0" >>>>>> pptdevs2=3D"72/0/0 73/0/0" >>>>>> >>>>>> this is showing the GPUs as pptdev on the pciconf >>>>>> >>>>>> ppt0@pci0:18:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d= 8 >>>>>> vendor =3D 'NVIDIA Corporation' >>>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>>> class =3D display >>>>>> subclass =3D 3D >>>>>> ppt1@pci0:19:0:0: class=3D0x030200 rev=3D0xa1 hdr=3D0x00 >>>>>> vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d= 8 >>>>>> vendor =3D 'NVIDIA Corporation' >>>>>> device =3D 'TU102GL [Quadro RTX 6000/8000]' >>>>>> class =3D display >>>>>> subclass =3D 3D >>>>>> >>>>>> as I am using vm-bhyve, I have put the configuration as below >>>>>> >>>>>> # cat /mnt/VMs/jagadish/jagadish.conf >>>>>> loader=3D"uefi" >>>>>> cpu=3D16 >>>>>> memory=3D128G >>>>>> xhci_mouse=3D"yes" >>>>>> debug=3D"true" >>>>>> >>>>>> graphics=3D"yes" >>>>>> graphics_listen=3D"127.0.0.1" >>>>>> graphics_port=3D"5920" >>>>>> graphics_res=3D"1024x768" >>>>>> graphics_wait=3D"no" >>>>>> #graphics_vga=3D"io" >>>>>> >>>>>> network0_type=3D"virtio-net" >>>>>> network0_switch=3D"swUNI" >>>>>> network0_mac=3D"58:9c:fc:06:3f:af" >>>>>> >>>>>> disk0_type=3D"nvme" >>>>>> disk0_name=3D"jagadish-disk0.img" >>>>>> #disk0_size=3D"128G" >>>>>> >>>>>> uuid=3D"966e909b-1293-11ef-a9a4-e4434bfe34de" >>>>>> >>>>>> passthru0=3D"19/0/0=3D6:0" >>>>>> >>>>>> bhyve_options=3D"-A -H -P" >>>>>> #END >>>>>> >>>>>> however, when I start the vm (for OS installation to start with), it >>>>>> shows it is running but I am unable to access the VNC for a while, t= hen >>>>>> when I finally manage to connect via VNC, there is nothing on the sc= reen, >>>>>> just blank dark screen >>>>>> >>>>>> the vm-bhyve.log shows >>>>>> >>>>>> Mar 24 13:59:26: initialising >>>>>> Mar 24 13:59:26: [loader: uefi] >>>>>> Mar 24 13:59:26: [cpu: 16] >>>>>> Mar 24 13:59:26: [memory: 128G] >>>>>> Mar 24 13:59:26: [hostbridge: standard] >>>>>> Mar 24 13:59:26: [com ports: com1] >>>>>> Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] >>>>>> Mar 24 13:59:26: [debug mode: true] >>>>>> Mar 24 13:59:26: [primary disk: jagadish-disk0.img] >>>>>> Mar 24 13:59:26: [primary disk dev: file] >>>>>> Mar 24 13:59:26: initialising network device tap0 >>>>>> Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) >>>>>> Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) >>>>>> Mar 24 13:59:26: booting >>>>>> Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l >>>>>> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U >>>>>> 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] >>>>>> Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s >>>>>> 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s >>>>>> 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -= s >>>>>> 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet] >>>>>> Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] >>>>>> Mar 24 13:59:26: [bhyve iso device: -s >>>>>> 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] >>>>>> Mar 24 13:59:26: starting bhyve (run 1) >>>>>> >>>>>> based on what I can see, and a little out of desparation, I decided >>>>>> to run the installer in the foreground, and below is what I got >>>>>> >>>>>> # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso >>>>>> Starting jagadish >>>>>> * found guest in /mnt/VMs/jagadish >>>>>> * booting... >>>>>> fbuf frame buffer base: 0x112245400000 [sz 16777216] >>>>>> >>>>>> it stays there for as long as I wait until I poweroff the vm. >>>>>> >>>>>> tried with Debian installer, >>>>>> # vm install -f jagadish debian-12.5.0-amd64-netinst.iso >>>>>> Starting jagadish >>>>>> * found guest in /mnt/VMs/jagadish >>>>>> * booting... >>>>>> fbuf frame buffer base: 0x2747e2400000 [sz 16777216] >>>>>> >>>>>> >>>>>> ideally I need to install Debian for my user base on these GPU based >>>>>> systems. But even that is proving to be impossible since the install= er does >>>>>> not move any further from where it is. >>>>>> >>>>>> Any thoughts/ideas/suggestions what I else I can try to make this >>>>>> work? Anything that you have tried that worked? have I missed someth= ing? >>>>>> >>>>>> Would appreciate any and all thoughts/suggestions >>>>>> >>>>>> best regards >>>>>> SS >>>>>> >>>>> >>>>> >>>>> -- >>>>> Mario. >>>>> >>>> >>>> >>>> -- >>>> Mario. >>>> >>> >>> >>> -- >>> Mario. >>> >> > > -- > Mario. > --=20 Mario. --000000000000b5df190631188bc5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ohhh man.

I forgot to s= ay that you need to apply the Corvin's patches if you want to pass your= gpu to a LINUX vm. And you should apply another patch if you want to pass = it to a Windows VM. It seems that the necessary patches have never been upl= oaded mainstream.

On Mon, Mar 24, 2025 at 4:38=E2=80=AFPM Mario= Marietto <m= arietto2008@gmail.com> wrote:
Hello Shamim.

You don't have any bhyve-win. You can use bhyve. I use bhyve-win= to pass through my GPU in a Windows vm,bhyve-lin for a Linux vm. They are = two different executables,because for some reason,my bhyve-lin executable i= s not able to pass my GPU in a Windows vm.=C2=A0

On Mon, Mar 24, 2025= at 4:29=E2=80=AFPM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Hi Ma= rio

I will try the options you have used later and let y= ou know how it goes. Not sure if this is intentional or typo, but I don'= ;t recall any bhyve-win binary in my system.

Best = regards
SS

On Mon, 24 Mar 2025 at 14:59, Mario Marietto <= marietto2008@gm= ail.com> wrote:
Oh sorry. I forgot to add the following bhyve = parameters necessary to passthru my GPU :D

-s 8:0,= passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \-s 8:3,passthru,2/0/3 \

On Mon, Mar 24, 2025 at 3:57=E2=80=AFPM Mario Marie= tto <mariett= o2008@gmail.com> wrote:
Usually I use this kind of script to l= aunch a bhyve vm :

#!/bin/sh

setxkbmap it
kld= load vmm.ko
vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax |= awk '/vncviewer [0]/{print $6}')"
kldload vmm.ko

if= ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
echo "rtx= 2080ti slot 2/0/0 is not attached to ppt,attaching..."
kldload nvi= dia-modeset
devctl clear driver -f pci0:2:0:0
devctl set driver -f pc= i0:2:0:0 ppt
else
echo "rtx 2080ti slot 2/0/0 is already attache= d to ppt"
fi

if ! pciconf -l pci0:2:0:1 | grep -q "^ppt= "; then
echo "rtx 2080ti slot 2/0/1 is not attached to ppt,att= aching..."
devctl clear driver -f pci0:2:0:1
devctl set driver -= f pci0:2:0:1 ppt
else
echo "rtx 2080ti slot 2/0/1 is already att= ached to ppt"
fi

if ! pciconf -l pci0:2:0:2 | grep -q "= ^ppt"; then
echo "rtx 2080ti slot 2/0/2 is not attached to ppt= ,attaching..."
devctl clear driver -f pci0:2:0:2
devctl set driv= er -f pci0:2:0:2 ppt
else
echo "rtx 2080ti slot 2/0/2 is already= attached to ppt"
fi

if ! pciconf -l pci0:2:0:3 | grep -q &q= uot;^ppt"; then
echo "rtx 2080ti slot 2/0/3 is not attached to= ppt,attaching..."
devctl clear driver -f pci0:2:0:3
devctl set = driver -f pci0:2:0:3 ppt
else
echo "rtx 2080ti slot 2/0/3 is alr= eady attached to ppt"
fi

echo "rtx 2080ti is fully atta= ched to ppt"

for vm in $vms; do
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${vm##*/}" =C2=A0
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "bhyve sessi= on =3D $session"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 echo "vnc session =3D $vncs"
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! printf '%s\n' "${vncs}&qu= ot; | grep "${session#vm}"; then =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 printf 'VNC session not found,destroying ghost vms\n'=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyvectl --vm=3D$session --destroy=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC session %s\n' &qu= ot;${session},no ghost vms found,not destroying them"
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
done

vmdisk0=3D`geo= m disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (AM13N4CZ)/ &= & d{print d}'`
echo "Seagate M3 Portable 1.8T ; $vmdisk0&qu= ot;

vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ = *ident: (2015040204055E)/ && d{print d}'`
echo "TOSHIBA= External USB 3.0 1.8T ; $vmdisk1"

vmdisk2=3D`geom disk list | = awk '/^Geom name: /{d=3D$NF} /^ *ident: (2027285F1175)/ && d{pr= int d}'`
echo "CT1000P1SSD8 ; $vmdisk2"

vmdisk3=3D`= geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (2013050800597= 6F)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 932= GB ; $vmdisk3"

vmdisk4=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (BE0191510218)/ && d{print d}'`
ech= o "G-DRIVE USB ; $vmdisk4"

vmdisk5=3D`geom disk list | awk= '/^Geom name: /{d=3D$NF} /^ *ident: (38234B4237354B45)/ && d{p= rint d}'`
echo "Elements 25A3 ; $vmdisk5"

vmdisk6= =3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X= 797309)/ && d{print d}'`
echo "WDC WD3200AAJS-00L7A0 ; = 298 GB ; $vmdisk6"

vmdisk7=3D`geom disk list | awk '/^Geom = name: /{d=3D$NF} /^ *ident: (20140108006C)/ && d{print d}'`
= echo "Corsair Force 3 SSD ; $vmdisk7"

vmdisk8=3D`geom disk= list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E50B2AE5)/ &&a= mp; d{print d}'`
echo "CT500MX500SSD4 ; $vmdisk8"

v= mdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (077= 4921DDC4200A6)/ && d{print d}'`
echo "SanDisk Cruzer-15= GB ; $vmdisk9"

vmdisk10=3D`geom disk list | awk '/^Geom nam= e: /{d=3D$NF} /^ *ident: (60A44D4138D8F311190A0149)/ && d{print d}&= #39;`
echo "Kingston DataTraveler 2.0 ; $vmdisk10"

vmdi= sk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57584= 5483038524844323238)/ && d{print d}'`
echo "WD 2500BMV = External ; $vmdisk11"

vmdisk12=3D`geom disk list | awk '/^G= eom name: /{d=3D$NF} /^ *ident: (57442E575845323039544433303334)/ &&= ; d{print d}'`
echo "WD 3200BMV External ; $vmdisk12"
<= br>vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:= (2414E989076B)/ && d{print d}'`
echo "CT500BX500SSD1 ;= $vmdisk13"

/usr/sbin/./bhyve-win -S -c sockets=3D4= ,cores=3D1,threads=3D1 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-h= d,/dev/$vmdisk13 \
-s 10,virtio-net,tap6 \
-s 11,virtio-9p,sharename= =3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5906,w=3D1600,h=3D950 \
-s 31,lpc \
-l bootrom,/usr/loc= al/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:6 < /dev/null & sl= eep 5 && vncviewer 0:6 &

As you can se= e,each vm has a unique ID (connected with the vncviewer instance) and the v= ncviewer window is launched as soon as the bhyve vm is called...and my Gefo= rce RTX 2080 ti is passed through inside the vm.


On Mon, Mar 24, 2= 025 at 3:47=E2=80=AFPM Mario Marietto <marietto2008@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">
Firs= t of all,you could use a more flexible technique than using the pptdevs on = /boot/loader.conf. I would use a script like this :

1)

if ! pciconf -l pci0:18:0:0 | grep -q "^= ppt"; then
echo "18/0/0 is not attached to ppt,attaching...&= quot;
devctl clear driver -f pci0:18:0:0
devctl set driver -f pci0:18= :0:0 ppt
else
echo "18/0/0 already attached to ppt"
fi

2) I would not like to use vm-bhyve. It adds only som= e confusion...

On Mon, Mar 24, 2025 at 3:22=E2=80=AFPM Shamim Shahriar= <shamim.= shahriar@gmail.com> wrote:
Good afternoon everyone.

I am try= ing to have VMs with GPU passthrough. The setup is a Dell server with NVidi= a Quadro RTX 6000/8000 installed already. I have checked the device IDs and= put pptdevs in place

# cat /boot/loader.conf
pptdevs=3D"18/= 0/0 19/0/0"
pptdevs2=3D"72/0/0 73/0/0"

this is sho= wing the GPUs as pptdev on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2= =A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D= 0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0= =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2= =A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt= 1@pci0:19:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 = vendor=3D0x10de device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
= =C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2= =A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'= ;
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 s= ubclass =C2=A0 =3D 3D

as I am using vm-bhyve, I have put the configu= ration as below

# cat /mnt/VMs/jagadish/jagadish.conf
loader=3D&q= uot;uefi"
cpu=3D16
memory=3D128G
xhci_mouse=3D"yes"=
debug=3D"true"

graphics=3D"yes"
graphics_= listen=3D"127.0.0.1"
graphics_port=3D"5920"
graph= ics_res=3D"1024x768"
graphics_wait=3D"no"
#graphi= cs_vga=3D"io"

network0_type=3D"virtio-net"
ne= twork0_switch=3D"swUNI"
network0_mac=3D"58:9c:fc:06:3f:af= "

disk0_type=3D"nvme"
disk0_name=3D"jagadish-= disk0.img"
#disk0_size=3D"128G"

uuid=3D"966e9= 09b-1293-11ef-a9a4-e4434bfe34de"

passthru0=3D"19/0/0=3D6:0= "

bhyve_options=3D"-A -H -P"
#END

however, = when I start the vm (for OS installation to start with), it shows it is run= ning but I am unable to access the VNC for a while, then when I finally man= age to connect via VNC, there is nothing on the screen, just blank dark scr= een

the vm-bhyve.log shows

Mar 24 13:59:26: initialising
M= ar 24 13:59:26: =C2=A0[loader: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]Mar 24 13:59:26: =C2=A0[memory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridg= e: standard]
Mar 24 13:59:26: =C2=A0[com ports: com1]
Mar 24 13:59:26= : =C2=A0[uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: = =C2=A0[debug mode: true]
Mar 24 13:59:26: =C2=A0[primary disk: jagadish-= disk0.img]
Mar 24 13:59:26: =C2=A0[primary disk dev: file]
Mar 24 13:= 59:26: initialising network device tap0
Mar 24 13:59:26: adding tap0 -&g= t; vm-swUNI (swUNI addm)
Mar 24 13:59:26: bring up tap0 -> vm-swUNI (= swUNI addm)
Mar 24 13:59:26: booting
Mar 24 13:59:26: =C2=A0[bhyve op= tions: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_= UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 1= 3:59:26: =C2=A0[bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/V= Ms/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:= af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920,w=3D1024,h=3D768 -s 8:0,xhci,tablet= ]
Mar 24 13:59:26: =C2=A0[bhyve console: -l com1,/dev/nmdm-jagadish.1A]<= br>Mar 24 13:59:26: =C2=A0[bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/l= ubuntu-24.04.1-desktop-amd64.iso,ro]
Mar 24 13:59:26: starting bhyve (ru= n 1)

based on what I can see, and a little out of desparation, I dec= ided to run the installer in the foreground, and below is what I got
# vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso
Starting = jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 * booting..= .
fbuf frame buffer base: 0x112245400000 [sz 16777216]

it stays t= here for as long as I wait until I poweroff the vm.

tried with Debia= n installer,
# vm install -f jagadish debian-12.5.0-amd64-netinst.isoStarting jagadish
=C2=A0 * found guest in /mnt/VMs/jagadish
=C2=A0 = * booting...
fbuf frame buffer base: 0x2747e2400000 [sz 16777216]

ideally I need to install Debian for my user base on these GPU based s= ystems. But even that is proving to be impossible since the installer does = not move any further from where it is.

Any thoughts/ideas/suggestion= s what I else I can try to make this work? Anything that you have tried tha= t worked? have I missed something?

Would appreciate any and all thou= ghts/suggestions

best regards
SS


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.


--
Ma= rio.
--000000000000b5df190631188bc5--