From nobody Mon Mar 24 14:21:48 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLwGy46GZz5rySt for ; Mon, 24 Mar 2025 14:22:02 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x1133.google.com (mail-yw1-x1133.google.com [IPv6:2607:f8b0:4864:20::1133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLwGx2qnYz3GY0 for ; Mon, 24 Mar 2025 14:22:01 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=SPMHKRoS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shamim.shahriar@gmail.com designates 2607:f8b0:4864:20::1133 as permitted sender) smtp.mailfrom=shamim.shahriar@gmail.com Received: by mail-yw1-x1133.google.com with SMTP id 00721157ae682-6ef9b8b4f13so38585877b3.2 for ; Mon, 24 Mar 2025 07:22:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742826120; x=1743430920; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=IDkkL/yqijTUoWappBqXP6FNcxQEhm+l6R9xn/rku8Y=; b=SPMHKRoSy+wiDZWYVd3rC3UYQJimyc2N/RkTsqKHc8Yid3mklqwuT1TyvCbFIAPScL BBmJHZYYiQrb6qgZe39gt7Vjdmubv0p3OSAdlhwQ36d5ALMztPGgo2x1LCs0hANyjCuX BeiebeNL6p0TwNCfViiR6+8wTGQm97etg8O0YQ/2FJ1zeBJ78rjAbD2xNv3QzITysXlO D1tlqnvneVM9X2wfn9TE2yW/nunkdpDBkgbXzx2YiwYy+JxNSQz6LvJjbkFrK/iUzy8n qYG1uXWPvbzHAxJVJzuML/Rdva36i8nWkDMlsbxFbS0IHNahs0WDdbPIB5b0MkGOB2Bf 45ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742826120; x=1743430920; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=IDkkL/yqijTUoWappBqXP6FNcxQEhm+l6R9xn/rku8Y=; b=hg5bsAMeSq6bW0X//S/3WLBaIFbk7MqOBG/gYyFBeLxUjDyr/z+dWBW51l2wECltdm iV6SI39GMuRFmIS48mBQHkT8bM6ZTJd1trcT25g8VJOCtT+8c9zArkKGU6C2mwkpstJV fhuzXfoJkvJ9Vx7LMIibam1wXR0rHc3jN5Lef4Xn1j2TGs94HZJpzLhaZxxJBxFSClc6 eMvFad5zx2yvyohTmVLfPuZM2WsuPRYSf1c053bx4jSkf0jlL/Rsye61/G31/bq/gclz t7lnLT9oicHSKac//G/5L8x/9Grjuh8e9vK9vR2vbfz3sdKBmPt4m4jOEzWRkNlXfihu Ig9g== X-Gm-Message-State: AOJu0YzI6I03cXKynKYx8MCtt7Tb27fSmbarRhIJJ44zWUO6uPkvVHIc X9FC3d/AYZZ7Icj6ZIY3YYoNtzyfmqd4864kPTEcfp3I0rDJBKAqsN+pPxcI/qfUxCfJTBvYanm bZTA1eDNpKIMarQMG6xspjZ6lfep+XOM= X-Gm-Gg: ASbGncuIEY0uMU8HObYUuMIyXqIl8DiREt8eUzLDpwomnsB+B8WtEXjnvuZs1LuU6Y/ OYYWJ6DcSmXMS0WhrIkAeJoDDhEwk9SHPPa0vQWZZpb285BVVb/aG2ZHfSo1fdyC9GIIOhzCl0/ tmwvB4ou0eHSmP8eRYMEM01iw= X-Google-Smtp-Source: AGHT+IHIHI6TItr5ZntpzkEV5I3v//iBE86+M983ON69M5NLE4SIXvVQQRlVtU1KdxyjBc1TvT2DguYSSuS8C4m0heA= X-Received: by 2002:a05:690c:c14:b0:700:a988:59dc with SMTP id 00721157ae682-700bad227e3mr165175657b3.31.1742826119948; Mon, 24 Mar 2025 07:21:59 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 From: Shamim Shahriar Date: Mon, 24 Mar 2025 14:21:48 +0000 X-Gm-Features: AQ5f1Jr8OMSWdJomsXDn9dA-bNtvlGMb6AlCdYBLpZbRoAn60Ulz8dyjDVRI7tw Message-ID: Subject: GPU Passthrough with FreeBSD 14.2 bhyve and NVidia Quadro RTX 6000/8000 To: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000c8f694063117538d" X-Spamd-Result: default: False [2.12 / 15.00]; HTTP_TO_IP(1.00)[]; URI_COUNT_ODD(1.00)[1]; NEURAL_SPAM_LONG(1.00)[1.000]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; NEURAL_HAM_SHORT(-0.88)[-0.882]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TAGGED_FROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1133:from]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4ZLwGx2qnYz3GY0 X-Spamd-Bar: ++ --000000000000c8f694063117538d Content-Type: text/plain; charset="UTF-8" Good afternoon everyone. I am trying to have VMs with GPU passthrough. The setup is a Dell server with NVidia Quadro RTX 6000/8000 installed already. I have checked the device IDs and put pptdevs in place # cat /boot/loader.conf pptdevs="18/0/0 19/0/0" pptdevs2="72/0/0 73/0/0" this is showing the GPUs as pptdev on the pciconf ppt0@pci0:18:0:0: class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8 vendor = 'NVIDIA Corporation' device = 'TU102GL [Quadro RTX 6000/8000]' class = display subclass = 3D ppt1@pci0:19:0:0: class=0x030200 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e78 subvendor=0x10de subdevice=0x13d8 vendor = 'NVIDIA Corporation' device = 'TU102GL [Quadro RTX 6000/8000]' class = display subclass = 3D as I am using vm-bhyve, I have put the configuration as below # cat /mnt/VMs/jagadish/jagadish.conf loader="uefi" cpu=16 memory=128G xhci_mouse="yes" debug="true" graphics="yes" graphics_listen="127.0.0.1" graphics_port="5920" graphics_res="1024x768" graphics_wait="no" #graphics_vga="io" network0_type="virtio-net" network0_switch="swUNI" network0_mac="58:9c:fc:06:3f:af" disk0_type="nvme" disk0_name="jagadish-disk0.img" #disk0_size="128G" uuid="966e909b-1293-11ef-a9a4-e4434bfe34de" passthru0="19/0/0=6:0" bhyve_options="-A -H -P" #END however, when I start the vm (for OS installation to start with), it shows it is running but I am unable to access the VNC for a while, then when I finally manage to connect via VNC, there is nothing on the screen, just blank dark screen the vm-bhyve.log shows Mar 24 13:59:26: initialising Mar 24 13:59:26: [loader: uefi] Mar 24 13:59:26: [cpu: 16] Mar 24 13:59:26: [memory: 128G] Mar 24 13:59:26: [hostbridge: standard] Mar 24 13:59:26: [com ports: com1] Mar 24 13:59:26: [uuid: 966e909b-1293-11ef-a9a4-e4434bfe34de] Mar 24 13:59:26: [debug mode: true] Mar 24 13:59:26: [primary disk: jagadish-disk0.img] Mar 24 13:59:26: [primary disk dev: file] Mar 24 13:59:26: initialising network device tap0 Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI addm) Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm) Mar 24 13:59:26: booting Mar 24 13:59:26: [bhyve options: -c 16 -m 128G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S] Mar 24 13:59:26: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadish-disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:06:3f:af -s 6:0,passthru,19/0/0 -s 7:0,fbuf,tcp=127.0.0.1:5920,w=1024,h=768 -s 8:0,xhci,tablet] Mar 24 13:59:26: [bhyve console: -l com1,/dev/nmdm-jagadish.1A] Mar 24 13:59:26: [bhyve iso device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro] Mar 24 13:59:26: starting bhyve (run 1) based on what I can see, and a little out of desparation, I decided to run the installer in the foreground, and below is what I got # vm install -f jagadish FreeBSD-14.2-RELEASE-amd64-disc1.iso Starting jagadish * found guest in /mnt/VMs/jagadish * booting... fbuf frame buffer base: 0x112245400000 [sz 16777216] it stays there for as long as I wait until I poweroff the vm. tried with Debian installer, # vm install -f jagadish debian-12.5.0-amd64-netinst.iso Starting jagadish * found guest in /mnt/VMs/jagadish * booting... fbuf frame buffer base: 0x2747e2400000 [sz 16777216] ideally I need to install Debian for my user base on these GPU based systems. But even that is proving to be impossible since the installer does not move any further from where it is. Any thoughts/ideas/suggestions what I else I can try to make this work? Anything that you have tried that worked? have I missed something? Would appreciate any and all thoughts/suggestions best regards SS --000000000000c8f694063117538d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Good afternoon everyone.

I am trying to have VMs wi= th GPU passthrough. The setup is a Dell server with NVidia Quadro RTX 6000/= 8000 installed already. I have checked the device IDs and put pptdevs in pl= ace

# cat /boot/loader.conf
pptdevs=3D"18/0/0 19/0/0"pptdevs2=3D"72/0/0 73/0/0"

this is showing the GPUs as p= ptdev on the pciconf

ppt0@pci0:18:0:0: =C2=A0 =C2=A0 =C2=A0 class=3D= 0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e78 subvendor=3D= 0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 vendor =C2=A0 =C2=A0 =3D 'NV= IDIA Corporation'
=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =3D 'TU102G= L [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0 class =C2=A0 =C2=A0 =C2=A0= =3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D 3D
ppt1@pci0:19:0:0: = =C2=A0 =C2=A0 =C2=A0 class=3D0x030200 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de= device=3D0x1e78 subvendor=3D0x10de subdevice=3D0x13d8
=C2=A0 =C2=A0 ven= dor =C2=A0 =C2=A0 =3D 'NVIDIA Corporation'
=C2=A0 =C2=A0 device = =C2=A0 =C2=A0 =3D 'TU102GL [Quadro RTX 6000/8000]'
=C2=A0 =C2=A0= class =C2=A0 =C2=A0 =C2=A0=3D display
=C2=A0 =C2=A0 subclass =C2=A0 =3D= 3D

as I am using vm-bhyve, I have put the configuration as below
# cat /mnt/VMs/jagadish/jagadish.conf
loader=3D"uefi"
= cpu=3D16
memory=3D128G
xhci_mouse=3D"yes"
debug=3D"= true"

graphics=3D"yes"
graphics_listen=3D"127= .0.0.1"
graphics_port=3D"5920"
graphics_res=3D"10= 24x768"
graphics_wait=3D"no"
#graphics_vga=3D"io&= quot;

network0_type=3D"virtio-net"
network0_switch=3D&q= uot;swUNI"
network0_mac=3D"58:9c:fc:06:3f:af"

disk= 0_type=3D"nvme"
disk0_name=3D"jagadish-disk0.img"#disk0_size=3D"128G"

uuid=3D"966e909b-1293-11ef-a9a4= -e4434bfe34de"

passthru0=3D"19/0/0=3D6:0"

bhyv= e_options=3D"-A -H -P"
#END

however, when I start the v= m (for OS installation to start with), it shows it is running but I am unab= le to access the VNC for a while, then when I finally manage to connect via= VNC, there is nothing on the screen, just blank dark screen

the vm-= bhyve.log shows

Mar 24 13:59:26: initialising
Mar 24 13:59:26: = =C2=A0[loader: uefi]
Mar 24 13:59:26: =C2=A0[cpu: 16]
Mar 24 13:59:26= : =C2=A0[memory: 128G]
Mar 24 13:59:26: =C2=A0[hostbridge: standard]
= Mar 24 13:59:26: =C2=A0[com ports: com1]
Mar 24 13:59:26: =C2=A0[uuid: 9= 66e909b-1293-11ef-a9a4-e4434bfe34de]
Mar 24 13:59:26: =C2=A0[debug mode:= true]
Mar 24 13:59:26: =C2=A0[primary disk: jagadish-disk0.img]
Mar = 24 13:59:26: =C2=A0[primary disk dev: file]
Mar 24 13:59:26: initialisin= g network device tap0
Mar 24 13:59:26: adding tap0 -> vm-swUNI (swUNI= addm)
Mar 24 13:59:26: bring up tap0 -> vm-swUNI (swUNI addm)
Mar= 24 13:59:26: booting
Mar 24 13:59:26: =C2=A0[bhyve options: -c 16 -m 12= 8G -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -H -P -= U 966e909b-1293-11ef-a9a4-e4434bfe34de -u -S]
Mar 24 13:59:26: =C2=A0[bh= yve devices: -s 0,hostbridge -s 31,lpc -s 4:0,nvme,/mnt/VMs/jagadish/jagadi= sh-disk0.img -s 5:0,virtio-net,tap0,mac=3D58:9c:fc:06:3f:af -s 6:0,passthru= ,19/0/0 -s 7:0,fbuf,tcp=3D127.0.0.1:5920<= /a>,w=3D1024,h=3D768 -s 8:0,xhci,tablet]
Mar 24 13:59:26: =C2=A0[bhyve c= onsole: -l com1,/dev/nmdm-jagadish.1A]
Mar 24 13:59:26: =C2=A0[bhyve iso= device: -s 3:0,ahci-cd,/mnt/VMs/.iso/lubuntu-24.04.1-desktop-amd64.iso,ro]=
Mar 24 13:59:26: starting bhyve (run 1)

based on what I can see,= and a little out of desparation, I decided to run the installer in the for= eground, and below is what I got

# vm install -f jagadish FreeBSD-14= .2-RELEASE-amd64-disc1.iso
Starting jagadish
=C2=A0 * found guest in = /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer base: 0x11224= 5400000 [sz 16777216]

it stays there for as long as I wait until I p= oweroff the vm.

tried with Debian installer,
# vm install -f jag= adish debian-12.5.0-amd64-netinst.iso
Starting jagadish
=C2=A0 * foun= d guest in /mnt/VMs/jagadish
=C2=A0 * booting...
fbuf frame buffer ba= se: 0x2747e2400000 [sz 16777216]


ideally I need to install Debia= n for my user base on these GPU based systems. But even that is proving to = be impossible since the installer does not move any further from where it i= s.

Any thoughts/ideas/suggestions what I else I can try to make this= work? Anything that you have tried that worked? have I missed something?
Would appreciate any and all thoughts/suggestions

best regards=
SS
--000000000000c8f694063117538d--