Date: Wed, 24 May 2023 15:27:18 +0200 From: Mario Marietto <marietto2008@gmail.com> To: Aryeh Friedman <aryeh.friedman@gmail.com>, Odhiambo Washington <odhiambo@gmail.com> Cc: FreeBSD Mailing List <freebsd-questions@freebsd.org> Subject: Re: booting linux on bhyve Message-ID: <CA%2B1FSiio3G3PiSj6XQM5hM8Z799s06ivZ%2BBAdy8RGJ4hbnKORg@mail.gmail.com> In-Reply-To: <CAGBxaXkxY4d1jdRHepT_DbhG-_ZvWXj2ODejZNg35_jYVKW6gg@mail.gmail.com> References: <CAGBxaXkhyoy5LMricm3NY92se4QkSA_R_kbJH8MFYHWqtmAZTA@mail.gmail.com> <CA%2B1FSiiwrQ_roPePfyK0wFsm7c9r-aJE43FjgsALpXWik2U-%2BQ@mail.gmail.com> <CAGBxaXkxY4d1jdRHepT_DbhG-_ZvWXj2ODejZNg35_jYVKW6gg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000a1a22505fc7078e8
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
I've created a script like the one below. Actually I use the same pattern
for every VM I want to use. Every VM has its own number associated. What
changes from one to another is only this line :
-s
1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex=3D=
1 \
That's the name and the location of the raw file. Inside the script I have
added a mechanism to kill a ghosted vm if it does not find the associated
viewer window active. That's because I don't find it useful to use a
ghosted vm. Yes,it is faster,but It causes also troubles in my case. In
addition the script automatically attaches and detaches the pci address
tied to my gpu,because I want to pass through my RTX 2080 ti within a Linux
VM. There is a bug that I want to fix : if I want to boot more than one
VM,the first one will be killed. Shouldn't be too hard to fix it,anyway.
#!/bin/sh
setxkbmap it
bhyvectl --vm=3Dvm0:19 --destroy
vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')"
echo $vms
echo $vncs
if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."
kldload nvidia-modeset
devctl detach pci0:2:0:0
devctl set driver pci0:2:0:0 ppt
else
echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
fi
if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."
devctl detach pci0:2:0:1
devctl set driver pci0:2:0:1 ppt
else
echo "rtx 2080ti slot 2/0/1 is already attached to ppt"
fi
if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
devctl detach pci0:2:0:2
devctl set driver pci0:2:0:2 ppt
else
echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
fi
if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
#devctl detach pci0:2:0:3
devctl set driver pci0:2:0:3 ppt
else
echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
fi
echo "rtx 2080ti is fully attached to ppt"
for vm in $vms; do
session=3D"${vm##*/}"
echo "bhyve session =3D $session"
echo "vnc session =3D $vncs"
if ! printf '%s\n' "${vncs}" | grep
"${session}"; then
printf 'VNC session not found,destroying
ghost vms\n'
bhyvectl --vm=3D$session --destroy
else
printf 'Found VNC session %s\n'
"${session},no ghost vms found,not destroying them"
fi
done
vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (NM13N4=
CZ)/
&& d{print d}'`
echo "Seagate M3 Portable 1.8 TB ; $vmdisk0"
vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(2022285F1175)/ && d{print d}'`
echo "CT1000P1SSD8 ; $vmdisk1"
vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(2015020204055E)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk2"
vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(20130506005976F)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 932G ; $vmdisk3"
vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(BE0191500218)/ && d{print d}'`
echo "G-DRIVE USB ; $vmdisk4"
vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(38434B4237354B45)/ && d{print d}'`
echo "Elements 25A3 ; $vmdisk5"
vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(WD-WCAV2X597309)/ && d{print d}'`
echo "WDC WD3200AAJS-00L7A0-298 GB ; $vmdisk6"
vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(0774911DDC4200A6)/ && d{print d}'`
echo "SanDisk Cruzer-15GB ; $vmdisk7"
vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(60A44C4138D8F311190A0149)/ && d{print d}'`
echo "Kingston DataTraveler 2.0 ; $vmdisk8"
vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(20140100006C)/ && d{print d}'`
echo "Corsair Force 3 SSD ; $vmdisk9"
vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(1924E20B2AE5)/ && d{print d}'`
echo "CT500MX500SSD4 ; $vmdisk10"
vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(575845583038524844323238)/ && d{print d}'`
echo "WD-2500BMV ; $vmdisk11"
vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(57442D575845323039544433303334)/ && d{print d}'`
echo "WD-3200BMV ; $vmdisk12"
vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident:
(S3Z2NB0KB99028V)/ && d{print d}'`
echo "Samsung SSD 860 EVO 500GB ; $vmdisk13"
pkexec zpool import -f -R /mnt/zroot2 zroot2
bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostbridge \
-s
1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex=3D=
1 \
-s 2,nvme,/dev/nvd0 \
-s 4,virtio-blk,/dev/$vmdisk11 \
-s 5,virtio-blk,/dev/$vmdisk12 \
-s 6,virtio-blk,/dev/$vmdisk13 \
-s 8,passthru,5/0/0 \
-s 13,virtio-net,tap19 \
-s 14,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5919,w=3D1600,h=3D950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19 < /dev/null & sleep 2 && vncviewer 0:19
On Wed, May 24, 2023 at 3:10=E2=80=AFPM Aryeh Friedman <aryeh.friedman@gmai=
l.com>
wrote:
> On Wed, May 24, 2023 at 8:33=E2=80=AFAM Mario Marietto <marietto2008@gmai=
l.com>
> wrote:
> >
> > I do it automatically using only some scripts. I don't like to use
> wrappers.
>
> Do you have the scripts... btw I finally gave up on getting GPT
> (actually more accurately Hugging Face Transformers) to run on FreeBSD
> and I did get them to work on this linsucks machine with the only snag
> benign it won't automatically boot... now that I have a better idea of
> what the whole working setup looks like I might try it again on
> FreeBSD when I get some free time.
>
> >
> > On Wed, May 24, 2023 at 1:28=E2=80=AFPM Aryeh Friedman <aryeh.friedman@=
gmail.com>
> wrote:
> >>
> >> I have it so I can get to a uefi shell prompt and manually boot but I
> >> am still missing how to make this all automatic
> >>
> >> --
> >> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
> >>
> >
> >
> > --
> > Mario.
>
>
>
> --
> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
>
--=20
Mario.
--000000000000a1a22505fc7078e8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>I've created a script like the one below. Actuall=
y I use the same pattern for every VM I want to use. Every VM has its own n=
umber associated. What changes from one to another is only this line :</div=
><div><br></div><div>-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubu=
ntu2210.img,bootindex=3D1 \</div><div><br></div><div>That's the name an=
d the location of the raw file. Inside the script I have added a mechanism =
to kill a ghosted vm if it does not find the associated viewer window activ=
e. That's because I don't find it useful to use a ghosted vm. Yes,i=
t is faster,but It causes also troubles in my case. In addition the script =
automatically attaches and detaches the pci address tied to my gpu,because =
I want to pass through my RTX 2080 ti within a Linux VM. There is a bug tha=
t I want to fix : if I want to boot more than one VM,the first one will be =
killed. Shouldn't be too hard to fix it,anyway. <br></div><div><br></di=
v><div>#!/bin/sh</div><br>setxkbmap it<br>bhyvectl --vm=3Dvm0:19 --destroy<=
br>vms=3D"$(ls /dev/vmm/*)"<br>vncs=3D"$(ps ax | awk '/v=
ncviewer [0]/{print $6}')"<br>echo $vms<br>echo $vncs<br><br>if ! =
pciconf -l pci0:2:0:0 | grep -q "^ppt"; then<br>echo "rtx 20=
80ti slot 2/0/0 is not attached to ppt,attaching..."<br>kldload nvidia=
-modeset<br>devctl detach pci0:2:0:0<br>devctl set driver pci0:2:0:0 ppt<br=
>else<br>echo "rtx 2080ti slot 2/0/0 is already attached to ppt"<=
br>fi<br><br>if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then<br=
>echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."=
<br>devctl detach pci0:2:0:1<br>devctl set driver pci0:2:0:1 ppt<br>else<br=
>echo "rtx 2080ti slot 2/0/1 is already attached to ppt"<br>fi<br=
><br>if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then<br>echo &q=
uot;rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."<br>devc=
tl detach pci0:2:0:2<br>devctl set driver pci0:2:0:2 ppt<br>else<br>echo &q=
uot;rtx 2080ti slot 2/0/2 is already attached to ppt"<br>fi<br><br>if =
! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then<br>echo "rtx =
2080ti slot 2/0/3 is not attached to ppt,attaching..."<br>#devctl deta=
ch pci0:2:0:3<br>devctl set driver pci0:2:0:3 ppt<br>else<br>echo "rtx=
2080ti slot 2/0/3 is already attached to ppt"<br>fi<br><br>echo "=
;rtx 2080ti is fully attached to ppt"<br><br>for vm in $vms; do<br>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${vm##=
*/}" =C2=A0 <br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 echo "bhyve session =3D $session"<br>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "vnc session =3D $vncs" =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 i=
f ! printf '%s\n' "${vncs}" | grep "${session}"=
; then<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'VNC session no=
t found,destroying ghost vms\n'<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 bhyvectl --vm=3D$session --destroy =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else<br>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC session %s\n' =
"${session},no ghost vms found,not destroying them"<br>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi<br>done<br><br>vmdisk0=3D`=
geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (NM13N4CZ)/ &a=
mp;& d{print d}'`<br>echo "Seagate M3 Portable 1.8 TB ; $vmdis=
k0"<br><br>vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF=
} /^ *ident: (2022285F1175)/ && d{print d}'`<br>echo "CT10=
00P1SSD8 ; $vmdisk1"<br><br>vmdisk2=3D`geom disk list | awk '/^Geo=
m name: /{d=3D$NF} /^ *ident: (2015020204055E)/ && d{print d}'`=
<br>echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk2"<br><br>vmdi=
sk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (201305=
06005976F)/ && d{print d}'`<br>echo "TOSHIBA External USB =
3.0 932G ; $vmdisk3"<br><br>vmdisk4=3D`geom disk list | awk '/^Geo=
m name: /{d=3D$NF} /^ *ident: (BE0191500218)/ && d{print d}'`<b=
r>echo "G-DRIVE USB ; $vmdisk4"<br><br>vmdisk5=3D`geom disk list =
| awk '/^Geom name: /{d=3D$NF} /^ *ident: (38434B4237354B45)/ &&=
; d{print d}'`<br>echo "Elements 25A3 ; $vmdisk5"<br><br>vmdi=
sk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCA=
V2X597309)/ && d{print d}'`<br>echo "WDC WD3200AAJS-00L7A0=
-298 GB ; $vmdisk6"<br><br>vmdisk7=3D`geom disk list | awk '/^Geom=
name: /{d=3D$NF} /^ *ident: (0774911DDC4200A6)/ && d{print d}'=
`<br>echo "SanDisk Cruzer-15GB ; $vmdisk7"<br><br>vmdisk8=3D`geom=
disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44C4138D8F3111=
90A0149)/ && d{print d}'`<br>echo "Kingston DataTraveler 2=
.0 ; $vmdisk8"<br><br>vmdisk9=3D`geom disk list | awk '/^Geom name=
: /{d=3D$NF} /^ *ident: (20140100006C)/ && d{print d}'`<br>echo=
"Corsair Force 3 SSD ; $vmdisk9"<br><br>vmdisk10=3D`geom disk li=
st | awk '/^Geom name: /{d=3D$NF} /^ *ident: (1924E20B2AE5)/ &&=
d{print d}'`<br>echo "CT500MX500SSD4 ; $vmdisk10"<br><br>vmd=
isk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (5758=
45583038524844323238)/ && d{print d}'`<br>echo "WD-2500BMV=
; $vmdisk11"<br><br>vmdisk12=3D`geom disk list | awk '/^Geom name=
: /{d=3D$NF} /^ *ident: (57442D575845323039544433303334)/ && d{prin=
t d}'`<br>echo "WD-3200BMV ; $vmdisk12"<br><br>vmdisk13=3D`ge=
om disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (S3Z2NB0KB99028V=
)/ && d{print d}'`<br>echo "Samsung SSD 860 EVO 500GB ; $v=
mdisk13"<br><br>pkexec zpool import -f -R /mnt/zroot2 zroot2<br><br>bh=
yve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \<br>-s 0,hostbr=
idge \<br>-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img=
,bootindex=3D1 \<br>-s 2,nvme,/dev/nvd0 \<br>-s 4,virtio-blk,/dev/$vmdisk11=
\<br>-s 5,virtio-blk,/dev/$vmdisk12 \<br>-s 6,virtio-blk,/dev/$vmdisk13 \<=
br>-s 8,passthru,5/0/0 \<br>-s 13,virtio-net,tap19 \<br>-s 14,virtio-9p,sha=
rename=3D/ \<br>-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5919">0.0.0.0:59=
19</a>,w=3D1600,h=3D950,wait \<br>-s 30,xhci,tablet \<br>-s 31,lpc \<br>-l =
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \<br>vm0:19 < =
/dev/null & sleep 2 && vncviewer 0:19<br><br></div><br><div cla=
ss=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, May 24, 20=
23 at 3:10=E2=80=AFPM Aryeh Friedman <<a href=3D"mailto:aryeh.friedman@g=
mail.com">aryeh.friedman@gmail.com</a>> wrote:<br></div><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r=
gb(204,204,204);padding-left:1ex">On Wed, May 24, 2023 at 8:33=E2=80=AFAM M=
ario Marietto <<a href=3D"mailto:marietto2008@gmail.com" target=3D"_blan=
k">marietto2008@gmail.com</a>> wrote:<br>
><br>
> I do it automatically using only some scripts. I don't like to use=
wrappers.<br>
<br>
Do you have the scripts... btw I finally gave up on getting GPT<br>
(actually more accurately Hugging Face Transformers) to run on FreeBSD<br>
and I did get them to work on this linsucks machine with the only snag<br>
benign it won't automatically boot... now that I have a better idea of<=
br>
what the whole working setup looks like I might try it again on<br>
FreeBSD when I get some free time.<br>
<br>
><br>
> On Wed, May 24, 2023 at 1:28=E2=80=AFPM Aryeh Friedman <<a href=3D"=
mailto:aryeh.friedman@gmail.com" target=3D"_blank">aryeh.friedman@gmail.com=
</a>> wrote:<br>
>><br>
>> I have it so I can get to a uefi shell prompt and manually boot bu=
t I<br>
>> am still missing how to make this all automatic<br>
>><br>
>> --<br>
>> Aryeh M. Friedman, Lead Developer, <a href=3D"http://www.PetiteClo=
ud.org" rel=3D"noreferrer" target=3D"_blank">http://www.PetiteCloud.org</a>=
<br>
>><br>
><br>
><br>
> --<br>
> Mario.<br>
<br>
<br>
<br>
-- <br>
Aryeh M. Friedman, Lead Developer, <a href=3D"http://www.PetiteCloud.org" r=
el=3D"noreferrer" target=3D"_blank">http://www.PetiteCloud.org</a><br>;
</blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre=
fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Mario.<br></d=
iv>
--000000000000a1a22505fc7078e8--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSiio3G3PiSj6XQM5hM8Z799s06ivZ%2BBAdy8RGJ4hbnKORg>