Date: Wed, 24 May 2023 22:40:25 +0200 From: Mario Marietto <marietto2008@gmail.com> To: Aryeh Friedman <aryeh.friedman@gmail.com>, Odhiambo Washington <odhiambo@gmail.com> Cc: FreeBSD Mailing List <freebsd-questions@freebsd.org> Subject: Re: booting linux on bhyve Message-ID: <CA%2B1FSihSEteKSzeZPBhvBWLiqzQQDSCzN9xypQPdj4bLXdaHow@mail.gmail.com> In-Reply-To: <CA%2B1FSiio3G3PiSj6XQM5hM8Z799s06ivZ%2BBAdy8RGJ4hbnKORg@mail.gmail.com> References: <CAGBxaXkhyoy5LMricm3NY92se4QkSA_R_kbJH8MFYHWqtmAZTA@mail.gmail.com> <CA%2B1FSiiwrQ_roPePfyK0wFsm7c9r-aJE43FjgsALpXWik2U-%2BQ@mail.gmail.com> <CAGBxaXkxY4d1jdRHepT_DbhG-_ZvWXj2ODejZNg35_jYVKW6gg@mail.gmail.com> <CA%2B1FSiio3G3PiSj6XQM5hM8Z799s06ivZ%2BBAdy8RGJ4hbnKORg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000096438c05fc76852c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I've fixed the bug. This line : if ! printf '%s\n' "${vncs}" | grep "${session}"; then becomes like this : if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then and more vms can boot at the same time. On Wed, May 24, 2023 at 3:27=E2=80=AFPM Mario Marietto <marietto2008@gmail.= com> wrote: > I've created a script like the one below. Actually I use the same pattern > for every VM I want to use. Every VM has its own number associated. What > changes from one to another is only this line : > > -s > 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex= =3D1 \ > > That's the name and the location of the raw file. Inside the script I hav= e > added a mechanism to kill a ghosted vm if it does not find the associated > viewer window active. That's because I don't find it useful to use a > ghosted vm. Yes,it is faster,but It causes also troubles in my case. In > addition the script automatically attaches and detaches the pci address > tied to my gpu,because I want to pass through my RTX 2080 ti within a Lin= ux > VM. There is a bug that I want to fix : if I want to boot more than one > VM,the first one will be killed. Shouldn't be too hard to fix it,anyway. > > #!/bin/sh > > setxkbmap it > bhyvectl --vm=3Dvm0:19 --destroy > vms=3D"$(ls /dev/vmm/*)" > vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" > echo $vms > echo $vncs > > if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..." > kldload nvidia-modeset > devctl detach pci0:2:0:0 > devctl set driver pci0:2:0:0 ppt > else > echo "rtx 2080ti slot 2/0/0 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..." > devctl detach pci0:2:0:1 > devctl set driver pci0:2:0:1 ppt > else > echo "rtx 2080ti slot 2/0/1 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..." > devctl detach pci0:2:0:2 > devctl set driver pci0:2:0:2 ppt > else > echo "rtx 2080ti slot 2/0/2 is already attached to ppt" > fi > > if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then > echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..." > #devctl detach pci0:2:0:3 > devctl set driver pci0:2:0:3 ppt > else > echo "rtx 2080ti slot 2/0/3 is already attached to ppt" > fi > > echo "rtx 2080ti is fully attached to ppt" > > for vm in $vms; do > session=3D"${vm##*/}" > echo "bhyve session =3D $session" > echo "vnc session =3D $vncs" > if ! printf '%s\n' "${vncs}" | grep > "${session}"; then > printf 'VNC session not found,destroying > ghost vms\n' > bhyvectl --vm=3D$session --destroy > > else > printf 'Found VNC session %s\n' > "${session},no ghost vms found,not destroying them" > fi > done > > vmdisk0=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (NM13N4CZ)/ && d{print d}'` > echo "Seagate M3 Portable 1.8 TB ; $vmdisk0" > > vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2022285F1175)/ && d{print d}'` > echo "CT1000P1SSD8 ; $vmdisk1" > > vmdisk2=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (2015020204055E)/ && d{print d}'` > echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk2" > > vmdisk3=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (20130506005976F)/ && d{print d}'` > echo "TOSHIBA External USB 3.0 932G ; $vmdisk3" > > vmdisk4=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (BE0191500218)/ && d{print d}'` > echo "G-DRIVE USB ; $vmdisk4" > > vmdisk5=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (38434B4237354B45)/ && d{print d}'` > echo "Elements 25A3 ; $vmdisk5" > > vmdisk6=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (WD-WCAV2X597309)/ && d{print d}'` > echo "WDC WD3200AAJS-00L7A0-298 GB ; $vmdisk6" > > vmdisk7=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (0774911DDC4200A6)/ && d{print d}'` > echo "SanDisk Cruzer-15GB ; $vmdisk7" > > vmdisk8=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (60A44C4138D8F311190A0149)/ && d{print d}'` > echo "Kingston DataTraveler 2.0 ; $vmdisk8" > > vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (20140100006C)/ && d{print d}'` > echo "Corsair Force 3 SSD ; $vmdisk9" > > vmdisk10=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (1924E20B2AE5)/ && d{print d}'` > echo "CT500MX500SSD4 ; $vmdisk10" > > vmdisk11=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (575845583038524844323238)/ && d{print d}'` > echo "WD-2500BMV ; $vmdisk11" > > vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (57442D575845323039544433303334)/ && d{print d}'` > echo "WD-3200BMV ; $vmdisk12" > > vmdisk13=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: > (S3Z2NB0KB99028V)/ && d{print d}'` > echo "Samsung SSD 860 EVO 500GB ; $vmdisk13" > > pkexec zpool import -f -R /mnt/zroot2 zroot2 > > bhyve -S -c sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \ > -s 0,hostbridge \ > -s > 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex= =3D1 \ > -s 2,nvme,/dev/nvd0 \ > -s 4,virtio-blk,/dev/$vmdisk11 \ > -s 5,virtio-blk,/dev/$vmdisk12 \ > -s 6,virtio-blk,/dev/$vmdisk13 \ > -s 8,passthru,5/0/0 \ > -s 13,virtio-net,tap19 \ > -s 14,virtio-9p,sharename=3D/ \ > -s 29,fbuf,tcp=3D0.0.0.0:5919,w=3D1600,h=3D950,wait \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ > vm0:19 < /dev/null & sleep 2 && vncviewer 0:19 > > > On Wed, May 24, 2023 at 3:10=E2=80=AFPM Aryeh Friedman <aryeh.friedman@gm= ail.com> > wrote: > >> On Wed, May 24, 2023 at 8:33=E2=80=AFAM Mario Marietto <marietto2008@gma= il.com> >> wrote: >> > >> > I do it automatically using only some scripts. I don't like to use >> wrappers. >> >> Do you have the scripts... btw I finally gave up on getting GPT >> (actually more accurately Hugging Face Transformers) to run on FreeBSD >> and I did get them to work on this linsucks machine with the only snag >> benign it won't automatically boot... now that I have a better idea of >> what the whole working setup looks like I might try it again on >> FreeBSD when I get some free time. >> >> > >> > On Wed, May 24, 2023 at 1:28=E2=80=AFPM Aryeh Friedman < >> aryeh.friedman@gmail.com> wrote: >> >> >> >> I have it so I can get to a uefi shell prompt and manually boot but I >> >> am still missing how to make this all automatic >> >> >> >> -- >> >> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org >> >> >> > >> > >> > -- >> > Mario. >> >> >> >> -- >> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org >> > > > -- > Mario. > --=20 Mario. --00000000000096438c05fc76852c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>I've fixed the bug. This line :</div><div><br></d= iv><div>if ! printf '%s\n' "${vncs}" | grep "${sessi= on}"; then</div><div><br></div><div>becomes like this :</div><div><br>= </div><div>if ! printf '%s\n' "${vncs}" | grep "${se= ssion#vm}"; then</div><div><br></div><div>and more vms can boot at the= same time.<br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Wed, May 24, 2023 at 3:27=E2=80=AFPM Mario Marietto= <<a href=3D"mailto:marietto2008@gmail.com">marietto2008@gmail.com</a>&g= t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div d= ir=3D"ltr"><div>I've created a script like the one below. Actually I us= e the same pattern for every VM I want to use. Every VM has its own number = associated. What changes from one to another is only this line :</div><div>= <br></div><div>-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu221= 0.img,bootindex=3D1 \</div><div><br></div><div>That's the name and the = location of the raw file. Inside the script I have added a mechanism to kil= l a ghosted vm if it does not find the associated viewer window active. Tha= t's because I don't find it useful to use a ghosted vm. Yes,it is f= aster,but It causes also troubles in my case. In addition the script automa= tically attaches and detaches the pci address tied to my gpu,because I want= to pass through my RTX 2080 ti within a Linux VM. There is a bug that I wa= nt to fix : if I want to boot more than one VM,the first one will be killed= . Shouldn't be too hard to fix it,anyway. <br></div><div><br></div><div= >#!/bin/sh</div><br>setxkbmap it<br>bhyvectl --vm=3Dvm0:19 --destroy<br>vms= =3D"$(ls /dev/vmm/*)"<br>vncs=3D"$(ps ax | awk '/vncview= er [0]/{print $6}')"<br>echo $vms<br>echo $vncs<br><br>if ! pcicon= f -l pci0:2:0:0 | grep -q "^ppt"; then<br>echo "rtx 2080ti s= lot 2/0/0 is not attached to ppt,attaching..."<br>kldload nvidia-modes= et<br>devctl detach pci0:2:0:0<br>devctl set driver pci0:2:0:0 ppt<br>else<= br>echo "rtx 2080ti slot 2/0/0 is already attached to ppt"<br>fi<= br><br>if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then<br>echo = "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."<br>de= vctl detach pci0:2:0:1<br>devctl set driver pci0:2:0:1 ppt<br>else<br>echo = "rtx 2080ti slot 2/0/1 is already attached to ppt"<br>fi<br><br>i= f ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then<br>echo "rt= x 2080ti slot 2/0/2 is not attached to ppt,attaching..."<br>devctl det= ach pci0:2:0:2<br>devctl set driver pci0:2:0:2 ppt<br>else<br>echo "rt= x 2080ti slot 2/0/2 is already attached to ppt"<br>fi<br><br>if ! pcic= onf -l pci0:2:0:3 | grep -q "^ppt"; then<br>echo "rtx 2080ti= slot 2/0/3 is not attached to ppt,attaching..."<br>#devctl detach pci= 0:2:0:3<br>devctl set driver pci0:2:0:3 ppt<br>else<br>echo "rtx 2080t= i slot 2/0/3 is already attached to ppt"<br>fi<br><br>echo "rtx 2= 080ti is fully attached to ppt"<br><br>for vm in $vms; do<br>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 session=3D"${vm##*/}&= quot; =C2=A0 <br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ec= ho "bhyve session =3D $session"<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "vnc session =3D $vncs" =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ! = printf '%s\n' "${vncs}" | grep "${session}"; th= en<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 printf 'VNC session not foun= d,destroying ghost vms\n'<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bhyve= ctl --vm=3D$session --destroy =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<br>=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else<br>=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 printf 'Found VNC session %s\n' "${se= ssion},no ghost vms found,not destroying them"<br>=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi<br>done<br><br>vmdisk0=3D`geom disk = list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (NM13N4CZ)/ && d= {print d}'`<br>echo "Seagate M3 Portable 1.8 TB ; $vmdisk0"<b= r><br>vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *iden= t: (2022285F1175)/ && d{print d}'`<br>echo "CT1000P1SSD8 ;= $vmdisk1"<br><br>vmdisk2=3D`geom disk list | awk '/^Geom name: /{= d=3D$NF} /^ *ident: (2015020204055E)/ && d{print d}'`<br>echo &= quot;TOSHIBA External USB 3.0 1.8 TB ; $vmdisk2"<br><br>vmdisk3=3D`geo= m disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20130506005976F)= / && d{print d}'`<br>echo "TOSHIBA External USB 3.0 932G ;= $vmdisk3"<br><br>vmdisk4=3D`geom disk list | awk '/^Geom name: /{= d=3D$NF} /^ *ident: (BE0191500218)/ && d{print d}'`<br>echo &qu= ot;G-DRIVE USB ; $vmdisk4"<br><br>vmdisk5=3D`geom disk list | awk '= ;/^Geom name: /{d=3D$NF} /^ *ident: (38434B4237354B45)/ && d{print = d}'`<br>echo "Elements 25A3 ; $vmdisk5"<br><br>vmdisk6=3D`geo= m disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (WD-WCAV2X597309)= / && d{print d}'`<br>echo "WDC WD3200AAJS-00L7A0-298 GB ; = $vmdisk6"<br><br>vmdisk7=3D`geom disk list | awk '/^Geom name: /{d= =3D$NF} /^ *ident: (0774911DDC4200A6)/ && d{print d}'`<br>echo = "SanDisk Cruzer-15GB ; $vmdisk7"<br><br>vmdisk8=3D`geom disk list= | awk '/^Geom name: /{d=3D$NF} /^ *ident: (60A44C4138D8F311190A0149)/ = && d{print d}'`<br>echo "Kingston DataTraveler 2.0 ; $vmdi= sk8"<br><br>vmdisk9=3D`geom disk list | awk '/^Geom name: /{d=3D$N= F} /^ *ident: (20140100006C)/ && d{print d}'`<br>echo "Cor= sair Force 3 SSD ; $vmdisk9"<br><br>vmdisk10=3D`geom disk list | awk &= #39;/^Geom name: /{d=3D$NF} /^ *ident: (1924E20B2AE5)/ && d{print d= }'`<br>echo "CT500MX500SSD4 ; $vmdisk10"<br><br>vmdisk11=3D`g= eom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (57584558303852= 4844323238)/ && d{print d}'`<br>echo "WD-2500BMV ; $vmdisk= 11"<br><br>vmdisk12=3D`geom disk list | awk '/^Geom name: /{d=3D$N= F} /^ *ident: (57442D575845323039544433303334)/ && d{print d}'`= <br>echo "WD-3200BMV ; $vmdisk12"<br><br>vmdisk13=3D`geom disk li= st | awk '/^Geom name: /{d=3D$NF} /^ *ident: (S3Z2NB0KB99028V)/ &&a= mp; d{print d}'`<br>echo "Samsung SSD 860 EVO 500GB ; $vmdisk13&qu= ot;<br><br>pkexec zpool import -f -R /mnt/zroot2 zroot2<br><br>bhyve -S -c = sockets=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \<br>-s 0,hostbridge \<br>= -s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex= =3D1 \<br>-s 2,nvme,/dev/nvd0 \<br>-s 4,virtio-blk,/dev/$vmdisk11 \<br>-s 5= ,virtio-blk,/dev/$vmdisk12 \<br>-s 6,virtio-blk,/dev/$vmdisk13 \<br>-s 8,pa= ssthru,5/0/0 \<br>-s 13,virtio-net,tap19 \<br>-s 14,virtio-9p,sharename=3D/= \<br>-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5919" target=3D"_blank">0.= 0.0.0:5919</a>,w=3D1600,h=3D950,wait \<br>-s 30,xhci,tablet \<br>-s 31,lpc = \<br>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \<br>vm0:= 19 < /dev/null & sleep 2 && vncviewer 0:19<br><br></div><br>= <div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Ma= y 24, 2023 at 3:10=E2=80=AFPM Aryeh Friedman <<a href=3D"mailto:aryeh.fr= iedman@gmail.com" target=3D"_blank">aryeh.friedman@gmail.com</a>> wrote:= <br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, May 24,= 2023 at 8:33=E2=80=AFAM Mario Marietto <<a href=3D"mailto:marietto2008@= gmail.com" target=3D"_blank">marietto2008@gmail.com</a>> wrote:<br> ><br> > I do it automatically using only some scripts. I don't like to use= wrappers.<br> <br> Do you have the scripts... btw I finally gave up on getting GPT<br> (actually more accurately Hugging Face Transformers) to run on FreeBSD<br> and I did get them to work on this linsucks machine with the only snag<br> benign it won't automatically boot... now that I have a better idea of<= br> what the whole working setup looks like I might try it again on<br> FreeBSD when I get some free time.<br> <br> ><br> > On Wed, May 24, 2023 at 1:28=E2=80=AFPM Aryeh Friedman <<a href=3D"= mailto:aryeh.friedman@gmail.com" target=3D"_blank">aryeh.friedman@gmail.com= </a>> wrote:<br> >><br> >> I have it so I can get to a uefi shell prompt and manually boot bu= t I<br> >> am still missing how to make this all automatic<br> >><br> >> --<br> >> Aryeh M. Friedman, Lead Developer, <a href=3D"http://www.PetiteClo= ud.org" rel=3D"noreferrer" target=3D"_blank">http://www.PetiteCloud.org</a>= <br> >><br> ><br> ><br> > --<br> > Mario.<br> <br> <br> <br> -- <br> Aryeh M. Friedman, Lead Developer, <a href=3D"http://www.PetiteCloud.org" r= el=3D"noreferrer" target=3D"_blank">http://www.PetiteCloud.org</a><br> </blockquote></div><br clear=3D"all"><br><span>-- </span><br><div dir=3D"lt= r">Mario.<br></div> </blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre= fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Mario.<br></d= iv> --00000000000096438c05fc76852c--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSihSEteKSzeZPBhvBWLiqzQQDSCzN9xypQPdj4bLXdaHow>