Date: Wed, 24 May 2023 22:40:25 +0200 From: Mario Marietto <marietto2008@gmail.com> To: Aryeh Friedman <aryeh.friedman@gmail.com>, Odhiambo Washington <odhiambo@gmail.com> Cc: FreeBSD Mailing List <freebsd-questions@freebsd.org> Subject: Re: booting linux on bhyve Message-ID: <CA%2B1FSihSEteKSzeZPBhvBWLiqzQQDSCzN9xypQPdj4bLXdaHow@mail.gmail.com> In-Reply-To: <CA%2B1FSiio3G3PiSj6XQM5hM8Z799s06ivZ%2BBAdy8RGJ4hbnKORg@mail.gmail.com> References: <CAGBxaXkhyoy5LMricm3NY92se4QkSA_R_kbJH8MFYHWqtmAZTA@mail.gmail.com> <CA%2B1FSiiwrQ_roPePfyK0wFsm7c9r-aJE43FjgsALpXWik2U-%2BQ@mail.gmail.com> <CAGBxaXkxY4d1jdRHepT_DbhG-_ZvWXj2ODejZNg35_jYVKW6gg@mail.gmail.com> <CA%2B1FSiio3G3PiSj6XQM5hM8Z799s06ivZ%2BBAdy8RGJ4hbnKORg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
I've fixed the bug. This line :
if ! printf '%s\n' "${vncs}" | grep "${session}"; then
becomes like this :
if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then
and more vms can boot at the same time.
On Wed, May 24, 2023 at 3:27 PM Mario Marietto <marietto2008@gmail.com>
wrote:
> I've created a script like the one below. Actually I use the same pattern
> for every VM I want to use. Every VM has its own number associated. What
> changes from one to another is only this line :
>
> -s
> 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex=1 \
>
> That's the name and the location of the raw file. Inside the script I have
> added a mechanism to kill a ghosted vm if it does not find the associated
> viewer window active. That's because I don't find it useful to use a
> ghosted vm. Yes,it is faster,but It causes also troubles in my case. In
> addition the script automatically attaches and detaches the pci address
> tied to my gpu,because I want to pass through my RTX 2080 ti within a Linux
> VM. There is a bug that I want to fix : if I want to boot more than one
> VM,the first one will be killed. Shouldn't be too hard to fix it,anyway.
>
> #!/bin/sh
>
> setxkbmap it
> bhyvectl --vm=vm0:19 --destroy
> vms="$(ls /dev/vmm/*)"
> vncs="$(ps ax | awk '/vncviewer [0]/{print $6}')"
> echo $vms
> echo $vncs
>
> if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."
> kldload nvidia-modeset
> devctl detach pci0:2:0:0
> devctl set driver pci0:2:0:0 ppt
> else
> echo "rtx 2080ti slot 2/0/0 is already attached to ppt"
> fi
>
> if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."
> devctl detach pci0:2:0:1
> devctl set driver pci0:2:0:1 ppt
> else
> echo "rtx 2080ti slot 2/0/1 is already attached to ppt"
> fi
>
> if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."
> devctl detach pci0:2:0:2
> devctl set driver pci0:2:0:2 ppt
> else
> echo "rtx 2080ti slot 2/0/2 is already attached to ppt"
> fi
>
> if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then
> echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."
> #devctl detach pci0:2:0:3
> devctl set driver pci0:2:0:3 ppt
> else
> echo "rtx 2080ti slot 2/0/3 is already attached to ppt"
> fi
>
> echo "rtx 2080ti is fully attached to ppt"
>
> for vm in $vms; do
> session="${vm##*/}"
> echo "bhyve session = $session"
> echo "vnc session = $vncs"
> if ! printf '%s\n' "${vncs}" | grep
> "${session}"; then
> printf 'VNC session not found,destroying
> ghost vms\n'
> bhyvectl --vm=$session --destroy
>
> else
> printf 'Found VNC session %s\n'
> "${session},no ghost vms found,not destroying them"
> fi
> done
>
> vmdisk0=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (NM13N4CZ)/ && d{print d}'`
> echo "Seagate M3 Portable 1.8 TB ; $vmdisk0"
>
> vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (2022285F1175)/ && d{print d}'`
> echo "CT1000P1SSD8 ; $vmdisk1"
>
> vmdisk2=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (2015020204055E)/ && d{print d}'`
> echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk2"
>
> vmdisk3=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (20130506005976F)/ && d{print d}'`
> echo "TOSHIBA External USB 3.0 932G ; $vmdisk3"
>
> vmdisk4=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (BE0191500218)/ && d{print d}'`
> echo "G-DRIVE USB ; $vmdisk4"
>
> vmdisk5=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (38434B4237354B45)/ && d{print d}'`
> echo "Elements 25A3 ; $vmdisk5"
>
> vmdisk6=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (WD-WCAV2X597309)/ && d{print d}'`
> echo "WDC WD3200AAJS-00L7A0-298 GB ; $vmdisk6"
>
> vmdisk7=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (0774911DDC4200A6)/ && d{print d}'`
> echo "SanDisk Cruzer-15GB ; $vmdisk7"
>
> vmdisk8=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (60A44C4138D8F311190A0149)/ && d{print d}'`
> echo "Kingston DataTraveler 2.0 ; $vmdisk8"
>
> vmdisk9=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (20140100006C)/ && d{print d}'`
> echo "Corsair Force 3 SSD ; $vmdisk9"
>
> vmdisk10=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (1924E20B2AE5)/ && d{print d}'`
> echo "CT500MX500SSD4 ; $vmdisk10"
>
> vmdisk11=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (575845583038524844323238)/ && d{print d}'`
> echo "WD-2500BMV ; $vmdisk11"
>
> vmdisk12=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (57442D575845323039544433303334)/ && d{print d}'`
> echo "WD-3200BMV ; $vmdisk12"
>
> vmdisk13=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident:
> (S3Z2NB0KB99028V)/ && d{print d}'`
> echo "Samsung SSD 860 EVO 500GB ; $vmdisk13"
>
> pkexec zpool import -f -R /mnt/zroot2 zroot2
>
> bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
> -s 0,hostbridge \
> -s
> 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex=1 \
> -s 2,nvme,/dev/nvd0 \
> -s 4,virtio-blk,/dev/$vmdisk11 \
> -s 5,virtio-blk,/dev/$vmdisk12 \
> -s 6,virtio-blk,/dev/$vmdisk13 \
> -s 8,passthru,5/0/0 \
> -s 13,virtio-net,tap19 \
> -s 14,virtio-9p,sharename=/ \
> -s 29,fbuf,tcp=0.0.0.0:5919,w=1600,h=950,wait \
> -s 30,xhci,tablet \
> -s 31,lpc \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
> vm0:19 < /dev/null & sleep 2 && vncviewer 0:19
>
>
> On Wed, May 24, 2023 at 3:10 PM Aryeh Friedman <aryeh.friedman@gmail.com>
> wrote:
>
>> On Wed, May 24, 2023 at 8:33 AM Mario Marietto <marietto2008@gmail.com>
>> wrote:
>> >
>> > I do it automatically using only some scripts. I don't like to use
>> wrappers.
>>
>> Do you have the scripts... btw I finally gave up on getting GPT
>> (actually more accurately Hugging Face Transformers) to run on FreeBSD
>> and I did get them to work on this linsucks machine with the only snag
>> benign it won't automatically boot... now that I have a better idea of
>> what the whole working setup looks like I might try it again on
>> FreeBSD when I get some free time.
>>
>> >
>> > On Wed, May 24, 2023 at 1:28 PM Aryeh Friedman <
>> aryeh.friedman@gmail.com> wrote:
>> >>
>> >> I have it so I can get to a uefi shell prompt and manually boot but I
>> >> am still missing how to make this all automatic
>> >>
>> >> --
>> >> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
>> >>
>> >
>> >
>> > --
>> > Mario.
>>
>>
>>
>> --
>> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
>>
>
>
> --
> Mario.
>
--
Mario.
[-- Attachment #2 --]
<div dir="ltr"><div>I've fixed the bug. This line :</div><div><br></div><div>if ! printf '%s\n' "${vncs}" | grep "${session}"; then</div><div><br></div><div>becomes like this :</div><div><br></div><div>if ! printf '%s\n' "${vncs}" | grep "${session#vm}"; then</div><div><br></div><div>and more vms can boot at the same time.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 24, 2023 at 3:27 PM Mario Marietto <<a href="mailto:marietto2008@gmail.com">marietto2008@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I've created a script like the one below. Actually I use the same pattern for every VM I want to use. Every VM has its own number associated. What changes from one to another is only this line :</div><div><br></div><div>-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex=1 \</div><div><br></div><div>That's the name and the location of the raw file. Inside the script I have added a mechanism to kill a ghosted vm if it does not find the associated viewer window active. That's because I don't find it useful to use a ghosted vm. Yes,it is faster,but It causes also troubles in my case. In addition the script automatically attaches and detaches the pci address tied to my gpu,because I want to pass through my RTX 2080 ti within a Linux VM. There is a bug that I want to fix : if I want to boot more than one VM,the first one will be killed. Shouldn't be too hard to fix it,anyway. <br></div><div><br></div><div>#!/bin/sh</div><br>setxkbmap it<br>bhyvectl --vm=vm0:19 --destroy<br>vms="$(ls /dev/vmm/*)"<br>vncs="$(ps ax | awk '/vncviewer [0]/{print $6}')"<br>echo $vms<br>echo $vncs<br><br>if ! pciconf -l pci0:2:0:0 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/0 is not attached to ppt,attaching..."<br>kldload nvidia-modeset<br>devctl detach pci0:2:0:0<br>devctl set driver pci0:2:0:0 ppt<br>else<br>echo "rtx 2080ti slot 2/0/0 is already attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:1 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/1 is not attached to ppt,attaching..."<br>devctl detach pci0:2:0:1<br>devctl set driver pci0:2:0:1 ppt<br>else<br>echo "rtx 2080ti slot 2/0/1 is already attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:2 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/2 is not attached to ppt,attaching..."<br>devctl detach pci0:2:0:2<br>devctl set driver pci0:2:0:2 ppt<br>else<br>echo "rtx 2080ti slot 2/0/2 is already attached to ppt"<br>fi<br><br>if ! pciconf -l pci0:2:0:3 | grep -q "^ppt"; then<br>echo "rtx 2080ti slot 2/0/3 is not attached to ppt,attaching..."<br>#devctl detach pci0:2:0:3<br>devctl set driver pci0:2:0:3 ppt<br>else<br>echo "rtx 2080ti slot 2/0/3 is already attached to ppt"<br>fi<br><br>echo "rtx 2080ti is fully attached to ppt"<br><br>for vm in $vms; do<br> session="${vm##*/}" <br> echo "bhyve session = $session"<br> echo "vnc session = $vncs" <br> if ! printf '%s\n' "${vncs}" | grep "${session}"; then<br> printf 'VNC session not found,destroying ghost vms\n'<br> bhyvectl --vm=$session --destroy <br> else<br> printf 'Found VNC session %s\n' "${session},no ghost vms found,not destroying them"<br> fi<br>done<br><br>vmdisk0=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (NM13N4CZ)/ && d{print d}'`<br>echo "Seagate M3 Portable 1.8 TB ; $vmdisk0"<br><br>vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (2022285F1175)/ && d{print d}'`<br>echo "CT1000P1SSD8 ; $vmdisk1"<br><br>vmdisk2=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (2015020204055E)/ && d{print d}'`<br>echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk2"<br><br>vmdisk3=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (20130506005976F)/ && d{print d}'`<br>echo "TOSHIBA External USB 3.0 932G ; $vmdisk3"<br><br>vmdisk4=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (BE0191500218)/ && d{print d}'`<br>echo "G-DRIVE USB ; $vmdisk4"<br><br>vmdisk5=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (38434B4237354B45)/ && d{print d}'`<br>echo "Elements 25A3 ; $vmdisk5"<br><br>vmdisk6=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (WD-WCAV2X597309)/ && d{print d}'`<br>echo "WDC WD3200AAJS-00L7A0-298 GB ; $vmdisk6"<br><br>vmdisk7=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (0774911DDC4200A6)/ && d{print d}'`<br>echo "SanDisk Cruzer-15GB ; $vmdisk7"<br><br>vmdisk8=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (60A44C4138D8F311190A0149)/ && d{print d}'`<br>echo "Kingston DataTraveler 2.0 ; $vmdisk8"<br><br>vmdisk9=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (20140100006C)/ && d{print d}'`<br>echo "Corsair Force 3 SSD ; $vmdisk9"<br><br>vmdisk10=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (1924E20B2AE5)/ && d{print d}'`<br>echo "CT500MX500SSD4 ; $vmdisk10"<br><br>vmdisk11=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (575845583038524844323238)/ && d{print d}'`<br>echo "WD-2500BMV ; $vmdisk11"<br><br>vmdisk12=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (57442D575845323039544433303334)/ && d{print d}'`<br>echo "WD-3200BMV ; $vmdisk12"<br><br>vmdisk13=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (S3Z2NB0KB99028V)/ && d{print d}'`<br>echo "Samsung SSD 860 EVO 500GB ; $vmdisk13"<br><br>pkexec zpool import -f -R /mnt/zroot2 zroot2<br><br>bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \<br>-s 0,hostbridge \<br>-s 1,virtio-blk,/mnt/zroot2/zroot2/bhyve/img/Linux/ubuntu2210.img,bootindex=1 \<br>-s 2,nvme,/dev/nvd0 \<br>-s 4,virtio-blk,/dev/$vmdisk11 \<br>-s 5,virtio-blk,/dev/$vmdisk12 \<br>-s 6,virtio-blk,/dev/$vmdisk13 \<br>-s 8,passthru,5/0/0 \<br>-s 13,virtio-net,tap19 \<br>-s 14,virtio-9p,sharename=/ \<br>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5919" target="_blank">0.0.0.0:5919</a>,w=1600,h=950,wait \<br>-s 30,xhci,tablet \<br>-s 31,lpc \<br>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \<br>vm0:19 < /dev/null & sleep 2 && vncviewer 0:19<br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 24, 2023 at 3:10 PM Aryeh Friedman <<a href="mailto:aryeh.friedman@gmail.com" target="_blank">aryeh.friedman@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, May 24, 2023 at 8:33 AM Mario Marietto <<a href="mailto:marietto2008@gmail.com" target="_blank">marietto2008@gmail.com</a>> wrote:<br>
><br>
> I do it automatically using only some scripts. I don't like to use wrappers.<br>
<br>
Do you have the scripts... btw I finally gave up on getting GPT<br>
(actually more accurately Hugging Face Transformers) to run on FreeBSD<br>
and I did get them to work on this linsucks machine with the only snag<br>
benign it won't automatically boot... now that I have a better idea of<br>
what the whole working setup looks like I might try it again on<br>
FreeBSD when I get some free time.<br>
<br>
><br>
> On Wed, May 24, 2023 at 1:28 PM Aryeh Friedman <<a href="mailto:aryeh.friedman@gmail.com" target="_blank">aryeh.friedman@gmail.com</a>> wrote:<br>
>><br>
>> I have it so I can get to a uefi shell prompt and manually boot but I<br>
>> am still missing how to make this all automatic<br>
>><br>
>> --<br>
>> Aryeh M. Friedman, Lead Developer, <a href="http://www.PetiteCloud.org" rel="noreferrer" target="_blank">http://www.PetiteCloud.org</a><br>;
>><br>
><br>
><br>
> --<br>
> Mario.<br>
<br>
<br>
<br>
-- <br>
Aryeh M. Friedman, Lead Developer, <a href="http://www.PetiteCloud.org" rel="noreferrer" target="_blank">http://www.PetiteCloud.org</a><br>;
</blockquote></div><br clear="all"><br><span>-- </span><br><div dir="ltr">Mario.<br></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSihSEteKSzeZPBhvBWLiqzQQDSCzN9xypQPdj4bLXdaHow>