Date: Sat, 12 Oct 2024 18:22:59 +0000 From: Vasily Postnicov <shamaz.mazum@gmail.com> To: Peter Grehan <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: Running Mezzano in bhyve Message-ID: <CADnZ6BmjGzHygqJSNY=wpuy-6Z4YiAMpt-gBx0f%2Bi%2BrXBfBvaQ@mail.gmail.com> In-Reply-To: <e395fc30-0582-4d51-b1b3-cf5157bdd3a9@freebsd.org> References: <CADnZ6B=ex24mbGN3du6UuS84akJZAxTcG5xqt0HB0RN5S262cQ@mail.gmail.com> <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <CADnZ6BkWd-v=y0L9%2BGiu=ys_Cuk5nm6djApSXYLufYuv=WnQWQ@mail.gmail.com> <CADnZ6B=LwZyiBTvXGek37e23t_e3ub4K%2BE96QaahukPbobkHhg@mail.gmail.com> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> <CADnZ6BkKh5V9_Y%2BTGrGpc=vTW2q81pdWJn8MUVvWNOiV35nBFw@mail.gmail.com> <CADnZ6BkHkNBD5LaEZCeSy7QnfquwB-Wv3sYu4S=P58ZyVGrDQQ@mail.gmail.com> <e395fc30-0582-4d51-b1b3-cf5157bdd3a9@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000027764c06244bb20d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I was able to boot Mezzano completely (though without a working mouse, just like on my real PC). Both ahci and virtio disks work. This is a list of issues I found: 1) The problem with PIT. Can be solved as you proposed or by patching Mezzano. 2) Mezzano assumes that Intel AHCI controllers report no more than 6 ports. Can be solved by patching Mezzano or defining MAX_PORTS to be 6 in usr.sbin/bhyve/pci_ahci.c 3) According to https://wiki.osdev.org/PCI#Message_Signaled_Interrupts, interrupt line config register must be RW. Bhyve does not support writing to it. I do not know a correct fix, this [1] workaround helps, however. 4) Finally, I had a random deadlock in interrupt handling for the virtio-net device. Likewise, I do not know how to fix it correctly, but this [2] patch helped. Used patches: [1] Workaround for the interrupt line. Forbid 1 byte writes to 0x3c config register (the write is performed by UEFI firmware or bootloader). diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_emul.c index e91b4d0a1e20..6e9d5885a6b1 100644 --- a/usr.sbin/bhyve/pci_emul.c +++ b/usr.sbin/bhyve/pci_emul.c @@ -160,8 +160,11 @@ static __inline void CFGWRITE(struct pci_devinst *pi, int coff, uint32_t val, int bytes) { - if (bytes =3D=3D 1) - pci_set_cfgdata8(pi, coff, val); + if (bytes =3D=3D 1) { + if (coff !=3D PCIR_INTLINE) { + pci_set_cfgdata8(pi, coff, val); + } + } else if (bytes =3D=3D 2) pci_set_cfgdata16(pi, coff, val); else [2] Workaround for a hang up in interrupt handling code: diff --git a/usr.sbin/bhyve/virtio.h b/usr.sbin/bhyve/virtio.h index 4c6c8004b2d1..d1a901ff0069 100644 --- a/usr.sbin/bhyve/virtio.h +++ b/usr.sbin/bhyve/virtio.h @@ -355,13 +355,13 @@ vi_interrupt(struct virtio_softc *vs, uint8_t isr, uint16_t msix_idx) if (pci_msix_enabled(vs->vs_pi)) pci_generate_msix(vs->vs_pi, msix_idx); else { - VS_LOCK(vs); + //VS_LOCK(vs); vs->vs_isr |=3D isr; - pci_generate_msi(vs->vs_pi, 0); + //pci_generate_msi(vs->vs_pi, 0); #ifdef __amd64__ pci_lintr_assert(vs->vs_pi); #endif - VS_UNLOCK(vs); + //VS_UNLOCK(vs); } } Do you have any ideas how to make proper patches for bhyve from these workarounds? P.S. After all 4 issues are addressed to, you can build Demo 5 image, but only after replacing a bootloader. See https://github.com/froggey/Mezzano/issues/173 If you wish, I can send the image to you by some means (Telegram?) Or you can build the bootloader from here: https://github.com/froggey/kboot/tree/mezzano-loader (requires Linux, I guess). I'll try virtio-input to see if the mouse works this way. =D1=81=D0=B1, 12 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 06:12, Pet= er Grehan <grehan@freebsd.org>: > > I suspect PCI interrupts are not functioning correctly. > > > > Look at this code: > > ;; Attach interrupt handler. > > (sup:debug-print-line "Handler: " (ahci-irq-handler ahci)) > > (sup:irq-attach (sup:platform-irq (pci:pci-intr-line location)) > > (ahci-irq-handler-function ahci) > > ahci) > > > > and this > > > > (defun pci-intr-line (device) > > (pci-config/8 device +pci-config-intr-line+)) ;; comment by me: the > > constant is #x3c > > > > I found that "PCI 0x3c" means PCI interrupt pin. AFAIK, interrupt pins > > are not supported by bhyve, is that correct? If it's true, I need eithe= r > > to teach bhyve how to deal with legacy interrupts or to teach Mezzano t= o > > understand MSI. What would be easier in your opinion? > > Legacy interrupts should work fine in bhyve for emulated devices. I'd > suspect this would be much easier to debug/enhance as opposed to adding > MSI (and likely MSI-x). > > later, > > Peter. > --00000000000027764c06244bb20d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I was able to boot Mezzano completely (though without a wo= rking mouse, just like on my real PC). Both ahci and virtio disks work. Thi= s is a list of issues I found:<div><br></div><div>1) The problem with PIT. = Can be solved as you proposed or by patching Mezzano.</div><div>2) Mezzano = assumes that Intel AHCI controllers report no more than 6 ports. Can be sol= ved by patching Mezzano or defining MAX_PORTS to be 6 in usr.sbin/bhyve/pci= _ahci.c</div><div>3) According=C2=A0to=C2=A0<a href=3D"https://wiki.osdev.o= rg/PCI#Message_Signaled_Interrupts">https://wiki.osdev.org/PCI#Message_Sign= aled_Interrupts</a>, interrupt line config register must be RW. Bhyve does = not support writing to it. I do not know a correct fix, this [1] workaround= helps, however.</div><div>4) Finally, I had a random deadlock in interrupt= handling for the virtio-net device. Likewise, I do not know how to fix it = correctly, but this [2] patch helped.</div><div><br></div><div>Used patches= :</div><div>[1] Workaround for the interrupt line. Forbid 1 byte writes to = 0x3c config register (the write is performed by UEFI firmware or bootloader= ).</div><div>diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_em= ul.c<br>index e91b4d0a1e20..6e9d5885a6b1 100644<br>--- a/usr.sbin/bhyve/pci= _emul.c<br>+++ b/usr.sbin/bhyve/pci_emul.c<br>@@ -160,8 +160,11 @@ static _= _inline void<br>=C2=A0CFGWRITE(struct pci_devinst *pi, int coff, uint32_t v= al, int bytes)<br>=C2=A0{<br><br>- =C2=A0 =C2=A0 =C2=A0 if (bytes =3D=3D 1)= <br>- =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pci_set_cfgdata8(pi,= coff, val);<br>+ =C2=A0 =C2=A0 =C2=A0 if (bytes =3D=3D 1) {<br>+ =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (coff !=3D PCIR_INTLINE) {<br>= + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 pci_set_cfgdata8(pi, coff, val);<br>+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 }<br>+ =C2=A0 =C2=A0 =C2=A0 }<br>=C2=A0 =C2=A0 =C2=A0 =C2= =A0 else if (bytes =3D=3D 2)<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 pci_set_cfgdata16(pi, coff, val);<br>=C2=A0 =C2=A0 =C2=A0 =C2= =A0 else<br></div><div><br></div><div>[2] Workaround for a hang up in inter= rupt handling code:</div><div>diff --git a/usr.sbin/bhyve/virtio.h b/usr.sb= in/bhyve/virtio.h<br>index 4c6c8004b2d1..d1a901ff0069 100644<br>--- a/usr.s= bin/bhyve/virtio.h<br>+++ b/usr.sbin/bhyve/virtio.h<br>@@ -355,13 +355,13 @= @ vi_interrupt(struct virtio_softc *vs, uint8_t isr, uint16_t msix_idx)<br>= =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (pci_msix_enabled(vs->vs_pi))<br>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pci_generate_msix(vs->v= s_pi, msix_idx);<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 else {<br>- =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VS_LOCK(vs);<br>+ =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 //VS_LOCK(vs);<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 vs->vs_isr |=3D isr;<br>- =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pci_generate_msi(vs->vs_pi, 0);<br>+ =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 //pci_generate_msi(vs->vs_= pi, 0);<br>=C2=A0#ifdef __amd64__<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 pci_lintr_assert(vs->vs_pi);<br>=C2=A0#endif<br>- =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VS_UNLOCK(vs);<br>+ =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 //VS_UNLOCK(vs);<br>=C2=A0 =C2=A0= =C2=A0 =C2=A0 }<br>=C2=A0}<br></div><div><br></div><div>Do you have any id= eas how to make proper patches for bhyve from these workarounds?</div><div>= P.S. After all 4 issues are addressed to, you can build Demo 5 image, but o= nly after replacing a bootloader. See=C2=A0<a href=3D"https://github.com/fr= oggey/Mezzano/issues/173">https://github.com/froggey/Mezzano/issues/173</a>= If you wish, I can send the image to you by some means (Telegram?) Or you = can build the bootloader from here:=C2=A0<a href=3D"https://github.com/frog= gey/kboot/tree/mezzano-loader">https://github.com/froggey/kboot/tree/mezzan= o-loader</a> (requires Linux, I guess).</div><div><br></div><div>I'll t= ry virtio-input to see if the mouse works this way.</div></div><br><div cla= ss=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">=D1=81=D0=B1, 12 = =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 06:12, Peter Grehan <<a = href=3D"mailto:grehan@freebsd.org">grehan@freebsd.org</a>>:<br></div><bl= ockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-lef= t-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padd= ing-left:1ex">> I suspect PCI interrupts are not functioning correctly.<= br> > <br> > Look at this code:<br> >=C2=A0 =C2=A0 =C2=A0 ;; Attach interrupt handler.<br> >=C2=A0 =C2=A0 =C2=A0 (sup:debug-print-line "Handler: " (ahci-= irq-handler ahci))<br> >=C2=A0 =C2=A0 =C2=A0 (sup:irq-attach (sup:platform-irq (pci:pci-intr-li= ne location))<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 (ahci-irq-handler-function ahci)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 ahci)<br> > <br> > and this<br> > <br> > (defun pci-intr-line (device)<br> >=C2=A0 =C2=A0 (pci-config/8 device +pci-config-intr-line+)) ;; comment = by me: the <br> > constant is #x3c<br> > <br> > I found that "PCI 0x3c" means PCI interrupt pin. AFAIK, inte= rrupt pins <br> > are not supported by bhyve, is that=C2=A0correct? If it's true, I = need either <br> > to teach bhyve how to deal with legacy interrupts or to teach Mezzano = to <br> > understand MSI. What would be easier in your=C2=A0opinion?<br> <br> =C2=A0 Legacy interrupts should work fine in bhyve for emulated devices. I&= #39;d <br> suspect this would be much easier to debug/enhance as opposed to adding <br= > MSI (and likely MSI-x).<br> <br> later,<br> <br> Peter.<br> </blockquote></div> --00000000000027764c06244bb20d--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADnZ6BmjGzHygqJSNY=wpuy-6Z4YiAMpt-gBx0f%2Bi%2BrXBfBvaQ>