Date: Sat, 12 Oct 2024 18:22:59 +0000 From: Vasily Postnicov <shamaz.mazum@gmail.com> To: Peter Grehan <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: Running Mezzano in bhyve Message-ID: <CADnZ6BmjGzHygqJSNY=wpuy-6Z4YiAMpt-gBx0f%2Bi%2BrXBfBvaQ@mail.gmail.com> In-Reply-To: <e395fc30-0582-4d51-b1b3-cf5157bdd3a9@freebsd.org> References: <CADnZ6B=ex24mbGN3du6UuS84akJZAxTcG5xqt0HB0RN5S262cQ@mail.gmail.com> <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <CADnZ6BkWd-v=y0L9%2BGiu=ys_Cuk5nm6djApSXYLufYuv=WnQWQ@mail.gmail.com> <CADnZ6B=LwZyiBTvXGek37e23t_e3ub4K%2BE96QaahukPbobkHhg@mail.gmail.com> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> <CADnZ6BkKh5V9_Y%2BTGrGpc=vTW2q81pdWJn8MUVvWNOiV35nBFw@mail.gmail.com> <CADnZ6BkHkNBD5LaEZCeSy7QnfquwB-Wv3sYu4S=P58ZyVGrDQQ@mail.gmail.com> <e395fc30-0582-4d51-b1b3-cf5157bdd3a9@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] I was able to boot Mezzano completely (though without a working mouse, just like on my real PC). Both ahci and virtio disks work. This is a list of issues I found: 1) The problem with PIT. Can be solved as you proposed or by patching Mezzano. 2) Mezzano assumes that Intel AHCI controllers report no more than 6 ports. Can be solved by patching Mezzano or defining MAX_PORTS to be 6 in usr.sbin/bhyve/pci_ahci.c 3) According to https://wiki.osdev.org/PCI#Message_Signaled_Interrupts, interrupt line config register must be RW. Bhyve does not support writing to it. I do not know a correct fix, this [1] workaround helps, however. 4) Finally, I had a random deadlock in interrupt handling for the virtio-net device. Likewise, I do not know how to fix it correctly, but this [2] patch helped. Used patches: [1] Workaround for the interrupt line. Forbid 1 byte writes to 0x3c config register (the write is performed by UEFI firmware or bootloader). diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_emul.c index e91b4d0a1e20..6e9d5885a6b1 100644 --- a/usr.sbin/bhyve/pci_emul.c +++ b/usr.sbin/bhyve/pci_emul.c @@ -160,8 +160,11 @@ static __inline void CFGWRITE(struct pci_devinst *pi, int coff, uint32_t val, int bytes) { - if (bytes == 1) - pci_set_cfgdata8(pi, coff, val); + if (bytes == 1) { + if (coff != PCIR_INTLINE) { + pci_set_cfgdata8(pi, coff, val); + } + } else if (bytes == 2) pci_set_cfgdata16(pi, coff, val); else [2] Workaround for a hang up in interrupt handling code: diff --git a/usr.sbin/bhyve/virtio.h b/usr.sbin/bhyve/virtio.h index 4c6c8004b2d1..d1a901ff0069 100644 --- a/usr.sbin/bhyve/virtio.h +++ b/usr.sbin/bhyve/virtio.h @@ -355,13 +355,13 @@ vi_interrupt(struct virtio_softc *vs, uint8_t isr, uint16_t msix_idx) if (pci_msix_enabled(vs->vs_pi)) pci_generate_msix(vs->vs_pi, msix_idx); else { - VS_LOCK(vs); + //VS_LOCK(vs); vs->vs_isr |= isr; - pci_generate_msi(vs->vs_pi, 0); + //pci_generate_msi(vs->vs_pi, 0); #ifdef __amd64__ pci_lintr_assert(vs->vs_pi); #endif - VS_UNLOCK(vs); + //VS_UNLOCK(vs); } } Do you have any ideas how to make proper patches for bhyve from these workarounds? P.S. After all 4 issues are addressed to, you can build Demo 5 image, but only after replacing a bootloader. See https://github.com/froggey/Mezzano/issues/173 If you wish, I can send the image to you by some means (Telegram?) Or you can build the bootloader from here: https://github.com/froggey/kboot/tree/mezzano-loader (requires Linux, I guess). I'll try virtio-input to see if the mouse works this way. сб, 12 окт. 2024 г. в 06:12, Peter Grehan <grehan@freebsd.org>: > > I suspect PCI interrupts are not functioning correctly. > > > > Look at this code: > > ;; Attach interrupt handler. > > (sup:debug-print-line "Handler: " (ahci-irq-handler ahci)) > > (sup:irq-attach (sup:platform-irq (pci:pci-intr-line location)) > > (ahci-irq-handler-function ahci) > > ahci) > > > > and this > > > > (defun pci-intr-line (device) > > (pci-config/8 device +pci-config-intr-line+)) ;; comment by me: the > > constant is #x3c > > > > I found that "PCI 0x3c" means PCI interrupt pin. AFAIK, interrupt pins > > are not supported by bhyve, is that correct? If it's true, I need either > > to teach bhyve how to deal with legacy interrupts or to teach Mezzano to > > understand MSI. What would be easier in your opinion? > > Legacy interrupts should work fine in bhyve for emulated devices. I'd > suspect this would be much easier to debug/enhance as opposed to adding > MSI (and likely MSI-x). > > later, > > Peter. > [-- Attachment #2 --] <div dir="ltr">I was able to boot Mezzano completely (though without a working mouse, just like on my real PC). Both ahci and virtio disks work. This is a list of issues I found:<div><br></div><div>1) The problem with PIT. Can be solved as you proposed or by patching Mezzano.</div><div>2) Mezzano assumes that Intel AHCI controllers report no more than 6 ports. Can be solved by patching Mezzano or defining MAX_PORTS to be 6 in usr.sbin/bhyve/pci_ahci.c</div><div>3) According to <a href="https://wiki.osdev.org/PCI#Message_Signaled_Interrupts">https://wiki.osdev.org/PCI#Message_Signaled_Interrupts</a>, interrupt line config register must be RW. Bhyve does not support writing to it. I do not know a correct fix, this [1] workaround helps, however.</div><div>4) Finally, I had a random deadlock in interrupt handling for the virtio-net device. Likewise, I do not know how to fix it correctly, but this [2] patch helped.</div><div><br></div><div>Used patches:</div><div>[1] Workaround for the interrupt line. Forbid 1 byte writes to 0x3c config register (the write is performed by UEFI firmware or bootloader).</div><div>diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_emul.c<br>index e91b4d0a1e20..6e9d5885a6b1 100644<br>--- a/usr.sbin/bhyve/pci_emul.c<br>+++ b/usr.sbin/bhyve/pci_emul.c<br>@@ -160,8 +160,11 @@ static __inline void<br> CFGWRITE(struct pci_devinst *pi, int coff, uint32_t val, int bytes)<br> {<br><br>- if (bytes == 1)<br>- pci_set_cfgdata8(pi, coff, val);<br>+ if (bytes == 1) {<br>+ if (coff != PCIR_INTLINE) {<br>+ pci_set_cfgdata8(pi, coff, val);<br>+ }<br>+ }<br> else if (bytes == 2)<br> pci_set_cfgdata16(pi, coff, val);<br> else<br></div><div><br></div><div>[2] Workaround for a hang up in interrupt handling code:</div><div>diff --git a/usr.sbin/bhyve/virtio.h b/usr.sbin/bhyve/virtio.h<br>index 4c6c8004b2d1..d1a901ff0069 100644<br>--- a/usr.sbin/bhyve/virtio.h<br>+++ b/usr.sbin/bhyve/virtio.h<br>@@ -355,13 +355,13 @@ vi_interrupt(struct virtio_softc *vs, uint8_t isr, uint16_t msix_idx)<br> if (pci_msix_enabled(vs->vs_pi))<br> pci_generate_msix(vs->vs_pi, msix_idx);<br> else {<br>- VS_LOCK(vs);<br>+ //VS_LOCK(vs);<br> vs->vs_isr |= isr;<br>- pci_generate_msi(vs->vs_pi, 0);<br>+ //pci_generate_msi(vs->vs_pi, 0);<br> #ifdef __amd64__<br> pci_lintr_assert(vs->vs_pi);<br> #endif<br>- VS_UNLOCK(vs);<br>+ //VS_UNLOCK(vs);<br> }<br> }<br></div><div><br></div><div>Do you have any ideas how to make proper patches for bhyve from these workarounds?</div><div>P.S. After all 4 issues are addressed to, you can build Demo 5 image, but only after replacing a bootloader. See <a href="https://github.com/froggey/Mezzano/issues/173">https://github.com/froggey/Mezzano/issues/173</a>; If you wish, I can send the image to you by some means (Telegram?) Or you can build the bootloader from here: <a href="https://github.com/froggey/kboot/tree/mezzano-loader">https://github.com/froggey/kboot/tree/mezzano-loader</a>; (requires Linux, I guess).</div><div><br></div><div>I'll try virtio-input to see if the mouse works this way.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">сб, 12 окт. 2024 г. в 06:12, Peter Grehan <<a href="mailto:grehan@freebsd.org">grehan@freebsd.org</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">> I suspect PCI interrupts are not functioning correctly.<br> > <br> > Look at this code:<br> > ;; Attach interrupt handler.<br> > (sup:debug-print-line "Handler: " (ahci-irq-handler ahci))<br> > (sup:irq-attach (sup:platform-irq (pci:pci-intr-line location))<br> > (ahci-irq-handler-function ahci)<br> > ahci)<br> > <br> > and this<br> > <br> > (defun pci-intr-line (device)<br> > (pci-config/8 device +pci-config-intr-line+)) ;; comment by me: the <br> > constant is #x3c<br> > <br> > I found that "PCI 0x3c" means PCI interrupt pin. AFAIK, interrupt pins <br> > are not supported by bhyve, is that correct? If it's true, I need either <br> > to teach bhyve how to deal with legacy interrupts or to teach Mezzano to <br> > understand MSI. What would be easier in your opinion?<br> <br> Legacy interrupts should work fine in bhyve for emulated devices. I'd <br> suspect this would be much easier to debug/enhance as opposed to adding <br> MSI (and likely MSI-x).<br> <br> later,<br> <br> Peter.<br> </blockquote></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADnZ6BmjGzHygqJSNY=wpuy-6Z4YiAMpt-gBx0f%2Bi%2BrXBfBvaQ>