From nobody Sat Oct 12 18:22:59 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XQsLT3kHSz5Ymmf for ; Sat, 12 Oct 2024 18:23:13 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XQsLS5ZJQz4Cxk; Sat, 12 Oct 2024 18:23:12 +0000 (UTC) (envelope-from shamaz.mazum@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12e.google.com with SMTP id 2adb3069b0e04-539e59dadebso1092280e87.0; Sat, 12 Oct 2024 11:23:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728757391; x=1729362191; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=stOsPLRCaB+WHjtRfQCe1Wu26pvs+AXcyPVMkEzWBXI=; b=nb2Nzjzni08XIi1mSSLMoy4zDfbi1s9VmPLrm15Yl9MMnnVdtagPdvvDqP7eiaonuq P7JXzrZjmCNqjSmBvVHwPcuqv9AXMx0lk8lYs9GDUzhc3XDfu6tsl0nX/lNBqViR7N/C 45PcSeZRGbmyoimA/Ecte3MeSmyMt9xMAABTxeUzKUL5Q3GCaCa0VH59RqqX+71zlglc 9epc61gw0RhnVuqTYThIzy1oIlEnL91e99AA5OzM690SXSqekRtWebH29Igdf1BJyz+E OonYjFu1m3OfjnxKXMcM7hGljs89NGvSJjhECUfIxuS3FVrZEHl9+CC2nTyVfnHMC4k8 u5yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728757391; x=1729362191; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=stOsPLRCaB+WHjtRfQCe1Wu26pvs+AXcyPVMkEzWBXI=; b=GOAipTnZfMa59/vkLnfw7kqtn74nVAJt7wq7ENHNPxFZGixJqZglZigP50LT7pkl56 kIBBc86FtdYQYwa3jcqpmJ+Y95UGa38CQaM35vTjDweAWI3N/fBkZqA+Vxwp4AjHDbwk 4K1CiAYmTQ9xL1Zo16MwfDGq+cfUmMFBW0MnE07UIMbKFVxjP56FEsDpf/wtFHbrlur5 2x03nRIHEMv+ibnm0DjxEzZoVctWD8zJ+t7Fs2GipX6rfvpvQ8zN5JydOUbqCPSBxvDD pYaNDLN0SEZwxM0ykrlYFDt2sjFprOdZQC035tDSpKCDDc2Aw1jdfCFIOTDOVa6eQncA 8JYQ== X-Gm-Message-State: AOJu0YyLqomZ3JOlKibO+FKgnu96iZXn6eVoXEvcWd3FAjJVs2SvOzzu oaj3qKyTq804x+0i8v05xxNtNFwaVDRsH/oUHut1SLOUGr3vkjo3lBDBG8Z68LyLeKTwpG59yAI APsRbCQhPPM/9AwoGM+Ms5lrZ7vKUNlbcTFbDwg== X-Google-Smtp-Source: AGHT+IECEJOBMr3ZCBsRbcHv9RqViBp+/+krBQsDz2NEfeN6j/stmZU+kcoZXy7f3ZJ7/rNoWbHujuqOEdwIHAStB7M= X-Received: by 2002:a05:6512:1588:b0:539:93b2:1383 with SMTP id 2adb3069b0e04-539da5661c8mr2943149e87.46.1728757390339; Sat, 12 Oct 2024 11:23:10 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: <17f4077d-647d-4848-9d6f-97f9886ef636@freebsd.org> <8b249b64-d041-4f12-b6cb-fdb528837f22@freebsd.org> In-Reply-To: From: Vasily Postnicov Date: Sat, 12 Oct 2024 18:22:59 +0000 Message-ID: Subject: Re: Running Mezzano in bhyve To: Peter Grehan Cc: freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="00000000000027764c06244bb20d" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4XQsLS5ZJQz4Cxk X-Spamd-Bar: ---- --00000000000027764c06244bb20d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I was able to boot Mezzano completely (though without a working mouse, just like on my real PC). Both ahci and virtio disks work. This is a list of issues I found: 1) The problem with PIT. Can be solved as you proposed or by patching Mezzano. 2) Mezzano assumes that Intel AHCI controllers report no more than 6 ports. Can be solved by patching Mezzano or defining MAX_PORTS to be 6 in usr.sbin/bhyve/pci_ahci.c 3) According to https://wiki.osdev.org/PCI#Message_Signaled_Interrupts, interrupt line config register must be RW. Bhyve does not support writing to it. I do not know a correct fix, this [1] workaround helps, however. 4) Finally, I had a random deadlock in interrupt handling for the virtio-net device. Likewise, I do not know how to fix it correctly, but this [2] patch helped. Used patches: [1] Workaround for the interrupt line. Forbid 1 byte writes to 0x3c config register (the write is performed by UEFI firmware or bootloader). diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_emul.c index e91b4d0a1e20..6e9d5885a6b1 100644 --- a/usr.sbin/bhyve/pci_emul.c +++ b/usr.sbin/bhyve/pci_emul.c @@ -160,8 +160,11 @@ static __inline void CFGWRITE(struct pci_devinst *pi, int coff, uint32_t val, int bytes) { - if (bytes =3D=3D 1) - pci_set_cfgdata8(pi, coff, val); + if (bytes =3D=3D 1) { + if (coff !=3D PCIR_INTLINE) { + pci_set_cfgdata8(pi, coff, val); + } + } else if (bytes =3D=3D 2) pci_set_cfgdata16(pi, coff, val); else [2] Workaround for a hang up in interrupt handling code: diff --git a/usr.sbin/bhyve/virtio.h b/usr.sbin/bhyve/virtio.h index 4c6c8004b2d1..d1a901ff0069 100644 --- a/usr.sbin/bhyve/virtio.h +++ b/usr.sbin/bhyve/virtio.h @@ -355,13 +355,13 @@ vi_interrupt(struct virtio_softc *vs, uint8_t isr, uint16_t msix_idx) if (pci_msix_enabled(vs->vs_pi)) pci_generate_msix(vs->vs_pi, msix_idx); else { - VS_LOCK(vs); + //VS_LOCK(vs); vs->vs_isr |=3D isr; - pci_generate_msi(vs->vs_pi, 0); + //pci_generate_msi(vs->vs_pi, 0); #ifdef __amd64__ pci_lintr_assert(vs->vs_pi); #endif - VS_UNLOCK(vs); + //VS_UNLOCK(vs); } } Do you have any ideas how to make proper patches for bhyve from these workarounds? P.S. After all 4 issues are addressed to, you can build Demo 5 image, but only after replacing a bootloader. See https://github.com/froggey/Mezzano/issues/173 If you wish, I can send the image to you by some means (Telegram?) Or you can build the bootloader from here: https://github.com/froggey/kboot/tree/mezzano-loader (requires Linux, I guess). I'll try virtio-input to see if the mouse works this way. =D1=81=D0=B1, 12 =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 06:12, Pet= er Grehan : > > I suspect PCI interrupts are not functioning correctly. > > > > Look at this code: > > ;; Attach interrupt handler. > > (sup:debug-print-line "Handler: " (ahci-irq-handler ahci)) > > (sup:irq-attach (sup:platform-irq (pci:pci-intr-line location)) > > (ahci-irq-handler-function ahci) > > ahci) > > > > and this > > > > (defun pci-intr-line (device) > > (pci-config/8 device +pci-config-intr-line+)) ;; comment by me: the > > constant is #x3c > > > > I found that "PCI 0x3c" means PCI interrupt pin. AFAIK, interrupt pins > > are not supported by bhyve, is that correct? If it's true, I need eithe= r > > to teach bhyve how to deal with legacy interrupts or to teach Mezzano t= o > > understand MSI. What would be easier in your opinion? > > Legacy interrupts should work fine in bhyve for emulated devices. I'd > suspect this would be much easier to debug/enhance as opposed to adding > MSI (and likely MSI-x). > > later, > > Peter. > --00000000000027764c06244bb20d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I was able to boot Mezzano completely (though without a wo= rking mouse, just like on my real PC). Both ahci and virtio disks work. Thi= s is a list of issues I found:

1) The problem with PIT. = Can be solved as you proposed or by patching Mezzano.
2) Mezzano = assumes that Intel AHCI controllers report no more than 6 ports. Can be sol= ved by patching Mezzano or defining MAX_PORTS to be 6 in usr.sbin/bhyve/pci= _ahci.c
3) According=C2=A0to=C2=A0https://wiki.osdev.org/PCI#Message_Sign= aled_Interrupts, interrupt line config register must be RW. Bhyve does = not support writing to it. I do not know a correct fix, this [1] workaround= helps, however.
4) Finally, I had a random deadlock in interrupt= handling for the virtio-net device. Likewise, I do not know how to fix it = correctly, but this [2] patch helped.

Used patches= :
[1] Workaround for the interrupt line. Forbid 1 byte writes to = 0x3c config register (the write is performed by UEFI firmware or bootloader= ).
diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_em= ul.c
index e91b4d0a1e20..6e9d5885a6b1 100644
--- a/usr.sbin/bhyve/pci= _emul.c
+++ b/usr.sbin/bhyve/pci_emul.c
@@ -160,8 +160,11 @@ static _= _inline void
=C2=A0CFGWRITE(struct pci_devinst *pi, int coff, uint32_t v= al, int bytes)
=C2=A0{

- =C2=A0 =C2=A0 =C2=A0 if (bytes =3D=3D 1)=
- =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pci_set_cfgdata8(pi,= coff, val);
+ =C2=A0 =C2=A0 =C2=A0 if (bytes =3D=3D 1) {
+ =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (coff !=3D PCIR_INTLINE) {
= + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 pci_set_cfgdata8(pi, coff, val);
+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 }
+ =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 else if (bytes =3D=3D 2)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 pci_set_cfgdata16(pi, coff, val);
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 else

[2] Workaround for a hang up in inter= rupt handling code:
diff --git a/usr.sbin/bhyve/virtio.h b/usr.sb= in/bhyve/virtio.h
index 4c6c8004b2d1..d1a901ff0069 100644
--- a/usr.s= bin/bhyve/virtio.h
+++ b/usr.sbin/bhyve/virtio.h
@@ -355,13 +355,13 @= @ vi_interrupt(struct virtio_softc *vs, uint8_t isr, uint16_t msix_idx)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (pci_msix_enabled(vs->vs_pi))
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pci_generate_msix(vs->v= s_pi, msix_idx);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 else {
- =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VS_LOCK(vs);
+ =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 //VS_LOCK(vs);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 vs->vs_isr |=3D isr;
- =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pci_generate_msi(vs->vs_pi, 0);
+ =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 //pci_generate_msi(vs->vs_= pi, 0);
=C2=A0#ifdef __amd64__
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 pci_lintr_assert(vs->vs_pi);
=C2=A0#endif
- =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VS_UNLOCK(vs);
+ =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 //VS_UNLOCK(vs);
=C2=A0 =C2=A0= =C2=A0 =C2=A0 }
=C2=A0}

Do you have any id= eas how to make proper patches for bhyve from these workarounds?
= P.S. After all 4 issues are addressed to, you can build Demo 5 image, but o= nly after replacing a bootloader. See=C2=A0https://github.com/froggey/Mezzano/issues/173= If you wish, I can send the image to you by some means (Telegram?) Or you = can build the bootloader from here:=C2=A0https://github.com/froggey/kboot/tree/mezzan= o-loader (requires Linux, I guess).

I'll t= ry virtio-input to see if the mouse works this way.

=D1=81=D0=B1, 12 = =D0=BE=D0=BA=D1=82. 2024=E2=80=AF=D0=B3. =D0=B2 06:12, Peter Grehan <grehan@freebsd.org>:
> I suspect PCI interrupts are not functioning correctly.<= br> >
> Look at this code:
>=C2=A0 =C2=A0 =C2=A0 ;; Attach interrupt handler.
>=C2=A0 =C2=A0 =C2=A0 (sup:debug-print-line "Handler: " (ahci-= irq-handler ahci))
>=C2=A0 =C2=A0 =C2=A0 (sup:irq-attach (sup:platform-irq (pci:pci-intr-li= ne location))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 (ahci-irq-handler-function ahci)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 ahci)
>
> and this
>
> (defun pci-intr-line (device)
>=C2=A0 =C2=A0 (pci-config/8 device +pci-config-intr-line+)) ;; comment = by me: the
> constant is #x3c
>
> I found that "PCI 0x3c" means PCI interrupt pin. AFAIK, inte= rrupt pins
> are not supported by bhyve, is that=C2=A0correct? If it's true, I = need either
> to teach bhyve how to deal with legacy interrupts or to teach Mezzano = to
> understand MSI. What would be easier in your=C2=A0opinion?

=C2=A0 Legacy interrupts should work fine in bhyve for emulated devices. I&= #39;d
suspect this would be much easier to debug/enhance as opposed to adding MSI (and likely MSI-x).

later,

Peter.
--00000000000027764c06244bb20d--