Date: Sat, 25 Nov 2023 23:20:33 +0100 From: Mario Marietto <marietto2008@gmail.com> To: Warner Losh <imp@bsdimp.com> Cc: freebsd-hackers <freebsd-hackers@freebsd.org>, freebsd-arm@freebsd.org, FreeBSD Current <freebsd-current@freebsd.org>, FreeBSD Mailing List <freebsd-questions@freebsd.org>, freebsd-xen@freebsd.org, royger@freebsd.org Subject: Re: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How? Message-ID: <CA%2B1FSijGW9Mxp6N5H2q%2BjPJmozEtDmtvqr_LQf1SHvQnO33LaQ@mail.gmail.com> In-Reply-To: <CANCZdfrt9xBN4QfU8qnxKyMHUVVD37w84Dah4NpG-FLmOncYoA@mail.gmail.com> References: <CA%2B1FSii=D_X1s7UdH1Gn5W1EQnUp-0QKhpL-X-3Od6fDGw7YQA@mail.gmail.com> <CA%2B1FSiivBQoyjWcn69huq%2BeJNyoLH6KdB7uC6gkeooPWBeUOEg@mail.gmail.com> <CANCZdfrt9xBN4QfU8qnxKyMHUVVD37w84Dah4NpG-FLmOncYoA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000005ff95f060b017cde Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I really don't know where to start. It sounds too technical for me. I don't even know what to ask to understand where to start. mmm...can someone provide some documentation ? but not too much dense with technicalities. Thanks. On Sat, Nov 25, 2023 at 6:02=E2=80=AFPM Warner Losh <imp@bsdimp.com> wrote: > > > On Sat, Nov 25, 2023 at 4:41=E2=80=AFAM Mario Marietto <marietto2008@gmai= l.com> > wrote: > >> Hello to everyone. >> >> we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As >> host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It >> works great. But our goal is different. We want to virtualize FreeBSD as >> domU. Can we have a working Xen PV network driver for a FreeBSD arm gues= t >> ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on ar= m. >> I would like to know if Julien's work was accepted upstream by FreeBSD, = in >> which case FreeBSD as a Xen guest on arm should work if we enable the Xe= n >> PV drivers in the FreeBSD on arm kernel. If Julien's work was not accept= ed >> upstream by FreeBSD, we will have to find his patches and apply them >> ourselves to the FreeBSD on arm kernel. >> >> We found these slides : >> >> >> https://events.static.linuxfound.org/sites/events/files/slides/Porting%2= 0FreeBSD%20on%20Xen%20on%20ARM%20.pdf >> >> Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what >> we want to find. >> >> It looks like when that slide presentation was written, there were some >> limitations on FreeBSD Xen guests. For example, for our debian bookworm >> guest, I am using vcpus =3D '2' to match the number of real cpus on our >> Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD >> guest, so I will need to change that vcpus =3D '1' in the FreeBSD guest >> config unless support for 2 or more vcpus was added later, which is >> possible because that slide presentation is 9 years old. >> >> Here is where I would expect to find the XENHVM FreeBSD on arm kernel >> config file: >> >> https://cgit.freebsd.org/src/tree/sys/arm/conf >> >> But it is not there unless I am not understanding something correctly. >> For now, unfortunately conclude that the support for Xen on arm that Jul= ien >> Grall mentioned in that slide presentation 9 years ago was never added t= o >> the official FreeBSD source code. I am searching the web now to see if t= he >> patches that Julien Grall wrote are still posted somewhere online. If we >> cannot find them, we can ask here and on the xen-users mailing list. Jul= ien >> regularly reads that list and responds to question about Xen on arm, so = I >> think he will tell us how to find the patches if we cannot find them onl= ine. >> >> According to this page from the FreeBSD wiki: >> >> https://wiki.freebsd.org/Xen >> >> I think FreeBSD only supports Xen on x86, not arm. So this is going to b= e >> a bit of a challenge to get a Xen FreeBSD guest on arm working. We know >> Julien Grall has some patches that made it work in the past ! >> >> I found a slightly newer slide presentation by Julien here: >> >> https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd >> >> It is about the same, but it mentions the GENERIC FreeBSD kernel support= s >> Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen o= n >> arm 32 bit, which I haven't found online yet. >> >> Please,take a look at this output of the linux kernel that can boot on >> Xen, and the FreeBSD kernel that cannot : >> >> >> % file zImage-6.1.59-stb-xen-cbe+ >> zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (lit= tle-endian) >> >> % file FREEBSD-XENVIRT >> FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV),= dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), = not stripped >> >> >> The FreeBSD kernel that won't boot is in ELF format but the Linux kernel >> that does boot is in zImage format. >> >> I spent time reading the docs on xenbits.xenproject.org, and according >> to those docs Xen on arm only knows how to boot a kernel in the zImage >> format, so the FreeBSD kernel is in a format that modern Xen incorrectly >> detects as an x86 kernel. >> >> I also watched Julien Grall's 30 minute video presentation of his work t= o >> boot FreeBSD/arm on Xen at FOSDEM 2014 here : >> >> https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/ >> >> In that video, and in other places, Julien mentions that the boot ABI fo= r >> FreeBSD/arm on Xen was not yet developed and he was getting occasional >> crashes and needed to investigate the problem. He mentioned the zImage A= BI >> that Linux uses, but pointed out FreeBSD does not use that format, and b= ack >> then it was an open question which format to use to boot FreeBSD/arm on >> Xen. Unfortunately, nine years later, the only supported format is still >> the zImage format that Linux uses. >> >> It looks like Julien's work back then was using an ELF binary to boot >> FreeBSD/arm on Xen instead of the supported zImage format that Linux use= s >> and the modern Xen toolstack exits with an error when trying to boot the >> FreeBSD ELF formatted binary that Julien's patch creates. So the best >> solution would be to try to port the rules to build a FreeBSD kernel in = the >> zImage format instead of the ELF format. I have been studying the Makefi= les >> in Linux to see how Linux builds the Linux arm kernel in the zImage form= at, >> but it is not trivial to understand >> > > Look at kernel.bin in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. I= t > should be easy to adapt the target to build that. I've done similar thing= s > with u-boot formats in the past, but that was 4 employers and 20 years ag= o > now. > > This path is not well trod. I do know that arm64 virtualization with bhyv= e > is hitting the tree. I'm not sure how easy/hard this will be to modernize= . > I'm interested to see how your explorations go. > > Warner > --=20 Mario. --0000000000005ff95f060b017cde Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I really don't know where to start. It sounds too tech= nical for me. I don't even know what to ask to understand where to star= t. mmm...can someone provide some documentation ? but not too much dense wi= th technicalities. Thanks. <br></div><br><div class=3D"gmail_quote"><div di= r=3D"ltr" class=3D"gmail_attr">On Sat, Nov 25, 2023 at 6:02=E2=80=AFPM Warn= er Losh <<a href=3D"mailto:imp@bsdimp.com">imp@bsdimp.com</a>> wrote:= <br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr= "><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote"><div dir=3D"ltr= " class=3D"gmail_attr">On Sat, Nov 25, 2023 at 4:41=E2=80=AFAM Mario Mariet= to <<a href=3D"mailto:marietto2008@gmail.com" target=3D"_blank">marietto= 2008@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" st= yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd= ing-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr">= Hello to everyone.<div><div><p>we have just virtualized Debian 12 on our ar= m (32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It=20 works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm=20 guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by= =20 FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we=20 enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's wor= k was not accepted upstream by FreeBSD, we will have to find his patches=20 and apply them ourselves to the FreeBSD on arm kernel.</p> <p>We found these slides :</p> <p><a href=3D"https://events.static.linuxfound.org/sites/events/files/slide= s/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf" rel=3D"nofollow noreferre= r" target=3D"_blank">https://events.static.linuxfound.org/sites/events/file= s/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf</a></p> <p>Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what = we want to find.</p> <p>It looks like when that slide presentation was written, there were=20 some limitations on FreeBSD Xen guests. For example, for our debian=20 bookworm guest, I am using vcpus =3D '2' to match the number of rea= l cpus=20 on our Chromebook, but slide 13 mentions support for only 1 VCPU with a=20 FreeBSD guest, so I will need to change that vcpus =3D '1' in the F= reeBSD=20 guest config unless support for 2 or more vcpus was added later, which=20 is possible because that slide presentation is 9 years old.</p> <p>Here is where I would expect to find the XENHVM FreeBSD on arm kernel co= nfig file:</p> <p><a href=3D"https://cgit.freebsd.org/src/tree/sys/arm/conf" rel=3D"nofoll= ow noreferrer" target=3D"_blank">https://cgit.freebsd.org/src/tree/sys/arm/= conf</a></p> <p>But it is not there unless I am not understanding something=20 correctly. For now, unfortunately conclude that the support for Xen on=20 arm that Julien Grall mentioned in that slide presentation 9 years ago=20 was never added to the official FreeBSD source code. I am searching the=20 web now to see if the patches that Julien Grall wrote are still posted=20 somewhere online. If we cannot find them, we can ask here and on the=20 xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the=20 patches if we cannot find them online.</p> <p>According to this page from the FreeBSD wiki:</p> <p><a href=3D"https://wiki.freebsd.org/Xen" rel=3D"nofollow noreferrer" tar= get=3D"_blank">https://wiki.freebsd.org/Xen</a></p>; <p>I think FreeBSD only supports Xen on x86, not arm. So this is going=20 to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !</p> <p>I found a slightly newer slide presentation by Julien here:</p> <p><a href=3D"https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-por= t-your-bsd" rel=3D"nofollow noreferrer" target=3D"_blank">https://www.slide= share.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd</a></p> <p>It is about the same, but it mentions the GENERIC FreeBSD kernel=20 supports Xen on arm64, but still says we need the XENHVM FreeBSD config=20 for Xen on arm 32 bit, which I haven't found online yet.</p> <p>Please,take a look at this output of the linux kernel that can boot on X= en, and the FreeBSD kernel that cannot :</p><p><br></p> <pre><code>% file zImage-6.1.59-stb-xen-cbe+ zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little= -endian) % file FREEBSD-XENVIRT =20 FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dy= namically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not= stripped </code></pre> <p><br></p><p>The FreeBSD kernel that won't boot is in ELF format but t= he Linux kernel that does boot is in zImage format.</p> <p>I spent time reading the docs on <a href=3D"http://xenbits.xenproject.or= g" target=3D"_blank">xenbits.xenproject.org</a>, and=20 according to those docs Xen on arm only knows how to boot a kernel in=20 the zImage format, so the FreeBSD kernel is in a format that modern Xen=20 incorrectly detects as an x86 kernel.</p> <p>I also watched Julien Grall's 30 minute video presentation of his wo= rk to boot FreeBSD/arm on Xen at FOSDEM 2014 here :</p> <p><a href=3D"https://archive.fosdem.org/2014/schedule/event/freebsd_xen_ar= m/" rel=3D"nofollow noreferrer" target=3D"_blank">https://archive.fosdem.or= g/2014/schedule/event/freebsd_xen_arm/</a></p> <p>In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting=20 occasional crashes and needed to investigate the problem. He mentioned=20 the zImage ABI that Linux uses, but pointed out FreeBSD does not use=20 that format, and back then it was an open question which format to use=20 to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only=20 supported format is still the zImage format that Linux uses.</p> <p>It looks like Julien's work back then was using an ELF binary to boo= t FreeBSD/arm on Xen instead of the supported zImage format that Linux=20 uses and the modern Xen toolstack exits with an error when trying to=20 boot the FreeBSD ELF formatted binary that Julien's patch creates. So= =20 the best solution would be to try to port the rules to build a FreeBSD=20 kernel in the zImage format instead of the ELF format. I have been=20 studying the Makefiles in Linux to see how Linux builds the Linux arm=20 kernel in the zImage format, but it is not trivial to understand</p></div><= /div></div></div></div></blockquote><div><br></div><div>Look at kernel.bin = in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be e= asy to adapt the target to build that. I've done similar things with u-= boot formats in the past, but that was 4 employers and 20 years ago now.<br= ></div><div><br></div><div>This path is not well trod. I do know that arm64= virtualization with bhyve is hitting the tree. I'm not sure how easy/h= ard this will be to modernize. I'm interested to see how your explorati= ons go.<br></div><div><br></div><div>Warner <br></div></div></div> </blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre= fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Mario.<br></d= iv> --0000000000005ff95f060b017cde--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSijGW9Mxp6N5H2q%2BjPJmozEtDmtvqr_LQf1SHvQnO33LaQ>