Date: Sat, 25 Nov 2023 10:02:43 -0700 From: Warner Losh <imp@bsdimp.com> To: Mario Marietto <marietto2008@gmail.com> Cc: freebsd-hackers <freebsd-hackers@freebsd.org>, freebsd-arm@freebsd.org, FreeBSD Current <freebsd-current@freebsd.org>, FreeBSD Mailing List <freebsd-questions@freebsd.org>, freebsd-xen@freebsd.org, royger@freebsd.org Subject: Re: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How? Message-ID: <CANCZdfrt9xBN4QfU8qnxKyMHUVVD37w84Dah4NpG-FLmOncYoA@mail.gmail.com> In-Reply-To: <CA%2B1FSiivBQoyjWcn69huq%2BeJNyoLH6KdB7uC6gkeooPWBeUOEg@mail.gmail.com> References: <CA%2B1FSii=D_X1s7UdH1Gn5W1EQnUp-0QKhpL-X-3Od6fDGw7YQA@mail.gmail.com> <CA%2B1FSiivBQoyjWcn69huq%2BeJNyoLH6KdB7uC6gkeooPWBeUOEg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000021c269060afd0a2c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Nov 25, 2023 at 4:41=E2=80=AFAM Mario Marietto <marietto2008@gmail.= com> wrote: > Hello to everyone. > > we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As hos= t > / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It works > great. But our goal is different. We want to virtualize FreeBSD as domU. > Can we have a working Xen PV network driver for a FreeBSD arm guest ?. I > found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I > would like to know if Julien's work was accepted upstream by FreeBSD, in > which case FreeBSD as a Xen guest on arm should work if we enable the Xen > PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepte= d > upstream by FreeBSD, we will have to find his patches and apply them > ourselves to the FreeBSD on arm kernel. > > We found these slides : > > > https://events.static.linuxfound.org/sites/events/files/slides/Porting%20= FreeBSD%20on%20Xen%20on%20ARM%20.pdf > > Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what w= e > want to find. > > It looks like when that slide presentation was written, there were some > limitations on FreeBSD Xen guests. For example, for our debian bookworm > guest, I am using vcpus =3D '2' to match the number of real cpus on our > Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD > guest, so I will need to change that vcpus =3D '1' in the FreeBSD guest > config unless support for 2 or more vcpus was added later, which is > possible because that slide presentation is 9 years old. > > Here is where I would expect to find the XENHVM FreeBSD on arm kernel > config file: > > https://cgit.freebsd.org/src/tree/sys/arm/conf > > But it is not there unless I am not understanding something correctly. Fo= r > now, unfortunately conclude that the support for Xen on arm that Julien > Grall mentioned in that slide presentation 9 years ago was never added to > the official FreeBSD source code. I am searching the web now to see if th= e > patches that Julien Grall wrote are still posted somewhere online. If we > cannot find them, we can ask here and on the xen-users mailing list. Juli= en > regularly reads that list and responds to question about Xen on arm, so I > think he will tell us how to find the patches if we cannot find them onli= ne. > > According to this page from the FreeBSD wiki: > > https://wiki.freebsd.org/Xen > > I think FreeBSD only supports Xen on x86, not arm. So this is going to be > a bit of a challenge to get a Xen FreeBSD guest on arm working. We know > Julien Grall has some patches that made it work in the past ! > > I found a slightly newer slide presentation by Julien here: > > https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd > > It is about the same, but it mentions the GENERIC FreeBSD kernel supports > Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on > arm 32 bit, which I haven't found online yet. > > Please,take a look at this output of the linux kernel that can boot on > Xen, and the FreeBSD kernel that cannot : > > > % file zImage-6.1.59-stb-xen-cbe+ > zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (litt= le-endian) > > % file FREEBSD-XENVIRT > FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), = dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), n= ot stripped > > > The FreeBSD kernel that won't boot is in ELF format but the Linux kernel > that does boot is in zImage format. > > I spent time reading the docs on xenbits.xenproject.org, and according to > those docs Xen on arm only knows how to boot a kernel in the zImage forma= t, > so the FreeBSD kernel is in a format that modern Xen incorrectly detects = as > an x86 kernel. > > I also watched Julien Grall's 30 minute video presentation of his work to > boot FreeBSD/arm on Xen at FOSDEM 2014 here : > > https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/ > > In that video, and in other places, Julien mentions that the boot ABI for > FreeBSD/arm on Xen was not yet developed and he was getting occasional > crashes and needed to investigate the problem. He mentioned the zImage AB= I > that Linux uses, but pointed out FreeBSD does not use that format, and ba= ck > then it was an open question which format to use to boot FreeBSD/arm on > Xen. Unfortunately, nine years later, the only supported format is still > the zImage format that Linux uses. > > It looks like Julien's work back then was using an ELF binary to boot > FreeBSD/arm on Xen instead of the supported zImage format that Linux uses > and the modern Xen toolstack exits with an error when trying to boot the > FreeBSD ELF formatted binary that Julien's patch creates. So the best > solution would be to try to port the rules to build a FreeBSD kernel in t= he > zImage format instead of the ELF format. I have been studying the Makefil= es > in Linux to see how Linux builds the Linux arm kernel in the zImage forma= t, > but it is not trivial to understand > Look at kernel.bin in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be easy to adapt the target to build that. I've done similar things with u-boot formats in the past, but that was 4 employers and 20 years ago now. This path is not well trod. I do know that arm64 virtualization with bhyve is hitting the tree. I'm not sure how easy/hard this will be to modernize. I'm interested to see how your explorations go. Warner --00000000000021c269060afd0a2c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Sat, Nov 25, 2023 at 4:41=E2=80=AF= AM Mario Marietto <<a href=3D"mailto:marietto2008@gmail.com">marietto200= 8@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"><div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr">Hel= lo to everyone.<div><div><p>we have just virtualized Debian 12 on our arm (= 32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It=20 works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm=20 guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by= =20 FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we=20 enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's wor= k was not accepted upstream by FreeBSD, we will have to find his patches=20 and apply them ourselves to the FreeBSD on arm kernel.</p> <p>We found these slides :</p> <p><a href=3D"https://events.static.linuxfound.org/sites/events/files/slide= s/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf" rel=3D"nofollow noreferre= r" target=3D"_blank">https://events.static.linuxfound.org/sites/events/file= s/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf</a></p> <p>Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what = we want to find.</p> <p>It looks like when that slide presentation was written, there were=20 some limitations on FreeBSD Xen guests. For example, for our debian=20 bookworm guest, I am using vcpus =3D '2' to match the number of rea= l cpus=20 on our Chromebook, but slide 13 mentions support for only 1 VCPU with a=20 FreeBSD guest, so I will need to change that vcpus =3D '1' in the F= reeBSD=20 guest config unless support for 2 or more vcpus was added later, which=20 is possible because that slide presentation is 9 years old.</p> <p>Here is where I would expect to find the XENHVM FreeBSD on arm kernel co= nfig file:</p> <p><a href=3D"https://cgit.freebsd.org/src/tree/sys/arm/conf" rel=3D"nofoll= ow noreferrer" target=3D"_blank">https://cgit.freebsd.org/src/tree/sys/arm/= conf</a></p> <p>But it is not there unless I am not understanding something=20 correctly. For now, unfortunately conclude that the support for Xen on=20 arm that Julien Grall mentioned in that slide presentation 9 years ago=20 was never added to the official FreeBSD source code. I am searching the=20 web now to see if the patches that Julien Grall wrote are still posted=20 somewhere online. If we cannot find them, we can ask here and on the=20 xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the=20 patches if we cannot find them online.</p> <p>According to this page from the FreeBSD wiki:</p> <p><a href=3D"https://wiki.freebsd.org/Xen" rel=3D"nofollow noreferrer" tar= get=3D"_blank">https://wiki.freebsd.org/Xen</a></p>; <p>I think FreeBSD only supports Xen on x86, not arm. So this is going=20 to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !</p> <p>I found a slightly newer slide presentation by Julien here:</p> <p><a href=3D"https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-por= t-your-bsd" rel=3D"nofollow noreferrer" target=3D"_blank">https://www.slide= share.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd</a></p> <p>It is about the same, but it mentions the GENERIC FreeBSD kernel=20 supports Xen on arm64, but still says we need the XENHVM FreeBSD config=20 for Xen on arm 32 bit, which I haven't found online yet.</p> <p>Please,take a look at this output of the linux kernel that can boot on X= en, and the FreeBSD kernel that cannot :</p><p><br></p> <pre><code>% file zImage-6.1.59-stb-xen-cbe+ zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little= -endian) % file FREEBSD-XENVIRT =20 FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dy= namically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not= stripped </code></pre> <p><br></p><p>The FreeBSD kernel that won't boot is in ELF format but t= he Linux kernel that does boot is in zImage format.</p> <p>I spent time reading the docs on <a href=3D"http://xenbits.xenproject.or= g" target=3D"_blank">xenbits.xenproject.org</a>, and=20 according to those docs Xen on arm only knows how to boot a kernel in=20 the zImage format, so the FreeBSD kernel is in a format that modern Xen=20 incorrectly detects as an x86 kernel.</p> <p>I also watched Julien Grall's 30 minute video presentation of his wo= rk to boot FreeBSD/arm on Xen at FOSDEM 2014 here :</p> <p><a href=3D"https://archive.fosdem.org/2014/schedule/event/freebsd_xen_ar= m/" rel=3D"nofollow noreferrer" target=3D"_blank">https://archive.fosdem.or= g/2014/schedule/event/freebsd_xen_arm/</a></p> <p>In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting=20 occasional crashes and needed to investigate the problem. He mentioned=20 the zImage ABI that Linux uses, but pointed out FreeBSD does not use=20 that format, and back then it was an open question which format to use=20 to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only=20 supported format is still the zImage format that Linux uses.</p> <p>It looks like Julien's work back then was using an ELF binary to boo= t FreeBSD/arm on Xen instead of the supported zImage format that Linux=20 uses and the modern Xen toolstack exits with an error when trying to=20 boot the FreeBSD ELF formatted binary that Julien's patch creates. So= =20 the best solution would be to try to port the rules to build a FreeBSD=20 kernel in the zImage format instead of the ELF format. I have been=20 studying the Makefiles in Linux to see how Linux builds the Linux arm=20 kernel in the zImage format, but it is not trivial to understand</p></div><= /div></div></div></div></blockquote><div><br></div><div>Look at kernel.bin = in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be e= asy to adapt the target to build that. I've done similar things with u-= boot formats in the past, but that was 4 employers and 20 years ago now.<br= ></div><div><br></div><div>This path is not well trod. I do know that arm64= virtualization with bhyve is hitting the tree. I'm not sure how easy/h= ard this will be to modernize. I'm interested to see how your explorati= ons go.<br></div><div><br></div><div>Warner <br></div></div></div> --00000000000021c269060afd0a2c--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfrt9xBN4QfU8qnxKyMHUVVD37w84Dah4NpG-FLmOncYoA>