From nobody Sat Nov 25 17:02:43 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ScypS4kdkz51jwb for ; Sat, 25 Nov 2023 17:02:56 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ScypR6fkGz3Fvc for ; Sat, 25 Nov 2023 17:02:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-a0029289b1bso403911566b.1 for ; Sat, 25 Nov 2023 09:02:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1700931773; x=1701536573; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=lJcyYwjFsGDCNUEF2SrPc1fBl800LPaPtK2E4VfkqSk=; b=Em3noFN9pbttYLD3ebSyUU2FmX67+VwjZcNMCmbA+67sOfb4IkdZQINaPY9jAaLvvG CnME70BylLfHioC2TETwbQuCqtJKzQQc5x/W7EuZ1NXwjclKy/dt4Vg0Y5L8DJAj3e2w xeiCNWSOdR0qwRYnz54Z6rEuUNorj/QMXmwNudqsAV5NLwXdz8G0SipvjKTRKONgCAIo tu4CFB4dXTRogK5vNuQBKjA5ZUXO7nS2aBDlvna6q3TmPIS4Tnm5OVZuKNr8UI3mi6Cf SsCuXfcdqL1MgzEcxnGsIDWHDCw/IFE9t8EV7rG1GQS8WXsTBoekDFC7pc6OEzDRq9T3 qwBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700931773; x=1701536573; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lJcyYwjFsGDCNUEF2SrPc1fBl800LPaPtK2E4VfkqSk=; b=DZZDHDNOjhHcVEQXS2fvqiL2i2k4c0b87HYkd1P0UByMTBRSCNJ05H0cx4BcOV1Gyl uqdM7BdnmhDKkte3KdfPeCJbd98LUn3c97Gm6Pikf8ftz/n2W6B7Pev5gdOmHVQv/CWk t8aYSaa56uCtz6/Me3fofmhNBSYyllGOef42s/RY2v7x8LHUr4RYK7Tl2HtLCQsFs9Gs 3eynuv6vkUqhaQVAcoyH0ftPl0l54ILtzvbvoF1VqNfUcucL18YKOM8ACZyDR5fyeOvX diEMX8VaaLdtDiL9d0iwvnASdqBa0HjX1La4SHmJ9NWYoNMTRT5FOzYxTyz/xDNjtfec stdQ== X-Gm-Message-State: AOJu0YwBK0gaOcCM0w1e8N7Mbgp7WS5au7TewgRXFeP1UpKdIK/8H8Nu b7b98av6Chg+7WYeMMsAvmiJzTHCezj7iJpakJP8fw== X-Google-Smtp-Source: AGHT+IGt9WzvZLY0tUhXeNt9BnWnm2hSaeduHPjx5ve9C1ZRnPuOtAra26+KL+N3o0hRlFoli27UFdGuH6gX0G+ZPys= X-Received: by 2002:a17:906:693:b0:9fc:9b28:7ff7 with SMTP id u19-20020a170906069300b009fc9b287ff7mr5534824ejb.60.1700931773247; Sat, 25 Nov 2023 09:02:53 -0800 (PST) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Sat, 25 Nov 2023 10:02:43 -0700 Message-ID: Subject: Re: Should we boot the FreeBSD kernel in ELF format or in zImage format ? How? To: Mario Marietto Cc: freebsd-hackers , freebsd-arm@freebsd.org, FreeBSD Current , FreeBSD Mailing List , freebsd-xen@freebsd.org, royger@freebsd.org Content-Type: multipart/alternative; boundary="00000000000021c269060afd0a2c" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4ScypR6fkGz3Fvc --00000000000021c269060afd0a2c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Nov 25, 2023 at 4:41=E2=80=AFAM Mario Marietto wrote: > Hello to everyone. > > we have just virtualized Debian 12 on our arm (32 bit) Chromebook. As hos= t > / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It works > great. But our goal is different. We want to virtualize FreeBSD as domU. > Can we have a working Xen PV network driver for a FreeBSD arm guest ?. I > found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I > would like to know if Julien's work was accepted upstream by FreeBSD, in > which case FreeBSD as a Xen guest on arm should work if we enable the Xen > PV drivers in the FreeBSD on arm kernel. If Julien's work was not accepte= d > upstream by FreeBSD, we will have to find his patches and apply them > ourselves to the FreeBSD on arm kernel. > > We found these slides : > > > https://events.static.linuxfound.org/sites/events/files/slides/Porting%20= FreeBSD%20on%20Xen%20on%20ARM%20.pdf > > Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what w= e > want to find. > > It looks like when that slide presentation was written, there were some > limitations on FreeBSD Xen guests. For example, for our debian bookworm > guest, I am using vcpus =3D '2' to match the number of real cpus on our > Chromebook, but slide 13 mentions support for only 1 VCPU with a FreeBSD > guest, so I will need to change that vcpus =3D '1' in the FreeBSD guest > config unless support for 2 or more vcpus was added later, which is > possible because that slide presentation is 9 years old. > > Here is where I would expect to find the XENHVM FreeBSD on arm kernel > config file: > > https://cgit.freebsd.org/src/tree/sys/arm/conf > > But it is not there unless I am not understanding something correctly. Fo= r > now, unfortunately conclude that the support for Xen on arm that Julien > Grall mentioned in that slide presentation 9 years ago was never added to > the official FreeBSD source code. I am searching the web now to see if th= e > patches that Julien Grall wrote are still posted somewhere online. If we > cannot find them, we can ask here and on the xen-users mailing list. Juli= en > regularly reads that list and responds to question about Xen on arm, so I > think he will tell us how to find the patches if we cannot find them onli= ne. > > According to this page from the FreeBSD wiki: > > https://wiki.freebsd.org/Xen > > I think FreeBSD only supports Xen on x86, not arm. So this is going to be > a bit of a challenge to get a Xen FreeBSD guest on arm working. We know > Julien Grall has some patches that made it work in the past ! > > I found a slightly newer slide presentation by Julien here: > > https://www.slideshare.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd > > It is about the same, but it mentions the GENERIC FreeBSD kernel supports > Xen on arm64, but still says we need the XENHVM FreeBSD config for Xen on > arm 32 bit, which I haven't found online yet. > > Please,take a look at this output of the linux kernel that can boot on > Xen, and the FreeBSD kernel that cannot : > > > % file zImage-6.1.59-stb-xen-cbe+ > zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (litt= le-endian) > > % file FREEBSD-XENVIRT > FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), = dynamically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), n= ot stripped > > > The FreeBSD kernel that won't boot is in ELF format but the Linux kernel > that does boot is in zImage format. > > I spent time reading the docs on xenbits.xenproject.org, and according to > those docs Xen on arm only knows how to boot a kernel in the zImage forma= t, > so the FreeBSD kernel is in a format that modern Xen incorrectly detects = as > an x86 kernel. > > I also watched Julien Grall's 30 minute video presentation of his work to > boot FreeBSD/arm on Xen at FOSDEM 2014 here : > > https://archive.fosdem.org/2014/schedule/event/freebsd_xen_arm/ > > In that video, and in other places, Julien mentions that the boot ABI for > FreeBSD/arm on Xen was not yet developed and he was getting occasional > crashes and needed to investigate the problem. He mentioned the zImage AB= I > that Linux uses, but pointed out FreeBSD does not use that format, and ba= ck > then it was an open question which format to use to boot FreeBSD/arm on > Xen. Unfortunately, nine years later, the only supported format is still > the zImage format that Linux uses. > > It looks like Julien's work back then was using an ELF binary to boot > FreeBSD/arm on Xen instead of the supported zImage format that Linux uses > and the modern Xen toolstack exits with an error when trying to boot the > FreeBSD ELF formatted binary that Julien's patch creates. So the best > solution would be to try to port the rules to build a FreeBSD kernel in t= he > zImage format instead of the ELF format. I have been studying the Makefil= es > in Linux to see how Linux builds the Linux arm kernel in the zImage forma= t, > but it is not trivial to understand > Look at kernel.bin in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be easy to adapt the target to build that. I've done similar things with u-boot formats in the past, but that was 4 employers and 20 years ago now. This path is not well trod. I do know that arm64 virtualization with bhyve is hitting the tree. I'm not sure how easy/hard this will be to modernize. I'm interested to see how your explorations go. Warner --00000000000021c269060afd0a2c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sat, Nov 25, 2023 at 4:41=E2=80=AF= AM Mario Marietto <marietto200= 8@gmail.com> wrote:
Hel= lo to everyone.

we have just virtualized Debian 12 on our arm (= 32 bit) Chromebook. As host / dom0 we have chosen Devuan 5,and for guest / domU,Debian 12. It=20 works great. But our goal is different. We want to virtualize FreeBSD as domU. Can we have a working Xen PV network driver for a FreeBSD arm=20 guest ?. I found that Julien Grall has ported the Xen drivers to FreeBSD on arm. I would like to know if Julien's work was accepted upstream by= =20 FreeBSD, in which case FreeBSD as a Xen guest on arm should work if we=20 enable the Xen PV drivers in the FreeBSD on arm kernel. If Julien's wor= k was not accepted upstream by FreeBSD, we will have to find his patches=20 and apply them ourselves to the FreeBSD on arm kernel.

We found these slides :

https://events.static.linuxfound.org/sites/events/file= s/slides/Porting%20FreeBSD%20on%20Xen%20on%20ARM%20.pdf

Slide 13 refers to a XENHVM FreeBSD on arm kernel config - that is what = we want to find.

It looks like when that slide presentation was written, there were=20 some limitations on FreeBSD Xen guests. For example, for our debian=20 bookworm guest, I am using vcpus =3D '2' to match the number of rea= l cpus=20 on our Chromebook, but slide 13 mentions support for only 1 VCPU with a=20 FreeBSD guest, so I will need to change that vcpus =3D '1' in the F= reeBSD=20 guest config unless support for 2 or more vcpus was added later, which=20 is possible because that slide presentation is 9 years old.

Here is where I would expect to find the XENHVM FreeBSD on arm kernel co= nfig file:

https://cgit.freebsd.org/src/tree/sys/arm/= conf

But it is not there unless I am not understanding something=20 correctly. For now, unfortunately conclude that the support for Xen on=20 arm that Julien Grall mentioned in that slide presentation 9 years ago=20 was never added to the official FreeBSD source code. I am searching the=20 web now to see if the patches that Julien Grall wrote are still posted=20 somewhere online. If we cannot find them, we can ask here and on the=20 xen-users mailing list. Julien regularly reads that list and responds to question about Xen on arm, so I think he will tell us how to find the=20 patches if we cannot find them online.

According to this page from the FreeBSD wiki:

https://wiki.freebsd.org/Xen

I think FreeBSD only supports Xen on x86, not arm. So this is going=20 to be a bit of a challenge to get a Xen FreeBSD guest on arm working. We know Julien Grall has some patches that made it work in the past !

I found a slightly newer slide presentation by Julien here:

https://www.slide= share.net/xen_com_mgr/bsdcan-2015-how-to-port-your-bsd

It is about the same, but it mentions the GENERIC FreeBSD kernel=20 supports Xen on arm64, but still says we need the XENHVM FreeBSD config=20 for Xen on arm 32 bit, which I haven't found online yet.

Please,take a look at this output of the linux kernel that can boot on X= en, and the FreeBSD kernel that cannot :


% file zImage-6.1.59-stb-xen-cbe+
zImage-6.1.59-stb-xen-cbe+: Linux kernel ARM boot executable zImage (little=
-endian)

% file FREEBSD-XENVIRT         =20
FREEBSD-XENVIRT: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dy=
namically linked, interpreter /red/herring, for FreeBSD 11.0 (1100048), not=
 stripped


The FreeBSD kernel that won't boot is in ELF format but t= he Linux kernel that does boot is in zImage format.

I spent time reading the docs on xenbits.xenproject.org, and=20 according to those docs Xen on arm only knows how to boot a kernel in=20 the zImage format, so the FreeBSD kernel is in a format that modern Xen=20 incorrectly detects as an x86 kernel.

I also watched Julien Grall's 30 minute video presentation of his wo= rk to boot FreeBSD/arm on Xen at FOSDEM 2014 here :

https://archive.fosdem.or= g/2014/schedule/event/freebsd_xen_arm/

In that video, and in other places, Julien mentions that the boot ABI for FreeBSD/arm on Xen was not yet developed and he was getting=20 occasional crashes and needed to investigate the problem. He mentioned=20 the zImage ABI that Linux uses, but pointed out FreeBSD does not use=20 that format, and back then it was an open question which format to use=20 to boot FreeBSD/arm on Xen. Unfortunately, nine years later, the only=20 supported format is still the zImage format that Linux uses.

It looks like Julien's work back then was using an ELF binary to boo= t FreeBSD/arm on Xen instead of the supported zImage format that Linux=20 uses and the modern Xen toolstack exits with an error when trying to=20 boot the FreeBSD ELF formatted binary that Julien's patch creates. So= =20 the best solution would be to try to port the rules to build a FreeBSD=20 kernel in the zImage format instead of the ELF format. I have been=20 studying the Makefiles in Linux to see how Linux builds the Linux arm=20 kernel in the zImage format, but it is not trivial to understand

<= /div>

Look at kernel.bin = in FreeBSD's kernel. It's enabled -DWITH_KERNEL_BIN. It should be e= asy to adapt the target to build that. I've done similar things with u-= boot formats in the past, but that was 4 employers and 20 years ago now.

This path is not well trod. I do know that arm64= virtualization with bhyve is hitting the tree. I'm not sure how easy/h= ard this will be to modernize. I'm interested to see how your explorati= ons go.

Warner
--00000000000021c269060afd0a2c--