Date: Sun, 18 Jun 2017 18:22:11 -0700 From: Mark Millard <markmi@dsl-only.net> To: Russell Haley <russ.haley@gmail.com> Cc: "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org> Subject: Re: Creating armv7 MACHINE_ARCH Message-ID: <D1C69755-C249-4CAC-BFDB-70B4BCAEB5C6@dsl-only.net> In-Reply-To: <CABx9NuR4o=mNosZs%2BtGLYSwo-NyB6DdRg35r%2BaG1Snh8iXFcJw@mail.gmail.com> References: <CANCZdfpUjPBRpxpmjtwK-wpiK=%2BwHscS4UmVeatrE7vrm260tw@mail.gmail.com> <20170612152808.6094931.74364.27128@gmail.com> <CANCZdfrxTo8vLsnjU_VerO%2B3%2BU=06cok7%2BuKba3FM8_nXFozhQ@mail.gmail.com> <B19EDB95-2A23-4F8F-8414-3F4E0E65AC4B@dsl-only.net> <CABx9NuQTOkf6HK=RacUCBR=W_WDfgZwbVHYwsRdx0YJd=zr51w@mail.gmail.com> <2A90A527-7DCA-4442-9322-0EA96236583C@dsl-only.net> <3CC8DE8A-CCF2-4856-A43E-6B259BDE8B2C@dsl-only.net> <CANCZdfp6cnXdHxSQGTXHq4Md4Jh6=u4Af_rDM3k_RgoN%2BFrXWA@mail.gmail.com> <CABx9NuRJLgc0cSsykrSbJi=fVq-jfOi7ZEiZ5LRHG5tqibZ_2Q@mail.gmail.com> <C5D08468-DB46-4972-BFE2-44559A5EAB88@dsl-only.net> <CABx9NuR4o=mNosZs%2BtGLYSwo-NyB6DdRg35r%2BaG1Snh8iXFcJw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
FYI: I'll provide the following example of the types of issues that can show up in trying to support old 32-bit code on an armv8 via AArch32 state mode. This is from OpenSuse's = https://bugzilla.opensuse.org/show_bug.cgi?id=3D1029158 : Tom Rini wrote on 2017-03-13 15:11:25 UTC : > If one wants to use legacy aarch32 binaries, the CONFIG_COMPAT option = needs to be set in the kernel, and today we do this. This in turn = allows some aarch32 binaries to run easily. However, there is a large = world of software that is "armhf" but rather than being tuned for ARMv7 = is tuned for ARMv6 (and the RPi). In order for this to run = ARMV8_DEPRECATED and then at least CP15_BARRIER_EMULATION needs to be = enabled (the other 2 options would also likely be helpful). >=20 > The real-world example here is that at least today, the Docker "armhf" = scratch builds (and in turn lots of stuff is based on this) do not just = run and you will get for example: > [ 1961.636326] python[3661]: undefined instruction: = pc=3D00000000f6965dbc > [ 1961.636336] Code: e5821060 e3a01000 e5821064 e5821068 (ee070fba)=20 > (this is also true of busybox). > And then with the options enabled: > [ 37.605796] "ls" (1464) uses deprecated CP15 Barrier instruction at = 0xc95dc > ... > [ 72.888570] "python" (1881) uses deprecated CP15 Barrier = instruction at 0xf6ab4dbc So full support can involve software emulation of some instructions that otherwise would be classified as illegal instructions or possibly as deprecated instructions. OpenSuse had a fix merged into the kernel near the end of 2017-Mar. =3D=3D=3D Mark Millard markmi at dsl-only.net On 2017-Jun-12, at 10:37 PM, Russell Haley <russ.haley at gmail.com> = wrote: Thanks Mark! I'll chew on that for a couple of days. :D Russ On Mon, Jun 12, 2017 at 7:01 PM, Mark Millard <markmi@dsl-only.net> = wrote: >=20 > On 2017-Jun-12, at 4:10 PM, Russell Haley <russ.haley at gmail.com> = wrote: >=20 >> Okay, feel free to ignore me, I'm not going to get the time drill = into >> the source code for my own questions so I don't expect anyone else >> too. However, I'll ask anyway. I'm too confused to try and inline >> these questions. Lets see if I understand: >>=20 >> - armv7 does not support 64 bit instructions (according to Wikipedia? >> I claim no expertise.) >=20 > It does not support AArch64 instructions but does support AArch32 > instructions (or is a good approximation). >=20 > AArch32 was actually created later to be armv7-a like. >=20 >> - FreeBSD has an armv6 "architecture" that is supports armv6 and = armv7 >> on Pre-Cortex-A-53 processors that is not supported on A-53 through >> the emulated AArch32. >=20 > There are actually architectures (by ARM's > definitions). . . >=20 > ARMv6, ARMv6T2, ARMv6Z, ARMv6K, ARMV6-M > ARMv7-M, ARMv7E-M, ARMv7-R, ARMv7-A > ARMv8-A, ARMv8.1-A ARMv8.2-Am ARMV8.3-A, ARMv8-R, ARMv8-M >=20 > I'm going to largely ignore much of > that variation in structure. >=20 > ARM has continued to produce/make new variations for both > 32-bit and 64-bit in overlapping time frames. So the > "Pre-Cortex-A53" presumes to much about relative timing. >=20 > Technically the old RPI2s (V1.1 and before) have > Cortex-A7's and the new V1.2's have Cortex-A53's. > RPI3 is also Cortex-A53 based. >=20 > Cortex-A7 includes an implementation of armv7-A architecture > (but is not the only one). > Cortex-A53 includes an implementation of armv8-A architecture > (but is not the only one). > (Both include other things as well. And the System On A Chip has even > more stuff than the Cortex-A<?>'s in question.) >=20 > Cortex-A<?>'s are specific CPUs/cores that are also > examples of the specific architecture(s) they implement. >=20 > If a kernel supports armv7-a it can support a variety of > CPUs that all implement armv7-a architecture. It may not > support things in the CPUs that are beyond that > architecture. >=20 > If a kernel supports armv8-a it can support a variety of > CPUs that all implement armv8-a architecture. It may not > support things in the CPUs that are beyond that > architecture. >=20 > Note there are also issues of support of the System On > a Chip involved as well. More than ARM is involved > overall. >=20 >=20 >=20 >> - Cortex A-53 can support armv8 (AArch64) and armv7 (AArch32) = instructions >=20 > As I remember armv8-a specifies: >=20 > A) Optional support: AArch64 > (Cortex-A32 is ARMv8-A archtecture but only 32-bit) > (ARMv8-R architecture is always only 32-bit as I understand) > (ARMv8-M architecture is always only 32-bit as I understand) > B) Optional support: AArch32 > (Cortex-A35/53/57/72/73 are ARMv8-A and have both) > (Cortex-A55/75 are ARMV8.2-A architecture and have both) > (But variants can be produced that omit AArch32.) > C) Various things specific to armv8-a > might go beyond what AArch64 and > AArch32 specify. >=20 > Note: AArch32 has 2 instruction sets > (A32/ARM and T32/Thumb), like armv7-a > does. AArch64 has A64. (I'm not giving > thumb version numbers here but there > are versioned vintages.) >=20 > So 1 architecture (armv8-a) has at least > 3 instruction sets when both AArch64 > and AArch32 are implemented. >=20 > AArch64 state: A64 >=20 > AArch32 state: A32 and T32 >=20 > There can be context switching between > these states as I understand. >=20 > (I'll not get into the NEON distinctions > and such so the above is simplified.) >=20 > armv8-a does not necessarily uniquely identify > the interrupt controller or its software > interface, as an example of variation that is > not an instruction set issue: register > interfaces to other hardware. ARM provides > GIC-400 and GIC-500 and others. But the > controller does not have to come from ARM. >=20 > While Cortex-A53 has a default/reference > interrupt controller(s), it is possible to > build variations that use other ones as > I understand. >=20 > Similar points go for Cortex-A7. >=20 > The other parts of a System On a Chip are for non-arm > aspects and may have some mixed degree on commonality > independently of ARM. But the kernel may deal with > such specifics as well. >=20 >> - The current proposal is to split the armv6 and armv7 into their own >> "architectures" >=20 > ARM defines these as distinct architectures. > That is not a FreeBSD specific terminology. > But much of the armv6 is common with armv7. >=20 > armv7 architecture has more than armv6 architecture does: > it is an update. Having armv7 implemented separately > in FreeBSD means more completely/correctly identifying > what is there compared to pretending it is an armv6. >=20 >> FreeBSD has an Arm 64 bit kernel build. I don't see what the >> TARGET_ARCH settings in the wiki and know little about it, but will >> conjecture that it doesn't have a TARGET_ARCH=3Darmv8 (please correct = me >> if I'm wrong). >=20 > FreeBSD has TARGET=3Darm64 TARGET_ARCH=3Daarch64 . That combination > is for armv8-A with AArch64 currently. Cortex-A32 is not > supported because it lacks AArch64. >=20 > FreeBSD's TARGET=3Darm64 TARGET_ARCH=3Daarch64 may grow to span > armv8.1 or such as well for all I know. Cortex-A53 is one > example of an armv8 implementation that the kernel supports > (presuming certain interrupt controllers and such). There are > others. >=20 >> What I was trying to ask was: is the kernel development moving in a >> direction that clearly indicates the differences in the instructions >> vs the architectures (or have I grossly simplified the problem)? Will >> it be possible to target a Cortex-A53 and build for 32 or 64 bit >> support? Or is this just to fix RPi? >=20 > Instruction sets are part of an architecture but not all of it. > They are not independent of architecture. Talking of an overall > architecture includes covering the instruction set(s). One > architecture can have multiple instruction sets as part of it. >=20 > TARGET=3Darm64 TARGET_ARCH=3Daarch64 may get a "lib32" (compat32) > implementation someday if someone is motivated to do the > work. That would enable AArch32 user processes. >=20 > What exists now and will in the future is mostly tied to > choices of volunteers that say "I'm going to provide > <whatever>" and then manage to complete whatever it was. > (Clearly a "we are going to" is also an option.) >=20 > A similar points would go for TARGET_ARCH=3Darmv7 supporting > something like Cortex-A53 in AArch32 mode (Raspian like). >=20 > So at this point no one knows if or when as far as I know. >=20 > armv6 and armv7 architectures are strongly overlapping for what > is in armv6. The GENERIC kernel for what is currently a > TARGET_ARCH=3Darmv6 build actually builds for armv7 and will not > work on armv6. One has to use a different kernel configuration > than GENERIC for armv6 (such as the one for the older RPI vintage > that was armv6 based). >=20 > Identifying an armv7-a build as a armv6 build has odd > consequences for software that tries to figure out what > can be depended on. >=20 >> In terms of Raspbian, I had assumed they were just supporting Aarch32 >> across both processor models. Many of the drivers would be the same >> source if they share components so I would think it would be = "simple". >> I didn't see anything in my brief look at it today to indicate >> otherwise. >=20 > For Cortex-A53: Raspian supports just AArch32 state, > not AArch64 state for rpi2 V1.2 and RPI3. (I do not > know if early boot is temporarily AArch64 or not.) >=20 > For Cortex-A7: Cortex-A7 predates AArch32 and is armv7-a > directly but AArch32 was designed to appear to be be > nearly the same as armv7-a. So Raspian is supporting > armv7-a directly for rpi2 V1.1 and V1.0. >=20 > =3D=3D=3D > Mark Millard > markmi at dsl-only.net >=20 > On Mon, Jun 12, 2017 at 2:07 PM, Warner Losh <imp@bsdimp.com> wrote: >>=20 >>=20 >> On Mon, Jun 12, 2017 at 2:35 PM, Mark Millard <markmi@dsl-only.net> = wrote: >>>=20 >>>=20 >>> On 2017-Jun-12, at 1:00 PM, Mark Millard <markmi at dsl-only.net> = wrote: >>>=20 >>>> On Mon, Jun 12, 2017 at 1:00 PM, Mark Millard <markmi at = dsl-only.net> >>>> wrote: >>>>> On 2017-Jun-12, at 12:16 PM, Russell Haley <russ.haley at = gmail.com> >>>>> wrote: >>>>>=20 >>>>>> On Mon, Jun 12, 2017 at 10:36 AM, Mark Millard <markmi at >>>>>> dsl-only.net> wrote: >>>>>>>=20 >>>>>>> On 2017-Jun-12, at 8:39 AM, Warner Losh <imp at bsdimp.com> = wrote: >>>>>>>=20 >>>>>>>> . . . >>>>>>>>=20 >>>>>>>> Plus, we aren't quite doing what Ian wanted. He wanted a full >>>>>>>> rename. The >>>>>>>> proposal on the able is to add an armv7 TARGET_ARCH in 12. Not = to >>>>>>>> rename or >>>>>>>> remove armv6. Sadly, that will still be there since the RPI >>>>>>>> foundation >>>>>>>> keeps finding new ways to repackage the rpi into new boards = that are >>>>>>>> just >>>>>>>> too cheap to ignore. >>>>>>>=20 >>>>>>> On 2017-Jun-12, at 6:59 AM, Andrew Turner <andrew at = fubar.geek.nz> >>>>>>> wrote: >>>>>>>=20 >>>>>>>> I like this. My understanding is adding armv7 would also fix = many of >>>>>>>> the currently broken ports that assume they are being built for = armv7 as >>>>>>>> many Linux distros target ARMv7+. >>>>>>>>=20 >>>>>>>> It should also be noted the GENERIC kernel is likely to only = ever >>>>>>>> target ARMv7+ even without an armv7 TARGET_ARCH. >>>>>>>=20 >>>>>>>=20 >>>>>>> Hopefully the choices related to TARGET and TARGET_ARCH >>>>>>> for armv7 end up identifying the context to port builds >>>>>>> so that many would just automatically do the right thing. >>>>>>>=20 >>>>>>>=20 >>>>>>> As for GENERIC: >>>>>>>=20 >>>>>>> powerpc has. . . >>>>>>>=20 >>>>>>> TARGET=3Dpowerpc TARGET_ARCH=3Dpowerpc and GENERIC >>>>>>> TARGET=3Dpowerpc TARGET_ARCH=3Dpowerpc64 and GENERIC64 >>>>>>>=20 >>>>>>> So there is precedent for more than one GENERIC* >>>>>>> for a family, with which one being appropriate >>>>>>> being based on TARGET_ARCH. >>>>>>>=20 >>>>>>> For powerpc TARGET=3Dpowerpc implicitly uses >>>>>>> TARGET_ARCH=3Dpowerpc when TARGET_ARCH is not >>>>>>> specified (if I remember right). Which should >>>>>>> be the default for armv6 vs. armv7 might go >>>>>>> the other direction (TARGET_ARCH=3Darmv7 by >>>>>>> default). >>>>>>>=20 >>>>>>>=20 >>>>>>> Side note: >>>>>>>=20 >>>>>>> A caution about talking about "rpi2" as >>>>>>> an example. . . >>>>>>>=20 >>>>>>> Raspberry Pi 2 Model B V1.2 is Cortex-A53 based >>>>>>> (so arm64/aarch64). (A BCM2837, not a BCM2836.) >>>>>>> This dates about to something like 2014 based >>>>>>> on the pictures showing the (c) notice on the >>>>>>> boards. >>>>>>>=20 >>>>>>> V1.1 and before were armv7 (BCM2836) based. >>>>>>>=20 >>>>>>> Unless a kernel and world are made that can >>>>>>> also configure/handle a Cortex-A53 in a >>>>>>> armv7-like manor there will be two different >>>>>>> GENERIC builds in order to span the "rpi2" >>>>>>> family, based on just V1.2+ vs. V1.1 and >>>>>>> before. >>>>>>>=20 >>>>>>> (A single, modern distribution of the official >>>>>>> Raspbian software for the rpi2 does support >>>>>>> all the V1.x boards if I understand right.) >>>>>>=20 >>>>>> I am confused. I don't see any documentation about Raspbian = supporting >>>>>> 64 bit? >>>>>=20 >>>>> 64-bit Cortex-A53's can be configure to operate in a >>>>> 32-bit mode (AArch32). Raspian does that for RPI2 V1.2 >>>>> and for RPI3. >>>>>=20 >>>>> Raspian does not (yet?) support a 64-bit mode (AArch64). >>>>>=20 >>>>> The Cortex-A53 can support either. As I understand it >>>>> is possible for an OS to allow a user processes to be >>>>> one or the other, different processes using the different >>>>> modes. That does not mean that all operating systems >>>>> bother to. >>>>>=20 >>>>> If I remember right the official Ubuntu for an ODroid-C2 >>>>> allows both AArch64 and AArch32 user programs (and >>>>> so processes, with shared library types tracking). >>>>>=20 >>>>>> =46rom Arm at >>>>>> = https://www.arm.com/products/processors/cortex-a/cortex-a53-processor.php:= >>>>>> "The Cortex-A53 supports the full ARMv8-A architecture. It not = only >>>>>> runs 64-bit applications also seamlessly and efficiently runs = legacy >>>>>> ARM 32-bit applications." >>>>>>=20 >>>>>> I assume that means it handles armv7-A without issue? (In fact, = many >>>>>> on this board know it does) >>>>>=20 >>>>> I've not gone through the details but targeting AArch32 >>>>> probably means targeting a subset of armv7. Or may be >>>>> to support both requires targeting a common subset of both. >>>>> (My guess is that AArch32 is the definition of a common >>>>> subset for 32-bit, at least for user processes.) >>>>>=20 >>>>> Raspian targets just AArch32 on armv7 and Cortex-A53 >>>>> (user space). (If I've got the definition of AArch32 >>>>> right: otherwise a common subset.) >>>>>=20 >>>>> FreeBSD targets armv7 and AArch64 separately (via >>>>> separate GENERIC kernels). I'm not aware of FreeBSD >>>>> targeting AArch32 at all on cores capable of AArch64. >>>>> FreeBSD possibly does not restrict itself to AArch32 >>>>> (user processes) on armv7 if AArch32 is really a >>>>> subset. >>>>=20 >>>> I thought all 64 bit Arm instructions are defined in armv8? >>>=20 >>> (I assume you are not trying to indicate armv8.1, armv8.2 >>> and such. Cortex-A53 is older than those and so does not >>> have the newer things involved.) >>>=20 >>> That Cortex-A53 allows armv8 64-bit arm code is not in >>> dispute. But the operating system in involved in setting >>> up what will actually work based on how it configures >>> things and operates. Much of this is the kernel. >>=20 >>=20 >> Correct. >>=20 >>>=20 >>> Cortex-A53 also supports AArch32, i.e., 32 bit instructions. >>> So that the 64-bit instructions all being there does not >>> of itself prevent using a 32-bit mode instead. >>>=20 >>> (The kernel likely has to deal with more specifics of >>> processor variations than user code does not. My notes >>> are really about the user process model, not the all >>> the kernel details.) >>>=20 >>> Raspian deals with armv7's that have no 64-bit support >>> and with Cortex-A53's that do. It presents a user-process >>> model that is 32-bit only, even on Cortex-A53's that allow >>> for 64-bit (but do not required user programs to be AArch64 >>> code). >>>=20 >>> Ubuntu for ODroid-C2 does not deal with armv7's but does >>> allow both 64-bit (AArch64) and 32-bit (AArch32) user >>> processes as I remember, on its Cortex-A53's. >>>=20 >>> FreeBSD armv7 does not support Cortext-A53 or anything >>> that allows 64-bit (that allows AArch64). This is a kernel >>> level issue. >>=20 >>=20 >> Not a hugely difficult issue to fix, but one nobody had fixed... >>=20 >>>=20 >>> FreeBSD aarch64 does not support having AArch32 user >>> processes. Nor does its kernel support processors that >>> do not support aarch64 (so it does not support armv7). >>=20 >>=20 >> Executing a 32-bit /bin/cat on aarch64 level support exists outside = the >> tree, according to the hallway track at BSDcan, so it will only be a = matter >> of time before compat32 exists there I think. >>=20 >> There's a further complication is that the aarch32 unit of aarch64 >> processors is optional. Not all of them have it, so that can be a = problem... >> IIRC, the early aarch64 targets didn't have this feature... >>=20 >>>=20 >>> These are simply examples of different choices about >>> what combinations of the technical possibilities to >>> put effort into supporting in the kernels (and >>> possibly elsewhere). None of the alternatives is >>> automatic. None are independent of software choices >>> that must be made by each operating system. >>=20 >>=20 >> Yes. They all require somebody to be interested in doing the work. >>=20 >> Warner >>=20 >>=20 >>=20 >=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D1C69755-C249-4CAC-BFDB-70B4BCAEB5C6>