Date: Tue, 29 Dec 2020 22:49:09 -0800 From: Mark Millard <marklmi@yahoo.com> To: bob prohaska <fbsd@www.zefox.net> Cc: freebsd-arm@freebsd.org Subject: Re: Migrating from -current to stable/12 on RPI2B (ARMv7) Message-ID: <CC9C9212-6F7B-429C-8846-E0FF1333E388@yahoo.com> In-Reply-To: <20201230035447.GA50440@www.zefox.net> References: <619A02CC-0EBA-4B50-A3BB-C326996AE706@yahoo.com> <B6A3C3C8-4AB9-43B5-A93F-8F2B6DE5E5E9@yahoo.com> <CB0E5ECD-CDDA-46A7-812B-744AF7645A78@yahoo.com> <20201229010220.GA36311@www.zefox.net> <78A4DEC3-421F-419D-ABDE-9F3724E44C8D@yahoo.com> <7E0A320A-4C81-4C7B-B5D0-E6681FFA24FC@yahoo.com> <DB6C87A3-19DC-4979-AC17-FD1261782803@yahoo.com> <67E8786A-79E7-409A-BF4D-738F4FEB5EFF@yahoo.com> <20201229235701.GA49529@www.zefox.net> <D562F29B-7154-49DC-B1FC-1ACA20695897@yahoo.com> <20201230035447.GA50440@www.zefox.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2020-Dec-29, at 19:54, bob prohaska <fbsd at www.zefox.net> wrote: > On Tue, Dec 29, 2020 at 05:28:55PM -0800, Mark Millard wrote: >>=20 >>=20 >> On 2020-Dec-29, at 15:57, bob prohaska <fbsd at www.zefox.net> wrote: >>=20 >>> On Tue, Dec 29, 2020 at 02:50:07PM -0800, Mark Millard wrote: >>>> [clang.full built this time.] >>>>=20 >>>>=20 >>>> With the use of -O2 instead of -O , the bootstrap clang.full >>>> linked and the later activities in building the bootstrap >>>> clang worked as well. The build has also built the bootstrap >>>> lld and is off doing things that use the bootstrap clang >>>> and lld. >>>>=20 >>>> I've not checked if -O2 usage would be a sufficient >>>> change by itself. For one, various other aspects of my >>>> normal builds vs. yours could be different: I've not >>>> replicated your context in any detail. >>>=20 >>> Is adding CFLAGS=3DO2 to /etc/make.conf worth a try? The machine >>> is otherwise idle at the moment.=20 >>=20 >> I'll deal with controlling -O2 use later in this note. It is a bit >> messy in my experience so it is not near a one word response. For more background I'll quote from 2 clang documents, one each for 10 vs. 11 of clang: = https://releases.llvm.org/10.0.0/tools/clang/docs/CommandGuide/clang.html = : -O Equivalent to -O2. = https://releases.llvm.org/11.0.0/tools/clang/docs/CommandGuide/clang.html = : -O Equivalent to -O1. So -O in stable/12 (clang 10) and in FreeBSD 13 (clang 11) are not equivalent. FreeBSD 13 does not use -O any more: it uses -O2 . Using -O2 notation makes the clang compilers in both FreeBSD's behave similarly. -O does not. I've no clue why stable/12 has not had some approximate MFC of the clang -O2 notation. You might be able to get some committer to take that point of view and update stable/12 . I've been writing for as things are presently. That will continue some below. As things are, you are out of the bounds that FreeBSD currently supports when you try to build stable/12 on an armv7 FreeBSD 13 (or armv6 FreeBSD 13) with the built-in toolchain (counting stable/12's Makefile*'s). (The same likely applies to 32-bit powerpc.) >>> As an aside, I've tried building stable/12 sources on a Pi3B running = -current. >>> It's already gotten past the point of failure and is now building = libraries. >>> Whatever is wrong, it's not present, or at least not visible, on = aarch64.=20 >>> Is there any hint where this bug (or feature) might reside? >>=20 >> If you run armv7 FreeBSD on the RPi3B it will behave like the RPi2 = v1.1 does: >> it will fail. You must be running the aarch64 FreeBSD on the RPi3B = for what >> you state above to be true. The RPi2 v1.1 can not run the aarch64 = FreeBSD. >>=20 > Yes, the success case ran aarch64 on the Pi3B. >=20 >> So: it is not a bug. armv7 user processes are each limited to = (virtual) 2 GiBytes >> (or somewhat less) user-process-address-space (32 bit addressing, = with a bit >> reserved).=20 >=20 > Doesn't that make it a bug specific to armv7? Perhaps not global, but = a bug > so far as users of ld would see it.=20 That the armv7 (/armv6) always has a "small" address space is a hardware issue (instruction set architecture), not software. That -O1 (and clang 11's -O) requires more than 2 GiByte per-process spaces to build clang.full is the consequence of (mostly) LLVM choices. FreeBSD chooses to be LLVM based and FreeBSD 13 chooses to be LLVM 11 based (until LLVM 12?). The closet things to a "bug" here is stable/12's use of -O . But FreeBSD does not define its correctness relative to using armv7, armv6, or 32-bit powerpc as you are trying to using it. (That does not mean an MFC of -O2 use to stable/12 is out of the question. It is just not on a main line of development for FreeBSD.) I'll note that cross-building from machines with a larger maximum per-process-space size still allows targeting armv7 and the like with -O in use. You could do a cross-build on the RPi3B from an arm64 FreeBSD context. Such is one of the reasons the LLVM folks do not=20 worry as much about builds that can not finish in a 2 GiByte maxmimum per-process context. >> aarch64 user processes are not each limited to a (virtual) 2 GiByte >> user-process-address-space: the user-process-address-space bound is = much larger >> so the effective bound is based on other things. aarch64 still = usually ends up >> with a larger effective user-process size limit than 2 GiByte = (depending on >> RAM size and swap/paging space configuration, for example). >>=20 >> What -O2 is doing is inlining functions and the like, cutting the = amount >> of linking of functions and the like at ld time: -O2 make the linking >> less memory-size-needed intensive. >>=20 >>=20 >>=20 >> Back to building vs. assigning CFLAGS . . . >>=20 >> Directly assigning CFLAGS (CFLAGS=3D or CFLAGS+=3D style) can be = problematical, >> with that definition possibly overriding internal definitions by = preventing >> internal CFLAGS?=3D usage from picking up material or other such in = contexts=20 >> one is not trying to control (nested contexts). I have run into such = issues >> historically. >>=20 >> Here we are trying to control something normally supplied internal to = the >> build infrastructure, only in places where that stable/12 = infrsuctcture's >> normal >>=20 >> CFLAGS?=3D -O -pipe >>=20 >> would have done the assignment. We do not want to be adjusting other >> contexts that do things with CFLAGS ( including, possibly, other uses >> of -O in nested contexts). >>=20 >> We probably also do not want to analyze the whole build = infrastructure >> looking for places to worry about inappropriate overriding and = figuring >> out what to do for them. >>=20 >> Such also applies to using, say, CFLAGS.clang+=3D that would add >> separate text to the command lines without disabling the CFLAGS?=3D = usage >> (for example). Then things get into order of conflicting options and >> which "wins" and if that is always an appropriate result. >>=20 >> So the only simple technique that I know of is to change the actual = file >> ( share/mk/sys.mk ) that has the above line. That limits itself to = the >> correct context directly. Only the share/mk/sys.mk copy in the file = system >> needs to be changed: no need to have, say, a git branch of your own = (unless >> you want such). >=20 > Looking at /usr/freebsd-src/share/mk/sys.mk I don't find a = CFLAGS.clang line. I was not trying to imply that you would find such in FreeBSD files. = (There is code that enables such notation to be used.) I have such .clang notation in use in my own files, not in modified = FreeBSD files. You likely do not want the self-support tradeoffs involved, = including the build environment knowledge gathering involved for figuring out when the notation will work sufficiently for your purposes. > There is a snippet containing > .if ${MACHINE_CPUARCH} =3D=3D "arm" || ${MACHINE_CPUARCH} =3D=3D = "mips" > CFLAGS ?=3D -O -pipe > .else > CFLAGS ?=3D -O2 -pipe > .endif >=20 > Indeed, I don't see anything that distinguishes armv7 from arm64.=20 True. armv7/armv6/arm64 have been using -O in stable/12 and now use -O2 in 13. (But -O means very different things in stable/12 vs. in 13, because of the different clang versions involved.) 13 uses -O2 for everything now. The reference to the armv6/armv7/arm64 change is: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D3502= 10 which shows in the sys.mk log as: Revision 350210 - (view) (download) (annotate) - [select for diffs]=20 Modified Mon Jul 22 10:17:59 2019 UTC (17 months, 1 week ago) by manu=20 File length: 8664 byte(s)=20 Diff to previous 335452 arm: Use -O2 instead of -O as optimization flag When using Clang -O is equivalent to -O2, change it -O2 to make it consistent with other platforms. Reference:=20 = https://clang.llvm.org/docs/ClangCommandLineReference.html#optimization-le= vel Submitted by: Daniel Engberg (daniel.engberg.lists@pyret.net) Reviewed by: emaste Differential Revision:=09 https://reviews.freebsd.org/D21021 Note the implication of the description: at the time -O and -O2 were equivalent for armv7/armv6/arm64. That is no longer true of clang 11 and FreeBSD 13. Later ( -r366664 ) mips was also changed. That is when having = conditional logic was removed. By this later time frame clang 11 had changed -O to mean -O1 instead of -O2. So to get the historical standard output, -O2 had to be used instead when clang 11 was doing the compiling. > Am I looking at the wrong file? That is the correct file. armv7 was not the only type of arm target changed: armv7 is just an example for the change. The earlier code that you quote is the same text from the same file that I showed in a git diff that showed the change I made to do my test. It was a one character addition for how I did it for my test. As of mips also changing, 13 had deleted the first 3 lines that you show and the .endif line as well. (Such is equivalent to what I did by instead adding one character to what stable/12 had there.) What 13 did should be fine for your use if you want to do such: equivalent. > /usr/freebsd-src is the stable/12 > build directory, cloned by git. Or, is that the problem, that armv7=20 > and arm64 are treated the same way?=20 arm64 builds clang.full either way in FreeBSD 13 but -O2 will produce faster LLVM toolchain programs. armv7 does not build clang.full when -O is in use on FreeBSD 13 and -O2 can work. When FreeBSD 13 builds stable/12 it uses stable/12's option to build clang.full , and that option is still -O as things are. Thus FreeBSD 13 does not support the operation as-is. >>=20 >> I've run into this before and have used my own share/mk/sys.mk = variant >> as needed. I actually use CFLAGS.clang+=3D and the like for something = but >> would not use it for this -O2 vs. -O issue. >>=20 >> If you still try your own CFLAGS assignment, include the -pipe as = well: >>=20 >> CFLAGS=3D -O2 -pipe >>=20 >> Otherwise the -pipe will end up missing. >>=20 >=20 > Are you referring to /etc/make.conf? WARNING: /etc/make.conf and /etc/src.conf are not limited to buildworld buildkernel use. For example, port builds can also use the likes of /etc/make.conf . Thus one ends up managing the content of those files as one switches what type of thing is being built. (The port build technique matters here, I'll not get into the details.) Yet, even for just buildworld buildkernel contexts: As an illustration of why such is bad: if CFLAGS already has a value the use of CFLAGS=3D would destroy the value if executed. (Is /etc/make.conf only included once for over the whole buildworld build kernel? At exactly which stage(s) relative to share/mk/sys.mk ? Do you want to analyze such things?) By contrast: CFLAGS?=3D -O2 -pipe avoids that issue but then what you get is dependent on which CFLAGS?=3D happens first in the detailed ordering of operations whenever /etc/make.conf is involved/executed. > That seems preferable to tampering > with makefiles.=20 The point of my notes was that, in this type of context, that is not necessarily true. I have reported that I have had example problems controlling -O2 use (or similar) and what my investigation indicated to me as what was simpler that would work --and that is what I use when I do not want to do what FreeBSD does for -O2 type options. (I do not normally build stable/12 or other such and so the issue does not normally show up for me in modern times.) Hopefully the notes will help if you try things and have problems. But the simplest thing at this point is to add the one character to stable/12's share/mk/sys.mk ( until/unless you get someone to MFC -O2 use into stable/12 ). > With my thanks (and apologies for dumb questions)! >=20 What dumb questions? Each point has at some point in the past been an investigation for me. None of it was magically obvious. You are welcome. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CC9C9212-6F7B-429C-8846-E0FF1333E388>