Date: Wed, 30 Dec 2020 10:30:52 -0800 From: bob prohaska <fbsd@www.zefox.net> To: Mark Millard <marklmi@yahoo.com> Cc: freebsd-arm@freebsd.org, bob prohaska <fbsd@www.zefox.net> Subject: Re: Migrating from -current to stable/12 on RPI2B (ARMv7) Message-ID: <20201230183052.GA56823@www.zefox.net> In-Reply-To: <CC9C9212-6F7B-429C-8846-E0FF1333E388@yahoo.com> References: <CB0E5ECD-CDDA-46A7-812B-744AF7645A78@yahoo.com> <20201229010220.GA36311@www.zefox.net> <78A4DEC3-421F-419D-ABDE-9F3724E44C8D@yahoo.com> <7E0A320A-4C81-4C7B-B5D0-E6681FFA24FC@yahoo.com> <DB6C87A3-19DC-4979-AC17-FD1261782803@yahoo.com> <67E8786A-79E7-409A-BF4D-738F4FEB5EFF@yahoo.com> <20201229235701.GA49529@www.zefox.net> <D562F29B-7154-49DC-B1FC-1ACA20695897@yahoo.com> <20201230035447.GA50440@www.zefox.net> <CC9C9212-6F7B-429C-8846-E0FF1333E388@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Dec 29, 2020 at 10:49:09PM -0800, Mark Millard wrote: > > > On 2020-Dec-29, at 19:54, bob prohaska <fbsd at www.zefox.net> wrote: > > > On Tue, Dec 29, 2020 at 05:28:55PM -0800, Mark Millard wrote: > >> > >> > >> On 2020-Dec-29, at 15:57, bob prohaska <fbsd at www.zefox.net> wrote: > >> > >>> On Tue, Dec 29, 2020 at 02:50:07PM -0800, Mark Millard wrote: > >>>> [clang.full built this time.] > >>>> > >>>> > >>>> With the use of -O2 instead of -O , the bootstrap clang.full > >>>> linked and the later activities in building the bootstrap > >>>> clang worked as well. The build has also built the bootstrap > >>>> lld and is off doing things that use the bootstrap clang > >>>> and lld. > >>>> > >>>> I've not checked if -O2 usage would be a sufficient > >>>> change by itself. For one, various other aspects of my > >>>> normal builds vs. yours could be different: I've not > >>>> replicated your context in any detail. > >>> > >>> Is adding CFLAGS=O2 to /etc/make.conf worth a try? The machine > >>> is otherwise idle at the moment. > >> > >> I'll deal with controlling -O2 use later in this note. It is a bit > >> messy in my experience so it is not near a one word response. > > For more background I'll quote from 2 clang documents, one each for > 10 vs. 11 of clang: > > https://releases.llvm.org/10.0.0/tools/clang/docs/CommandGuide/clang.html : > > -O Equivalent to -O2. > > https://releases.llvm.org/11.0.0/tools/clang/docs/CommandGuide/clang.html : > > -O Equivalent to -O1. > > So -O in stable/12 (clang 10) and in FreeBSD 13 (clang 11) are not > equivalent. FreeBSD 13 does not use -O any more: it uses -O2 . > > Using -O2 notation makes the clang compilers in both FreeBSD's behave > similarly. -O does not. > > I've no clue why stable/12 has not had some approximate MFC of the clang > -O2 notation. You might be able to get some committer to take that point > of view and update stable/12 . > > I've been writing for as things are presently. That will continue some > below. > > As things are, you are out of the bounds that FreeBSD currently supports > when you try to build stable/12 on an armv7 FreeBSD 13 (or armv6 FreeBSD > 13) with the built-in toolchain (counting stable/12's Makefile*'s). (The > same likely applies to 32-bit powerpc.) > It's worse than that: I'm trying to turn back time. I wanted to test the stable-12 upgrade path for armv7 and used the 13-current box because it was handy and expected backward compatibility. Makes no sense in the context of an OS. It makes more sense to wait for stable/13 to emerge and test that, given that 13-current seems to work with armv7. The handbook warns against crossing major versions, doing it in reverse compounds the trouble, since developers have no reason to expect anybody in their right mind _wanting_ to do it. > >>> As an aside, I've tried building stable/12 sources on a Pi3B running -current. > >>> It's already gotten past the point of failure and is now building libraries. > >>> Whatever is wrong, it's not present, or at least not visible, on aarch64. > >>> Is there any hint where this bug (or feature) might reside? > >> > >> If you run armv7 FreeBSD on the RPi3B it will behave like the RPi2 v1.1 does: > >> it will fail. You must be running the aarch64 FreeBSD on the RPi3B for what > >> you state above to be true. The RPi2 v1.1 can not run the aarch64 FreeBSD. > >> > > Yes, the success case ran aarch64 on the Pi3B. > > > >> So: it is not a bug. armv7 user processes are each limited to (virtual) 2 GiBytes > >> (or somewhat less) user-process-address-space (32 bit addressing, with a bit > >> reserved). > > > > Doesn't that make it a bug specific to armv7? Perhaps not global, but a bug > > so far as users of ld would see it. > > That the armv7 (/armv6) always has a "small" address space is a hardware > issue (instruction set architecture), not software. That -O1 (and clang > 11's -O) requires more than 2 GiByte per-process spaces to build > clang.full is the consequence of (mostly) LLVM choices. FreeBSD chooses > to be LLVM based and FreeBSD 13 chooses to be LLVM 11 based (until LLVM > 12?). > Given the continued "expansion" of LLVM, it seems likely that before long it will become incompatible with armv7, despite the use of optimization. Does this seem correct? > The closet things to a "bug" here is stable/12's use of -O . But FreeBSD > does not define its correctness relative to using armv7, armv6, or > 32-bit powerpc as you are trying to using it. (That does not mean an MFC > of -O2 use to stable/12 is out of the question. It is just not on a main > line of development for FreeBSD.) > > I'll note that cross-building from machines with a larger maximum > per-process-space size still allows targeting armv7 and the like with > -O in use. You could do a cross-build on the RPi3B from an arm64 > FreeBSD context. Such is one of the reasons the LLVM folks do not > worry as much about builds that can not finish in a 2 GiByte > maxmimum per-process context. > > >> aarch64 user processes are not each limited to a (virtual) 2 GiByte > >> user-process-address-space: the user-process-address-space bound is much larger > >> so the effective bound is based on other things. aarch64 still usually ends up > >> with a larger effective user-process size limit than 2 GiByte (depending on > >> RAM size and swap/paging space configuration, for example). > >> > >> What -O2 is doing is inlining functions and the like, cutting the amount > >> of linking of functions and the like at ld time: -O2 make the linking > >> less memory-size-needed intensive. > >> > >> > >> > >> Back to building vs. assigning CFLAGS . . . > >> > >> Directly assigning CFLAGS (CFLAGS= or CFLAGS+= style) can be problematical, > >> with that definition possibly overriding internal definitions by preventing > >> internal CFLAGS?= usage from picking up material or other such in contexts > >> one is not trying to control (nested contexts). I have run into such issues > >> historically. > >> > >> Here we are trying to control something normally supplied internal to the > >> build infrastructure, only in places where that stable/12 infrsuctcture's > >> normal > >> > >> CFLAGS?= -O -pipe > >> > >> would have done the assignment. We do not want to be adjusting other > >> contexts that do things with CFLAGS ( including, possibly, other uses > >> of -O in nested contexts). > >> > >> We probably also do not want to analyze the whole build infrastructure > >> looking for places to worry about inappropriate overriding and figuring > >> out what to do for them. > >> > >> Such also applies to using, say, CFLAGS.clang+= that would add > >> separate text to the command lines without disabling the CFLAGS?= usage > >> (for example). Then things get into order of conflicting options and > >> which "wins" and if that is always an appropriate result. > >> > >> So the only simple technique that I know of is to change the actual file > >> ( share/mk/sys.mk ) that has the above line. That limits itself to the > >> correct context directly. Only the share/mk/sys.mk copy in the file system > >> needs to be changed: no need to have, say, a git branch of your own (unless > >> you want such). > > > > Looking at /usr/freebsd-src/share/mk/sys.mk I don't find a CFLAGS.clang line. > > I was not trying to imply that you would find such in FreeBSD files. (There > is code that enables such notation to be used.) > > I have such .clang notation in use in my own files, not in modified FreeBSD > files. You likely do not want the self-support tradeoffs involved, including > the build environment knowledge gathering involved for figuring out when > the notation will work sufficiently for your purposes. > > > There is a snippet containing > > .if ${MACHINE_CPUARCH} == "arm" || ${MACHINE_CPUARCH} == "mips" > > CFLAGS ?= -O -pipe > > .else > > CFLAGS ?= -O2 -pipe > > .endif > > > > Indeed, I don't see anything that distinguishes armv7 from arm64. > > True. armv7/armv6/arm64 have been using -O in stable/12 and now > use -O2 in 13. (But -O means very different things in stable/12 > vs. in 13, because of the different clang versions involved.) > > 13 uses -O2 for everything now. The reference to the armv6/armv7/arm64 > change is: https://svnweb.freebsd.org/base?view=revision&revision=350210 > which shows in the sys.mk log as: > > Revision 350210 - (view) (download) (annotate) - [select for diffs] > Modified Mon Jul 22 10:17:59 2019 UTC (17 months, 1 week ago) by manu > File length: 8664 byte(s) > Diff to previous 335452 > arm: Use -O2 instead of -O as optimization flag > > When using Clang -O is equivalent to -O2, change it -O2 to make it > consistent with other platforms. > > Reference: > https://clang.llvm.org/docs/ClangCommandLineReference.html#optimization-level > > > Submitted by: Daniel Engberg (daniel.engberg.lists@pyret.net) > Reviewed by: emaste > Differential Revision: > https://reviews.freebsd.org/D21021 > > Note the implication of the description: at the time -O and > -O2 were equivalent for armv7/armv6/arm64. That is no longer > true of clang 11 and FreeBSD 13. > > Later ( -r366664 ) mips was also changed. That is when having conditional > logic was removed. By this later time frame clang 11 had changed -O to > mean -O1 instead of -O2. So to get the historical standard output, -O2 > had to be used instead when clang 11 was doing the compiling. > > > > Am I looking at the wrong file? > > That is the correct file. armv7 was not the only type of arm target > changed: armv7 is just an example for the change. > > The earlier code that you quote is the same text from the same file > that I showed in a git diff that showed the change I made to do my > test. It was a one character addition for how I did it for my test. > > As of mips also changing, 13 had deleted the first 3 lines that you > show and the .endif line as well. (Such is equivalent to what I did > by instead adding one character to what stable/12 had there.) What > 13 did should be fine for your use if you want to do such: equivalent. > > > /usr/freebsd-src is the stable/12 > > build directory, cloned by git. Or, is that the problem, that armv7 > > and arm64 are treated the same way? > > arm64 builds clang.full either way in FreeBSD 13 but -O2 will produce > faster LLVM toolchain programs. > > armv7 does not build clang.full when -O is in use on FreeBSD 13 and > -O2 can work. When FreeBSD 13 builds stable/12 it uses stable/12's > option to build clang.full , and that option is still -O as things > are. Thus FreeBSD 13 does not support the operation as-is. > > >> > >> I've run into this before and have used my own share/mk/sys.mk variant > >> as needed. I actually use CFLAGS.clang+= and the like for something but > >> would not use it for this -O2 vs. -O issue. > >> > >> If you still try your own CFLAGS assignment, include the -pipe as well: > >> > >> CFLAGS= -O2 -pipe > >> > >> Otherwise the -pipe will end up missing. > >> > > > > Are you referring to /etc/make.conf? > > WARNING: /etc/make.conf and /etc/src.conf are not limited > to buildworld buildkernel use. For example, port builds > can also use the likes of /etc/make.conf . Thus one ends > up managing the content of those files as one switches > what type of thing is being built. (The port build > technique matters here, I'll not get into the details.) > > Yet, even for just buildworld buildkernel contexts: > > As an illustration of why such is bad: if CFLAGS already > has a value the use of CFLAGS= would destroy the value if > executed. (Is /etc/make.conf only included once for over the > whole buildworld build kernel? At exactly which stage(s) > relative to share/mk/sys.mk ? Do you want to analyze such > things?) > > By contrast: > > CFLAGS?= -O2 -pipe > > avoids that issue but then what you get is dependent on which > CFLAGS?= happens first in the detailed ordering of operations > whenever /etc/make.conf is involved/executed. > > > > That seems preferable to tampering > > with makefiles. > > The point of my notes was that, in this type of context, that is > not necessarily true. > > I have reported that I have had example problems controlling -O2 > use (or similar) and what my investigation indicated to me as what > was simpler that would work --and that is what I use when I do not > want to do what FreeBSD does for -O2 type options. (I do not > normally build stable/12 or other such and so the issue does not > normally show up for me in modern times.) > > Hopefully the notes will help if you try things and have problems. > But the simplest thing at this point is to add the one character > to stable/12's share/mk/sys.mk ( until/unless you get someone to > MFC -O2 use into stable/12 ). That seems pointless as I now understand the situation. Nobody has a practical incentive to compile old OS versions with newer ones. I tried it only out of laziness. When the arm64 Pi3 came out it seemed likely that FreeBSD's support for 32 bit arm would erode. As LLVM gets larger the erosion can only speed up, particularly for self-hosting. I wonder when armv7 will be "too small". Your notes and comments have been profoundly enlightening, Thank you, bob prohaska
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20201230183052.GA56823>