Date: Wed, 20 Jan 2021 09:51:33 -0800 From: Mark Millard <marklmi@yahoo.com> To: bob prohaska <fbsd@www.zefox.net> Cc: Current FreeBSD <freebsd-current@freebsd.org>, freebsd-arm@freebsd.org Subject: Re: Silent hang in buildworld, was Re: Invoking -v for clang during buildworld Message-ID: <A6150F56-062F-4582-853A-319C1EE4DDCB@yahoo.com> In-Reply-To: <056845FE-7131-4951-96AF-805D07F7BE0D@yahoo.com> References: <20210116043740.GA19523@www.zefox.net> <ED26508F-282D-439D-8A6A-65A136C76C84@yahoo.com> <20210116155538.GA24259@www.zefox.net> <F62027C8-0813-4E6F-934A-3088F966AF8C@yahoo.com> <20210116220334.GA26756@www.zefox.net> <C75D3D9C-4284-4BE3-B2CD-5DC6BBB60843@yahoo.com> <20210117174006.GA30728@www.zefox.net> <85889EAE-F579-4220-9185-944D9AA5075A@yahoo.com> <20210118015009.GA31353@www.zefox.net> <60CCCDE8-E3D3-4920-9FC0-A945330F6830@yahoo.com> <A0427375-5515-4D3C-AF2A-915E60A836A7@yahoo.com> <00104FAD-E32B-4DDE-80DD-FCEF14CEC06B@yahoo.com> <D9878BB6-2693-4A04-9E1C-126E0D378F7B@yahoo.com> <056845FE-7131-4951-96AF-805D07F7BE0D@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2021-Jan-19, at 17:42, Mark Millard <marklmi at yahoo.com> wrote:
> On 2021-Jan-18, at 21:12, Mark Millard <marklmi at yahoo.com> wrote:
>=20
>> On 2021-Jan-18, at 19:19, Mark Millard <marklmi at yahoo.com> wrote:
>>=20
>>> . . .
>>>> FYI: I re-established my access to a RPi2B V1.1 and made
>>>> it report: "maximum recommended amount (468832 pages)"
>>>>=20
>>>> (The figure can vary some from release to release.)
>>>>=20
>>>> 468832*4096 =3D=3D 1920335872 or a little over 1831 MiBytes
>>>>=20
>>>> For the 4096 Byte pages, that means that the following from
>>>> gpart fits without complaint (size is in blocks, not pages):
>>>>=20
>>>> 413140992 3686400 da0p2 freebsd-swap (1.8G)
>>>>=20
>>>> 3686400*512 is a little over 1.75 GiByte or 1800 MiByte. So
>>>> I've left some room below 1831 MiBytes, but not a lot.
>>>>=20
>>>> FYI about my build experiment that is running:
>>>>=20
>>>> # sysctl hw.physmem
>>>> hw.physmem: 979042304
>>>>=20
>>>> which, in recent times for armv7, I can (and did) set in
>>>> /boot/loader.conf on a faster cortex-A7 SBC (that can boot
>>>> the same media but has more RAM).
>>>>=20
>>>> So I tried a -j4 build, but with LDFLAGS.lld+=3D -Wl,--threads=3D1
>>>> in use and my other particular src.conf/make.conf like content
>>>> (so the builds do likely differ from yours in various ways).
>>>> My build is producing a non-debug build (but with -g symbols).
>>>> Somewhat after where your buildworld.log stops, my odd variant
>>>> of top was reporting:
>>>>=20
>>>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892732Ki =
MaxObs(Act+Wir)
>>>> Swap: . . . , 145832Ki MaxObsUsed
>>>>=20
>>>> and top was also showing lots of processes as having "0B" RES
>>>> in STATE "wait" or "nanslp" (so, apparently swapped out, not =
paging).
>>>> ("MaxObs" is short for "maximum observed".)
>>>>=20
>>>> For comparison, your swapscript.log reported a maximum total of
>>>> 346192 KiBytes "Used" for swap, about 98% into the log file.
>>>>=20
>>>> (Time goes by . . .)
>>>>=20
>>>> It finished with building libllvm and is part way into building
>>>> libclang. This is probably well past where your hangup happened,
>>>> given that your published buildworldlog file stopped with
>>>> libllvm's Target/ARM/ARMMCInstLower.o . My odd top now shows:
>>>>=20
>>>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892732Ki =
MaxObs(Act+Wir)
>>>> Swap: . . . , 392328Ki MaxObsUsed
>>>>=20
>>>> The build continues to run. I'll let you know how it goes.
>>>> . . .
>>>=20
>>> Just after libclang finished my odd top showed:
>>>=20
>>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892736Ki =
MaxObs(Act+Wir)
>>> Swap: . . . , 537588Ki MaxObsUsed
>>>=20
>>> After liblldb:
>>>=20
>>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 899276Ki =
MaxObs(Act+Wir)
>>> Swap: . . . , 537588Ki MaxObsUsed
>>>=20
>>> Much later, after the lldb program had been built:
>>>=20
>>> Mem: . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki =
MaxObs(Act+Wir)
>>> Swap: . . . , 537588Ki MaxObsUsed
>>>=20
>>>>>> World build completed on Mon Jan 18 19:10:08 PST 2021
>>>>>> World built in 72960 seconds, ncpu: 4, make -j4
>>>=20
>>> This was building from scratch what was already installed:
>>>=20
>>> # ~/fbsd-based-on-what-freebsd-main.sh=20
>>> merge-base: 818390ce0ca539300dd15d7a817784f1e3f7a9b8
>>> merge-base: CommitDate: 2021-01-13 21:27:44 +0000
>>> 4180404713ec (HEAD -> mm-src) mm-src snapshot for mm's patched build =
in git context.
>>> 818390ce0ca5 (freebsd/main, freebsd/HEAD, pure-src, main) arm64: fix =
early devmap assertion
>>> FreeBSD OPiP2E_RPi2v11 13.0-CURRENT FreeBSD 13.0-CURRENT =
mm-src-c255938-g4180404713ec GENERIC-NODBG arm armv7 1300135 1300135
>>>=20
>>> This suggests that you should be able to build on the RPi2B v1.1,
>>> using -j4, with appropriate configuration for what and how to build.
>>>=20
>>>=20
>>> It is now building the matching kernel, my GENERIC-NODBG style.
>>=20
>> Done:
>>=20
>>>>> Kernel build for GENERIC-NODBG completed on Mon Jan 18 20:33:26 =
PST 2021
>>>>> Kernel(s) GENERIC-NODBG built in 4998 seconds, ncpu: 4, make -j4
>>=20
>> So, World+Kernel in somewhat under 22 hours.
>>=20
>> The "MaxObs*" figures were unchanged, so:
>>=20
>> Mem: . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki =
MaxObs(Act+Wir)
>> Swap: . . . , 537588Ki MaxObsUsed
>>=20
>> This suggests that, for now, 800 MiByte of swap would be something
>> more than 1.5 times what it actually used and 1050 MiBytes would
>> be something like 2.0 times what it actually used, so leaving some
>> notable margin for variations in peek usage, at least when linker
>> threading is avoided.
>>=20
>>=20
>>=20
>> As for what I used to control "what and how to build" . . .
>>=20
>> # more =
~/sys_build_scripts.armv7-host/make_armv7_nodebug_clang_bootstrap-armv7-ho=
st.sh=20
>> kldload -n filemon && \
>> script =
~/sys_typescripts/typescript_make_armv7_nodebug_clang_bootstrap-armv7-host=
-$(date +%Y-%m-%d:%H:%M:%S) \
>> env __MAKE_CONF=3D"/root/src.configs/make.conf" SRCCONF=3D"/dev/null" =
SRC_ENV_CONF=3D"/root/src.configs/src.conf.armv7-clang-bootstrap.armv7-hos=
t" \
>> WITH_META_MODE=3Dyes \
>> WORLD_FLAGS=3D"${WORLD_FLAGS} UBLDR_LOADADDR=3D0x42000000" \
>> MAKEOBJDIRPREFIX=3D"/usr/obj/armv7_clang/arm.armv7" \
>> make $*
>>=20
>> (In my context, UBLDR_LOADADDR is ignored by anything that
>> can not use the figure given. So I've no bothered to be
>> more selective about having it in the armv7 builds.)
>>=20
>> # more ~/src.configs/make.conf
>> LDFLAGS.lld+=3D -Wl,--threads=3D1
>>=20
>> # more ~/src.configs/src.conf.armv7-clang-bootstrap.armv7-host
>> TO_TYPE=3Darmv7
>> #
>> KERNCONF=3DGENERIC-NODBG
>> TARGET=3Darm
>> .if ${.MAKE.LEVEL} =3D=3D 0
>> TARGET_ARCH=3D${TO_TYPE}
>> .export TARGET_ARCH
>> .endif
>> #
>> #WITH_CROSS_COMPILER=3D
>> WITH_SYSTEM_COMPILER=3D
>> WITH_SYSTEM_LINKER=3D
>> #
>> WITH_LIBCPLUSPLUS=3D
>> WITHOUT_BINUTILS_BOOTSTRAP=3D
>> WITH_ELFTOOLCHAIN_BOOTSTRAP=3D
>> #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=3D
>> WITHOUT_LLVM_TARGET_AARCH64=3D
>> WITH_LLVM_TARGET_ARM=3D
>> WITHOUT_LLVM_TARGET_MIPS=3D
>> WITHOUT_LLVM_TARGET_POWERPC=3D
>> WITHOUT_LLVM_TARGET_RISCV=3D
>> WITHOUT_LLVM_TARGET_X86=3D
>> WITH_CLANG=3D
>> WITH_CLANG_IS_CC=3D
>> WITH_CLANG_FULL=3D
>> WITH_CLANG_EXTRAS=3D
>> WITH_LLD=3D
>> WITH_LLD_IS_LD=3D
>> WITHOUT_BINUTILS=3D
>> #
>> WITH_LLDB=3D
>> #
>> WITH_BOOT=3D
>> WITHOUT_LIB32=3D
>> #
>> #
>> WITHOUT_WERROR=3D
>> #WERROR=3D
>> MALLOC_PRODUCTION=3D
>> WITH_MALLOC_PRODUCTION=3D
>> WITHOUT_ASSERT_DEBUG=3D
>> WITHOUT_LLVM_ASSERTIONS=3D
>> #
>> # Avoid stripping but do not control host -g status as well:
>> DEBUG_FLAGS+=3D
>> #
>> WITH_REPRODUCIBLE_BUILD=3D
>> WITH_DEBUG_FILES=3D
>> #
>> # Use of the .clang 's here avoids
>> # interfering with other C<?>FLAGS
>> # usage, such as ?=3D usage.
>> CFLAGS.clang+=3D -mcpu=3Dcortex-a7
>> CXXFLAGS.clang+=3D -mcpu=3Dcortex-a7
>> CPPFLAGS.clang+=3D -mcpu=3Dcortex-a7
>>=20
>> (I do not claim that you would want WITH_REPRODUCIBLE_BUILD .
>> I just happen to have been experimenting with it. You might
>> not want to be explicit about the cpu to target. You might
>> not want WITH_CLANG_EXTRAS .)
>>=20
>> # more /usr/fbsd/mm-src/sys/arm/conf/GENERIC-NODBG
>> include "GENERIC"
>>=20
>> ident GENERIC-NODBG
>>=20
>> makeoptions DEBUG=3D-g # Build kernel with gdb(1) =
debug symbols
>>=20
>> options AUDIT # Not enabled by default in =
armv7/v6 kernels
>> # Enabled here to allow kyua =
test runs to
>> # possibly report auditing =
works.
>>=20
>> options ALT_BREAK_TO_DEBUGGER
>>=20
>> options KDB # Enable kernel debugger =
support
>>=20
>> # For minimum debugger support (stable branch) use:
>> options KDB_TRACE # Print a stack trace for a =
panic
>> options DDB # Enable the kernel debugger
>>=20
>> # Extra stuff:
>> #options VERBOSE_SYSINIT=3D0 # Enable verbose sysinit =
messages
>> #options BOOTVERBOSE=3D1
>> #options BOOTHOWTO=3DRB_VERBOSE
>> options ALT_BREAK_TO_DEBUGGER # Enter debugger on keyboard =
escape sequence
>> options KLD_DEBUG
>> #options KTR
>> #options KTR_MASK=3DKTR_TRAP
>> ##options KTR_CPUMASK=3D0xF
>> #options KTR_VERBOSE
>>=20
>> # Disable any extra checking for. . .
>> nooptions INVARIANTS # Enable calls of extra =
sanity checking
>> nooptions INVARIANT_SUPPORT # Extra sanity checks of =
internal structures, required by INVARIANTS
>> nooptions WITNESS # Enable checks to detect =
deadlocks and cycles
>> nooptions WITNESS_SKIPSPIN # Don't run witness on =
spinlocks for speed
>> nooptions DEADLKRES # Enable the deadlock =
resolver
>> nooptions MALLOC_DEBUG_MAXZONES # Separate malloc(9) zones
>> nooptions DIAGNOSTIC
>> nooptions BUF_TRACKING
>> nooptions FULL_BUF_TRACKING
>> nooptions USB_DEBUG
>> nooptions USB_REQ_DEBUG
>> nooptions USB_VERBOSE
>>=20
>> The /boot/loader.conf file and the /etc/sysctl.conf files
>> both contained:
>>=20
>> vm.pageout_oom_seq=3D120
>> vm.pfault_oom_attempts=3D-1
>>=20
>> (The hw.physmem=3D979042304 in /boot/loader.conf was very-special,
>> to better approximate your environment. I also controlled the
>> cpu frequency used via a line in /etc/sysctl.conf . I do not
>> bother with such non-default frequency usage [or related settings]
>> for RPi*'s with the pre-RPi4B style power connections but do
>> control the frequency for the OPi+2E.)
>=20
> The following had been left implicit about my context and
> how it manages memory space use.
>=20
> I'll note that I do not use tmpfs or other such memory based
> file system techniques that could compete for RAM/swap. What
> is in use for the only file system involved is just the
> root file system:
>=20
> # df -m
> Filesystem 1M-blocks Used Avail Capacity Mounted on
> /dev/gpt/BPIM3root 195378 63940 115808 36% /
> devfs 0 0 0 100% /dev
>=20
> It is a USB SSD. The swap partition is also on that same
> media. (The BPIM3 based name dates back to before the
> BPI-M3 power connection failed and I switched to the
> OPi+2E.)
>=20
> I'll note that I've started a new from-scratch build without
> LDFLAGS.lld+=3D -Wl,--threads=3D1 . So at some point I'll have
> information about how much of a difference (+/-) in swap
> usage it actually made for with vs. without, if any.
Looks like, for such 4-core contexts, that bothering
with LDFLAGS.lld+=3D -Wl,--threads=3D1 is typically a
waste of effort for both swap usage and time . . .
With LDFLAGS.lld+=3D -Wl,--threads=3D1 :
Mem: . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki =
MaxObs(Act+Wir)
Swap: . . . , 537588Ki MaxObsUsed
without:
Mem: . . ., 715756Ki MaxObsActive, 194816Ki MaxObsWired, 903132Ki =
MaxObs(Act+Wir)
Swap: . . ., 557208Ki MaxObsUsed
With LDFLAGS.lld+=3D -Wl,--threads=3D1 :
World built in 72960 seconds, ncpu: 4, make -j4
Kernel(s) GENERIC-NODBG built in 4998 seconds, ncpu: 4, make -j4
without:
World built in 72804 seconds, ncpu: 4, make -j4
Kernel(s) GENERIC-NODBG built in 4824 seconds, ncpu: 4, make -j4
So, just not that much of a difference compared to the overall
sizes or times involved.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A6150F56-062F-4582-853A-319C1EE4DDCB>