Date: Wed, 20 Jan 2021 18:33:58 -0800 From: bob prohaska <fbsd@www.zefox.net> To: Mark Millard <marklmi@yahoo.com> Cc: Current FreeBSD <freebsd-current@freebsd.org>, freebsd-arm@freebsd.org Subject: Re: Silent hang in buildworld, was Re: Invoking -v for clang during buildworld Message-ID: <20210121023358.GA58854@www.zefox.net> In-Reply-To: <A6150F56-062F-4582-853A-319C1EE4DDCB@yahoo.com> References: <C75D3D9C-4284-4BE3-B2CD-5DC6BBB60843@yahoo.com> <20210117174006.GA30728@www.zefox.net> <85889EAE-F579-4220-9185-944D9AA5075A@yahoo.com> <20210118015009.GA31353@www.zefox.net> <60CCCDE8-E3D3-4920-9FC0-A945330F6830@yahoo.com> <A0427375-5515-4D3C-AF2A-915E60A836A7@yahoo.com> <00104FAD-E32B-4DDE-80DD-FCEF14CEC06B@yahoo.com> <D9878BB6-2693-4A04-9E1C-126E0D378F7B@yahoo.com> <056845FE-7131-4951-96AF-805D07F7BE0D@yahoo.com> <A6150F56-062F-4582-853A-319C1EE4DDCB@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 20, 2021 at 09:51:33AM -0800, Mark Millard wrote:
> On 2021-Jan-19, at 17:42, Mark Millard <marklmi at yahoo.com> wrote:
>
> > On 2021-Jan-18, at 21:12, Mark Millard <marklmi at yahoo.com> wrote:
> >
> >> On 2021-Jan-18, at 19:19, Mark Millard <marklmi at yahoo.com> wrote:
> >>
> >>> . . .
> >>>> FYI: I re-established my access to a RPi2B V1.1 and made
> >>>> it report: "maximum recommended amount (468832 pages)"
> >>>>
> >>>> (The figure can vary some from release to release.)
> >>>>
> >>>> 468832*4096 == 1920335872 or a little over 1831 MiBytes
> >>>>
> >>>> For the 4096 Byte pages, that means that the following from
> >>>> gpart fits without complaint (size is in blocks, not pages):
> >>>>
> >>>> 413140992 3686400 da0p2 freebsd-swap (1.8G)
> >>>>
> >>>> 3686400*512 is a little over 1.75 GiByte or 1800 MiByte. So
> >>>> I've left some room below 1831 MiBytes, but not a lot.
> >>>>
> >>>> FYI about my build experiment that is running:
> >>>>
> >>>> # sysctl hw.physmem
> >>>> hw.physmem: 979042304
> >>>>
> >>>> which, in recent times for armv7, I can (and did) set in
> >>>> /boot/loader.conf on a faster cortex-A7 SBC (that can boot
> >>>> the same media but has more RAM).
> >>>>
> >>>> So I tried a -j4 build, but with LDFLAGS.lld+= -Wl,--threads=1
> >>>> in use and my other particular src.conf/make.conf like content
> >>>> (so the builds do likely differ from yours in various ways).
> >>>> My build is producing a non-debug build (but with -g symbols).
> >>>> Somewhat after where your buildworld.log stops, my odd variant
> >>>> of top was reporting:
> >>>>
> >>>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892732Ki MaxObs(Act+Wir)
> >>>> Swap: . . . , 145832Ki MaxObsUsed
> >>>>
> >>>> and top was also showing lots of processes as having "0B" RES
> >>>> in STATE "wait" or "nanslp" (so, apparently swapped out, not paging).
> >>>> ("MaxObs" is short for "maximum observed".)
> >>>>
> >>>> For comparison, your swapscript.log reported a maximum total of
> >>>> 346192 KiBytes "Used" for swap, about 98% into the log file.
> >>>>
> >>>> (Time goes by . . .)
> >>>>
> >>>> It finished with building libllvm and is part way into building
> >>>> libclang. This is probably well past where your hangup happened,
> >>>> given that your published buildworldlog file stopped with
> >>>> libllvm's Target/ARM/ARMMCInstLower.o . My odd top now shows:
> >>>>
> >>>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892732Ki MaxObs(Act+Wir)
> >>>> Swap: . . . , 392328Ki MaxObsUsed
> >>>>
> >>>> The build continues to run. I'll let you know how it goes.
> >>>> . . .
> >>>
> >>> Just after libclang finished my odd top showed:
> >>>
> >>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892736Ki MaxObs(Act+Wir)
> >>> Swap: . . . , 537588Ki MaxObsUsed
> >>>
> >>> After liblldb:
> >>>
> >>> Mem: . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 899276Ki MaxObs(Act+Wir)
> >>> Swap: . . . , 537588Ki MaxObsUsed
> >>>
> >>> Much later, after the lldb program had been built:
> >>>
> >>> Mem: . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki MaxObs(Act+Wir)
> >>> Swap: . . . , 537588Ki MaxObsUsed
> >>>
> >>>>>> World build completed on Mon Jan 18 19:10:08 PST 2021
> >>>>>> World built in 72960 seconds, ncpu: 4, make -j4
> >>>
> >>> This was building from scratch what was already installed:
> >>>
> >>> # ~/fbsd-based-on-what-freebsd-main.sh
> >>> merge-base: 818390ce0ca539300dd15d7a817784f1e3f7a9b8
> >>> merge-base: CommitDate: 2021-01-13 21:27:44 +0000
> >>> 4180404713ec (HEAD -> mm-src) mm-src snapshot for mm's patched build in git context.
> >>> 818390ce0ca5 (freebsd/main, freebsd/HEAD, pure-src, main) arm64: fix early devmap assertion
> >>> FreeBSD OPiP2E_RPi2v11 13.0-CURRENT FreeBSD 13.0-CURRENT mm-src-c255938-g4180404713ec GENERIC-NODBG arm armv7 1300135 1300135
> >>>
> >>> This suggests that you should be able to build on the RPi2B v1.1,
> >>> using -j4, with appropriate configuration for what and how to build.
> >>>
> >>>
> >>> It is now building the matching kernel, my GENERIC-NODBG style.
> >>
> >> Done:
> >>
> >>>>> Kernel build for GENERIC-NODBG completed on Mon Jan 18 20:33:26 PST 2021
> >>>>> Kernel(s) GENERIC-NODBG built in 4998 seconds, ncpu: 4, make -j4
> >>
> >> So, World+Kernel in somewhat under 22 hours.
> >>
> >> The "MaxObs*" figures were unchanged, so:
> >>
> >> Mem: . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki MaxObs(Act+Wir)
> >> Swap: . . . , 537588Ki MaxObsUsed
> >>
> >> This suggests that, for now, 800 MiByte of swap would be something
> >> more than 1.5 times what it actually used and 1050 MiBytes would
> >> be something like 2.0 times what it actually used, so leaving some
> >> notable margin for variations in peek usage, at least when linker
> >> threading is avoided.
> >>
> >>
> >>
> >> As for what I used to control "what and how to build" . . .
> >>
> >> # more ~/sys_build_scripts.armv7-host/make_armv7_nodebug_clang_bootstrap-armv7-host.sh
> >> kldload -n filemon && \
> >> script ~/sys_typescripts/typescript_make_armv7_nodebug_clang_bootstrap-armv7-host-$(date +%Y-%m-%d:%H:%M:%S) \
> >> env __MAKE_CONF="/root/src.configs/make.conf" SRCCONF="/dev/null" SRC_ENV_CONF="/root/src.configs/src.conf.armv7-clang-bootstrap.armv7-host" \
> >> WITH_META_MODE=yes \
> >> WORLD_FLAGS="${WORLD_FLAGS} UBLDR_LOADADDR=0x42000000" \
> >> MAKEOBJDIRPREFIX="/usr/obj/armv7_clang/arm.armv7" \
> >> make $*
> >>
> >> (In my context, UBLDR_LOADADDR is ignored by anything that
> >> can not use the figure given. So I've no bothered to be
> >> more selective about having it in the armv7 builds.)
> >>
> >> # more ~/src.configs/make.conf
> >> LDFLAGS.lld+= -Wl,--threads=1
> >>
> >> # more ~/src.configs/src.conf.armv7-clang-bootstrap.armv7-host
> >> TO_TYPE=armv7
> >> #
> >> KERNCONF=GENERIC-NODBG
> >> TARGET=arm
> >> .if ${.MAKE.LEVEL} == 0
> >> TARGET_ARCH=${TO_TYPE}
> >> .export TARGET_ARCH
> >> .endif
> >> #
> >> #WITH_CROSS_COMPILER=
> >> WITH_SYSTEM_COMPILER=
> >> WITH_SYSTEM_LINKER=
> >> #
> >> WITH_LIBCPLUSPLUS=
> >> WITHOUT_BINUTILS_BOOTSTRAP=
> >> WITH_ELFTOOLCHAIN_BOOTSTRAP=
> >> #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=
> >> WITHOUT_LLVM_TARGET_AARCH64=
> >> WITH_LLVM_TARGET_ARM=
> >> WITHOUT_LLVM_TARGET_MIPS=
> >> WITHOUT_LLVM_TARGET_POWERPC=
> >> WITHOUT_LLVM_TARGET_RISCV=
> >> WITHOUT_LLVM_TARGET_X86=
> >> WITH_CLANG=
> >> WITH_CLANG_IS_CC=
> >> WITH_CLANG_FULL=
> >> WITH_CLANG_EXTRAS=
> >> WITH_LLD=
> >> WITH_LLD_IS_LD=
> >> WITHOUT_BINUTILS=
> >> #
> >> WITH_LLDB=
> >> #
> >> WITH_BOOT=
> >> WITHOUT_LIB32=
> >> #
> >> #
> >> WITHOUT_WERROR=
> >> #WERROR=
> >> MALLOC_PRODUCTION=
> >> WITH_MALLOC_PRODUCTION=
> >> WITHOUT_ASSERT_DEBUG=
> >> WITHOUT_LLVM_ASSERTIONS=
> >> #
> >> # Avoid stripping but do not control host -g status as well:
> >> DEBUG_FLAGS+=
> >> #
> >> WITH_REPRODUCIBLE_BUILD=
> >> WITH_DEBUG_FILES=
> >> #
> >> # Use of the .clang 's here avoids
> >> # interfering with other C<?>FLAGS
> >> # usage, such as ?= usage.
> >> CFLAGS.clang+= -mcpu=cortex-a7
> >> CXXFLAGS.clang+= -mcpu=cortex-a7
> >> CPPFLAGS.clang+= -mcpu=cortex-a7
> >>
> >> (I do not claim that you would want WITH_REPRODUCIBLE_BUILD .
> >> I just happen to have been experimenting with it. You might
> >> not want to be explicit about the cpu to target. You might
> >> not want WITH_CLANG_EXTRAS .)
> >>
> >> # more /usr/fbsd/mm-src/sys/arm/conf/GENERIC-NODBG
> >> include "GENERIC"
> >>
> >> ident GENERIC-NODBG
> >>
> >> makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
> >>
> >> options AUDIT # Not enabled by default in armv7/v6 kernels
> >> # Enabled here to allow kyua test runs to
> >> # possibly report auditing works.
> >>
> >> options ALT_BREAK_TO_DEBUGGER
> >>
> >> options KDB # Enable kernel debugger support
> >>
> >> # For minimum debugger support (stable branch) use:
> >> options KDB_TRACE # Print a stack trace for a panic
> >> options DDB # Enable the kernel debugger
> >>
> >> # Extra stuff:
> >> #options VERBOSE_SYSINIT=0 # Enable verbose sysinit messages
> >> #options BOOTVERBOSE=1
> >> #options BOOTHOWTO=RB_VERBOSE
> >> options ALT_BREAK_TO_DEBUGGER # Enter debugger on keyboard escape sequence
> >> options KLD_DEBUG
> >> #options KTR
> >> #options KTR_MASK=KTR_TRAP
> >> ##options KTR_CPUMASK=0xF
> >> #options KTR_VERBOSE
> >>
> >> # Disable any extra checking for. . .
> >> nooptions INVARIANTS # Enable calls of extra sanity checking
> >> nooptions INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS
> >> nooptions WITNESS # Enable checks to detect deadlocks and cycles
> >> nooptions WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed
> >> nooptions DEADLKRES # Enable the deadlock resolver
> >> nooptions MALLOC_DEBUG_MAXZONES # Separate malloc(9) zones
> >> nooptions DIAGNOSTIC
> >> nooptions BUF_TRACKING
> >> nooptions FULL_BUF_TRACKING
> >> nooptions USB_DEBUG
> >> nooptions USB_REQ_DEBUG
> >> nooptions USB_VERBOSE
> >>
> >> The /boot/loader.conf file and the /etc/sysctl.conf files
> >> both contained:
> >>
> >> vm.pageout_oom_seq=120
> >> vm.pfault_oom_attempts=-1
> >>
> >> (The hw.physmem=979042304 in /boot/loader.conf was very-special,
> >> to better approximate your environment. I also controlled the
> >> cpu frequency used via a line in /etc/sysctl.conf . I do not
> >> bother with such non-default frequency usage [or related settings]
> >> for RPi*'s with the pre-RPi4B style power connections but do
> >> control the frequency for the OPi+2E.)
> >
> > The following had been left implicit about my context and
> > how it manages memory space use.
> >
> > I'll note that I do not use tmpfs or other such memory based
> > file system techniques that could compete for RAM/swap. What
> > is in use for the only file system involved is just the
> > root file system:
> >
> > # df -m
> > Filesystem 1M-blocks Used Avail Capacity Mounted on
> > /dev/gpt/BPIM3root 195378 63940 115808 36% /
> > devfs 0 0 0 100% /dev
> >
> > It is a USB SSD. The swap partition is also on that same
> > media. (The BPIM3 based name dates back to before the
> > BPI-M3 power connection failed and I switched to the
> > OPi+2E.)
> >
> > I'll note that I've started a new from-scratch build without
> > LDFLAGS.lld+= -Wl,--threads=1 . So at some point I'll have
> > information about how much of a difference (+/-) in swap
> > usage it actually made for with vs. without, if any.
>
> Looks like, for such 4-core contexts, that bothering
> with LDFLAGS.lld+= -Wl,--threads=1 is typically a
> waste of effort for both swap usage and time . . .
>
> With LDFLAGS.lld+= -Wl,--threads=1 :
>
> Mem: . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki MaxObs(Act+Wir)
> Swap: . . . , 537588Ki MaxObsUsed
>
> without:
>
> Mem: . . ., 715756Ki MaxObsActive, 194816Ki MaxObsWired, 903132Ki MaxObs(Act+Wir)
> Swap: . . ., 557208Ki MaxObsUsed
>
>
> With LDFLAGS.lld+= -Wl,--threads=1 :
>
> World built in 72960 seconds, ncpu: 4, make -j4
> Kernel(s) GENERIC-NODBG built in 4998 seconds, ncpu: 4, make -j4
>
> without:
>
> World built in 72804 seconds, ncpu: 4, make -j4
> Kernel(s) GENERIC-NODBG built in 4824 seconds, ncpu: 4, make -j4
>
>
> So, just not that much of a difference compared to the overall
> sizes or times involved.
>
A first OS build/install cycle on armv7 (RPI2) using meta mode
finished without trouble. Sources were a day or two newer than
the kernel, -j4 buildworld took 157121 seconds. Peak swap use
was half again as much at 732932. No constraints on ld.lld
beyond defaults. I'm a little surprised at the extreme slowness,
but this was a fully-debug'd-current kernel and sources were
slightly newer than existing world.
In case there's interest I've put what log files I could gather at
http://www.zefox.net/~fbsd/rpi2/buildworld/main-c950-gff1a307801/
Thanks for your attention and help!!
bob prohaska
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20210121023358.GA58854>