Date: Tue, 6 Aug 2024 10:56:29 +0200 From: meloun.michal@gmail.com To: Mark Millard <marklmi@yahoo.com>, Dimitry Andric <dim@FreeBSD.org> Cc: FreeBSD Toolchain <freebsd-toolchain@freebsd.org>, FreeBSD ARM List <freebsd-arm@freebsd.org> Subject: Re: Any known way to build devel/llvm* ( such as devel/llvm19 ) with --threads=1 for its linker activity during the build? Message-ID: <0853529b-eb45-43c7-9957-8fe23001c94a@freebsd.org> In-Reply-To: <EF58D898-1188-4F74-AE7F-DC100E074D50@yahoo.com> References: <4FFD603F-E67C-4B62-B91B-8BE365EAA050@yahoo.com> <82E78798-C376-45C4-80FE-96AD14229419@yahoo.com> <dcfa36c0-8ba6-4e8f-937d-17a99d8b23cf@freebsd.org> <F65EFFEF-FD93-49AB-B0E0-7BF880760EA8@yahoo.com> <E8E2166F-06DD-42FF-B54E-215BC507B3C3@yahoo.com> <0b3b532c-ae94-439c-81aa-9e80a08af43f@freebsd.org> <EFEE572A-B34D-4A02-AA15-D7E15F12A826@yahoo.com> <4b5a6fa1-3b08-4b8b-8577-c548116d2115@freebsd.org> <EF58D898-1188-4F74-AE7F-DC100E074D50@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 05.08.2024 23:35, Mark Millard wrote:
> On Aug 5, 2024, at 08:57, meloun.michal@gmail.com wrote:
>
>> On 05.08.2024 11:09, Mark Millard wrote:
>>> On Aug 5, 2024, at 00:44, meloun.michal@gmail.com wrote:
>>>>> . . .
>>>> Tegra has 4 Cortex-A15 cores and 2 GB of RAM.
>>> OrangePi+ 2ed: Cortex-A7 with 4 cores and 2 GiBytes of RAM.
>>> I wonder if the 2483 MiBytes would end up being about the
>>> same on the Tegra variation indicated.
>>
>> Yep, it must be +/-same. The 2/2 GB for userland/kernel is defined by HW. Only size of shared libraries may affect (lower) usable user space for given program.
>>>> All ports are built with default options. The only non-standard item is the swap size -> I have 16GB of swap on a swap partition on the SSD.
>>> Wow, 16 GiBYtes of swap space for 2 GiBytes of RAM. I guess
>>> when the swap is added that you get a notice-pair of the
>>> structure:
>>> QUOTE
>>> warning: total configured swap (. . . pages) exceeds maximum recommended amount (. . . pages).
>>> warning: increase kern.maxswzone or reduce amount of swap.
>>> END QUOTE
>>> with a rather large difference between the two ". . ." figures.
>>> Do you make other adjustments to deal with the otherwise-reported
>>> potential mistuning? It appears to make tradeoffs in the kernel
>>> internal memory handling, if I understand right.
>> The above message should be interpreted as: warning, the kernel may in word, rear case need to allocate additional
>> memory when swapping some object(memory) out. This may leads to deadlock/panic. But again, event is this warning valid,
>> resulting deadlock/panic is very rare. I newer see it in past many years...
>
> Interesting.
>
>>>> But I guess that's not important in this case.
>>> At least for my context, it appears that memory allocations
>>> are failing to find a big enough free area inside the
>>> process's address space --without running out of system
>>> RAM+SWAP space overall.
>>> For the OrangePi+ 2ed ( and devel/llvm18 18.1.7 ) it was
>>> during the earlier linker run for:
>>> FAILED: bin/lli-child-target
>>> . . .
>>> LLVM ERROR: out of memory
>>> Allocation failed
>>> That much finished just fine on the Windows DevKit
>>> 2023 used via a armv7 jail ( devel/llvm18 18.1.8_1 ).
>>> The failure point was in a later link ( matching what
>>> I saw via devel/llvm19 ).
>>>> I just started build of llvm19 - but it takes few hours to complete..
>>> Probably fewer hours than on the OrangePi+ 2ed but
>>> more than on the Windows DevKit 2023 (if they were
>>> completing, anyway).
>>
>> The native build is still running (60% in fact), arm32 jail build has been stopped on my Honeycomb (killed by OOM).Unfortunately this is an old problem and is common on all platforms. The current LLVM cannot be built without additional tricks on machines that have less than 2GB (RAM + swap) per core.....
>
> FYI: A small RAM+SWAP test for amd64-as-amd64 for devel/llvm18 18.1.7 . . .
>
> USE_TMPFS=no
>
> Targetting BE_NATIVE without BE_AMDGPU and without MLIR. Avoiding being stripped.
> (MLIR is resource and time expensive to build but I make no use of it.)
> (I also make no use of BE_AMDGPU materials.)
> (I've not been doing any cross builds in a very long time.)
> (Not stripping can make for better failure reporting.)
>
> (Leading whitespace might notn b e preserved in the copy of the diff:)
> # git -C /usr/ports/ diff devel/llvm18
> diff --git a/devel/llvm18/Makefile b/devel/llvm18/Makefile
> index 1b1f759ba50e..779ddf3bea7e 100644
> --- a/devel/llvm18/Makefile
> +++ b/devel/llvm18/Makefile
> @@ -87,7 +87,7 @@ CMAKE_ARGS+= -DLLVM_ENABLE_TERMINFO=OFF
> CMAKE_ARGS+= -DLLVM_VERSION_SUFFIX=
> OPTIONS_DEFINE= BE_AMDGPU BE_WASM CLANG COMPILER_RT DOCS LLD STATIC_LIBS
> -OPTIONS_DEFAULT= BE_AMDGPU BE_WASM CLANG LLD
> +OPTIONS_DEFAULT= BE_WASM CLANG LLD
> OPTIONS_SINGLE= BACKENDS
> OPTIONS_SINGLE_BACKENDS=BE_FREEBSD BE_NATIVE BE_STANDARD
> OPTIONS_EXCLUDE_armv6= COMPILER_RT
> @@ -103,7 +103,7 @@ OPTIONS_DEFINE_powerpc= GOLD
> OPTIONS_DEFINE_powerpc64= GOLD
> OPTIONS_DEFINE_powerpc64le= GOLD
> -OPTIONS_DEFAULT+= BE_STANDARD COMPILER_RT EXTRAS LIT LLDB MLIR OPENMP \
> +OPTIONS_DEFAULT+= BE_NATIVE COMPILER_RT EXTRAS LIT LLDB OPENMP \
> PYCLANG POLLY STATIC_LIBS
> OPTIONS_DEFAULT_amd64= GOLD
> OPTIONS_DEFAULT_powerpc= GOLD
> @@ -211,8 +211,8 @@ CONFLICTS_INSTALL= ${PORTNAME}${LLVM_SUFFIX} ${PORTNAME}${LLVM_SUFFIX}-lite
> .if defined(WITH_DEBUG)
> CMAKE_BUILD_TYPE= RelWithDebInfo
> -STRIP=
> .endif
> +STRIP=
> PLIST_SUB+= LLVM_MAJOR_MINOR=${LLVM_MAJOR_MINOR} \
> LLVM_MAJOR=${LLVM_MAJOR} \
> @@ -224,10 +224,10 @@ FIRST_COMMAND= ${COMMANDS:C/^/XXXX/1:MXXXX*:C/^XXXX//}
> MAN1SRCS+= ${LLVM_MAN1SRCS}
> -STRIP_LIBS= BugpointPasses.so \
> - LLVMHello.so \
> - ${LIBNAME}.0 \
> - libLTO.so
> +#STRIP_LIBS= BugpointPasses.so \
> +# LLVMHello.so \
> +# ${LIBNAME}.0 \
> +# libLTO.so
> EXTRAS_LIBS= \
> libclangApplyReplacements \
>
> Note: This amd64 testing is not using --threads=1 for linking.
>
> A Hyper-V context for amd64-as-amd64 set up for small memory testing with 4 Hyper-V cores:
>
> real memory = 2147483648 (2048 MB)
> avail memory = 2029309952 (1935 MB)
>
> Swap: 3584Mi Total
>
> So: AVAIL_RAM+SWAP == 5519 MiBytes
>
> Building just llvm18-18.1.7 :
>
> [00:00:22] [01] [00:00:00] Building devel/llvm18@default | llvm18-18.1.7
> [00:55:27] [01] [00:55:05] Finished devel/llvm18@default | llvm18-18.1.7: Success ending TMPFS: 0.00 GiB
>
> (My poudriere-devel is patched to report the ending TMPFS figures.)
>
> From my patched-up top's output from monitoring during the build:
>
> Mem: . . . , 1462Mi MaxObsActive, 582432Ki MaxObsWired, 1940Mi MaxObs(Act+Wir+Lndry)
>
> Swap: 3584Mi Total , . . . , 1090Mi MaxObsUsed, 2455Mi MaxObs(Act+Lndry+SwapUsed), 2995Mi MaxObs(A+Wir+L+SU),
> 2996Mi (A+W+L+SU+InAct)
>
> So, slightly under 3 GiBytes RAM+SWAP observed as used, well under 8 GiBytes.
>
> 3GiByte[RAM+SWAP]/4Core == 0.75 GiByte[RAM+SWAP]/Core, well under 2 GiByte[RAM+SWAP]/Core.
>
> From this and the like of your OOM reporting, I get an idea how much MLIR, BE_AMDGPU, and BE_STANDARD contribute to the RAM+SWAP use. (I've not tested such directly in a long time.)
>
>
> For reference:
>
> # uname -apKU
> FreeBSD 7950X3D-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT #144 main-n271413-1f7df7570174-dirty: Sat Jul 27 16:01:52 UTC 2024 root@7950X3D-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1500021 1500021
>
> # ~/fbsd-based-on-what-commit.sh -C /usr/main-src
> 1f7df7570174 (HEAD -> main, freebsd/main, freebsd/HEAD) LinuxKPI: move __kmalloc from slab.h to slab.c
> Author: Bjoern A. Zeeb <bz@FreeBSD.org>
> Commit: Bjoern A. Zeeb <bz@FreeBSD.org>
> CommitDate: 2024-07-26 11:51:04 +0000
> branch: main
> merge-base: 1f7df757017404011732196e65981d9325f7a89f
> merge-base: CommitDate: 2024-07-26 11:51:04 +0000
> n271413 (--first-parent --count for merge-base)
>
> # ~/fbsd-based-on-what-commit.sh -C /usr/ports
> 4e0616a2066d (HEAD -> main, freebsd/main, freebsd/HEAD) graphics/libavif: drop maintainership
> Author: Jan Beich <jbeich@FreeBSD.org>
> Commit: Jan Beich <jbeich@FreeBSD.org>
> CommitDate: 2024-07-13 01:31:18 +0000
> branch: main
> merge-base: 4e0616a2066dc9e45cef47de812efdea4e09709c
> merge-base: CommitDate: 2024-07-13 01:31:18 +0000
> n671165 (--first-parent --count for merge-base)
>
> Note: This /usr/ports/ is from before devel/llvm19 or /devel/freebsd-gcc14 . I've not synchronized this UFS environment with my experimention with a more recent ports vintage. devel/llvm18 is at 18.1.7 still here.
>
> Hyper-V is actually using the same UFS boot media as the native UFS FreeBSD boot does --but is using a smaller swap partition.
>
So, I'm finally able to build llvm19, on native armv7 and also on arm32
jail.
First It needs the not yet promoted patch
https://www.mail-archive.com/lldb-commits@lists.llvm.org/msg95950.html
But the more serious problem is that llvm's memory requirements are out
of control.
llvm ports with default configuration causes OOM on a 4-core armv7 with
4G RAM/16G swap, on a 4-core aarch64 with 4G RAM/16G swap and on a
16-core aarch64 with 16G RAM/16G swap.
It's very hard to name this as correct/expected behavior. (Added dim@
just FYI).
Mark, I think that root case of your problems is that you're trying to
link with libraries (shared or local) with debug information
(unstriped). This causes a huge memory consumption. I don't remember if
it just wastes address space (because mmapping a large library but
never accessing the debugging information) or if it accessing the
debugging information (so the link needs physical memory for it as well).
Michal
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0853529b-eb45-43c7-9957-8fe23001c94a>