Date: Mon, 10 Mar 2025 13:52:28 -0700 From: Mark Millard <marklmi@yahoo.com> To: FreeBSD Mailing List <freebsd-ports@freebsd.org> Cc: Bryan Drewery <bdrewery@FreeBSD.org> Subject: Re: Something to avoid for using tmpfs for inactive poudriere(-devel) builders? Message-ID: <B5977932-F6B7-41E3-9CF6-95F86BA7A212@yahoo.com> In-Reply-To: <03CE95FB-83A9-4649-8F07-0387A1F92072@yahoo.com> References: <58D16ED7-486A-4F68-B925-28E9C90E384D@yahoo.com> <690968E6-85D7-4725-843E-F126EE802B03@yahoo.com> <03CE95FB-83A9-4649-8F07-0387A1F92072@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[I was curious about what had large or huge install sizes --and
how many for my example thresholds.]
On Mar 9, 2025, at 15:45, Mark Millard <marklmi@yahoo.com> wrote:
> On Mar 9, 2025, at 12:34, Mark Millard <marklmi@yahoo.com> wrote:
>=20
>> On Mar 9, 2025, at 01:02, Mark Millard <marklmi@yahoo.com> wrote:
>>=20
>>> In looking for why the builders used notable tmpfs space
>>> even when only one builder was left that was active,
>>> I discovered that, for example, each builder ends up
>>> with its own copy of /usr/local/poudriere/data/.m/*/*/usr/
>>> (and more) that does not end up being cleared out while
>>> the builder is inactive. This looked to be a systematic
>>> contribution to the tmpfs usage during times when various
>>> builders are inactive.
>>>=20
>>> df -m /usr/local/poudriere/data/.m/main-aarch64-default/*/*/ | sed =
-e 's@/[0-9][0-9]/@/*/@' | sort -k1,1 -k6,6 -k3,3 -k1,6 -u
>>> /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue =
1114846 498917 526741 49% =
/usr/local/poudriere/data/.m/main-aarch64-default/*/rescue
>>> /usr/local/poudriere/data/packages/main-aarch64-default/.building =
1114846 498917 526741 49% =
/usr/local/poudriere/data/.m/main-aarch64-default/*/packages
>>> /usr/local/poudriere/data/packages/main-aarch64-default/.building =
1114846 498917 526741 49% =
/usr/local/poudriere/data/.m/main-aarch64-default/ref/packages
>>> /usr/local/poudriere/jails/main-aarch64/rescue =
1114846 498917 526741 49% =
/usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue
>>> /usr/ports/distfiles =
1114846 498917 526741 49% =
/usr/local/poudriere/data/.m/main-aarch64-default/*/distfiles
>>> /usr/ports/distfiles =
1114846 498917 526741 49% =
/usr/local/poudriere/data/.m/main-aarch64-default/ref/distfiles
>>> Filesystem =
1M-blocks Used Avail Capacity Mounted on
>>> devfs =
0 0 0 0% =
/usr/local/poudriere/data/.m/main-aarch64-default/*/dev
>>> devfs =
0 0 0 0% =
/usr/local/poudriere/data/.m/main-aarch64-default/ref/dev
>>> procfs =
0 0 0 0% =
/usr/local/poudriere/data/.m/main-aarch64-default/*/proc
>>> procfs =
0 0 0 0% =
/usr/local/poudriere/data/.m/main-aarch64-default/ref/proc
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/01
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/02
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/03
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/04
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/05
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/06
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/07
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/08
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/09
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/10
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/11
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/12
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/13
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/14
>>> tmpfs =
66539 1140 65398 2% =
/usr/local/poudriere/data/.m/main-aarch64-default/ref
>>>=20
>>> For the above, only ref/ and one other were active at the time.
>>> Imagine having 32 builders or 128 builders or even more with
>>> 1140 MiBytes for each inactive one. For the, above all the
>>> builders actually reported:
>>>=20
>>> Inspecting . . .: determining shlib requirements
>>>=20
>>> for each package build it tried. No actual builds were done.
>>>=20
>>> [I'll note that the world directory trees for the booted
>>> system and for the poudriere jail are from official
>>> PkgBase builds that were installed. Also, the system
>>> is UFS based, not ZFS based.]
>>>=20
>>> For reference (note the use of -x):
>>>=20
>>> # du -xsAm /usr/local/poudriere/data/.m/main-*-default/*/[uv][sa]r/ =
| sed -e 's@/[0-9][0-9]/@/*/@' | sort -k1,2 -u
>>> 376 /usr/local/poudriere/data/.m/main-aarch64-default/*/var/
>>> 376 /usr/local/poudriere/data/.m/main-aarch64-default/ref/var/
>>> 713 /usr/local/poudriere/data/.m/main-aarch64-default/*/usr/
>>> 713 /usr/local/poudriere/data/.m/main-aarch64-default/ref/usr/
>>>=20
>>> So 1089 MiBytes or so of the 1140 MiBytes for each such tmpfs
>>> is contributed overall by the combination of var/ and usr/ for
>>> the example.
>>>=20
>>> # poudriere jail -l
>>> JAILNAME VERSION OSVERSION ARCH METHOD =
TIMESTAMP PATH
>>> . . .
>>> main-aarch64 15.0-CURRENT aarch64 pkgbase =
2025-02-12 22:49:27 /usr/local/poudriere/jails/main-aarch64
>>> . . .
>>>=20
>>> There is more to look into here, in that, historically, larger
>>> builders leave behind larger tmpfs usage until the next builder
>>> reuse (if any). Having a few of lang/rust , devel/llvm20 , etc.
>>> finish, but those builders not starting something new for a
>>> notable time, can lead to huge RAM+SWAPSPACE usage for those
>>> inactive builders for that time for USE_TMPFS=3Dall without
>>> TMPFS_BLACKLIST=3D in use.
>>=20
>> Further explorations in a ZFS context indicate that the following
>> stick around in tmpfs for inactive builders for the USE_TMPFS=3Dall
>> type of context:
>>=20
>> /usr/local/poudriere/data/.m/*/*/usr/local/ (varying sizes, unlike =
the rest of the below)
>>=20
>> Note: the first * replaced text of the pattern [0-9][0-9] .
>> In this case, 32 possibilities matched.
>>=20
>> The ones with lines having "32" as the count of matching lines
>> in the later supporting material (largest size per to smallest):
>>=20
>> /usr/local/poudriere/data/.m/*/*/usr/bin/
>> /usr/local/poudriere/data/.m/*/*/var/cache/
>> /usr/local/poudriere/data/.m/*/*/usr/lib/
>> /usr/local/poudriere/data/.m/*/*/usr/lib32/
>> /usr/local/poudriere/data/.m/*/*/usr/share/
>> /usr/local/poudriere/data/.m/*/*/usr/include/
>> /usr/local/poudriere/data/.m/*/*/usr/sbin/
>>=20
>> When a:
>>=20
>> /usr/local/poudriere/data/.m/*/*/usr/local/
>>=20
>> has something like a lang/rust install that is large
>> or even huge, the inactive builder is also individually
>> large or huge as well. But there is no uniform "32"
>> [or whatever] multiplier. My test did not happen to
>> generate such an example usr/local/ .
>=20
> Well, rust's 1 GiBytes or so does appear to show up. Huge
> compared to 1 GiByte or installing multiple large packages
> into a usr/local/ did not happen but can happen depending
> on what all is being built.
>=20
>> For reference material from my test:
>> (The below avoids usr/src/ and usr/ports/ .)
>> (/ref/ is not replaced, unlike /[0-9][0-9]/ becoming /*/ .)
>>=20
>> # du -xsAm =
/usr/local/poudriere/data/.m/*-default/*/[uvw]*/*[a-bd-rt-zA_Z0-9]/ | =
sed -e 's@/[0-9][0-9]/@/*/@' | sort -k1,2 | uniq -c | sort -n -k2,3 | =
tail -40 ; pou
>> driere status -b
>> 32 24 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/sbin/
>> 1 27 =
/usr/local/poudriere/data/.m/official-amd64-default/ref/usr/include/
>> 32 27 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/include/
>> 1 29 /usr/local/poudriere/data/.m/official-amd64-default/*/var/db/
>> 3 51 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 54 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 3 62 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 63 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 65 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 82 =
/usr/local/poudriere/data/.m/official-amd64-default/ref/usr/share/
>> 32 82 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/share/
>> 1 122 =
/usr/local/poudriere/data/.m/official-amd64-default/*/wrkdirs/usr/
>> 1 129 =
/usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib32/
>> 32 129 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib32/
>> 1 259 =
/usr/local/poudriere/data/.m/official-amd64-default/ref/usr/lib/
>> 32 259 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/lib/
>> 1 262 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 2 265 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 270 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 272 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 288 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 332 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 339 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 355 =
/usr/local/poudriere/data/.m/official-amd64-default/ref/var/cache/
>> 32 355 =
/usr/local/poudriere/data/.m/official-amd64-default/*/var/cache/
>> 1 379 =
/usr/local/poudriere/data/.m/official-amd64-default/ref/usr/bin/
>> 32 379 /usr/local/poudriere/data/.m/official-amd64-default/*/usr/bin/
>> 1 381 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 423 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 446 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 462 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 520 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 522 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 532 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 3 546 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 548 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 657 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 794 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 1073 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> 1 1393 =
/usr/local/poudriere/data/.m/official-amd64-default/*/usr/local/
>> =3D>> [official-amd64-default] [2025-03-09_11h09m56s] =
[parallel_build] Time: 01:00:03
>> Queued: 450 Inspected: 0 Ignored: 2 Built: 440 Failed: 1 Skipped: 5 =
Fetched: 0 Remaining: 2
>> ID TOTAL ORIGIN PKGNAME =
PHASE TIME TMPFS CPU% MEM%
>> [01] 00:00:07 x11-toolkits/wxgtk30@gtk3 | wx30-gtk3-3.0.5.1_6 =
configure 00:00:02 2.40 GiB 0.1% 0.1%
>> =3D>> Logs: =
/usr/local/poudriere/data/logs/bulk/official-amd64-default/2025-03-09_11h0=
9m56s
I used an amd64 context for the below.
# pkg rquery -e '%s >=3D 1073741824' -rFreeBSD '%sb (%sh): %o' | wc -l
63
# pkg rquery -e '%s >=3D 1073741824' -rFreeBSD '%sb (%sh): %o' | sort =
-nr
6312637268 (5.88GiB): x11-fonts/nerd-fonts
5028065806 (4.68GiB): cad/kicad-library-packages3d-devel
5014656473 (4.67GiB): cad/kicad-library-packages3d
4023994200 (3.75GiB): science/quantum-espresso-pseudopotentials
3759663181 (3.50GiB): print/texlive-docs
3523340789 (3.28GiB): games/0ad
3519431692 (3.28GiB): science/geant4
3498302532 (3.26GiB): java/intellij-ultimate
3456464186 (3.22GiB): games/ufoai-data
3419032211 (3.18GiB): print/texlive-texmf
3392802952 (3.16GiB): games/diaspora
3364341955 (3.13GiB): x11-fonts/iosevka
2849694498 (2.65GiB): devel/llvm-devel
2688064435 (2.50GiB): games/flightgear-data
2600325941 (2.42GiB): devel/pycharm-pro
2291266787 (2.13GiB): devel/jetbrains-clion
2265796966 (2.11GiB): x11-fonts/iosevka-comfy
2204358709 (2.05GiB): devel/jetbrains-goland
2125363178 (1.98GiB): devel/jetbrains-webstorm
2124118296 (1.98GiB): devel/jetbrains-phpstorm
2103384613 (1.96GiB): science/afni
2072316177 (1.93GiB): devel/jetbrains-rustrover
2031255963 (1.89GiB): graphics/piglit
2029504371 (1.89GiB): lang/ghc
1962446365 (1.83GiB): games/alienarena-data
1926448239 (1.79GiB): security/seclists
1910595369 (1.78GiB): games/urbanterror-data
1903938256 (1.77GiB): devel/llvm19
1856224925 (1.73GiB): devel/llvm18
1848424990 (1.72GiB): devel/llvm20
1697277047 (1.58GiB): devel/llvm17
1657044767 (1.54GiB): lang/rust-nightly
1654848172 (1.54GiB): lang/ghc810
1632416131 (1.52GiB): www/onlyoffice-documentserver
1615499118 (1.50GiB): devel/pycharm-ce
1606980295 (1.50GiB): editors/linux-r7-office
1596435146 (1.49GiB): devel/llvm14
1595739363 (1.49GiB): devel/rubymine
1552335509 (1.45GiB): science/openmolcas
1518593666 (1.41GiB): devel/llvm13
1496606231 (1.39GiB): devel/llvm16
1360483588 (1.27GiB): lang/linux-dotnet-sdk
1316472542 (1.23GiB): biology/cytoscape
1293585141 (1.20GiB): lang/ghc94
1247648556 (1.16GiB): lang/ghc92
1247211549 (1.16GiB): editors/lazarus-qt5-devel
1246863609 (1.16GiB): editors/lazarus-qt6-devel
1241195659 (1.16GiB): editors/lazarus-devel
1221647684 (1.14GiB): graphics/blender-doc
1214602305 (1.13GiB): japanese/dams
1196058036 (1.11GiB): games/pioneer
1195741547 (1.11GiB): x11-fonts/google-fonts
1194904485 (1.11GiB): lang/linux-dotnet11-sdk
1192270465 (1.11GiB): games/xonotic
1189280335 (1.11GiB): lang/dotnet
1172734857 (1.09GiB): games/quake-maps
1149121461 (1.07GiB): lang/rust
1133648487 (1.06GiB): editors/lazarus-qt5
1133294170 (1.06GiB): editors/lazarus-qt6
1126663720 (1.05GiB): editors/lazarus
1114054879 (1.04GiB): science/chrono
1085741063 (1.01GiB): games/redeclipse-data
1074153804 (1.00GiB): math/gap
For reference, for >=3D 512 MiBytes, for >=3D 256 MiBytes, and
for >=3D 128 MiBytes:
# pkg rquery -e '%s >=3D 536870912' -rFreeBSD '%sb (%sh): %o' | wc -l
147
# pkg rquery -e '%s >=3D 268435456' -rFreeBSD '%sb (%sh): %o' | wc -l
331
# pkg rquery -e '%s >=3D 134217728' -rFreeBSD '%sb (%sh): %o' | wc -l
638
Of course, folks building there own that might add, say,
line number debug information being kept, could have bigger
figures for the same packages for the same platform.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B5977932-F6B7-41E3-9CF6-95F86BA7A212>