Date: Sun, 28 Apr 2024 18:48:35 -0700 To: FreeBSD Mailing List <freebsd-ports@freebsd.org>, FreeBSD Toolchain <freebsd-toolchain@freebsd.org> Subject: I've started collecting tmpfs usage figures from a poudriere-devel bulk -a for later publishing some of the top ones (handy for TMPFS_BLACKLIST judgments) Message-ID: <97E1D1BA-CBA1-42C6-8202-B6D654941C28@yahoo.com> References: <97E1D1BA-CBA1-42C6-8202-B6D654941C28.ref@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I've modified my local poudriere-devel to have Success and Failure lines =
also
report the tmpfs size at that point. Using, say, script to log the =
output to
a file allows later sorting and listing of the TMPFS usage filgures. =
(The
context is an amd64 one that has the resources to do a full bulk -a with
USE_TMPFS=3Dall invovled.) An example (from an in-progress bulk -a that =
has a
long way to go):
# grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -n -r -k11
[00:36:37] [18] [00:26:53] Finished print/texlive-docs | =
texlive-docs-20230313: Success ending TMPFS: 10.67 GiB
[01:04:41] [07] [00:06:58] Finished net-mgmt/telegraf | =
telegraf-1.30.1_1: Success ending TMPFS: 10.52 GiB
[01:03:32] [25] [00:06:09] Finished security/trivy | trivy-0.50.1_1: =
Success ending TMPFS: 10.10 GiB
. . .
[01:15:56] [20] [00:00:54] Finished databases/pg_tileserv | =
pg_tileserv-1.0.9_12: Failed: build TMPFS: 2.61 GiB
. . .
Note that the design is for sort with -k11 to work for Success and
for Failure. (This is why "ending" is in place for Success.) I
choose to use poudriere -N (no coloring) for this kind of activity.
This helps for figuring out what all is appropriate for listing in
TMPFS_BLACKLIST for a poudriere-devel configuration to avoid tmpfs
competing too much for RAM+SWAP. (But approraite free file system
space is needed.)
"<" below is what is new, ">" is what was original, in
/usr/local/share/poudriere/common.sh :
5928,5934d5927
< tmpfs_at_end=3D"$(env BLOCKSIZE=3D512 df -t tmpfs \
< ${MASTERMNTROOT}/${MY_JOBID}/ \
< ${MASTERMNTROOT}/${MY_JOBID}/.p/ \
< ${MASTERMNTROOT}/${MY_JOBID}/usr/local/ \
< 2>/dev/null | tail -3 \
< | awk '{ tmpfs_use +=3D $3; } END { printf "%s %.2f %s", "TMPFS:", =
tmpfs_use*512/(1024**3), "GiB" }')"
<=20
5942c5935
< "Success${COLOR_RESET} ending ${tmpfs_at_end}"
---
> "Success"
5968c5961
< "Failed: ${COLOR_PHASE}${failed_phase}${COLOR_RESET} =
${tmpfs_at_end}"
---
> "Failed: ${COLOR_PHASE}${failed_phase}"
The form of use that I've done also involves (over?) use of
MUTUALLY_EXCLUSIVE_BUILD_PACKAGES . It is not as good of data
for this other use, but the same .txt file can be processed
with:
# grep TMPFS: ~/mmjnk-bulk-a-output.txt | sort -r -k3 | more
[01:42:09] [04] [00:48:16] Finished lang/erlang-runtime21 | =
erlang-runtime21-21.3.8.24_3: Success ending TMPFS: 1.92 GiB
[01:38:39] [28] [00:44:41] Finished lang/erlang-runtime22 | =
erlang-runtime22-22.3.4.27: Success ending TMPFS: 1.92 GiB
[01:05:41] [02] [00:34:54] Finished lang/erlang-runtime26 | =
erlang-runtime26-26.2.4: Success ending TMPFS: 2.02 GiB
. . .
to find longer running package builds. This is subject to
significant variation based on what other builders are
running in parallel at the time and what sort of load
averages are involved over period in question. The
MUTUALLY_EXCLUSIVE_BUILD_PACKAGES that I've used will
limit that to some extent. But the result is comparisons
of some builds that have no activity in parallel by other
builders vs. builds that have extensive parallel activity
by other builders (of not-huge packages).
Note: In modern times the [1D: notation and the like for
what the -k3 compares are not well placed in the overall
list compared to the likes of, say, [20: . The day vs.
hour comparison is not a straight forward thing to sort
on.
Hopefully in a few days I'll be able to list off example
top tmpfs usage for USE_TMPFS=3Dall and top build times as
well (such as they are).
Note:
This is from my personal environment. I've not tried to
simulate how FreeBSD's official package builders are set
up to operate.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?97E1D1BA-CBA1-42C6-8202-B6D654941C28>