Date: Sun, 9 Mar 2025 01:02:51 -0800 From: Mark Millard <marklmi@yahoo.com> To: FreeBSD Mailing List <freebsd-ports@freebsd.org> Cc: Bryan Drewery <bdrewery@FreeBSD.org> Subject: Something to avoid for using tmpfs for inactive poudriere(-devel) builders? Message-ID: <58D16ED7-486A-4F68-B925-28E9C90E384D@yahoo.com> References: <58D16ED7-486A-4F68-B925-28E9C90E384D.ref@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In looking for why the builders used notable tmpfs space even when only one builder was left that was active, I discovered that, for example, each builder ends up with its own copy of /usr/local/poudriere/data/.m/*/*/usr/ (and more) that does not end up being cleared out while the builder is inactive. This looked to be a systematic contribution to the tmpfs usage during times when various builders are inactive. df -m /usr/local/poudriere/data/.m/main-aarch64-default/*/*/ | sed -e = 's@/[0-9][0-9]/@/*/@' | sort -k1,1 -k6,6 -k3,3 -k1,6 -u /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue = 1114846 498917 526741 49% = /usr/local/poudriere/data/.m/main-aarch64-default/*/rescue /usr/local/poudriere/data/packages/main-aarch64-default/.building = 1114846 498917 526741 49% = /usr/local/poudriere/data/.m/main-aarch64-default/*/packages /usr/local/poudriere/data/packages/main-aarch64-default/.building = 1114846 498917 526741 49% = /usr/local/poudriere/data/.m/main-aarch64-default/ref/packages /usr/local/poudriere/jails/main-aarch64/rescue = 1114846 498917 526741 49% = /usr/local/poudriere/data/.m/main-aarch64-default/ref/rescue /usr/ports/distfiles = 1114846 498917 526741 49% = /usr/local/poudriere/data/.m/main-aarch64-default/*/distfiles /usr/ports/distfiles = 1114846 498917 526741 49% = /usr/local/poudriere/data/.m/main-aarch64-default/ref/distfiles Filesystem = 1M-blocks Used Avail Capacity Mounted on devfs = 0 0 0 0% = /usr/local/poudriere/data/.m/main-aarch64-default/*/dev devfs = 0 0 0 0% = /usr/local/poudriere/data/.m/main-aarch64-default/ref/dev procfs = 0 0 0 0% = /usr/local/poudriere/data/.m/main-aarch64-default/*/proc procfs = 0 0 0 0% = /usr/local/poudriere/data/.m/main-aarch64-default/ref/proc tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/01 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/02 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/03 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/04 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/05 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/06 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/07 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/08 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/09 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/10 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/11 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/12 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/13 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/14 tmpfs = 66539 1140 65398 2% = /usr/local/poudriere/data/.m/main-aarch64-default/ref For the above, only ref/ and one other were active at the time. Imagine having 32 builders or 128 builders or even more with 1140 MiBytes for each inactive one. For the, above all the builders actually reported: Inspecting . . .: determining shlib requirements for each package build it tried. No actual builds were done. [I'll note that the world directory trees for the booted system and for the poudriere jail are from official PkgBase builds that were installed. Also, the system is UFS based, not ZFS based.] For reference (note the use of -x): # du -xsAm /usr/local/poudriere/data/.m/main-*-default/*/[uv][sa]r/ | = sed -e 's@/[0-9][0-9]/@/*/@' | sort -k1,2 -u 376 /usr/local/poudriere/data/.m/main-aarch64-default/*/var/ 376 /usr/local/poudriere/data/.m/main-aarch64-default/ref/var/ 713 /usr/local/poudriere/data/.m/main-aarch64-default/*/usr/ 713 /usr/local/poudriere/data/.m/main-aarch64-default/ref/usr/ So 1089 MiBytes or so of the 1140 MiBytes for each such tmpfs is contributed overall by the combination of var/ and usr/ for the example. # poudriere jail -l JAILNAME VERSION OSVERSION ARCH METHOD = TIMESTAMP PATH . . . main-aarch64 15.0-CURRENT aarch64 pkgbase = 2025-02-12 22:49:27 /usr/local/poudriere/jails/main-aarch64 . . . There is more to look into here, in that, historically, larger builders leave behind larger tmpfs usage until the next builder reuse (if any). Having a few of lang/rust , devel/llvm20 , etc. finish, but those builders not starting something new for a notable time, can lead to huge RAM+SWAPSPACE usage for those inactive builders for that time for USE_TMPFS=3Dall without TMPFS_BLACKLIST=3D in use. =3D=3D=3D Mark Millard marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?58D16ED7-486A-4F68-B925-28E9C90E384D>