Date: Mon, 22 Jan 2024 11:56:34 -0800 From: Mark Millard <marklmi@yahoo.com> To: Baptiste Daroussin <bapt@FreeBSD.org>, dev-commits-src-main@freebsd.org Subject: Re: git: 636592343c3e - main - tmpfs: increase memory reserve to a percent of available memory + swap Message-ID: <BEC5B876-00F1-4517-9167-1AC29D1E9FDB@yahoo.com> In-Reply-To: <33F50320-DA8F-4913-9197-5AED3D916E7A@yahoo.com> References: <33F50320-DA8F-4913-9197-5AED3D916E7A@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jan 22, 2024, at 11:37, Mark Millard <marklmi@yahoo.com> wrote: >=20 > Baptiste Daroussin <bapt_at_freebsd.org> wrote on > Date: Mon, 22 Jan 2024 13:23:37 UTC : >=20 >> On Mon, Jan 22, 2024 at 07:15:22AM -0600, Mike Karels wrote: >>> On 21 Jan 2024, at 21:48, Alexey Dokuchaev wrote: >>>=20 >>>> On Mon, Jan 22, 2024 at 03:27:57AM +0000, Jessica Clarke wrote: >>>>> On 19 Dec 2023, at 15:34, Mike Karels <karels@FreeBSD.org> wrote: >>>>>> commit 636592343c3ec0feb61a4d8043676381384420dd >>>>>>=20 >>>>>> tmpfs: increase memory reserve to a percent of available memory + = swap >>>>>>=20 >>>>>> [...] >>>>>>=20 >>>>>> Add sysctl for vfs.tmpfs.memory_percent and the pre-existing >>>>>> vfs.tmpfs.memory_reserved to tmpfs(5). >>>>>>=20 >>>>>> PR: 275436 >>>>>> MFC after: 1 month >>>>>> Reviewed by: rgrimes >>>>>> Differential Revision: https://reviews.freebsd.org/D43011 >>>>>=20 >>>>> I just got: >>>>>=20 >>>>> make[6]: Could not create temporary file /tmp/makeDl4i9S: >>>>> No space left on device >>>>>=20 >>>>> after doing a buildworld -j4 on a 4 core, 16 GiB system where /tmp = is a >>>>> tmpfs, with the prior buildworld having taken me past this commit, >>>>> which I strongly suspect to be the cause. Whilst I understand the >>>>> motivation behind your commits, this is a sad regression to have; = it's >>>>> not exactly a particularly strange situation. >>>=20 >>> Bug 276402 is about the interaction of this change with ZFS ARC, = which >>> seems to be problematical. I spent some time investigating = yesterday. >>> Depending on the dynamics, I get different results. If tmpfs grows, >>> the ARC will be reduced if needed. But if the ARC grows to the point >>> that tmpfs sets free space to 0 or quite low, attempts to write to >>> tmpfs fail without any reduction in ARC. >>>=20 >>> Jess, two quesions: >>>=20 >>> 1. Are you using ZFS on this system? >>>=20 >>> 2. Can you try with vfs.tmpfs.memory_percent set to 100? >>>=20 >>>> FWIW, I've seen two people complain about this last week, = apparently >>>> this kernel OOM protection doesn't work very well. :-/ >>>=20 >>> So far, I have only seen OOM kills when memory + swap are quite = short, >>> with a tmpfs memory_percent above 95. But with ZFS, it can happen >>> when the ARC could have been reduced. >>>=20 >>> I will investigate more today. If I don't find a way to resolve = this, >>> I'll probably commit a change to set vfs.tmpfs.memory_percent to 100 >>> by default, at least for now, and if that works around the problem. >>>=20 >> This is easily reproducible with poudriere, if you try to build some >> ports/packages which are big memory consumers, for example any chrome = variant or >> just lang/rust, I have a machine with 64G of ram >=20 > How many hardware threads? > How much swap space? For got to ask: How many parallel builders allowed in the bulk run at once? > You specified: USE_TMPFS=3Dall > ALLOW_MAKE_JOBS=3Dyes ? > MAKE_JOBS_NUMBER use (or the like)? > I assume ZFS but any ARC specific tuning? >=20 > Not that you would want to do what I do, but it > illustrates why I ask about the contexts explored. >=20 >=20 > 16 hardware thread aarch64 with 64 GiBytes of RAM context: >=20 > I've historically used: Forgot to say: Number of parallel builders allowed in the bulk run equal to the count of hardware threads, so 16 here. > ZFS untuned ARC context (but I also do the same in UFS contexts) > USE_TMPFS=3Dall > ALLOW_MAKE_JOBS=3Dyes > no MAKE_JOBS_NUMBER other than for editors/lapce* > RAM+SWAP =3D=3D 310 GiBytes (so a little over RAM+SWAP=3D=3D4.8*RAM) >=20 > Basically SWAP is preventing tmpfs use from running > out of space, despite the likes of 25 GiByte+ use > of tmpfs by the rust build, for example. >=20 > Historically multiple toolchains building in parallel > in the resulting high load average context have > completed just fine, including when rust is one of > them. I've not come close to running out of RAM+SWAP > so far. >=20 > Although my testing of "bulk -a" is very rare, such > has worked when I tried it, also using the hig load > average style. >=20 > Why ~4.8? Because somewhere between there and 5.0 the > binding of the swap space starts puttting out a notice > about potentially being mistuned. (It reports on a > potential change that I do not know the other tradeoffs > for. So I stay in the range where no notice is made.) >=20 >> and it is impossible to get >> lang/rust build in poudriere (USE_TMPFS=3Dall) without setting >> vfs.tmpfs.memory_percent to 100. >=20 > I'm guessing this is implicitly: without having a huge > RAM+SWAP space. >=20 >> the poudriere dies with plenty of no space left on device. >=20 > I will note that I've not tested my configuration > with the change yet, holding back on updating FreeBSD > for other reasons. But I doubt that I'd run out of > RAM+SWAP, given the huge RAM+SWAP that I've > historically used for the context. >=20 > I do analogously on 2 other systems: > 32 GiBytes of RAM and 8 hardware threads > 192 GiBytes of RAM and 32 hardware threads > (Both having both ZFS and UFS media.) >=20 > On the various small RAM systems, I use > USE_TMPFS=3Ddata or USE_TMPFS=3Dno and avoid > ZFS. >=20 > I also always avoid spinning rust. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BEC5B876-00F1-4517-9167-1AC29D1E9FDB>