Date: Sat, 25 Jan 2020 13:41:16 -0800 From: Cy Schubert <Cy.Schubert@cschubert.com> To: freebsd-current@freebsd.org, Mark Millard <marklmi@yahoo.com>, yasu@utahime.org, FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: After update to r357104 build of poudriere jail fails with 'out of swap space' Message-ID: <A6C8D5FD-EC99-4846-8F43-ABE7B6D27D03@cschubert.com> In-Reply-To: <DAA3A910-25F2-447A-B540-C35985DA822E@yahoo.com> References: <DAA3A910-25F2-447A-B540-C35985DA822E.ref@yahoo.com> <DAA3A910-25F2-447A-B540-C35985DA822E@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On January 25, 2020 12:02:07 PM PST, Mark Millard <marklmi@yahoo=2Ecom> wro= te: >Yasuhiro KIMURA yasu at utahime=2Eorg wrote on >Sat Jan 25 14:45:13 UTC 2020 : > >> I use VirtualBox to run 13-CURRENT=2E Host is 64bit Windows 10 1909 and >> spec of VM is as following=2E >>=20 >> * 4 CPU >> * 8GB memory >> * 100GB disk >> - 92GB ZFS pool (zroot) >> - 8GB swap >>=20 >> Today I updated this VM to r357104=2E And after that I tried to update >> poudriere jail with `poudriere jail -u -j jailname -b`=2E But it failed >> at install stage=2E After the failure I found following message is >> written to syslog=2E >>=20 >> Jan 25 19:18:25 rolling-vm-freebsd1 kernel: pid 7963 (strip), jid 0, >uid 0, was killed: out of swap space > >This message text's detailed wording is a misnomer=2E >Do you also have any messages of the form: > >=2E =2E =2E sentinel kernel: swap_pager_getswapspace(32): failed > >If yes: you really were out of swap space=2E >If no: you were not out of swap space, > or at least it is highly unlikely that you were=2E > >FreeBSD kills processes for multiple potential reasons=2E >For example: > >a) Still low on free RAM after a number of tries to increase it above a >threshold=2E >b) Slow paging I/O=2E >c) =2E =2E =2E (I do not know the full list) =2E =2E =2E > >Unfortunately, FreeBSD is not explicit about the category >of problem that leads to the kill activity that happens=2E > >You might learn more by watching how things are going >via top or some such program or other way of monitoring=2E > > >Below are some notes about specific tunables that might >or might not be of help=2E (There may be more tunables >that can help that I do not know about=2E) > >For (a) there is a way to test if it is the issue by >adding to the number of tries before it gives up and >starts killing things=2E That will either: > >1) let it get more done before kills start >2) let it complete before the count is reached >3) make no significant difference > >(3) would imply that (b) or (c) are involved instead=2E > >(1) might be handled by having it do even more tries=2E > >For delaying how long free RAM staying low is >tolerated, one can increase vm=2Epageout_oom_seq from >12 to larger=2E The management of slow paging I've >less experience with but do have some notes about >below=2E > >Examples follow that I use in contexts with >sufficient RAM that I do not have to worry about >out of swap/page space=2E These I've set in >/etc/sysctl=2Econf =2E (Of course, I'm not trying to >deliberately run out of RAM=2E) > ># ># Delay when persisstent low free RAM leads to ># Out Of Memory killing of processes: >vm=2Epageout_oom_seq=3D120 > >I'll note that figures like 1024 or 1200 or >even more are possible=2E This is controlling how >many tries at regaining sufficient free RAM >that that level would be tolerated long-term=2E >After that it starts Out Of Memory kills to get >some free RAM=2E > >No figure is designed to make the delay >unbounded=2E There may be large enough figures to >effectively be bounded beyond any reasonable >time to wait=2E > > >As for paging I/O (this is specific to 13, >or was last I checked): > ># ># For plunty of swap/paging space (will not ># run out), avoid pageout delays leading to ># Out Of Memory killing of processes: >vm=2Epfault_oom_attempts=3D-1 > >(Note: In my context "plunty" really means >sufficient RAM that paging is rare=2E But >others have reported on using the -1 in >contexts where paging was heavy at times and >OOM kills had been happening that were >eliminated by the assignment=2E) > >I've no experience with the below alternative >to that -1 use: > ># ># For possibly insufficient swap/paging space ># (might run out), increase the pageout delay ># that leads to Out Of Memory killing of ># processes: >#vm=2Epfault_oom_attempts=3D ??? >#vm=2Epfault_oom_wait=3D ??? ># (The multiplication is the total but there ># are other potential tradoffs in the factors ># multiplied, even for nearly the same total=2E) > > >I'm not claiming that these 3 vm=2E???_oom_??? >figures are always sufficient=2E Nor am I >claiming that tunables are always available >that would be sufficient=2E Nor that it is easy >to find the ones that do exist that might >help for specific OOM kill issues=2E > >I have seen reports of OOM kills for other >reasons when both vm=2Epageout_oom_seq and >vm=2Epfault_oom_attempts=3D-1 were in use=2E >As I understand, FreeBSD did not report >what kind of condition lead to the >decision to do an OOM kill=2E > >So the above notes may or may-not help you=2E > >> To make sure I shutdown both VM and host, restarted them and tried >> update of jail again=2E Then the problem was reproduced=2E > > >=3D=3D=3D >Mark Millard >marklmi at yahoo=2Ecom >( dsl-only=2Enet went >away in early 2018-Mar) > >_______________________________________________ >freebsd-current@freebsd=2Eorg mailing list >https://lists=2Efreebsd=2Eorg/mailman/listinfo/freebsd-current >To unsubscribe, send any mail to >"freebsd-current-unsubscribe@freebsd=2Eorg" It's not just poudeiere=2E Standard port builds of chromium, rust and thun= derbird also fail on my machines with less than 8 GB=2E --=20 Pardon the typos and autocorrect, small keyboard in use=2E=20 Cy Schubert <Cy=2ESchubert@cschubert=2Ecom> FreeBSD UNIX: <cy@FreeBSD=2Eorg> Web: https://www=2EFreeBSD=2Eorg The need of the many outweighs the greed of the few=2E Sent from my Android device with K-9 Mail=2E Please excuse my brevity=2E
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A6C8D5FD-EC99-4846-8F43-ABE7B6D27D03>