From owner-freebsd-current@freebsd.org Mon Jan 27 22:26:06 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A4D40230CC2 for ; Mon, 27 Jan 2020 22:26:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48648Y68sfz4HDX for ; Mon, 27 Jan 2020 22:26:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: GqyJprAVM1kZNFHRHv0OpGLFHAi0IqEegE10oxPMNVpTEJCSPi9BuyCDsFZeA2f 1k.Z4dCGqOILRTDXWj_xzVtFvPitkHpDuw9oT_Ei7Ok28fiKWdi6iSbY1RKPxzG.o54IqZFoEs8q XgJHfI6qcLTVpjIJSo9Zwd19snBcReHCwinj4J78oqlL2GZLq_JbBMWmMk4OyvvKGvuVGG2vYBj8 YW_ItwviwOnlWXPQrtxksKPjkYLuI4ukVrdvgHr0AmprDjAmRqhy1WSS6P22o11Phm8EbupQ3_cP j6Pt.EudG8WkqGFiDhLaS4LBLkI3aduws2C.adPC6gWbv9_N.sDr.8cuLZydjZO.aPgC1My.yzIV 2PQQiyccmWhoVDyMUg5i3og0CAvT_W9_irznF25YXW0kHBHUBHogIxoTYBvGV1QGzHjDRT4h.F7O BRBMoukru8pYpLOjcHPxzBAcVQuIGitGRPdoIonfYWJ0BUiuNf2FlZvDmDnmCwIE9Wz9JY4AHJJ2 XrwA7pRwELsS.Nl_FHkH9aWW0HioTHyivk8_ON2XsY3J0eKi2V..RMWkHy3_fPNjf6AhE0UH5zDM cde8y2P_Pv0f._jjSX0t9C8W2oVDTgTiCK5jd6KpzK29jYih0N.75EUwB9Wp2nW2fgv3A9wiT.wi SbIDC8uq830.HkNHsUoSD7.IZU3zz1pjwQDzW5_WsKPPu6MRNJgg4E1S5yxlOA_FpKil8JIsHR_k xJP0HlAmuzxDtGhsLdAb1UMzzzM1YHBW32VCjl48k6JLNTXMkpTcEswvemNy_copWcd72T3wZcPP fVSW95AZk929GRh4osl4AmiouW9ecnovaEsrdfjQpkj232vZC3ljC01leYBM86KXuZuITbvT_qHC Wi8qaThF8Y9PYjGSu5e5DTo2ebCzpqSQqRJTvvJph_2eKLedXK0YzrpXAXBXvndsR7_Mm6kIrlP5 qO4a_Fy1cF5v1ge2ymYO5lkSTDqWfUvtmPuVwAesdJaCiow.VsK3m3FaGaFSgiflPl2TcUc.ykZP AgYybS64pMEXdsXHU982n_z8AZcQ.oouDYybA.2MUJDsrL.VU1SU8t1r9wMmg0AvBWHHPgD6OpB7 UbEWquona.sQVU81hJCOWq3B63xGVW8ZJfZhBpt62JmEO3WxfICf2t0guWNajzqYROuGLuwRPCtk 50qi2_.pDWkEql8v0B4y0rWpq_d9g90ph4Q5zO7e1I13Us9t2cho9xcdewhqSewdpbdRfWzw8T36 td1.ejmGCi6WcSzwBI2jDd7QOwSSB7MIWsiEBggToyGVhfYSGtOyA5GLdn.6Bao1A_ZVJ9DBDvaC pS1MFkbzpEQ97bAaVnF12JE5lUV.VZVhpstV.dJLXo16r11joYPVdmwdgDQGmgaTfkqY9U853Ep9 HULi2tk1Y2ci4zqPRbNK_k76kZRq90mgqWUAshh5hzgWUkQTg8GPvzd_9NJg46jed.Mx2M2w- Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 27 Jan 2020 22:26:04 +0000 Received: by smtp431.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 3fd107ed9dde4ed448e14cbf5ccc608b; Mon, 27 Jan 2020 22:26:01 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: After update to r357104 build of poudriere jail fails with 'out of swap space' From: Mark Millard In-Reply-To: <202001272048.00RKmiZs006726@slippy.cwsent.com> Date: Mon, 27 Jan 2020 14:25:59 -0800 Cc: "Rodney W. Grimes" , sgk@troutmask.apl.washington.edu, freebsd-current@freebsd.org, yasu@utahime.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <202001261745.00QHjkuW044006@gndrsh.dnsmgr.net> <202001271309.00RD96nr005876@slippy.cwsent.com> <202001272048.00RKmiZs006726@slippy.cwsent.com> To: Cy Schubert X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 48648Y68sfz4HDX X-Spamd-Bar: / X-Spamd-Result: default: False [-0.52 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; RCPT_COUNT_FIVE(0.00)[5]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.28)[-0.276,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (4.00), ipnet: 98.137.64.0/21(0.84), asn: 36647(0.67), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.25)[0.255,0]; RCVD_IN_DNSWL_NONE(0.00)[205.69.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[205.69.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jan 2020 22:26:06 -0000 On 2020-Jan-27, at 12:48, Cy Schubert = wrote: > In message , Mark = Millard=20 > write > s: >>=20 >>=20 >>=20 >> On 2020-Jan-27, at 10:20, Cy Schubert = wrote: >>=20 >>> On January 27, 2020 5:09:06 AM PST, Cy Schubert = >> wrote: >>>>> . . .=20 >>>>=20 >>>> Setting a lower arc_max at boot is unlikely to help. Rust was = building >>>> on=20 >>>> the 8 GB and 5 GB 4 core machines last night. It completed = successfully >>>> on=20 >>>> the 8 GB machine, while using 12 MB of swap. ARC was at 1307 MB. >>>>=20 >>>> On the 5 GB 4 core machine the rust build died of OOM. 328 KB swap = was=20 >>>> used. ARC was reported at 941 MB. arc_min on this machine is 489.2 = MB. >>>=20 >>> MAKE_JOBS_NUMBER=3D3 worked building rust on the 5 GB 4 core = machine. ARC is >> at 534 MB with 12 MB swap used. >>=20 >> If you increase vm.pageout_oom_seq to, say, 10 times what you now = use, >> does MAKE_JOBS_NUMBER=3D4 complete --or at least go notably longer = before >> getting OOM behavior from the system? (The default is 12 last I = checked. >> So that might be what you are now using.) >=20 > It's already 4096 (default is 12). Wow. Then the count of tries to get free RAM above the threshold does not seem likely to be the source of the OOM kills. >>=20 >> Have you tried also having: vm.pfault_oom_attempts=3D"-1" (Presuming >> you are not worried about actually running out of swap/page space, >> or can tolerate a deadlock if it does run out.) This setting presumes >> head, not release or stable. (Last I checked anyway.) >=20 > Already there. Then page-out delay does not seem likely to be the source of the OOM = kills. > The box is a sandbox with remote serial console access so deadlocks = are ok. >=20 >>=20 >> It would be interesting to know what difference those two settings >> together might make for your context: it seems to be a good context >> for testing in this area. (But you might already have set them. >> If so, it would be good to report the figures in use.) >>=20 >> Of course, my experiment ideas need not be your actions. >=20 > It's a sandbox machine. We already know 8 GB works with 4 threads on = as=20 > many cores. And, 5 GB works with 3 threads on 4 cores. It would be nice to find out what category of issue in the kernel is driving the OOM kills for your 5GB context with MAKE_JOBS_NUMBER=3D4. Too bad the first kill does not report a backtrace spanning the code choosing to do the kill (or otherwise report the type of issue leading the the kill). Your is consistent with the small arm board folks reporting that = recently contexts that were doing buildworld and the like fine under somewhat older kernels have started getting OOM kills, despite the two settings. At the moment I'm not sure how to find the category(s) of issue(s) that is(are) driving these OOM kills. Thanks for reporting what settings you were using. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)