From nobody Thu Jan 27 20:12:20 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3C12D1972E55 for ; Thu, 27 Jan 2022 20:12:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JlBb60Pk9z3LmZ for ; Thu, 27 Jan 2022 20:12:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643314347; bh=BhBUn9y0Ghnghc33Q3v4jnYGPqOUdMh89FgQCq6hh4w=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=U7nOtCJCTQXu1LHJoNjWCxYnRkKixCtoaQBYq42TPbYLoNR+tOjiL9zjAvuZ7lh8bSyH8tMyIdufUKBjc3P6ySsyu9aOkGnh6yxUIFO9ocw6mWr5pfNz6HCZb5zNnLIo6RSoWLLQuUjLz/kIm7Hy8EiMFe2uybnVEqsFuJgv9GoEztJXnWRvj03Eww7lZrJfed4qVMWanQUoj/9dtwfTnQoXcGgX58PWJbO5bRjM50sLx3DzumsejQuBlPRCmm5KnJqIySoulLQ2TuTSTqxVtmUnyt73ptP4IRerffVakal0QWaw9el4jIpouAKLqvVbhkjJoebDD4TjY5kaPlxW3A== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643314347; bh=h5v++PVQuXIZfZmc5q/pe4+4E17QJJipulSogNEewQ4=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=T6D0hqxbqdEekU8Z51t/uyxvPy5cDNnaRb/70KyIRoznUab/qkea8k3t4VKbsGYlE/2Ir3eGQl51TOt7g2xhuy+HGIUKvuBV48kTP6wuJABkgBCJJRig3zCDb/9Bh7Kv+eHdpwnN3HWGwZAbMIOgkKjyHeJH337hiAAc8K3LlcoS0mg8x7bnHC1RoGnqLUjlBzaTVP+ZKeWockj5l6/Pz+uWkV1V6f/cprc1PviOnAM0vhEnr6dIyImQA5Vxq0YMxNPLRv7MrH3yF1Xquw1jTGraJt80XP1wExqBB8hg/EnsgpU4GRR13iGffc2n+GB/V7l/mBu1SJ3tMdw3kx7r6g== X-YMail-OSG: 86dyfagVM1kKjZlDIhPiS.7pzSpq2DKJ9SE7dJu694pRS60593L8ooaiEUQsiUS _0LMZ.Lk2HnVUTeJCkuX0r40HL6KaqIvrPMr5J36aSGZmm4iC2YM9jt99xx_luVp38sO_gH0LHRv rgiBR8.PlruV67ow6JQ1Vlakl8E5ZC9sYIx0DeTqtJ49b7qVqvh_D1XWxP7nqhoo6AEbD9sOlICx Z.6JmNBEkm6MzSi6zU4p3tHNoPZDhM3jddkXvGNacfs412g89XbBXYzM9xFVswpBIl2yLXbjXlKU PEoU4_gW8aXenVrq4oBSR1pubwD0AhgjBZt9Iu927kuyFD.ShwyPDQhg0KJk22xUJ5CNH13f7VL4 upwma9A_YKXwjwhu6SInlzI3fpxhtkW7xYt9Q6sgHF6ieLAaX0afW7S6.4zWM_VoM0yH5nECmaQZ vbgalxUWR9K_5KP3V6LZUsBF8cyhkBMNrpN2Am9QjNkXjeIQZBhl_cYk.BHRltOG5l5hT1dc5eCo LgSn6YzI8UJ7M6dM42R_xZ0c2nOJGKOw51UbN9nYO4B.OuQMDWDQXB.eiAnFSRotNY4G3fWC85f3 3xev4Lj7byljlHtih0cMnG8aF3O8K91wXJAuyBHEGzDt.Ame88OX8wk7u9cRgCsj3LImgqqUsVWn m56MemOOJNP3xgMiFpccnfdEkWLpyLMorjc_NjpIeiiMGbCWXA8UepQadsnwl35wRSAAOaamVO34 yszrWWbgZ7zPH98F4JWOEgoJXtTN7ERaqIIU5w9N2pSB06YSBnU6kAVIUHI541LLmKE5aKEPyG0j v1dfd9Ndk9v3CA0WiGWfkWj.1j_m0njXPPRpyMs7jHYge1c5tL_7yGOwTZ92XNZMZ6gvhrW_1uVQ GYrJLMse7bp4R7tdiwVdMRp2H7Gt5IpgJKnV5x.2WUI4.oEXY.5iNeAauOrcF8eD3AEad30cyDtD M9Ed7imAdciG_4Xb_a5aySEdIDV6EA0of4Z8ESLURx0gu4UN74lgUX0toRQWn1sz.iGYP7jHdB.t ULNHt_711yjqlTpGBxwKD7aWmbzDvcVyYCXcL7rigGQCkY_rkAj5fn6.AIxj8ofR4jK6eYv46xtT 74v6j1thTku6UkTTa6iWIGQ9JBh2XiRFLE5.jppjdc6JaxYjUVv7NpYakdtBhLWs3JoilPGVy4K_ D4evO.CJ73Z97Nppj4wljS0j_AZx9laCiK2j.Lb5sd8id7HEpMCSq6njUfYqp1WOIHDRtxaOg1nh RbfDn7hxaXBSQe3lNdzXWnIsLZkjSaTFWyakBpGlbQT7ZVUzB4pZiCXZWuoJ92_lFCKL6p72t74O _wICNtaHneBSwEDGCGZcFXfjtHzP7OdehEFY_7l8uPPOzpRbZhu0vK5cnJAXj5KHon8GlTI8SeFu 6oDXerQ9NvBNRqYTvxCfPq5itDOYFI43sGPqGXMiYxEk9qt47srxk8iQ7Esa6fDTgdiCAkzsOF1I QZzjE09e_BLq79n8lqoP_uODi0205zqpOgHCowW5i6kxae0OaE7.ETdILTro7ig_.iJKcO6thuiZ jktVbU_vnl9YVOVt67YLCmKNdChfLW7OfHVZ3tz86kYhk7SV0VnnU5EbsAec.gxRTWcKz2xYgvsh s75Y7xvZGgXMTZXYRV8Pv2JMv0jlIhvFvEHXWHptsEjiYSmLGYw5mAl4x_1cB0W_tRosUHQscIn0 cQ5BWW9SwS2Llje1hGUQtA9J8kDHNKjRWYODPMECyMxid_ZIB5ACW6BJUrA6jHjWe8Efu25IT6KV C7.ey8dFdorBZl6kG1C0NZaH1MIttJnEuUrewXFfvWSyRG_kjIZ1hAyfE4PHDxHESl.f9cPOx347 _JxGyq3QkFMNW9NVlRUCyfrf0EHOGk.SC.4XemxfbHh0QKHVaYMiNpvOFK0nT8OhIICw61_bD_S5 DgOm1.a899bGzMM5oSGTlLHPfUVGMv69IkdN8DgXElAQq.UtWeFNIEIpOscS5glWy_K.ZL6yYhX7 CBGkPto0wwlGWdKdQX0IIjDJEzwItJsWXcZaT.0Wx4anoJW3xLUcwSaINqUkPOdAwGiKcTn0E5XX VnYFSuGI5cuuLIlmLkB2fcXtiAguHX17x1SBfrEn_6npYgy3f8EDNl1ZxNTwLO6OMK.yalU18rkL OExUnQ42EWZKCDXUG2yPnqOXw76x_Upfft93vWqNsCxOoFXIiLqAThuiBwbK9fk5M7M2ovwqDzYO CNfGlh_M6UcNAkLADoowOTd7FLRQpxMVXQZE.A9Se7QLKB.lg3XShv07i2XCwiDIk.cXBB41gjjd .tSE9UXfc3z6ENE8SmN5vaQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Thu, 27 Jan 2022 20:12:27 +0000 Received: by kubenode507.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID c78ab6bab38316f9a6d63bf794595527; Thu, 27 Jan 2022 20:12:22 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current From: Mark Millard In-Reply-To: Date: Thu, 27 Jan 2022 12:12:20 -0800 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2C7E741F-4703-4E41-93FE-72E1F16B60E2@yahoo.com> References: <20220127164512.GA51200@www.zefox.net> To: bob prohaska , Mark Johnston X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4JlBb60Pk9z3LmZ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=U7nOtCJC; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.148:from]; MLMMJ_DEST(0.00)[freebsd-arm]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.148:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Status: O Content-Length: 4588 Lines: 126 On 2022-Jan-27, at 11:31, Mark Millard wrote: > On 2022-Jan-27, at 08:45, bob prohaska wrote: >=20 >> Attempts to compile devel/llvm13 on a Pi4 running -current (updated >> on 20220126) with 8 GB of RAM and 8 GB of swap has failed on two = occasions using=20 >> make -DBATCH > make.log &=20 >> in /usr/ports/devel/llvm13 using the system compiler. The system is >> self-hosted.=20 Context question: ZFS? UFS? (In things involving memory usage issues, knowing which is always appropriate because of differences in memory use patterns.) >> The first failure reported clang error 139, but the second >> was different, reporting only: >> FAILED: = tools/flang/lib/Evaluate/CMakeFiles/obj.FortranEvaluate.dir/check-expressi= on.cpp.o >> along with a console report of >> +swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1258432, size: = 4096 >> +swap_pager: indefinite wait buffer: bufobj: 0, blkno: 627221, size: = 8192 >> +swap_pager: indefinite wait buffer: bufobj: 0, blkno: 240419, size: = 4096 >> +swap_pager: out of swap space >=20 > In recent builds, such as yours, the above "out of swap" is a > misnomer but is very interesting for what it is actually about. >=20 > Mark Johnston later wrote on 2022-Jan-15 about his "git: > 4a864f624a70 - main - vm_pageout: Print a more accurate message > to the console before an OOM kill" that produced the above report > of "out of swap space": >=20 > QUOTE > Hmm, those cases should likely be changed from "out of swap space" to > "failed to allocate swap metadata" or something like that. > END QUOTE >=20 > Your context proves the metadata problem really happens, so > the messaging should be fixed to not be misleading. >=20 > In my builds I've code that is more explicit: >=20 > diff --git a/sys/vm/swap_pager.c b/sys/vm/swap_pager.c > index 01cf9233329f..280621ca51be 100644 > --- a/sys/vm/swap_pager.c > +++ b/sys/vm/swap_pager.c > @@ -2091,6 +2091,7 @@ swp_pager_meta_build(vm_object_t object, = vm_pindex_t pindex, daddr_t swapblk) > 0, 1)) > printf("swap blk zone exhausted, = " > "increase = kern.maxswzone\n"); > + printf("swp_pager_meta_build: swap blk = uma zone exhausted\n"); > vm_pageout_oom(VM_OOM_SWAPZ); > pause("swzonxb", 10); > } else > @@ -2121,6 +2122,7 @@ swp_pager_meta_build(vm_object_t object, = vm_pindex_t pindex, daddr_t swapblk) > 0, 1)) > printf("swap pctrie zone = exhausted, " > "increase = kern.maxswzone\n"); > + printf("swp_pager_meta_build: swap = pctrie uma zone exhausted\n"); > vm_pageout_oom(VM_OOM_SWAPZ); > pause("swzonxp", 10); > } else >=20 > The "metadata" is the "swap blk uma zone" and "swap pctrie > uma zone". Unfortuantely, which got the failure is not still > indicated in the standard builds. >=20 >> +swp_pager_getswapspace(12): failed >> +pid 61012 (c++), jid 0, uid 0, was killed: failed to reclaim memory >=20 > Abssent being able to swap, it tries to reclaim --and that > too failed. That finally leads to the kills. >=20 >> Swap use peaked a little over 50%. >=20 > So at around 50% "swap blk uma zone" and/or "swap pctrie uma zone" > had problems, probably fragmentation related problems. >=20 >> After the first failure a restart >> of make using MAKE_JOBS_UNSAFE=3Dyes ran to completion with one = thread. >>=20 >> A copy of the build log, logging script and other notes is at >> http://www.zefox.net/~fbsd/rpi4/20220127/ >>=20 >> Clang error 139 has been seen several times during make buildworld on = a Pi3 running >> stable/13 with 2 GB of swap as well. Perhaps the two failures are = related. The Pi3=20 >> failures didn't report out of swap, all were clang error 139 with = "failed to reclaim=20 >> memory". Even with only 1 thread (j1) the failure reproduced. >>=20 >=20 > Note in your report above: obj.FortranEvaluate.dir >=20 > If you use the options to disable building flang (a.k.a., > the Fortran compiler build), your builds on the RPi4B > will likely work in the current configuration. >=20 > But it looks like you have identified a test context > for the "swap blk uma zone" and "swap pctrie uma zone" > handling. =3D=3D=3D Mark Millard marklmi at yahoo.com