From nobody Fri Jan 28 06:00:15 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AEE861979859 for ; Fri, 28 Jan 2022 06:00:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-8.consmr.mail.gq1.yahoo.com (sonic315-8.consmr.mail.gq1.yahoo.com [98.137.65.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JlRdM65vPz3rd2 for ; Fri, 28 Jan 2022 06:00:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643349617; bh=Xc/Sjwo55C7wrJUbZsvLfQUUEohHdeptFDoVnforKew=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=f40cQQgjXJ3OtpXmVf4IK8W6urOzUXrZQ5Z4L/7UihgjTon+qZNREzZcGnO914bha6ldZUCtNInmCGIoWysM7b0/ShvFHqmb+iIfkMLr8Uc391b33/tPHlsMxFPJwQ6U6Q/gVoIyDarE1sEPnLH3/sZA0fduSYHZFFTdkVbb7R8FEFxLmfRd+5qimdjHkCmsZZZx30/IRj9OQehaObxl0mOpSPjapLaM9vB26RGb9qMBF5GGiWuAjJ0eEexFYF50HoPbH1ozRKW4fM3viyeNOubCiEbUcVILQMQUrjKp3eHvgpgAqI7YeeVPUOI/xYIr5ch3386JmE65Thu15tk9Wg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1643349617; bh=Z2ui3o8IenWMDgEbh1ZFnrekgEtdTSqc1/giaY+8E1h=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=E0OrDo/dbVaaZkaaGMwG9f9UF2Slnv4l5J0AJ3s7NGY6Zfe2F2i6Ui6LRNLcQ139oxCDkKtsjahLnTg2ATU6WErNuxaaN83SiKxAWEj8T4vFENPrHVrsd8jQas0XA+pLckEqinspEkfeR6yxcmobiPmEhAyyDHJG2G++BOVd+39D21y5C5XlUgesXISUg9r1gRcHIAplOhO9GyyIHTluX0s4i4kfUUc+0Bt9k/tCQx+mOvC2NIgoWtvmaz3kbqLuVsu26AobXsKBOy7RQyNL2SJP2vUuxdhfCh/OKEtKVq+eFzj1EY57Wt7GNEXUJDz5HtNSQ/+gZhsqWRl97LoyCg== X-YMail-OSG: QhqM4N4VM1mYmnCrpz83emOnBsRNB7iJVTTHZ9QtZzVVtoGwquS8jLUiVe.1b5a QEbnQuQrKYfmLfO1hcgibYjKcNnVTaK7EgXS8_z182MyjbHKjkAnZKTF5tj_u4R.iJz6oX602oC_ gRSsMZEw76Tyf4Nmem4ZDBrQ9sG78IdWmjoVbAnOWiqQTQ3B4m5hbqxiSfT8nLu5Tofs2fbv0XpY C5wmVLd.IsKrsvPZ.nxfYueLWcNdJeQhafriZoTAO4gGqVnEA4oxXow2SQOe8d1mY9xIajKtAtnJ SotXeDRwqM6cLrbVj_.5DWKez3DuaaUtAse3sglOLVh5Z7NJXSEIbv8hPnB45zE.iBiNI01DsoJU rlopy9m8I5GUIvuomkJaWaW0TdH8zkjXWy9ctmCXhheZkiS.DLb4xZro1tUO.93xP.P5xDzwnyLO Ndnzy06JWDKxXhGheSu07nOHPfisGSEZ2pi3Gjc2XainJql0I8PFZuUzwhOqPbEH_fhM7La3NRUN EvhUQ7zEpctCyB0kBQUSko9qj13wz.CQ4usTxsrqFYoeBhBx3GfaS7uaneqdChANUOusA33wFclP CAirQ_AJy.UK2f8a8xpBJukHbzJfvr5oWYH851k3vOC0aJDhDNbT564pX0ZzwJE4jzOvVpdDRFDb dzd9UXR1FKYRSo1dJ6DrNooe6pmUeLWvKKOLrcUtwFNDmhKSQN4n.isnIxP0FzMS5fvgDz.34MHj 2JEhedKWQbjpk26yBXNT7uPF5Su7bcxySURpVyn1UTHmhfqtdB6UMPRiAeAgvjcCM_jiT_f_KTDo JeG.mvsqhFQiPguBLvPj7dzevHU01.f.z1TgPJICrxJV_fcL0HWedj5Y4aR6.FER0uUHfzBxWuPR 3vpbrcqZIu8Ow2pC3JghOOl_49S5yA2dSijudCkTvO42kAmPgJHbjNJ83t.25nxofQtAgs8Bt1XK lD0TLc7_gM77oS11ySdMiDnL976uAq9YjINRLPbUFA9XhlEEuqjdQysXvnoXWTksAWsYfxr8H7Kr QU314wJ32tRXpj.NalOHFmWZCsGpT7X9xo3W08E.WZFlrUcTkaYU6OfrQJEluYUUcMWbcZphf1EN GVWIk66dcAqfMTCdsBSzMnk7XFS5P.abkkdHMXyLf2P3s2ETzq0hl.WDrPIbJOE5S_VpGB7mZ1cD 25nhdkAKEbOVjbinwmdqQrNPQP2s5N.Q6siEIZSmfMlj07dAVN2coH1Wg9dTojlITGKhKa8uIhLB U1CcyTJuvf78Ug3C3HzlRjxD3WPfywwV3Ggby0fPKg5B2DsURFAbMg5PwfklWXwgVMOi1_7gGqDp PkZ7gphN1Yy8.ctmlS7VLxxlNU4lIdf5kXzW1kceryjR54mfZB54azkqgHfAoZDgGhppSHNGKe2U DprMOYCBnj5r7p.qC7d66DPGdfCPdUYCCYz8Tqela5quimUX2pTefE3ZyCB0v7xo0H5Z4LeEedyI OmghrhtmUIDTIRcG1cz6dvIdOmzhPLVMI3VY8TQkqsDNehF4ZyQFTbt4mtv15Q9vOAbfSKtHClIv 63CnJ7ptz.cdbP1Gi5SWdjN4I7aj6KM2YJJCFyGv2Nv5N1ANAdZDI_Vjo6mD.vqVEl7X1QbHR086 hH7bAurjozo9tSKGgwdDsCn4y_CCwHBdnqtaK8tHGvczXtYarygsY1ZcRwdHP.ouS_lBHFQlBWVN my02Fft6ojpQjb7ix.o3Ses0ChQeHjqcQHZErY7AEsnwCkf0rT7IVNL72FyRk4dNF3tHbBGhyBwF 4jKWaArXvpe6L2QtxF1n_J8YJaohcI.Ra9WDLs17mMje0H__nTpokOiLQWMZ98pzCA01zp3JISjT OWmh7cM6S8hrJG3KomTAVjqXHN1bWcKgiGruSw3ZR2CWqzUUqUzZTWrsRprFOmVb2ZYsFGWbJk7Y 3G88VKoMrcnMsCJNwKP1D0YIiHoRwwdm_UqHKK1d6ygc8HhfXVe3o6u3aOz70cc5_dUZGyil.r8U .L2oOI3UojyoexTYT6tkDIVxV.n_7hBirJdA3VYeSpFXmg7086G5t4lkTZ3C0rPcfWIFKAqU6g6r MlXKfLTG_6u13f30Ft.qlvZxdhzRnAiv1cgFZLkY9oHFmBrMr5c1KUAHVl0keg1w6Txf92jAy41o DwN_T0ppXuQWXa9WXLtApPRw5D5kW.h2TyvUaWRF2Vo9tyC9k7oM7B96svMCwhuWECpaGIm3fMaG GIlFB0Q5tDNpoIFmSIYJUSTnUda5c.WGuxz8367xWFM2h5KpADYIsWWgmvOSS7gqoq.bgbJyIVyg Y1GeGIKJlXLu_bjCEIA-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Fri, 28 Jan 2022 06:00:17 +0000 Received: by kubenode548.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 82e3c88be288bd77624934644cc343b5; Fri, 28 Jan 2022 06:00:16 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current [ZFS context: used the whole swap space] From: Mark Millard In-Reply-To: <10B4E2F0-6219-4674-875F-A7B01CA6671C@yahoo.com> Date: Thu, 27 Jan 2022 22:00:15 -0800 Cc: Free BSD , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: <54CD0806-3902-4B9C-AA30-5ED003DE4D41@yahoo.com> References: <20220127164512.GA51200@www.zefox.net> <2C7E741F-4703-4E41-93FE-72E1F16B60E2@yahoo.com> <20220127214801.GA51710@www.zefox.net> <5E861D46-128A-4E09-A3CF-736195163B17@yahoo.com> <20220127233048.GA51951@www.zefox.net> <6528ED25-A3C6-4277-B951-1F58ADA2D803@yahoo.com> <10B4E2F0-6219-4674-875F-A7B01CA6671C@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4JlRdM65vPz3rd2 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=f40cQQgj; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.32:from]; MLMMJ_DEST(0.00)[freebsd-arm]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.32:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Status: O Content-Length: 8173 Lines: 223 On 2022-Jan-27, at 21:55, Mark Millard wrote: > On 2022-Jan-27, at 17:43, Mark Millard wrote: >=20 >> On 2022-Jan-27, at 15:30, bob prohaska wrote: >>=20 >>> On Thu, Jan 27, 2022 at 02:21:44PM -0800, Mark Millard wrote: >>>>=20 >>>> Okay. I just started a poudriere bulk devel/llvm13 build >>>> in a ZFS context: >>>>=20 >>>> . . . >>>> [00:00:37] Pkg: +BE_AMDGPU -BE_FREEBSD +BE_NATIVE -BE_STANDARD = +BE_WASM +CLANG +DOCS +EXTRAS -FLANG +LIT +LLD +LLDB +MLIR -OPENMP = -PYCLANG >>>> [00:00:37] New: +BE_AMDGPU -BE_FREEBSD -BE_NATIVE +BE_STANDARD = +BE_WASM +CLANG +DOCS +EXTRAS +FLANG +LIT +LLD +LLDB +MLIR +OPENMP = +PYCLANG >>>> . . . >>>> [00:01:27] [01] [00:00:00] Building devel/llvm13 | llvm13-13.0.0_3 >>>>=20 >>>=20 >>> Is this ARM hardware, or an emulator? >>=20 >> 8 GiByte RPi4B, USB3 NVMe media with a ZFS partition. The content >> is a slightly modified copy of the HoneyComb's PCIe slot Optane >> media. >>=20 >> The UFS-based 8 GiByte RPi4B is also based on copying from the >> same Optane media, both for the system materials and various >> ports/packages/pouriere related materials. (Not, necessarily, >> other things.) >>=20 >>> I've been using plain old make in /usr/ports/devel,=20 >>> might it be informative to try a poudriere build as well? >>=20 >> The Pkg:, New:, and llvm13 lines I listed are poudriere(-devel) >> output. I am doing my builds via poudriere. ALLOW_PARALLEL_JOBS=3D >> and USE_TMPFS=3D"data" in use. >>=20 >> I have a context in which almost all prerequisites had already >> been built. (The change in options lead to 2 very small ports >> to build before devel/llvm13's started in a builder.) >>=20 >> (You might not have a jail that already has the prerequisites.) >>=20 >>> One would expect the added overhead to increase memory use. >>>=20 >>=20 >> Well, from the context I started in, only devel/llvm13 is being >> built once it starts. Once it gets to the build phase (after >> dependencies and such are set up), there is not much overhead >> because the only activity is the one builder and it is only >> building llvm13 --via make in the builder. At the end there >> would be extra activity as poudriere finishes up. During the >> build phase, I only expect minor overhead from poudriere >> monitoring the build logs and such. >>=20 >> I expect that the mere fact that a poudriere jail is in use >> for the builder to execute in does not contribute to >> significantly increasing the system's memory use or changing >> the system's memory use pattern. >>=20 >>=20 >> There are some other differences my context. The instances of >> main [so: 14] are non-debug builds (but with symbols). The >> builds are optimized for the RPi4B (and others) via use of >> -mcpu=3Dcortex-a72 usage. My /usr/main-src/ does have some >> personal changes in it. (Some messaging about the kills is >> part of that.) >>=20 >> The RPi4B's are using: >>=20 >> over_voltage=3D6=20 >> arm_freq=3D2000=20 >> sdram_freq_min=3D3200=20 >> force_turbo=3D1=20 >>=20 >> (There are heat-sinks, fans, and good power supplies.) >>=20 >> The media in use are USB3 1 TB Samsung Portable SSD T7 >> Touch's. I'm unlikely to see "swap_pager: indefinite >> wait buffer:" notices if the cause was based on the >> media performance. (You have spinning rust, if I >> remember right.) >>=20 >> I do not have a monitoring script making a huge log file >> during the build. So less is competing for media access >> or leading to other overheads. (But, as I remember, >> you have gotten the problem without having such a script >> running.) >=20 >=20 > ZFS context: >=20 > Well, the ZFS example used up all the swap space, according > to my patched top. This means that my setting of > vm.pfault_oom_attempts is not appropriate for this context: >=20 > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 > # > # For plunty of swap/paging space (will not > # run out), avoid pageout delays leading to > # Out Of Memory killing of processes: > vm.pfault_oom_attempts=3D-1 > # > # For possibly insufficient swap/paging space > # (might run out), increase the pageout delay > # that leads to Out Of Memory killing of > # processes (showing defaults at the time): > #vm.pfault_oom_attempts=3D 3 > #vm.pfault_oom_wait=3D 10 > # (The multiplication is the total but there > # are other potential tradoffs in the factors > # multiplied, even for nearly the same total.) >=20 > I'll need to retest with something more like the > commented out vm.pfault_oom_attempts and > vm.pfault_oom_wait figures in order to see the > intended handling of the test case. >=20 > What are you using for each of: > vm.pageout_oom_seq ? > vm.pfault_oom_attempts ? > vm.pfault_oom_wait ? >=20 >=20 > For reference, for ZFS: >=20 > last pid: 380; load averages: 1.50, 3.07, 3.93 MaxObs: = 5.71, 4.92, 4.76 = up 0+07:23:14 21:23:43 > 68 threads: 1 running, 65 sleeping, 2 waiting, 19 MaxObsRunning > CPU: 13.3% user, 0.0% nice, 4.9% system, 0.9% interrupt, 80.8% idle > Mem: 4912Mi Active, 167936B Inact, 1193Mi Laundry, 1536Mi Wired, = 40960B Buf, 33860Ki Free, 6179Mi MaxObsActive, 6476Mi MaxObsWired, = 7820Mi MaxObs(Act+Wir+Lndry) > ARC: 777086Ki Total, 132156Ki MFU, 181164Ki MRU, 147456B Anon, 5994Ki = Header, 457626Ki Other > 59308Ki Compressed, 254381Ki Uncompressed, 4.29:1 Ratio > Swap: 8192Mi Total, 8192Mi Used, K Free, 100% Inuse, 19572Ki In, = 3436Ki Out, 8192Mi MaxObsUsed, 14458Mi MaxObs(Act+Lndry+SwapUsed), = 15993Mi MaxObs(Act+Wir+Lndry+SwapUsed) >=20 > Console: > (Looks like I misremembered adjusting the "out of swap space" > wording for the misnomer message.) >=20 > swap_pager: out of swap space > swp_pager_getswapspace(18): failed > swap_pager: out of swap space > swp_pager_getswapspace(1): failed > swp_pager_getswapspace(1): failed > swap_pager: out of swap space > swp_pager_getswapspace(1): failed > swp_pager_getswapspace(7): failed > swp_pager_getswapspace(24): failed > swp_pager_getswapspace(3): failed > swp_pager_getswapspace(18): failed > swp_pager_getswapspace(17): failed > swp_pager_getswapspace(1): failed > swp_pager_getswapspace(12): failed > swp_pager_getswapspace(23): failed > swp_pager_getswapspace(30): failed > swp_pager_getswapspace(3): failed > swp_pager_getswapspace(2): failed >=20 > . . . Then a bunch of time with no messages . . . >=20 > swp_pager_getswapspace(5): failed > swp_pager_getswapspace(28): failed >=20 > . . . Then a bunch of time with no messages . . . >=20 >=20 > Top again: >=20 > last pid: 382; load averages: 0.73, 1.00, 2.40 MaxObs: = 5.71, 4.92, 4.76 = up 0+07:31:26 21:31:55 > 70 threads: 1 running, 65 sleeping, 4 waiting, 19 MaxObsRunning > CPU: 0.1% user, 0.0% nice, 5.6% system, 0.0% interrupt, 94.3% idle > Mem: 3499Mi Active, 4096B Inact, 2612Mi Laundry, 1457Mi Wired, 40960B = Buf, 34676Ki Free, 6179Mi MaxObsActive, 6476Mi MaxObsWired, 7820Mi = MaxObs(Act+Wir+Lndry) > ARC: 777154Ki Total, 135196Ki MFU, 178330Ki MRU, 5995Ki Header, = 457631Ki Other > 59520Ki Compressed, 254231Ki Uncompressed, 4.27:1 Ratio > Swap: 8192Mi Total, 8192Mi Used, K Free, 100% Inuse, 409600B In, 4096B = Out, 8192Mi MaxObsUsed, 14458Mi MaxObs(Act+Lndry+SwapUsed), 15993Mi = MaxObs(Act+Wir+Lndry+SwapUsed) >=20 >=20 > I then used top to kill ninja and the 4 large compiles > that were going on. I'll change: >=20 > vm.pfault_oom_attempts > vm.pfault_oom_wait >=20 > and reboot and start over. >=20 >=20 > I expect that the ongoing UFS test will likely end up > similarly and that similar adjustments and restarts > will be needed because of actually running out of > swap space. >=20 I forgot to report: [00:01:27] [01] [00:00:00] Building devel/llvm13 | llvm13-13.0.0_3 [07:49:17] [01] [07:47:50] Finished devel/llvm13 | llvm13-13.0.0_3: = Failed: build So the swap space filling happened somewhat before that much time had passed. =3D=3D=3D Mark Millard marklmi at yahoo.com