Date: Mon, 21 Nov 2022 17:29:25 +0800 From: Archimedes Gaviola <archimedes.gaviola@gmail.com> To: Mark Millard <marklmi@yahoo.com> Cc: Ronald Klop <ronald-lists@klop.ws>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build Message-ID: <CAJFbk7HXC3-jK6HxDUN77ZFsOeTHyW4Q_AVn_xer_%2BygA-B1mA@mail.gmail.com> In-Reply-To: <320C43A9-74FF-4848-9035-98BA1B71C44E@yahoo.com> References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> <CAJFbk7FfYPSe3eF00HgDdebW70HKp5zKR0JaChTVniUDPG2qxQ@mail.gmail.com> <CA350C16-3604-4D88-9C14-040A45F6F125@yahoo.com> <CAJFbk7Hxvr9gs7GnniWtJ-QEH4yjYbB9S-vKVLjipa8v5VHa%2Bw@mail.gmail.com> <CAJFbk7GQhvgQS-23jFfFf=ershgGNKByHR8ug08Df8merwkh6Q@mail.gmail.com> <A11AA52B-BDB0-43C7-BF85-3C252B276F17@yahoo.com> <CAJFbk7HnFTdzANtdERqvgX30hcHmufAZmrNvbfEWORkUJJ7_3w@mail.gmail.com> <CAJFbk7HJ1WA5Qc0LNEZpKgv78yiM0w7ex=gjgpjjTf3chhHhiQ@mail.gmail.com> <CANCZdfruoDPQE%2ByqebDomg8w7YKCkbFPvrPweJQUqoaOMN3rwA@mail.gmail.com> <1722758786.127406.1667909706281@localhost> <1E33E6A3-ABB8-4804-B2A2-0E95E853C860@yahoo.com> <CAJFbk7GXw3pQxgcWfwYzVkF5ff=S8=oorRVJW5wY_6sEz81StQ@mail.gmail.com> <CAJFbk7G05iJgCADDJPPr-3qu8J9GcvS35aUOkWt5DC_UNcDYFA@mail.gmail.com> <320C43A9-74FF-4848-9035-98BA1B71C44E@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000026194e05edf7b5f2 Content-Type: text/plain; charset="UTF-8" On Mon, Nov 21, 2022 at 12:24 PM Mark Millard <marklmi@yahoo.com> wrote: > On Nov 20, 2022, at 19:48, Archimedes Gaviola < > archimedes.gaviola@gmail.com> wrote: > > > On Wed, Nov 9, 2022 at 10:15 AM Archimedes Gaviola < > archimedes.gaviola@gmail.com> wrote: > > On Wed, Nov 9, 2022 at 1:37 AM Mark Millard <marklmi@yahoo.com> wrote: > > On Nov 8, 2022, at 04:15, Ronald Klop <ronald-lists@klop.ws> wrote: > > > > > Van: Warner Losh <imp@bsdimp.com> > > > Datum: dinsdag, 8 november 2022 04:28 > > > Aan: Archimedes Gaviola <archimedes.gaviola@gmail.com > > > . . . > > > ... > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: > 4096 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: > 40960 > > > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to > allocate a page > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: > 28672 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: > 4096 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: > 8192 > > > This means that paging to the swap partition and/or swap file took > too long (> 30 seconds... that's all that indefinite means). It also means > that it can't write to backing store dirty pages to give to another > process... > > > Typical reason is that the disk / flash is not responsive to writes > for some reason. You'll need to find why... I'd look at trims. > > > Or.... if you can't change the disk... you need to put less memory > pressure on it.. > > > Warner > > > > > . . . > > > > Hi Mark, > > > > As a recap on the kernel tunables, the changes are the following, > > > > root@generic:~ # sysctl -a | grep oom > > vm.pageout_oom_seq: 120 > > vm.pfault_oom_wait: 10 > Hi, > > FYI . . . > > As long as: > > vm.pfault_oom_attempts == -1 > > vm.pfault_oom_wait is ignored. It also likely does > nothing for: > > vm.pfault_oom_attempts == 0 > > vm.pfault_oom_wait gets involved for: > > 0 < vm.pfault_oom_attempts . > Okay, noted on this. > > > vm.pfault_oom_attempts: -1 > > > > With -j1 and -j2 options, both were able to complete the kernel and > buildworld compilation in 103 and 84 hours respectively. Though I still > could see messages on "swap_pager: indefinite wait buffer: bufobj" but > definitely it's ignorable as it survived the compilation process. With the > -j3 option, it failed along the course of compilation, it encountered the > previous error on "failed to reclaim memory" but this time this error is > not that relevant as -j1 and -j2 already works. Preferably with -j2 as the > appropriate choice for my RPi 3B build setup. > > Glad you got it working in your context. > > Thanks for the report. I too am glad about the outcome of the testing. In the context of -j1 and -j2 build options, the oom kernel tunable settings were very effective. Thanks a lot for your help! > My media does not lead to the > conditions and, so, does not lead to learning the > behavior when "swap_pager: indefinite wait buffer: > bufobj" is significantly involved (for the time scale > of waits that you got into). > Ah okay, as you are using a more powerful media so there's no manifestation of such messages. Nice and good to know. > > The implication of the result is that you would > need a larger vm.pageout_oom_seq value in order > for -j3 to finish normally. Oh I see I will take note of this one. Thanks for the further hint! I can explore this soon. > Based on my media, > I've never had to use larger values, but, I knew > it was a technical possibility to need such. I do > not know how to pre-calculate what value would > work. > > (I'm not suggesting any more -j3 experiments.) > It's alright, if I have spare time I will explore the -j3 settings further. For now -j2 is sufficient. To Warner and Ronald, thanks as well to your inputs! Best regards, Archimedes > > === > Mark Millard > marklmi at yahoo.com > > --00000000000026194e05edf7b5f2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Mon, Nov 21, 2022 at 12:24 PM Mark= Millard <<a href=3D"mailto:marklmi@yahoo.com" target=3D"_blank">marklmi= @yahoo.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex">On Nov 20, 2022, at 19:48, Archimedes Gaviola <<a href=3D"mai= lto:archimedes.gaviola@gmail.com" target=3D"_blank">archimedes.gaviola@gmai= l.com</a>> wrote:<br> <br> > On Wed, Nov 9, 2022 at 10:15 AM Archimedes Gaviola <<a href=3D"mail= to:archimedes.gaviola@gmail.com" target=3D"_blank">archimedes.gaviola@gmail= .com</a>> wrote:<br> > On Wed, Nov 9, 2022 at 1:37 AM Mark Millard <<a href=3D"mailto:mark= lmi@yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>> wrote:<br> > On Nov 8, 2022, at 04:15, Ronald Klop <<a href=3D"mailto:ronald-lis= ts@klop.ws" target=3D"_blank">ronald-lists@klop.ws</a>> wrote:<br> > <br> > > Van: Warner Losh <<a href=3D"mailto:imp@bsdimp.com" target=3D"= _blank">imp@bsdimp.com</a>><br> > > Datum: dinsdag, 8 november 2022 04:28<br> > > Aan: Archimedes Gaviola <<a href=3D"mailto:archimedes.gaviola@= gmail.com" target=3D"_blank">archimedes.gaviola@gmail.com</a><br> > > . . .<br> > > ...<br> > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, siz= e: 4096<br> > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size:= 4096<br> > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, siz= e: 40960<br> > > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too lo= ng to allocate a page<br> > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, siz= e: 28672<br> > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size:= 8192<br> > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, siz= e: 4096<br> > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, siz= e: 8192<br> > >=C2=A0 =C2=A0This means that paging to the swap partition and/or s= wap file took too long (> 30 seconds... that's all that indefinite m= eans). It also means that it can't write to backing store dirty pages t= o give to another process...<br> > >=C2=A0 =C2=A0Typical reason is that the disk / flash is not respon= sive to writes for some reason. You'll need to find why... I'd look= at trims.<br> > >=C2=A0 =C2=A0Or.... if you can't change the disk... you need t= o put less memory pressure on it..<br> > >=C2=A0 =C2=A0Warner<br> > >=C2=A0 =C2=A0<br> > . . .<br> > <br> > Hi Mark,<br> > <br> > As a recap on the kernel tunables, the changes are the following,<br> > <br> > root@generic:~ # sysctl -a | grep oom<br> > vm.pageout_oom_seq: 120<br> > vm.pfault_oom_wait: 10<br></blockquote><div><br></div><div>Hi,</div><d= iv>=C2=A0<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> FYI . . .<br> <br> As long as:<br> <br> vm.pfault_oom_attempts =3D=3D -1 <br> <br> vm.pfault_oom_wait is ignored. It also likely does<br> nothing for:<br> <br> vm.pfault_oom_attempts =3D=3D 0<br> <br> vm.pfault_oom_wait gets involved for:<br> <br> 0 < vm.pfault_oom_attempts .<br></blockquote><div><br></div><div>Okay, n= oted on this.</div><div>=C2=A0<br></div><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad= ding-left:1ex"> <br> > vm.pfault_oom_attempts: -1<br> > <br> > With -j1 and -j2 options, both were able to complete the kernel and bu= ildworld compilation in 103 and 84 hours respectively. Though I still could= see messages on "swap_pager: indefinite wait buffer: bufobj" but= definitely it's ignorable as it survived the compilation process. With= the -j3 option, it failed along the course of compilation, it encountered = the previous error on "failed to reclaim memory" but this time th= is error is not that relevant as -j1 and -j2 already works. Preferably with= -j2 as the appropriate choice for my RPi 3B build setup.<br> <br> Glad you got it working in your context.<br> <br> Thanks for the report. </blockquote><div><br></div><div>I too am glad about= the outcome of the testing. In the context of -j1 and -j2 build options, t= he oom kernel tunable settings were very effective. Thanks a lot for your h= elp!<br></div><div>=C2=A0<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex">My media does not lead to the<br> conditions and, so, does not lead to learning the<br> behavior when "swap_pager: indefinite wait buffer:<br> bufobj" is significantly involved (for the time scale<br> of waits that you got into).<br></blockquote><div><br></div><div>Ah okay, a= s you are using a more powerful media so there's no manifestation of su= ch messages. Nice and good to know.</div><div>=C2=A0<br></div><blockquote c= lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli= d rgb(204,204,204);padding-left:1ex"> <br> The implication of the result is that you would<br> need a larger vm.pageout_oom_seq value in order<br> for -j3 to finish normally.</blockquote><div><br></div><div>Oh I see I will= take note of this one. Thanks for the further hint! I can explore this soo= n.<br></div><div>=C2=A0 <br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex">Based on my media,<br> I've never had to use larger values, but, I knew<br> it was a technical possibility to need such. I do<br> not know how to pre-calculate what value would<br> work.<br> <br> (I'm not suggesting any more -j3 experiments.)<br></blockquote><div><br= ></div><div>It's alright, if I have spare time I will explore the -j3 s= ettings further. For now -j2 is sufficient.<br></div><div><br></div><div>To= Warner and Ronald, thanks as well to your inputs!</div><div><br></div><div= >Best regards,<br></div><div>Archimedes<br></div><div>=C2=A0<br></div><bloc= kquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:= 1px solid rgb(204,204,204);padding-left:1ex"> <br> =3D=3D=3D<br> Mark Millard<br> marklmi at <a href=3D"http://yahoo.com" rel=3D"noreferrer" target=3D"_blank= ">yahoo.com</a><br> <br> </blockquote></div></div> --00000000000026194e05edf7b5f2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJFbk7HXC3-jK6HxDUN77ZFsOeTHyW4Q_AVn_xer_%2BygA-B1mA>