Date: Mon, 21 Nov 2022 11:48:58 +0800 From: Archimedes Gaviola <archimedes.gaviola@gmail.com> To: Mark Millard <marklmi@yahoo.com> Cc: Ronald Klop <ronald-lists@klop.ws>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build Message-ID: <CAJFbk7G05iJgCADDJPPr-3qu8J9GcvS35aUOkWt5DC_UNcDYFA@mail.gmail.com> In-Reply-To: <CAJFbk7GXw3pQxgcWfwYzVkF5ff=S8=oorRVJW5wY_6sEz81StQ@mail.gmail.com> References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> <CAJFbk7FfYPSe3eF00HgDdebW70HKp5zKR0JaChTVniUDPG2qxQ@mail.gmail.com> <CA350C16-3604-4D88-9C14-040A45F6F125@yahoo.com> <CAJFbk7Hxvr9gs7GnniWtJ-QEH4yjYbB9S-vKVLjipa8v5VHa%2Bw@mail.gmail.com> <CAJFbk7GQhvgQS-23jFfFf=ershgGNKByHR8ug08Df8merwkh6Q@mail.gmail.com> <A11AA52B-BDB0-43C7-BF85-3C252B276F17@yahoo.com> <CAJFbk7HnFTdzANtdERqvgX30hcHmufAZmrNvbfEWORkUJJ7_3w@mail.gmail.com> <CAJFbk7HJ1WA5Qc0LNEZpKgv78yiM0w7ex=gjgpjjTf3chhHhiQ@mail.gmail.com> <CANCZdfruoDPQE%2ByqebDomg8w7YKCkbFPvrPweJQUqoaOMN3rwA@mail.gmail.com> <1722758786.127406.1667909706281@localhost> <1E33E6A3-ABB8-4804-B2A2-0E95E853C860@yahoo.com> <CAJFbk7GXw3pQxgcWfwYzVkF5ff=S8=oorRVJW5wY_6sEz81StQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000007e5df705edf2f368 Content-Type: text/plain; charset="UTF-8" On Wed, Nov 9, 2022 at 10:15 AM Archimedes Gaviola < archimedes.gaviola@gmail.com> wrote: > > > On Wed, Nov 9, 2022 at 1:37 AM Mark Millard <marklmi@yahoo.com> wrote: > >> On Nov 8, 2022, at 04:15, Ronald Klop <ronald-lists@klop.ws> wrote: >> >> > Van: Warner Losh <imp@bsdimp.com> >> > Datum: dinsdag, 8 november 2022 04:28 >> > Aan: Archimedes Gaviola <archimedes.gaviola@gmail.com >> > . . . >> > ... >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: 4096 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: >> 40960 >> > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to >> allocate a page >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: >> 28672 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 4096 >> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 8192 >> > This means that paging to the swap partition and/or swap file took >> too long (> 30 seconds... that's all that indefinite means). It also means >> that it can't write to backing store dirty pages to give to another >> process... >> > Typical reason is that the disk / flash is not responsive to writes >> for some reason. You'll need to find why... I'd look at trims. >> > Or.... if you can't change the disk... you need to put less memory >> pressure on it.. >> > Warner >> > >> > >> > >> > NB: a way to put less memory pressure on it is not using -j3, but -j2 >> or -j1 in your make command. >> > >> > > Hi Mark, > > >> Extending Ronold's comment: If things are really taking this >> long for the paging I/O, you might actually find, say, -j2 >> takes less elapsed time than -j3 because of the latencies >> involved in -j3 causing more overall delay. >> > > Yes I'll take these options on lowering down N in the -jN parameter as my > next steps. So far so good with -j3, ongoing build is still observed for 17 > hours now. > > >> >> vm.pfault_oom_attempts=-1 would still be appropriate for avoiding >> I/O kills at any -jN: the smaller -jN just makes the issue less >> likely, not impossible. (Again, presuming sufficient swap/paging >> space if deadlock is to be well avoided.) >> > > The ongoing build is at the moment on > /usr/src/contrib/llvm-project/llvm/lib/*. I'm observing from time-to-time > if the error will occur again. > > >> (I use NVMe or SSD USB media that do not get such long delays but >> fit the power limitations of the context. I have about as little >> on microsd card media as I can get away with in my context. I also >> avoid spinning rust. Thus I've only gotten "indefinite wait buffer" >> or the like back before such was true, long ago.) >> > > Okay thanks for sharing this one. Keeping this in my mind just in case I > needed these types of media soon. > > Thanks and best regards, > Archimedes > Hi Mark, As a recap on the kernel tunables, the changes are the following, root@generic:~ # sysctl -a | grep oom vm.pageout_oom_seq: 120 vm.pfault_oom_wait: 10 vm.pfault_oom_attempts: -1 With -j1 and -j2 options, both were able to complete the kernel and buildworld compilation in 103 and 84 hours respectively. Though I still could see messages on "swap_pager: indefinite wait buffer: bufobj" but definitely it's ignorable as it survived the compilation process. With the -j3 option, it failed along the course of compilation, it encountered the previous error on "failed to reclaim memory" but this time this error is not that relevant as -j1 and -j2 already works. Preferably with -j2 as the appropriate choice for my RPi 3B build setup. Thanks and best regards, Archimedes --0000000000007e5df705edf2f368 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Wed, Nov 9, 2022 at 10:15 AM Archi= medes Gaviola <<a href=3D"mailto:archimedes.gaviola@gmail.com">archimede= s.gaviola@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quot= e" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)= ;padding-left:1ex"><div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div cla= ss=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Nov 9, 202= 2 at 1:37 AM Mark Millard <<a href=3D"mailto:marklmi@yahoo.com" target= =3D"_blank">marklmi@yahoo.com</a>> wrote:<br></div><blockquote class=3D"= gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex">On Nov 8, 2022, at 04:15, Ronald Klop <<a h= ref=3D"mailto:ronald-lists@klop.ws" target=3D"_blank">ronald-lists@klop.ws<= /a>> wrote:<br> <br> > Van: Warner Losh <<a href=3D"mailto:imp@bsdimp.com" target=3D"_blan= k">imp@bsdimp.com</a>><br> > Datum: dinsdag, 8 november 2022 04:28<br> > Aan: Archimedes Gaviola <<a href=3D"mailto:archimedes.gaviola@gmail= .com" target=3D"_blank">archimedes.gaviola@gmail.com</a><br> > . . .<br> > ...<br> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: 40= 96<br> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096= <br> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: 40= 960<br> > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to= allocate a page<br> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: 28= 672<br> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192= <br> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 40= 96<br> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 81= 92<br> >=C2=A0 =C2=A0This means that paging to the swap partition and/or swap f= ile took too long (> 30 seconds... that's all that indefinite means)= . It also means that it can't write to backing store dirty pages to giv= e to another process...<br> >=C2=A0 =C2=A0Typical reason is that the disk / flash is not responsive = to writes for some reason. You'll need to find why... I'd look at t= rims.<br> >=C2=A0 =C2=A0Or.... if you can't change the disk... you need to put= less memory pressure on it..<br> >=C2=A0 =C2=A0Warner<br> >=C2=A0 =C2=A0<br> > <br> > <br> > NB: a way to put less memory pressure on it is not using -j3, but -j2 = or -j1 in your make command.<br> > <br></blockquote><div><br></div><div>Hi Mark,</div><div>=C2=A0<br></di= v><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;borde= r-left:1px solid rgb(204,204,204);padding-left:1ex"> Extending Ronold's comment: If things are really taking this<br> long for the paging I/O, you might actually find, say, -j2<br> takes less elapsed time than -j3 because of the latencies<br> involved in -j3 causing more overall delay.<br></blockquote><div><br></div>= <div>Yes I'll take these options on lowering down N in the -jN paramete= r as my next steps. So far so good with -j3, ongoing build is still observe= d for 17 hours now.</div><div>=C2=A0 <br></div><blockquote class=3D"gmail_q= uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex"> <br> vm.pfault_oom_attempts=3D-1 would still be appropriate for avoiding<br> I/O kills at any -jN: the smaller -jN just makes the issue less<br> likely, not impossible. (Again, presuming sufficient swap/paging<br> space if deadlock is to be well avoided.)<br></blockquote><div><br></div><d= iv>The ongoing build is at the moment on /usr/src/contrib/llvm-project/llvm= /lib/*. I'm observing from time-to-time if the error will occur again.<= br></div><div>=C2=A0<br></div><blockquote class=3D"gmail_quote" style=3D"ma= rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:= 1ex"> (I use NVMe or SSD USB media that do not get such long delays but<br> fit the power limitations of the context. I have about as little<br> on microsd card media as I can get away with in my context. I also<br> avoid spinning rust. Thus I've only gotten "indefinite wait buffer= "<br> or the like back before such was true, long ago.)<br></blockquote><div><br>= </div><div>Okay thanks for sharing this one. Keeping this in my mind just i= n case I needed these types of media soon.</div><div><br></div><div>Thanks = and best regards,</div><div>Archimedes<br></div></div></div></blockquote><d= iv><br></div><div>Hi Mark,</div><div><br></div><div>As a recap on the kerne= l tunables, the changes are the following,<br></div><div><br></div><div>roo= t@generic:~ # sysctl -a | grep oom<br>vm.pageout_oom_seq: 120<br>vm.pfault_= oom_wait: 10<br>vm.pfault_oom_attempts: -1</div><div><br></div><div>With -j= 1 and -j2 options, both were able to complete the kernel and buildworld com= pilation in 103 and 84 hours respectively. Though I still could see message= s on "swap_pager: indefinite wait buffer: bufobj" but definitely = it's ignorable as it survived the compilation process. With the -j3 opt= ion, it failed along the course of compilation, it encountered the previous= error on "failed to reclaim memory" but this time this error is = not that relevant as -j1 and -j2 already works. Preferably with -j2 as the = appropriate choice for my RPi 3B build setup.</div><div><br></div><div>Than= ks and best regards,</div><div>Archimedes<br></div></div></div> --0000000000007e5df705edf2f368--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJFbk7G05iJgCADDJPPr-3qu8J9GcvS35aUOkWt5DC_UNcDYFA>