Date: Tue, 8 Nov 2022 15:15:30 +0800 From: Archimedes Gaviola <archimedes.gaviola@gmail.com> To: Mark Millard <marklmi@yahoo.com> Cc: Warner Losh <imp@bsdimp.com>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build Message-ID: <CAJFbk7GpwhNV55kh%2Bny3kLpuSx8y9%2BzqGRYVkfm6hmK4vjS93Q@mail.gmail.com> In-Reply-To: <788B97E7-AA06-4A87-BB4A-CF2602DC1AD3@yahoo.com> References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> <CAJFbk7FfYPSe3eF00HgDdebW70HKp5zKR0JaChTVniUDPG2qxQ@mail.gmail.com> <CA350C16-3604-4D88-9C14-040A45F6F125@yahoo.com> <CAJFbk7Hxvr9gs7GnniWtJ-QEH4yjYbB9S-vKVLjipa8v5VHa%2Bw@mail.gmail.com> <CAJFbk7GQhvgQS-23jFfFf=ershgGNKByHR8ug08Df8merwkh6Q@mail.gmail.com> <A11AA52B-BDB0-43C7-BF85-3C252B276F17@yahoo.com> <CAJFbk7HnFTdzANtdERqvgX30hcHmufAZmrNvbfEWORkUJJ7_3w@mail.gmail.com> <CAJFbk7HJ1WA5Qc0LNEZpKgv78yiM0w7ex=gjgpjjTf3chhHhiQ@mail.gmail.com> <CANCZdfruoDPQE%2ByqebDomg8w7YKCkbFPvrPweJQUqoaOMN3rwA@mail.gmail.com> <788B97E7-AA06-4A87-BB4A-CF2602DC1AD3@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000ace67e05ecf04f5b
Content-Type: text/plain; charset="UTF-8"
On Tue, Nov 8, 2022 at 11:53 AM Mark Millard <marklmi@yahoo.com> wrote:
> On Nov 7, 2022, at 19:28, Warner Losh <imp@bsdimp.com> wrote:
> >
> > . . .
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: 40960
> > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to
> allocate a page
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: 28672
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 8192
> >
> > This means that paging to the swap partition and/or swap file took too
> long (> 30 seconds... that's all that indefinite means).
>
> FYI: I think the "indefinite wait buffer" bound that leads
> to those messages is 20 sec (the hz*20 below):
>
> /*
> * Wait for the pages we want to complete. VPO_SWAPINPROG is
> always
> * cleared on completion. If an I/O error occurs, SWAPBLK_NONE
> * is set in the metadata for each page in the request.
> */
> VM_OBJECT_WLOCK(object);
> /* This could be implemented more efficiently with aflags */
> while ((ma[0]->oflags & VPO_SWAPINPROG) != 0) {
> ma[0]->oflags |= VPO_SWAPSLEEP;
> VM_CNT_INC(v_intrans);
> if (VM_OBJECT_SLEEP(object, &object->handle, PSWP,
> "swread", hz * 20)) {
> printf(
> "swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: %ld\n",
> bp->b_bufobj, (intmax_t)bp->b_blkno,
> bp->b_bcount);
> }
> }
> VM_OBJECT_WUNLOCK(object);
>
> But the "was killed: a thread waited too long to allocate a page" is
> tied to a total of 30 sec (3*10sec) from:
>
> vm.pfault_oom_attempts= 3
> vm.pfault_oom_wait= 10
>
> (Presuming that you had defaults at the time.)
>
Confirming the default values.
root@generic:~ # sysctl -a | grep pfault_oom
vm.pfault_oom_wait: 10
vm.pfault_oom_attempts: 3
>
> > It also means that it can't write to backing store dirty pages to give
> to another process...
> >
> > Typical reason is that the disk / flash is not responsive to writes for
> some reason. You'll need to find why... I'd look at trims.
> >
> > Or.... if you can't change the disk... you need to put less memory
> pressure on it..
>
>
>
>
>
> ===
> Mark Millard
> marklmi at yahoo.com
>
>
--000000000000ace67e05ecf04f5b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Tue, Nov 8, 2022 at 11:53 AM Mark =
Millard <<a href=3D"mailto:marklmi@yahoo.com">marklmi@yahoo.com</a>> =
wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0=
px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Nov 7,=
2022, at 19:28, Warner Losh <<a href=3D"mailto:imp@bsdimp.com" target=
=3D"_blank">imp@bsdimp.com</a>> wrote:<br>
> <br>
> . . .<br>
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: 40=
96<br>
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096=
<br>
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: 40=
960<br>
> pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to=
allocate a page<br>
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: 28=
672<br>
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192=
<br>
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 40=
96<br>
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 81=
92<br>
> <br>
> This means that paging to the swap partition and/or swap file took too=
long (> 30 seconds... that's all that indefinite means).<br>
<br>
FYI: I think the "indefinite wait buffer" bound that leads<br>
to those messages is 20 sec (the hz*20 below):<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Wait for the pages we want to complete.=
=C2=A0 VPO_SWAPINPROG is always<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* cleared on completion.=C2=A0 If an I/O =
error occurs, SWAPBLK_NONE<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* is set in the metadata for each page in=
the request.<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 VM_OBJECT_WLOCK(object);<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /* This could be implemented more efficiently w=
ith aflags */<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 while ((ma[0]->oflags & VPO_SWAPINPROG) =
!=3D 0) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ma[0]->oflags |=
=3D VPO_SWAPSLEEP;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VM_CNT_INC(v_intran=
s);<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (VM_OBJECT_SLEEP=
(object, &object->handle, PSWP,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "=
;swread", hz * 20)) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 printf(<br>
"swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: %ld=
\n",<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 bp->b_bufobj, (intmax_t)bp->b_blkno, bp->=
b_bcount);<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 VM_OBJECT_WUNLOCK(object);<br>
<br>
But the "was killed: a thread waited too long to allocate a page"=
is<br>
tied to a total of 30 sec (3*10sec) from:<br>
<br>
vm.pfault_oom_attempts=3D 3<br>
vm.pfault_oom_wait=3D 10<br>
<br>
(Presuming that you had defaults at the time.)<br></blockquote><div><br></d=
iv><div>Confirming the default values.</div><div><br></div><div>root@generi=
c:~ # sysctl -a | grep pfault_oom<br>vm.pfault_oom_wait: 10<br>vm.pfault_oo=
m_attempts: 3</div><div></div><div></div><div>=C2=A0</div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">
<br>
> It also means that it can't write to backing store dirty pages to =
give to another process...<br>
> <br>
> Typical reason is that the disk / flash is not responsive to writes fo=
r some reason. You'll need to find why... I'd look at trims.<br>
> <br>
> Or.... if you can't change the disk... you need to put less memory=
pressure on it..<br>
<br>
<br>
<br>
<br>
<br>
=3D=3D=3D<br>
Mark Millard<br>
marklmi at <a href=3D"http://yahoo.com" rel=3D"noreferrer" target=3D"_blank=
">yahoo.com</a><br>
<br>
</blockquote></div></div>
--000000000000ace67e05ecf04f5b--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJFbk7GpwhNV55kh%2Bny3kLpuSx8y9%2BzqGRYVkfm6hmK4vjS93Q>