From nobody Mon Nov 21 09:29:25 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NG2Dr3rnZz4hTnT for ; Mon, 21 Nov 2022 09:30:36 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Received: from mail-yb1-xb2d.google.com (mail-yb1-xb2d.google.com [IPv6:2607:f8b0:4864:20::b2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NG2Dr1tTyz3kPd for ; Mon, 21 Nov 2022 09:30:36 +0000 (UTC) (envelope-from archimedes.gaviola@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb2d.google.com with SMTP id f201so12878386yba.12 for ; Mon, 21 Nov 2022 01:30:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=1Gie+xm1gIfpJZyDDwaucUkjBIkVZ37EZ6sCvP+t//A=; b=N8mnVKHmlOZvhPXJpehuveLBjB4cXd4nMTneLsIcQeL08EJuNEgI4gjFbRUbv9maDl l9n4tuDHb1USCLq2c/FtADTJgsa6OIPs6p/2LdrB5xM+gKlc3+IYbbzOgHLS7nKNz3pg 8X8wyl7VwheHRzChZQz1sKXLa11GZtCbCH5Q+bbhlVncRR2mYoLAOhI06+YF6lZIul3q 7VnuEnLitsRUr1SA1uAjFPAUk5J+lnv5j+dU48kTtg/4oYLYXbdnu7v9R8tvKoWdo2Wa /Nk+V261T7BEOM0KtHCmTnxUeAPfC6Tm2DAdck33gnXG3EB07A4T+hFcZUqhNc082U3M fT2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1Gie+xm1gIfpJZyDDwaucUkjBIkVZ37EZ6sCvP+t//A=; b=FabHrININc5clzFn6Cp5xqdHNNzmtHBFEIYNFGJWOnoD4wqHO3lZBqtygTTtYTcz2A B1Sy1R3hVNqodAJ2q3YS9D7aMFyuvQiojwK+OdJKIJEF1zxRvBxAc4/TfojOXWJdeGdu U4vQbcoNMuv2s1jlJp5oCibX6UMULpzWH8DNTS2WvsyVurDIJulZgBkGAUkdKvTd7c+J LsgRcIkwJDp68kbpnKRGYz5lqJDoDL6nm52/L8iWZlZcIWKbEtRiV/vvPp9/iat7TM+z rJYmCakElA60swn88SIaM9lIg7YwDdrDkrm9SZ6k76VB1sJFY/ETQgdGb96UcmsT2Tyd vY6A== X-Gm-Message-State: ANoB5pkiHd/MSaKDKEx3w1aCuxBaBnhvbl7SVULw/ILuagG7oC4FD7ks DObizdTKHqaD6LIU9XITcmvgGMivGvq6vI2/qKY= X-Google-Smtp-Source: AA0mqf6bEX2ZkEl3o8oQDzqb6BoO7lr7hwaSo1koVSfvAWr9UAbF/VGc/LMeocok8j8DWdzRhwLymdGHiordgZukD3Q= X-Received: by 2002:a25:cfcb:0:b0:6e6:d92d:e1 with SMTP id f194-20020a25cfcb000000b006e6d92d00e1mr15759153ybg.308.1669023035429; Mon, 21 Nov 2022 01:30:35 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> <1722758786.127406.1667909706281@localhost> <1E33E6A3-ABB8-4804-B2A2-0E95E853C860@yahoo.com> <320C43A9-74FF-4848-9035-98BA1B71C44E@yahoo.com> In-Reply-To: <320C43A9-74FF-4848-9035-98BA1B71C44E@yahoo.com> From: Archimedes Gaviola Date: Mon, 21 Nov 2022 17:29:25 +0800 Message-ID: Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build To: Mark Millard Cc: Ronald Klop , freebsd-current Content-Type: multipart/alternative; boundary="00000000000026194e05edf7b5f2" X-Rspamd-Queue-Id: 4NG2Dr1tTyz3kPd X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000026194e05edf7b5f2 Content-Type: text/plain; charset="UTF-8" On Mon, Nov 21, 2022 at 12:24 PM Mark Millard wrote: > On Nov 20, 2022, at 19:48, Archimedes Gaviola < > archimedes.gaviola@gmail.com> wrote: > > > On Wed, Nov 9, 2022 at 10:15 AM Archimedes Gaviola < > archimedes.gaviola@gmail.com> wrote: > > On Wed, Nov 9, 2022 at 1:37 AM Mark Millard wrote: > > On Nov 8, 2022, at 04:15, Ronald Klop wrote: > > > > > Van: Warner Losh > > > Datum: dinsdag, 8 november 2022 04:28 > > > Aan: Archimedes Gaviola > > . . . > > > ... > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: > 4096 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: > 40960 > > > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to > allocate a page > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: > 28672 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: > 4096 > > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: > 8192 > > > This means that paging to the swap partition and/or swap file took > too long (> 30 seconds... that's all that indefinite means). It also means > that it can't write to backing store dirty pages to give to another > process... > > > Typical reason is that the disk / flash is not responsive to writes > for some reason. You'll need to find why... I'd look at trims. > > > Or.... if you can't change the disk... you need to put less memory > pressure on it.. > > > Warner > > > > > . . . > > > > Hi Mark, > > > > As a recap on the kernel tunables, the changes are the following, > > > > root@generic:~ # sysctl -a | grep oom > > vm.pageout_oom_seq: 120 > > vm.pfault_oom_wait: 10 > Hi, > > FYI . . . > > As long as: > > vm.pfault_oom_attempts == -1 > > vm.pfault_oom_wait is ignored. It also likely does > nothing for: > > vm.pfault_oom_attempts == 0 > > vm.pfault_oom_wait gets involved for: > > 0 < vm.pfault_oom_attempts . > Okay, noted on this. > > > vm.pfault_oom_attempts: -1 > > > > With -j1 and -j2 options, both were able to complete the kernel and > buildworld compilation in 103 and 84 hours respectively. Though I still > could see messages on "swap_pager: indefinite wait buffer: bufobj" but > definitely it's ignorable as it survived the compilation process. With the > -j3 option, it failed along the course of compilation, it encountered the > previous error on "failed to reclaim memory" but this time this error is > not that relevant as -j1 and -j2 already works. Preferably with -j2 as the > appropriate choice for my RPi 3B build setup. > > Glad you got it working in your context. > > Thanks for the report. I too am glad about the outcome of the testing. In the context of -j1 and -j2 build options, the oom kernel tunable settings were very effective. Thanks a lot for your help! > My media does not lead to the > conditions and, so, does not lead to learning the > behavior when "swap_pager: indefinite wait buffer: > bufobj" is significantly involved (for the time scale > of waits that you got into). > Ah okay, as you are using a more powerful media so there's no manifestation of such messages. Nice and good to know. > > The implication of the result is that you would > need a larger vm.pageout_oom_seq value in order > for -j3 to finish normally. Oh I see I will take note of this one. Thanks for the further hint! I can explore this soon. > Based on my media, > I've never had to use larger values, but, I knew > it was a technical possibility to need such. I do > not know how to pre-calculate what value would > work. > > (I'm not suggesting any more -j3 experiments.) > It's alright, if I have spare time I will explore the -j3 settings further. For now -j2 is sufficient. To Warner and Ronald, thanks as well to your inputs! Best regards, Archimedes > > === > Mark Millard > marklmi at yahoo.com > > --00000000000026194e05edf7b5f2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Nov 21, 2022 at 12:24 PM Mark= Millard <marklmi= @yahoo.com> wrote:
On Nov 20, 2022, at 19:48, Archimedes Gaviola <archimedes.gaviola@gmai= l.com> wrote:

> On Wed, Nov 9, 2022 at 10:15 AM Archimedes Gaviola <archimedes.gaviola@gmail= .com> wrote:
> On Wed, Nov 9, 2022 at 1:37 AM Mark Millard <marklmi@yahoo.com> wrote:
> On Nov 8, 2022, at 04:15, Ronald Klop <ronald-lists@klop.ws> wrote:
>
> > Van: Warner Losh <imp@bsdimp.com>
> > Datum: dinsdag, 8 november 2022 04:28
> > Aan: Archimedes Gaviola <archimedes.gaviola@gmail.com
> > . . .
> > ...
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, siz= e: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size:= 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, siz= e: 40960
> > pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too lo= ng to allocate a page
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, siz= e: 28672
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size:= 8192
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, siz= e: 4096
> > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, siz= e: 8192
> >=C2=A0 =C2=A0This means that paging to the swap partition and/or s= wap file took too long (> 30 seconds... that's all that indefinite m= eans). It also means that it can't write to backing store dirty pages t= o give to another process...
> >=C2=A0 =C2=A0Typical reason is that the disk / flash is not respon= sive to writes for some reason. You'll need to find why... I'd look= at trims.
> >=C2=A0 =C2=A0Or.... if you can't change the disk... you need t= o put less memory pressure on it..
> >=C2=A0 =C2=A0Warner
> >=C2=A0 =C2=A0
> . . .
>
> Hi Mark,
>
> As a recap on the kernel tunables, the changes are the following,
>
> root@generic:~ # sysctl -a | grep oom
> vm.pageout_oom_seq: 120
> vm.pfault_oom_wait: 10

Hi,
=C2=A0

FYI . . .

As long as:

vm.pfault_oom_attempts =3D=3D -1

vm.pfault_oom_wait is ignored. It also likely does
nothing for:

vm.pfault_oom_attempts =3D=3D 0

vm.pfault_oom_wait gets involved for:

0 < vm.pfault_oom_attempts .

Okay, n= oted on this.
=C2=A0

> vm.pfault_oom_attempts: -1
>
> With -j1 and -j2 options, both were able to complete the kernel and bu= ildworld compilation in 103 and 84 hours respectively. Though I still could= see messages on "swap_pager: indefinite wait buffer: bufobj" but= definitely it's ignorable as it survived the compilation process. With= the -j3 option, it failed along the course of compilation, it encountered = the previous error on "failed to reclaim memory" but this time th= is error is not that relevant as -j1 and -j2 already works. Preferably with= -j2 as the appropriate choice for my RPi 3B build setup.

Glad you got it working in your context.

Thanks for the report.

I too am glad about= the outcome of the testing. In the context of -j1 and -j2 build options, t= he oom kernel tunable settings were very effective. Thanks a lot for your h= elp!
=C2=A0
My media does not lead to the
conditions and, so, does not lead to learning the
behavior when "swap_pager: indefinite wait buffer:
bufobj" is significantly involved (for the time scale
of waits that you got into).

Ah okay, a= s you are using a more powerful media so there's no manifestation of su= ch messages. Nice and good to know.
=C2=A0

The implication of the result is that you would
need a larger vm.pageout_oom_seq value in order
for -j3 to finish normally.

Oh I see I will= take note of this one. Thanks for the further hint! I can explore this soo= n.
=C2=A0
Based on my media,
I've never had to use larger values, but, I knew
it was a technical possibility to need such. I do
not know how to pre-calculate what value would
work.

(I'm not suggesting any more -j3 experiments.)
It's alright, if I have spare time I will explore the -j3 s= ettings further. For now -j2 is sufficient.

To= Warner and Ronald, thanks as well to your inputs!

Best regards,
Archimedes
=C2=A0

=3D=3D=3D
Mark Millard
marklmi at yahoo.com

--00000000000026194e05edf7b5f2--