Date: Tue, 8 Nov 2022 15:03:59 +0800 From: Archimedes Gaviola <archimedes.gaviola@gmail.com> To: Warner Losh <imp@bsdimp.com> Cc: Mark Millard <marklmi@yahoo.com>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build Message-ID: <CAJFbk7Gw-5C%2BMcH7rAajarn2tyynVOaGL8sHsqz6%2BfSHAhO6uA@mail.gmail.com> In-Reply-To: <CANCZdfruoDPQE%2ByqebDomg8w7YKCkbFPvrPweJQUqoaOMN3rwA@mail.gmail.com> References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> <CAJFbk7FfYPSe3eF00HgDdebW70HKp5zKR0JaChTVniUDPG2qxQ@mail.gmail.com> <CA350C16-3604-4D88-9C14-040A45F6F125@yahoo.com> <CAJFbk7Hxvr9gs7GnniWtJ-QEH4yjYbB9S-vKVLjipa8v5VHa%2Bw@mail.gmail.com> <CAJFbk7GQhvgQS-23jFfFf=ershgGNKByHR8ug08Df8merwkh6Q@mail.gmail.com> <A11AA52B-BDB0-43C7-BF85-3C252B276F17@yahoo.com> <CAJFbk7HnFTdzANtdERqvgX30hcHmufAZmrNvbfEWORkUJJ7_3w@mail.gmail.com> <CAJFbk7HJ1WA5Qc0LNEZpKgv78yiM0w7ex=gjgpjjTf3chhHhiQ@mail.gmail.com> <CANCZdfruoDPQE%2ByqebDomg8w7YKCkbFPvrPweJQUqoaOMN3rwA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000080e6fd05ecf0267f Content-Type: text/plain; charset="UTF-8" On Tue, Nov 8, 2022 at 11:29 AM Warner Losh <imp@bsdimp.com> wrote: > > > On Mon, Nov 7, 2022 at 7:40 PM Archimedes Gaviola < > archimedes.gaviola@gmail.com> wrote: > >> >> >> On Sat, Nov 5, 2022 at 5:28 PM Archimedes Gaviola < >> archimedes.gaviola@gmail.com> wrote: >> >>> >>> >>> >>> On Thu, Nov 3, 2022 at 7:52 AM Mark Millard <marklmi@yahoo.com> wrote: >>> >>>> On 2022-Nov-2, at 14:09, Archimedes Gaviola < >>>> archimedes.gaviola@gmail.com> wrote: >>>> >>>> > On Mon, Oct 31, 2022 at 1:47 PM Archimedes Gaviola < >>>> archimedes.gaviola@gmail.com> wrote: >>>> > >>>> > . . . >>>> > >>>> > . . . >>>> > >>>> > >>>> > Hi Mark, >>>> > >>>> > Just an update, as kernel and world compilation is ongoing with my >>>> RPi3B system (with swap partition) is doing so far, so good. It already >>>> surpassed the tough part that breaks the compilation process here. >>>> > ... >>>> > >>>> > llvm-tblgen -gen-asm-matcher -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenAsmMatcher.inc.d -o RISCVGenAsmMatcher.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-asm-writer -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenAsmWriter.inc.d -o RISCVGenAsmWriter.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-callingconv -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenCallingConv.inc.d -o RISCVGenCallingConv.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-compress-inst-emitter -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenCompressInstEmitter.inc.d -o RISCVGenCompressInstEmitter.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-dag-isel -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenDAGISel.inc.d -o RISCVGenDAGISel.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-disassembler -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenDisassemblerTables.inc.d -o RISCVGenDisassemblerTables.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-global-isel -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenGlobalISel.inc.d -o RISCVGenGlobalISel.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-instr-info -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenInstrInfo.inc.d -o RISCVGenInstrInfo.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-emitter -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenMCCodeEmitter.inc.d -o RISCVGenMCCodeEmitter.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-pseudo-lowering -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenMCPseudoLowering.inc.d -o RISCVGenMCPseudoLowering.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-register-bank -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenRegisterBank.inc.d -o RISCVGenRegisterBank.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-register-info -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenRegisterInfo.inc.d -o RISCVGenRegisterInfo.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-searchable-tables -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenSearchableTables.inc.d -o RISCVGenSearchableTables.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-subtarget -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenSubtargetInfo.inc.d -o RISCVGenSubtargetInfo.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > llvm-tblgen -gen-searchable-tables -I >>>> /usr/src/contrib/llvm-project/llvm/include -I >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV -d >>>> RISCVGenSystemOperands.inc.d -o RISCVGenSystemOperands.inc >>>> /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td >>>> > >>>> > Any thoughts why this part is quite a challenge when it comes to >>>> memory usage? The other architectures do not possess such behavior... just >>>> curious. >>>> >>> >>> Hi Mark, >>> >>> Sorry for the late response, I got fully occupied at work for the past >>> few days due to deliverables. Thanks for your feedback and further inputs! >>> >>> >>>> I've not done any monitoring of buildworld buildkernel build >>>> activity (RAM use, memory space use, swap partition use over >>>> time) on RPi3B class hardware in a very long time. >>>> >>> >>> It's alright, it so happened that I just observed that behavior on that >>> particular part as it requires more memory than other architectures while >>> compiling. The additional 3.5G swap partition really helps on that part >>> that's why I was so happy that the compilation continued and never broke. >>> Your input of having 3.5G swap allocation is very effective. >>> >>> Even on systems that I have monitored in more recent times, >>>> what I usually monitor tends to be builds with -jN (such as >>>> -j4 fora 4-hardware-thread system). (I once did have an >>>> example of -j3 taking less time than -j4 on a RPi4B. >>>> >>> >>> Wow, this is interesting this -jN. Let me explore this as well. I >>> usually build kernel the old way but recently since I have to include >>> building the world then I need to use the new way. >>> >>> >>>> Basically, the memory subsystem can be saturated without all >>>> the cores being in use. The extra interference made things >>>> take longer.) >>>> >>> >>> Oh I see so it's the reason. >>> >>> >>>> You had listed that you were using the likes of: >>>> >>>> # cd /usr/src ; make KERNCONF=ARM TARGET_ARCH=aarch64 \ >>>> buildkernel buildworld installkernel installworld distribution \ >>>> DESTDIR=/home/freebsd/rpi3b >>>> >>>> I'll note that the standard order of the first 2 is: >>>> >>>> buildworld buildkernel >>>> >>>> This is because buildworld builds some software that >>>> buildkernel does not build for itself but does use. >>>> >>> >>> Okay this is noted, thanks for clarifying and correcting me, I really >>> appreciate it. I'll reflect on the proper build sequence for much >>> efficiency. >>> >>> >>>> There is a kernel-toolchain target for avoiding the >>>> need to do a full buildworld just to buildkernel , so: >>>> >>>> kernel-toolchain buildkernel >>>> >>>> is an expected sequence. >>>> >>> >>> Okay I'll take note of this too. >>> >>> >>>> I do not know how long a from-scratch buildworld >>>> buildkernel without a -jN takes on a RPi3B these >>>> days. If I remember right, for -jN with 1<N, I last >>>> saw claims about such they were somewhere in the >>>> range 36hr..48hr. >>> >>> >>> There's an ongoing build at the moment, it's already taking 41 hours >>> since I started it. I took another build when I came back home from the >>> office. >>> >>> >>>> But I'm unsure of the specific N >>>> that was in use. Nor do I know the storage media >>>> type(s) involved, for example. I do not remember >>>> any reports for -j1. >>>> >>> >>> I'll try this with RPi 3B. The current build that I have will be my >>> baseline. >>> >>> Use of the likes of: vm.pageout_oom_seq=120 >>>> was essential to such -jN usage on a RPi3B as N >>>> gets larger. Of course, swap partition use for >>>> paging was also essential. >>>> >>> >>> Wow, that's great! I have this vm.pageout_oom_seq=120 configured in my >>> system now based on your previous inputs. >>> >>> Likewise use of: >>>> >>>> vm.swap_enabled=0 >>>> vm.swap_idle_enabled=0 >>>> >>>> can be important to not losing communication >>>> with the RPi3B. Those last 2 are not tunable >>>> but are writable: >>>> >>>> # sysctl -aT | grep swap_ >>>> # sysctl -aW | grep swap_ >>>> vm.swap_enabled: 0 >>>> . . . >>>> vm.swap_idle_enabled: 0 >>>> . . . >>>> >>>> (This means that they have fewer places where >>>> assignments can be made. For example, the >>>> loader can not make the assignments.) >>>> >>>> By contrast, vm.pageout_oom_seq is both >>>> writable and tunable: >>>> >>>> # sysctl -aW | grep oom >>>> . . . >>>> vm.pageout_oom_seq: 120 >>>> . . . >>>> >>>> # sysctl -aT | grep oom >>>> . . . >>>> vm.pageout_oom_seq: 120 >>>> . . . >>>> >>>> (So even the loader can make such assignments.) >>>> >>> >>> Yes, I have these two sysctl parameters configured in the system. Thanks >>> for the details and further inputs. >>> >>> >>>> I'll note that I've no interest in using arm hardware >>>> to build for other types of hardware. So I normally >>>> have the targeting of support for building for other >>>> architectures disabled when I build on aarch64 or >>>> armv7. (Basically, a less complete clang/clang++ >>>> related toolchain ends up being built.) >>>> >>> >>> Ah okay, so you mean to say that you disable these other architectures >>> by declaring and accomplishing it in the /etc/src.conf? >>> >>> I'll provide an update here once the build is done knowing how long it >>> takes to finish. >>> >> >> Hi Mark, >> >> With this set of build commands now, >> >> # cd /usr/src; make -j3 KERNCONF=ARM TARGET_ARCH=aarch64 buildworld >> kernel-toolchain buildkernel installworld installkernel distribution >> DESTDIR=/home/freebsd/rpi3b >> >> in RPi 3B, I encountered the other OOM error which is the 'thread waited >> too long to allocate a page'. This occurred from every build I conducted. >> Though the first error on 'failed to reclaim memory' was never experienced >> again. Below are the error logs. >> ... >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256929, size: 4096 >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3628, size: 4096 >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255839, size: 40960 >> pid 46153 (c++), jid 0, uid 0, was killed: a thread waited too long to >> allocate a page >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: 28672 >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192 >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 4096 >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 8192 >> > > This means that paging to the swap partition and/or swap file took too > long (> 30 seconds... that's all that indefinite means). It also means that > it can't write to backing store dirty pages to give to another process... > > Typical reason is that the disk / flash is not responsive to writes for > some reason. You'll need to find why... I'd look at trims. > > Or.... if you can't change the disk... you need to put less memory > pressure on it.. > Hi Warner, This is noted too, if the error persists I'll try replacing it with another USB flash drive. Thanks and best regards, Archimedes > > Warner > > Perhaps some further tweaks are needed in the system so I set aside my RPi >> 3B temporarily and switched over to my RPi 4B using the same microSD card >> and USB flash drive (3.5 GB swap partition device) and the build completed >> successfully. It took around 30 hours to complete. This RPi 4B has 2GB RAM >> capacity while the RPi 3B has 1GB. From here, I'll continue looking further >> for system tunables in RPi 3B which has lesser RAM capacity. >> >> Thanks and best regards, >> Archimedes >> >> >> >> --00000000000080e6fd05ecf0267f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Tue, Nov 8, 2022 at 11:29 AM Warne= r Losh <<a href=3D"mailto:imp@bsdimp.com">imp@bsdimp.com</a>> wrote:<= br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e= x;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"= ><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote"><div dir=3D"ltr"= class=3D"gmail_attr">On Mon, Nov 7, 2022 at 7:40 PM Archimedes Gaviola <= ;<a href=3D"mailto:archimedes.gaviola@gmail.com" target=3D"_blank">archimed= es.gaviola@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quo= te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204= );padding-left:1ex"><div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div cl= ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sat, Nov 5, 20= 22 at 5:28 PM Archimedes Gaviola <<a href=3D"mailto:archimedes.gaviola@g= mail.com" target=3D"_blank">archimedes.gaviola@gmail.com</a>> wrote:<br>= </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b= order-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><d= iv dir=3D"ltr"><br></div><div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><d= iv class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Nov = 3, 2022 at 7:52 AM Mark Millard <<a href=3D"mailto:marklmi@yahoo.com" ta= rget=3D"_blank">marklmi@yahoo.com</a>> wrote:<br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex">On 2022-Nov-2, at 14:09, Archimedes Gaviol= a <<a href=3D"mailto:archimedes.gaviola@gmail.com" target=3D"_blank">arc= himedes.gaviola@gmail.com</a>> wrote:<br> <br> > On Mon, Oct 31, 2022 at 1:47 PM Archimedes Gaviola <<a href=3D"mail= to:archimedes.gaviola@gmail.com" target=3D"_blank">archimedes.gaviola@gmail= .com</a>> wrote:<br> > <br> > . . .<br> > <br> > . . .<br> > <br> > <br> > Hi Mark,<br> > <br> > Just an update, as kernel and world compilation is ongoing with my RPi= 3B system (with swap partition) is doing so far, so good. It already surpas= sed the tough part that breaks the compilation process here.<br> > ...<br> > <br> > llvm-tblgen -gen-asm-matcher=C2=A0 -I /usr/src/contrib/llvm-project/ll= vm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d = RISCVGenAsmMatcher.inc.d -o RISCVGenAsmMatcher.inc=C2=A0 /usr/src/contrib/l= lvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-asm-writer=C2=A0 -I /usr/src/contrib/llvm-project/llv= m/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d R= ISCVGenAsmWriter.inc.d -o RISCVGenAsmWriter.inc=C2=A0 /usr/src/contrib/llvm= -project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-callingconv=C2=A0 -I /usr/src/contrib/llvm-project/ll= vm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d = RISCVGenCallingConv.inc.d -o RISCVGenCallingConv.inc=C2=A0 /usr/src/contrib= /llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-compress-inst-emitter=C2=A0 -I /usr/src/contrib/llvm-= project/llvm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV= =C2=A0 -d RISCVGenCompressInstEmitter.inc.d -o RISCVGenCompressInstEmitter.= inc=C2=A0 /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-dag-isel=C2=A0 -I /usr/src/contrib/llvm-project/llvm/= include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d RIS= CVGenDAGISel.inc.d -o RISCVGenDAGISel.inc=C2=A0 /usr/src/contrib/llvm-proje= ct/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-disassembler=C2=A0 -I /usr/src/contrib/llvm-project/l= lvm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d= RISCVGenDisassemblerTables.inc.d -o RISCVGenDisassemblerTables.inc=C2=A0 /= usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-global-isel=C2=A0 -I /usr/src/contrib/llvm-project/ll= vm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d = RISCVGenGlobalISel.inc.d -o RISCVGenGlobalISel.inc=C2=A0 /usr/src/contrib/l= lvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-instr-info=C2=A0 -I /usr/src/contrib/llvm-project/llv= m/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d R= ISCVGenInstrInfo.inc.d -o RISCVGenInstrInfo.inc=C2=A0 /usr/src/contrib/llvm= -project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-emitter=C2=A0 -I /usr/src/contrib/llvm-project/llvm/i= nclude -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d RISC= VGenMCCodeEmitter.inc.d -o RISCVGenMCCodeEmitter.inc=C2=A0 /usr/src/contrib= /llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-pseudo-lowering=C2=A0 -I /usr/src/contrib/llvm-projec= t/llvm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0= -d RISCVGenMCPseudoLowering.inc.d -o RISCVGenMCPseudoLowering.inc=C2=A0 /u= sr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-register-bank=C2=A0 -I /usr/src/contrib/llvm-project/= llvm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -= d RISCVGenRegisterBank.inc.d -o RISCVGenRegisterBank.inc=C2=A0 /usr/src/con= trib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-register-info=C2=A0 -I /usr/src/contrib/llvm-project/= llvm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -= d RISCVGenRegisterInfo.inc.d -o RISCVGenRegisterInfo.inc=C2=A0 /usr/src/con= trib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-searchable-tables=C2=A0 -I /usr/src/contrib/llvm-proj= ect/llvm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2= =A0 -d RISCVGenSearchableTables.inc.d -o RISCVGenSearchableTables.inc=C2=A0= /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-subtarget=C2=A0 -I /usr/src/contrib/llvm-project/llvm= /include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2=A0 -d RI= SCVGenSubtargetInfo.inc.d -o RISCVGenSubtargetInfo.inc=C2=A0 /usr/src/contr= ib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > llvm-tblgen -gen-searchable-tables=C2=A0 -I /usr/src/contrib/llvm-proj= ect/llvm/include -I /usr/src/contrib/llvm-project/llvm/lib/Target/RISCV=C2= =A0 -d RISCVGenSystemOperands.inc.d -o RISCVGenSystemOperands.inc=C2=A0 /us= r/src/contrib/llvm-project/llvm/lib/Target/RISCV/RISCV.td<br> > <br> > Any thoughts why this part is quite a challenge when it comes to memor= y usage? The other architectures do not possess such behavior... just curio= us.<br></blockquote><div><br></div><div>Hi Mark,</div><div><br></div><div>S= orry for the late response, I got fully occupied at work for the past few d= ays due to deliverables. Thanks for your feedback and further inputs!</div>= <div>=C2=A0<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> I've not done any monitoring of buildworld buildkernel build<br> activity (RAM use, memory space use, swap partition use over<br> time) on RPi3B class hardware in a very long time.<br></blockquote><div><br= ></div><div>It's alright, it so happened that I just observed that beha= vior on that particular part as it requires more memory than other architec= tures while compiling. The additional 3.5G swap partition really helps on t= hat part that's why I was so happy that the compilation continued and n= ever broke. Your input of having 3.5G swap allocation is very effective.</d= iv><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px= 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Even on systems that I have monitored in more recent times,<br> what I usually monitor tends to be builds with -jN (such as<br> -j4 fora 4-hardware-thread system). (I once did have an<br> example of -j3 taking less time than -j4 on a RPi4B.<br></blockquote><div><= br></div><div>Wow, this is interesting this -jN. Let me explore this as wel= l. I usually build kernel the old way but recently since I have to include = building the world then I need to use the new way. <br></div><div>=C2=A0<br= ></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;= border-left:1px solid rgb(204,204,204);padding-left:1ex"> Basically, the memory subsystem can be saturated without all<br> the cores being in use. The extra interference made things<br> take longer.)<br></blockquote><div><br></div><div>Oh I see so it's the = reason.</div><div>=C2=A0<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"> You had listed that you were using the likes of:<br> <br> # cd /usr/src ; make KERNCONF=3DARM TARGET_ARCH=3Daarch64 \<br> buildkernel buildworld installkernel installworld distribution \<br> DESTDIR=3D/home/freebsd/rpi3b<br> <br> I'll note that the standard order of the first 2 is:<br> <br> buildworld buildkernel<br> <br> This is because buildworld builds some software that<br> buildkernel does not build for itself but does use.<br></blockquote><div><b= r></div><div>Okay this is noted, thanks for clarifying and correcting me, I= really appreciate it. I'll reflect on the proper build sequence for mu= ch efficiency.<br></div><div>=C2=A0<br></div><blockquote class=3D"gmail_quo= te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204= );padding-left:1ex"> There is a kernel-toolchain target for avoiding the<br> need to do a full buildworld just to buildkernel , so:<br> <br> kernel-toolchain buildkernel<br> <br> is an expected sequence.<br></blockquote><div><br></div><div>Okay I'll = take note of this too.</div><div>=C2=A0<br></div><blockquote class=3D"gmail= _quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204= ,204);padding-left:1ex"> I do not know how long a from-scratch buildworld<br> buildkernel without a -jN takes on a RPi3B these<br> days. If I remember right, for -jN with 1<N, I last<br> saw claims about such they were somewhere in the<br> range 36hr..48hr.</blockquote><div><br></div><div>There's an ongoing bu= ild at the moment, it's already taking 41 hours since I started it. I t= ook another build when I came back home from the office.<br></div><div>=C2= =A0<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">But I'm = unsure of the specific N<br> that was in use. Nor do I know the storage media<br> type(s) involved, for example. I do not remember<br> any reports for -j1.<br></blockquote><div><br></div><div>I'll try this = with RPi 3B. The current build that I have will be my baseline.</div><div><= br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e= x;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Use of the likes of: vm.pageout_oom_seq=3D120<br> was essential to such -jN usage on a RPi3B as N<br> gets larger. Of course, swap partition use for<br> paging was also essential.<br></blockquote><div><br></div><div>Wow, that= 9;s great! I have this=20 vm.pageout_oom_seq=3D120 configured in my system now based on your previous= inputs.<br></div><div><br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le= ft:1ex"> Likewise use of:<br> <br> vm.swap_enabled=3D0<br> vm.swap_idle_enabled=3D0<br> <br> can be important to not losing communication<br> with the RPi3B. Those last 2 are not tunable<br> but are writable:<br> <br> # sysctl -aT | grep swap_<br> # sysctl -aW | grep swap_<br> vm.swap_enabled: 0<br> . . .<br> vm.swap_idle_enabled: 0<br> . . .<br> <br> (This means that they have fewer places where<br> assignments can be made. For example, the<br> loader can not make the assignments.)<br> <br> By contrast, vm.pageout_oom_seq is both<br> writable and tunable:<br> <br> # sysctl -aW | grep oom<br> . . .<br> vm.pageout_oom_seq: 120<br> . . .<br> <br> # sysctl -aT | grep oom<br> . . .<br> vm.pageout_oom_seq: 120<br> . . .<br> <br> (So even the loader can make such assignments.)<br></blockquote><div><br></= div><div>Yes, I have these two sysctl parameters configured in the system. = Thanks for the details and further inputs.<br></div><div>=C2=A0 <br></div><= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex"> I'll note that I've no interest in using arm hardware<br> to build for other types of hardware. So I normally<br> have the targeting of support for building for other<br> architectures disabled when I build on aarch64 or<br> armv7. (Basically, a less complete clang/clang++<br> related toolchain ends up being built.)<br></blockquote><div><br></div><div= >Ah okay, so you mean to say that you disable these other architectures by = declaring and accomplishing it in the /etc/src.conf?</div><div><br></div><d= iv>I'll provide an update here once the build is done knowing how long = it takes to finish.<br></div></div></div></div></blockquote><div><br></div>= </div><div class=3D"gmail_quote">Hi Mark,</div><div class=3D"gmail_quote"><= br></div><div class=3D"gmail_quote">With this set of build commands now,</d= iv><div class=3D"gmail_quote"><br></div><div class=3D"gmail_quote"># cd /us= r/src; make -j3 KERNCONF=3DARM TARGET_ARCH=3Daarch64 buildworld kernel-tool= chain buildkernel installworld installkernel distribution DESTDIR=3D/home/f= reebsd/rpi3b</div><div class=3D"gmail_quote"><br></div><div class=3D"gmail_= quote">in RPi 3B, I encountered the other OOM error which is the 'threa= d waited too long to allocate a page'. This occurred from every build I= conducted. Though the first error on 'failed to reclaim memory' wa= s never experienced again. Below are the error logs.<br></div><div class=3D= "gmail_quote">...<br></div><div class=3D"gmail_quote">swap_pager: indefinit= e wait buffer: bufobj: 0, blkno: 256929, size: 4096<br>swap_pager: indefini= te wait buffer: bufobj: 0, blkno: 3628, size: 4096<br>swap_pager: indefinit= e wait buffer: bufobj: 0, blkno: 255839, size: 40960<br>pid 46153 (c++), ji= d 0, uid 0, was killed: a thread waited too long to allocate a page<br>swap= _pager: indefinite wait buffer: bufobj: 0, blkno: 255857, size: 28672<br>sw= ap_pager: indefinite wait buffer: bufobj: 0, blkno: 3634, size: 8192<br>swa= p_pager: indefinite wait buffer: bufobj: 0, blkno: 256037, size: 4096<br>sw= ap_pager: indefinite wait buffer: bufobj: 0, blkno: 255320, size: 8192</div= ></div></blockquote><div><br></div><div>This means that paging to the swap = partition and/or swap file took too long (> 30 seconds... that's all= that indefinite means). It also means that it can't write to backing s= tore dirty pages to give to another process...</div><div><br></div><div>Typ= ical reason is that the disk / flash is not responsive to writes for some r= eason. You'll need to find why... I'd look at trims.</div><div><br>= </div><div>Or.... if you can't change the disk... you need to put less = memory pressure on it..</div></div></div></blockquote><div><br></div><div>H= i Warner,</div><div><br></div><div>This is noted too, if the error persists= I'll try replacing it with another USB flash drive.<br></div><div><br>= </div><div>Thanks and best regards,</div><div>Archimedes<br></div><div>=C2= =A0<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"= ltr"><div class=3D"gmail_quote"><div><br></div><div>Warner</div><div><br></= div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor= der-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div= class=3D"gmail_quote">Perhaps some further tweaks are needed in the system= so I set aside my RPi 3B temporarily and switched over to my RPi 4B using = the same microSD card and USB flash drive (3.5 GB swap partition device) an= d the build completed successfully. It took around 30 hours to complete. Th= is RPi 4B has 2GB RAM capacity while the RPi 3B has 1GB. From here, I'l= l continue looking further for system tunables in RPi 3B which has lesser R= AM capacity.</div><div class=3D"gmail_quote"><br></div><div class=3D"gmail_= quote">Thanks and best regards,</div><div class=3D"gmail_quote">Archimedes = <br></div><div class=3D"gmail_quote"><br></div><div class=3D"gmail_quote"><= br></div><div class=3D"gmail_quote"><br></div></div> </blockquote></div></div> </blockquote></div></div> --00000000000080e6fd05ecf0267f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJFbk7Gw-5C%2BMcH7rAajarn2tyynVOaGL8sHsqz6%2BfSHAhO6uA>