Date: Mon, 13 Aug 2018 03:16:48 -0700 From: Mark Millard <marklmi@yahoo.com> To: Mark Johnston <markj@FreeBSD.org>, Warner Losh <imp@bsdimp.com> Cc: Trev <freebsd-arm@sentry.org>, bob prohaska <fbsd@www.zefox.net>, John Kennedy <warlock@phouka.net>, Jamie Landeg-Jones <jamie@catflap.org>, "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org> Subject: Re: RPI3 swap experiments [a failure for a Pine64+ 2GB at head -r337400 without vm.pageout_oom_seq or #define adjustment, no I/O latency problem reported] Message-ID: <7AAC5B79-003C-4504-8860-26657E3A7D98@yahoo.com> In-Reply-To: <507FC6E0-6EDB-4A0D-A9A6-DD03B50A1DA3@yahoo.com> References: <EC74A5A6-0DF4-48EB-88DA-543FD70FEA07@yahoo.com> <20180806155837.GA6277@raichu> <20180808153800.GF26133@www.zefox.net> <20180808204841.GA19379@raichu> <20180809065648.GB30347@www.zefox.net> <20180809152152.GC68459@raichu> <CANCZdfpKOTBrxiNhaeHHRp-2iw5a4eXt%2Bmd_1LTD-c0%2BAE6qxg@mail.gmail.com> <20180809153710.GC30347@www.zefox.net> <CANCZdfrC0s8X-LxJmrDmkxmz%2BGUMNsHSMpBEQmp1S5ahcvptpg@mail.gmail.com> <20180810044426.GB32974@www.zefox.net> <20180811163209.GA38922@www.zefox.net> <64798536-4ba5-24e9-304b-30cfb5b702d0@sentry.org> <F43ED9FB-6AF2-4069-9A4E-15E134593174@yahoo.com> <507FC6E0-6EDB-4A0D-A9A6-DD03B50A1DA3@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[vm.pageout_oom_seq=3D120 worked in this Pine64+ 2GB environment.] On 2018-Aug-12, at 10:59 AM, Mark Millard <marklmi@yahoo.com> wrote: > [Quick top post: Possibly related bugzilla's for this area > are: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227609 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230402 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230454 > .] >=20 > On 2018-Aug-12, at 10:15 AM, Mark Millard <marklmi at yahoo.com> = wrote: >=20 >> Based on having all Mark Johnston's reporting-patches but >> not adjusting vm.pageout_oom_seq or the #define I got >> an OOM kill during buildworld on a Pine64+ 2GB (so 2 GiByte >> of RAM). >>=20 >> I'll give other environment characteristic later. >> But that no I/O latency problems were reported by the >> patches is not a surprise: the device with the root file >> system and swap partition has not historically shown >> latency problems and is not of a type usually used with >> such a system (better than normal). >>=20 >> I expect that this case shows that the problem does not >> require I/O latency problems to be involved. >>=20 >> The only console message was: >>=20 >> Aug 12 09:30:13 pine64 kernel: v_free_count: 7773, v_inactive_count: = 1 >> Aug 12 09:30:13 pine64 kernel: pid 80573 (c++), uid 0, was killed: = out of swap space >>=20 >> The build had been started at: 01:44:59 so the failure >> was around 7 hours 45 minutes into the build. >>=20 >> The build log shows: >>=20 >> Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/lib= clang/Sema/SemaDeclAttr.o >> Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/lib= clang/Sema/SemaDeclCXX.o >> Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/lib= clang/Sema/SemaDeclObjC.o >> --- Sema/SemaDecl.o --- >> c++: error: unable to execute command: Killed >> c++: error: clang frontend command failed due to signal (use -v to = see invocation) >> FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on = LLVM 6.0.1) >> Target: aarch64-unknown-freebsd12.0 >> Thread model: posix >> InstalledDir: /usr/bin >> c++: note: diagnostic msg: PLEASE submit a bug report to = https://bugs.freebsd.org/submit/ and include the crash backtrace, = preprocessed source, and associated run script. >> c++: note: diagnostic msg:=20 >> ******************** >>=20 >> PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: >> Preprocessed source(s) and associated run script(s) are located at: >> c++: note: diagnostic msg: /tmp/SemaDecl-44b104.cpp >> c++: note: diagnostic msg: /tmp/SemaDecl-44b104.sh >> c++: note: diagnostic msg:=20 >>=20 >> ******************** >> *** [Sema/SemaDecl.o] Error code 254 >>=20 >>=20 >> Description of the context . . . >>=20 >> I have access to a Pine64+ 2GB (that was updated to -r337400 >> via a cross build) and had it doing a self-hosted rebuild of >> -r337400 via -j4. (This was a jump from its last update over >> 10 months ago. I've not historically had OOM kill problems.) >>=20 >> The root file system is on a USB3.0-capable 240 GB SSD >> plugged in the lower slot, where it gets the 480Mbps >> USB 2.0 classification. The kernel was loaded from a >> microSDHC card that also supplied a /etc/fstab that >> redirects to the USB root file system. >>=20 >> (With an edit of that /etc/fstab the microSDHC can boot >> standalone: I keep it tracking.) >>=20 >> # usbconfig >> ugen0.1: <Allwinner EHCI root HUB> at usbus0, cfg=3D0 md=3DHOST = spd=3DHIGH (480Mbps) pwr=3DSAVE (0mA) >> ugen1.1: <Generic OHCI root HUB> at usbus1, cfg=3D0 md=3DHOST = spd=3DFULL (12Mbps) pwr=3DSAVE (0mA) >> ugen0.2: <OWC Envoy Pro mini> at usbus0, cfg=3D0 md=3DHOST spd=3DHIGH = (480Mbps) pwr=3DON (0mA) >>=20 >> Historically I've not observed latency problems for >> the USB SSD. None were reported by Mark Johnston's >> reporting patches, unlike for Bob P.'s context. >>=20 >> In top I'd seem over 1400 Mem Active, well over the amount >> of RAM in a rpi3 or rpi2. (Of course with more memory the >> relative timings of what is running will be different >> from what rpi3's/rpi2's get. That is not the only source of >> variation.) >>=20 >> Swap had been used, I've seen 19M Used shown (of 3 GiByte >> in the swap partition in use). (I do not have in place my >> prior changes to record and report the maximum observed >> swap used: I reverted to the normal version of top during >> top's development activity.) >>=20 >> I'm unlikely to have observed approximate the maximums for >> Mem Active or Swap Used. These are things I wish there were >> normally available for inspection from FreeBSD. >>=20 >> I did not have a parallel loop showing gstat output >> or other such, relying on Mark Johnston's patches >> for reporting any that are a problem for the subsystem(s) >> involved. (It is also less I/O to not be running the >> extra logs.) >>=20 vm.pageout_oom_seq=3D120 worked in this Pine64+ 2GB environment. There were no messages from Mark Johnston's reporting patches. It took a little under 13 hr 40 minutes for a from-scratch build that did not build the bootstrap compiler or bootstrap linker but did build the clang extras: # = ~/sys_build_scripts.aarch64-host/make_cortexA53_nodebug_clang_bootstrap-aa= rch64-host.sh -j4 buildworld buildkernel . . . --- buildworld --- make[1]: "/usr/src/Makefile.inc1" line 341: SYSTEM_COMPILER: Determined = that CC=3Dcc matches the source tree. Not bootstrapping a = cross-compiler. make[1]: "/usr/src/Makefile.inc1" line 346: SYSTEM_LINKER: Determined = that LD=3Dld matches the source tree. Not bootstrapping a cross-linker. . . . -------------------------------------------------------------- >>> Kernel build for GENERIC-NODBG completed on Mon Aug 13 00:44:10 PDT = 2018 -------------------------------------------------------------- Script done, output file is = /root/sys_typescripts/typescript_make_cortexA53_nodebug_clang_bootstrap-aa= rch64-host-2018-08-12:11:07:29 (The script file name has the start time in it.) That was based on: # more ~/src.configs/src.conf.cortexA53-clang-bootstrap.aarch64-host=20 TO_TYPE=3Daarch64 # KERNCONF=3DGENERIC-NODBG TARGET=3Darm64 .if ${.MAKE.LEVEL} =3D=3D 0 TARGET_ARCH=3D${TO_TYPE} .export TARGET_ARCH .endif # #WITH_CROSS_COMPILER=3D WITH_SYSTEM_COMPILER=3D WITH_SYSTEM_LINKER=3D # #CPUTYPE=3Dsoft WITH_LIBCPLUSPLUS=3D #WITH_LLD_BOOTSTRAP=3D WITHOUT_BINUTILS_BOOTSTRAP=3D WITH_ELFTOOLCHAIN_BOOTSTRAP=3D #WITH_CLANG_BOOTSTRAP=3D WITH_CLANG=3D WITH_CLANG_IS_CC=3D WITH_CLANG_FULL=3D WITH_CLANG_EXTRAS=3D WITH_LLD=3D WITH_LLD_IS_LD=3D WITHOUT_BINUTILS=3D WITH_LLDB=3D # WITH_BOOT=3D WITHOUT_LIB32=3D # WITHOUT_GCC_BOOTSTRAP=3D WITHOUT_GCC=3D WITHOUT_GCC_IS_CC=3D WITHOUT_GNUCXX=3D # NO_WERROR=3D #WERROR=3D MALLOC_PRODUCTION=3D # WITH_REPRODUCIBLE_BUILD=3D WITH_DEBUG_FILES=3D # # Use of the .clang 's here avoids # interfering with other C<?>FLAGS # usage, such as ?=3D usage. CFLAGS.clang+=3D -mcpu=3Dcortex-a53 CXXFLAGS.clang+=3D -mcpu=3Dcortex-a53 CPPFLAGS.clang+=3D -mcpu=3Dcortex-a53 ACFLAGS.arm64cpuid.S+=3D -mcpu=3Dcortex-a53+crypto ACFLAGS.aesv8-armx.S+=3D -mcpu=3Dcortex-a53+crypto ACFLAGS.ghashv8-armx.S+=3D -mcpu=3Dcortex-a53+crypto I used the following: # more monitor_ram_swap.sh=20 #!/bin/sh while true do sleep 1 ; date ; top -CawSores | egrep "(Mem|Swap):" done=20 to monitor part of the build, although I accidentally stopped it part way through clang/libclang being built and did not notice at the time. This was somewhat past where the prior vm.pageout_oom_seq=3D12 test had the supposedly-OOM kill. The peak observed mem-active was 1572M: (I manually added the blank lines before date output.) . . . Sun Aug 12 19:12:27 PDT 2018 Mem: 1473M Active, 64M Inact, 752K Laundry, 352M Wired, 202M Buf, 76M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:28 PDT 2018 Mem: 1478M Active, 65M Inact, 752K Laundry, 352M Wired, 202M Buf, 69M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:29 PDT 2018 Mem: 1540M Active, 39M Inact, 752K Laundry, 351M Wired, 202M Buf, 35M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:31 PDT 2018 Mem: 1568M Active, 12M Inact, 1192K Laundry, 342M Wired, 202M Buf, 41M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:32 PDT 2018 Mem: 1572M Active, 2780K Inact, 1512K Laundry, 342M Wired, 202M Buf, 49M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:33 PDT 2018 Mem: 1113M Active, 3828K Inact, 1612K Laundry, 342M Wired, 202M Buf, = 505M Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse . . . But this was not where the (earlier) peak observed swap-use of 222M Used was recorded: . . . Sun Aug 12 18:56:35 PDT 2018 Mem: 1118M Active, 209M Inact, 250M Laundry, 345M Wired, 202M Buf, 43M = Free Swap: 3072M Total, 216M Used, 2856M Free, 7% Inuse Sun Aug 12 18:56:36 PDT 2018 Mem: 1117M Active, 209M Inact, 250M Laundry, 345M Wired, 202M Buf, 44M = Free Swap: 3072M Total, 216M Used, 2856M Free, 7% Inuse Sun Aug 12 18:56:37 PDT 2018 Mem: 1123M Active, 208M Inact, 247M Laundry, 345M Wired, 202M Buf, 42M = Free Swap: 3072M Total, 219M Used, 2853M Free, 7% Inuse . . . (an omitted block of 219M Used examples) . . . Sun Aug 12 18:57:09 PDT 2018 Mem: 1114M Active, 217M Inact, 247M Laundry, 346M Wired, 202M Buf, 41M = Free Swap: 3072M Total, 219M Used, 2853M Free, 7% Inuse Sun Aug 12 18:57:10 PDT 2018 Mem: 1127M Active, 205M Inact, 244M Laundry, 347M Wired, 203M Buf, 42M = Free Swap: 3072M Total, 222M Used, 2850M Free, 7% Inuse Sun Aug 12 18:57:11 PDT 2018 Mem: 988M Active, 162M Inact, 148M Laundry, 345M Wired, 202M Buf, 320M = Free Swap: 3072M Total, 56M Used, 3016M Free, 1% Inuse . . . The first and last recorded was: Sun Aug 12 11:24:50 PDT 2018 Mem: 62M Active, 435M Inact, 4596K Laundry, 454M Wired, 202M Buf, 1014M = Free Swap: 3072M Total, 19M Used, 3053M Free . . . Sun Aug 12 19:47:06 PDT 2018 Mem: 782M Active, 83M Inact, 1068K Laundry, 347M Wired, 202M Buf, 753M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7AAC5B79-003C-4504-8860-26657E3A7D98>