From owner-freebsd-arm@freebsd.org Mon Aug 13 10:17:00 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDAC6106D534 for ; Mon, 13 Aug 2018 10:16:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-21.consmr.mail.gq1.yahoo.com (sonic312-21.consmr.mail.gq1.yahoo.com [98.137.69.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D242733A7 for ; Mon, 13 Aug 2018 10:16:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ib5dZ1AVM1lM8aGStNRpclcZHMG9Utd1Vo3_bzePEj426CrvazK28ycKq8rfdkb Pu6GYM2sFOtmTTZzPVT9a4K4HhSWAgzJVX8FxxIjIhOT_Eumh9w9ZPL4t1xlFQc5xpP.2H5NFV0h c95Fr1mnN4DYOQdZtirdvvbf_IQ1gIOaWTbFGKgD8THRUKLlSF_8_Ro4kJ4aj21_SbsN4U.TCKuT ot4ZiUaK1evwws7LqPFknJTkhTMzk2kjp.UyqM8udMMDIy71Oy5q_o_zGxTiU3_M0HMmoAWYXDh3 d_4d9zfXF7txbJWTVkeyACGbjBL_Wcm6SY1gNaSRkfQNnhdrYqinIjoREbqgOkjxCQ5hZfUp8skk dSF3_c2CMVbaaMhDBOvk5knYuds4_LS8c2rLTFrk.VRv9u8s1eDv_cXv_sRux2NVLzGYCXwpQP5O jWZHHdrg1XrQCyFtjqxMeWPkBIZwMumoYEml8LKoLBhgLWTWBzE0KwlgDnxPjT4HkV5wO.xCxzXy GPLXwc7Rgwk0ivO5rpmOd3Igzcb80a7hloILf9M143_gIbv5y7_5rsgwP7JhGKBkevls2VqN4KWz pjsY0KAsyI7z2JiZYO0Kk0AepKTHbES31awc20nbSJHrhDCNMzLiDzyvKqm2wmvckNdpBvRg5WZS 4vjnyPL4q95JHGjCsRcS6AYSkQCDCze.3GA6kJgTvQNsA610xTanV06i3KTV1RsTDMF_rMaIN9HR ZDNoUE5iiE8OEGJ1AA9GDaIJTMh07evtvU94uU7OPGATGcy85BUsgfssOwRX4hEqpykE_3jemVka jhOSGbH6ewk5gX5Lj09x4PUb5C7p5j81s599QePjtwd77WfN7V6xnpcp1DYBiz2L5QxAwpARDYIw VSWAn_ijLVNq33.ZD8dDCHgwS2hcE3KjkPhysuHIguU55GyK_zOb8SMqGAkRurX.DPm97kzPYmCO DgEysRjWO1P7iqzOV4r0J7IOJYRNMDl2Dz.NdnIACWWGtT4SSz7yuNFCJY172YgivpX77KX8Pzu8 khUkznjEBEjtQtf1kMm2G1P_Z Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Mon, 13 Aug 2018 10:16:51 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp412.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID db1ef0fe9ef849f14a4084fbf2932658; Mon, 13 Aug 2018 10:16:50 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: RPI3 swap experiments [a failure for a Pine64+ 2GB at head -r337400 without vm.pageout_oom_seq or #define adjustment, no I/O latency problem reported] From: Mark Millard In-Reply-To: <507FC6E0-6EDB-4A0D-A9A6-DD03B50A1DA3@yahoo.com> Date: Mon, 13 Aug 2018 03:16:48 -0700 Cc: Trev , bob prohaska , John Kennedy , Jamie Landeg-Jones , "freebsd-arm@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <7AAC5B79-003C-4504-8860-26657E3A7D98@yahoo.com> References: <20180806155837.GA6277@raichu> <20180808153800.GF26133@www.zefox.net> <20180808204841.GA19379@raichu> <20180809065648.GB30347@www.zefox.net> <20180809152152.GC68459@raichu> <20180809153710.GC30347@www.zefox.net> <20180810044426.GB32974@www.zefox.net> <20180811163209.GA38922@www.zefox.net> <64798536-4ba5-24e9-304b-30cfb5b702d0@sentry.org> <507FC6E0-6EDB-4A0D-A9A6-DD03B50A1DA3@yahoo.com> To: Mark Johnston , Warner Losh X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Aug 2018 10:17:00 -0000 [vm.pageout_oom_seq=3D120 worked in this Pine64+ 2GB environment.] On 2018-Aug-12, at 10:59 AM, Mark Millard wrote: > [Quick top post: Possibly related bugzilla's for this area > are: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227609 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230402 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D230454 > .] >=20 > On 2018-Aug-12, at 10:15 AM, Mark Millard = wrote: >=20 >> Based on having all Mark Johnston's reporting-patches but >> not adjusting vm.pageout_oom_seq or the #define I got >> an OOM kill during buildworld on a Pine64+ 2GB (so 2 GiByte >> of RAM). >>=20 >> I'll give other environment characteristic later. >> But that no I/O latency problems were reported by the >> patches is not a surprise: the device with the root file >> system and swap partition has not historically shown >> latency problems and is not of a type usually used with >> such a system (better than normal). >>=20 >> I expect that this case shows that the problem does not >> require I/O latency problems to be involved. >>=20 >> The only console message was: >>=20 >> Aug 12 09:30:13 pine64 kernel: v_free_count: 7773, v_inactive_count: = 1 >> Aug 12 09:30:13 pine64 kernel: pid 80573 (c++), uid 0, was killed: = out of swap space >>=20 >> The build had been started at: 01:44:59 so the failure >> was around 7 hours 45 minutes into the build. >>=20 >> The build log shows: >>=20 >> Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/lib= clang/Sema/SemaDeclAttr.o >> Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/lib= clang/Sema/SemaDeclCXX.o >> Building = /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/lib= clang/Sema/SemaDeclObjC.o >> --- Sema/SemaDecl.o --- >> c++: error: unable to execute command: Killed >> c++: error: clang frontend command failed due to signal (use -v to = see invocation) >> FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on = LLVM 6.0.1) >> Target: aarch64-unknown-freebsd12.0 >> Thread model: posix >> InstalledDir: /usr/bin >> c++: note: diagnostic msg: PLEASE submit a bug report to = https://bugs.freebsd.org/submit/ and include the crash backtrace, = preprocessed source, and associated run script. >> c++: note: diagnostic msg:=20 >> ******************** >>=20 >> PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: >> Preprocessed source(s) and associated run script(s) are located at: >> c++: note: diagnostic msg: /tmp/SemaDecl-44b104.cpp >> c++: note: diagnostic msg: /tmp/SemaDecl-44b104.sh >> c++: note: diagnostic msg:=20 >>=20 >> ******************** >> *** [Sema/SemaDecl.o] Error code 254 >>=20 >>=20 >> Description of the context . . . >>=20 >> I have access to a Pine64+ 2GB (that was updated to -r337400 >> via a cross build) and had it doing a self-hosted rebuild of >> -r337400 via -j4. (This was a jump from its last update over >> 10 months ago. I've not historically had OOM kill problems.) >>=20 >> The root file system is on a USB3.0-capable 240 GB SSD >> plugged in the lower slot, where it gets the 480Mbps >> USB 2.0 classification. The kernel was loaded from a >> microSDHC card that also supplied a /etc/fstab that >> redirects to the USB root file system. >>=20 >> (With an edit of that /etc/fstab the microSDHC can boot >> standalone: I keep it tracking.) >>=20 >> # usbconfig >> ugen0.1: at usbus0, cfg=3D0 md=3DHOST = spd=3DHIGH (480Mbps) pwr=3DSAVE (0mA) >> ugen1.1: at usbus1, cfg=3D0 md=3DHOST = spd=3DFULL (12Mbps) pwr=3DSAVE (0mA) >> ugen0.2: at usbus0, cfg=3D0 md=3DHOST spd=3DHIGH = (480Mbps) pwr=3DON (0mA) >>=20 >> Historically I've not observed latency problems for >> the USB SSD. None were reported by Mark Johnston's >> reporting patches, unlike for Bob P.'s context. >>=20 >> In top I'd seem over 1400 Mem Active, well over the amount >> of RAM in a rpi3 or rpi2. (Of course with more memory the >> relative timings of what is running will be different >> from what rpi3's/rpi2's get. That is not the only source of >> variation.) >>=20 >> Swap had been used, I've seen 19M Used shown (of 3 GiByte >> in the swap partition in use). (I do not have in place my >> prior changes to record and report the maximum observed >> swap used: I reverted to the normal version of top during >> top's development activity.) >>=20 >> I'm unlikely to have observed approximate the maximums for >> Mem Active or Swap Used. These are things I wish there were >> normally available for inspection from FreeBSD. >>=20 >> I did not have a parallel loop showing gstat output >> or other such, relying on Mark Johnston's patches >> for reporting any that are a problem for the subsystem(s) >> involved. (It is also less I/O to not be running the >> extra logs.) >>=20 vm.pageout_oom_seq=3D120 worked in this Pine64+ 2GB environment. There were no messages from Mark Johnston's reporting patches. It took a little under 13 hr 40 minutes for a from-scratch build that did not build the bootstrap compiler or bootstrap linker but did build the clang extras: # = ~/sys_build_scripts.aarch64-host/make_cortexA53_nodebug_clang_bootstrap-aa= rch64-host.sh -j4 buildworld buildkernel . . . --- buildworld --- make[1]: "/usr/src/Makefile.inc1" line 341: SYSTEM_COMPILER: Determined = that CC=3Dcc matches the source tree. Not bootstrapping a = cross-compiler. make[1]: "/usr/src/Makefile.inc1" line 346: SYSTEM_LINKER: Determined = that LD=3Dld matches the source tree. Not bootstrapping a cross-linker. . . . -------------------------------------------------------------- >>> Kernel build for GENERIC-NODBG completed on Mon Aug 13 00:44:10 PDT = 2018 -------------------------------------------------------------- Script done, output file is = /root/sys_typescripts/typescript_make_cortexA53_nodebug_clang_bootstrap-aa= rch64-host-2018-08-12:11:07:29 (The script file name has the start time in it.) That was based on: # more ~/src.configs/src.conf.cortexA53-clang-bootstrap.aarch64-host=20 TO_TYPE=3Daarch64 # KERNCONF=3DGENERIC-NODBG TARGET=3Darm64 .if ${.MAKE.LEVEL} =3D=3D 0 TARGET_ARCH=3D${TO_TYPE} .export TARGET_ARCH .endif # #WITH_CROSS_COMPILER=3D WITH_SYSTEM_COMPILER=3D WITH_SYSTEM_LINKER=3D # #CPUTYPE=3Dsoft WITH_LIBCPLUSPLUS=3D #WITH_LLD_BOOTSTRAP=3D WITHOUT_BINUTILS_BOOTSTRAP=3D WITH_ELFTOOLCHAIN_BOOTSTRAP=3D #WITH_CLANG_BOOTSTRAP=3D WITH_CLANG=3D WITH_CLANG_IS_CC=3D WITH_CLANG_FULL=3D WITH_CLANG_EXTRAS=3D WITH_LLD=3D WITH_LLD_IS_LD=3D WITHOUT_BINUTILS=3D WITH_LLDB=3D # WITH_BOOT=3D WITHOUT_LIB32=3D # WITHOUT_GCC_BOOTSTRAP=3D WITHOUT_GCC=3D WITHOUT_GCC_IS_CC=3D WITHOUT_GNUCXX=3D # NO_WERROR=3D #WERROR=3D MALLOC_PRODUCTION=3D # WITH_REPRODUCIBLE_BUILD=3D WITH_DEBUG_FILES=3D # # Use of the .clang 's here avoids # interfering with other CFLAGS # usage, such as ?=3D usage. CFLAGS.clang+=3D -mcpu=3Dcortex-a53 CXXFLAGS.clang+=3D -mcpu=3Dcortex-a53 CPPFLAGS.clang+=3D -mcpu=3Dcortex-a53 ACFLAGS.arm64cpuid.S+=3D -mcpu=3Dcortex-a53+crypto ACFLAGS.aesv8-armx.S+=3D -mcpu=3Dcortex-a53+crypto ACFLAGS.ghashv8-armx.S+=3D -mcpu=3Dcortex-a53+crypto I used the following: # more monitor_ram_swap.sh=20 #!/bin/sh while true do sleep 1 ; date ; top -CawSores | egrep "(Mem|Swap):" done=20 to monitor part of the build, although I accidentally stopped it part way through clang/libclang being built and did not notice at the time. This was somewhat past where the prior vm.pageout_oom_seq=3D12 test had the supposedly-OOM kill. The peak observed mem-active was 1572M: (I manually added the blank lines before date output.) . . . Sun Aug 12 19:12:27 PDT 2018 Mem: 1473M Active, 64M Inact, 752K Laundry, 352M Wired, 202M Buf, 76M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:28 PDT 2018 Mem: 1478M Active, 65M Inact, 752K Laundry, 352M Wired, 202M Buf, 69M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:29 PDT 2018 Mem: 1540M Active, 39M Inact, 752K Laundry, 351M Wired, 202M Buf, 35M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:31 PDT 2018 Mem: 1568M Active, 12M Inact, 1192K Laundry, 342M Wired, 202M Buf, 41M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:32 PDT 2018 Mem: 1572M Active, 2780K Inact, 1512K Laundry, 342M Wired, 202M Buf, 49M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse Sun Aug 12 19:12:33 PDT 2018 Mem: 1113M Active, 3828K Inact, 1612K Laundry, 342M Wired, 202M Buf, = 505M Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse . . . But this was not where the (earlier) peak observed swap-use of 222M Used was recorded: . . . Sun Aug 12 18:56:35 PDT 2018 Mem: 1118M Active, 209M Inact, 250M Laundry, 345M Wired, 202M Buf, 43M = Free Swap: 3072M Total, 216M Used, 2856M Free, 7% Inuse Sun Aug 12 18:56:36 PDT 2018 Mem: 1117M Active, 209M Inact, 250M Laundry, 345M Wired, 202M Buf, 44M = Free Swap: 3072M Total, 216M Used, 2856M Free, 7% Inuse Sun Aug 12 18:56:37 PDT 2018 Mem: 1123M Active, 208M Inact, 247M Laundry, 345M Wired, 202M Buf, 42M = Free Swap: 3072M Total, 219M Used, 2853M Free, 7% Inuse . . . (an omitted block of 219M Used examples) . . . Sun Aug 12 18:57:09 PDT 2018 Mem: 1114M Active, 217M Inact, 247M Laundry, 346M Wired, 202M Buf, 41M = Free Swap: 3072M Total, 219M Used, 2853M Free, 7% Inuse Sun Aug 12 18:57:10 PDT 2018 Mem: 1127M Active, 205M Inact, 244M Laundry, 347M Wired, 203M Buf, 42M = Free Swap: 3072M Total, 222M Used, 2850M Free, 7% Inuse Sun Aug 12 18:57:11 PDT 2018 Mem: 988M Active, 162M Inact, 148M Laundry, 345M Wired, 202M Buf, 320M = Free Swap: 3072M Total, 56M Used, 3016M Free, 1% Inuse . . . The first and last recorded was: Sun Aug 12 11:24:50 PDT 2018 Mem: 62M Active, 435M Inact, 4596K Laundry, 454M Wired, 202M Buf, 1014M = Free Swap: 3072M Total, 19M Used, 3053M Free . . . Sun Aug 12 19:47:06 PDT 2018 Mem: 782M Active, 83M Inact, 1068K Laundry, 347M Wired, 202M Buf, 753M = Free Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)