From owner-freebsd-arm@freebsd.org Mon Dec 28 18:56:26 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 107904BDF84 for ; Mon, 28 Dec 2020 18:56:26 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D4RbX5jSvz4ZVZ for ; Mon, 28 Dec 2020 18:56:24 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.16.1/8.15.2) with ESMTPS id 0BSIuM3d035433 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 28 Dec 2020 10:56:23 -0800 (PST) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.16.1/8.15.2/Submit) id 0BSIuMre035432; Mon, 28 Dec 2020 10:56:22 -0800 (PST) (envelope-from fbsd) Date: Mon, 28 Dec 2020 10:56:22 -0800 From: bob prohaska To: Mark Millard Cc: freebsd-arm , bob prohaska Subject: Re: Migrating from -current to stable/12 on RPI2B (ARMv7) Message-ID: <20201228185622.GB28380@www.zefox.net> References: <20201228044840.GA28380@www.zefox.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: X-Rspamd-Queue-Id: 4D4RbX5jSvz4ZVZ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net X-Spamd-Result: default: False [-1.10 / 15.00]; RCVD_TLS_ALL(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[50.1.20.27:from]; WWW_DOT_DOMAIN(0.50)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zefox.net]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[50.1.20.27:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm]; MID_RHS_WWW(0.50)[] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2020 18:56:26 -0000 On Sun, Dec 27, 2020 at 10:10:18PM -0800, Mark Millard wrote: >=20 >=20 > On 2020-Dec-27, at 20:48, bob prohaska wrote: >=20 > > Having a bit of trouble migrating to stable/12 from 13.0-CURRENT=20 > > FreeBSD 13.0-CURRENT #3 r368820: Sat Dec 26 19:01:50 PST 2020 =20 > > bob@www.zefox.com:/usr/obj/usr/src/arm.armv7/sys/GENERIC-MMCCAM arm > >=20 > > This is on a Pi2B v1.1 The MMCCAM kernel option was added for sake=20 > > of the experiment and hasn't, up to now, been accompanied by any > > visible problems. World and kernel have been built many times, no > > problems.=20 > >=20 > > Buildworld keeps stopping at > > --- clang.full --- > > c++ -O -pipe -fno-common -mlong-calls -I/usr/obj/usr/freebsd-src/arm.ar= mv7/tmp/obj-tools/lib/clang/libclang -I/usr/obj/usr/freebsd-src/arm.armv7/t= mp/obj-tools/lib/clang/libllvm -I/usr/freebsd-src/contrib/llvm-project/clan= g/include -I/usr/freebsd-src/lib/clang/include -I/usr/freebsd-src/contrib/l= lvm-project/llvm/include -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D= __STDC_LIMIT_MACROS -DHAVE_VCS_VERSION_INC -DNDEBUG -DLLVM_DEFAULT_TARGET_T= RIPLE=3D\"armv7-unknown-freebsd12.2-gnueabihf\" -DLLVM_HOST_TRIPLE=3D\"armv= 7-unknown-freebsd12.2\" -DDEFAULT_SYSROOT=3D\"/usr/obj/usr/freebsd-src/arm.= armv7/tmp\" -DLLVM_TARGET_ENABLE_ARM -DLLVM_NATIVE_ASMPARSER=3DLLVMInitiali= zeARMAsmParser -DLLVM_NATIVE_ASMPRINTER=3DLLVMInitializeARMAsmPrinter -DLLV= M_NATIVE_DISASSEMBLER=3DLLVMInitializeARMDisassembler -DLLVM_NATIVE_TARGET= =3DLLVMInitializeARMTarget -DLLVM_NATIVE_TARGETINFO=3DLLVMInitializeARMTarg= etInfo -DLLVM_NATIVE_TARGETMC=3DLLVMInitializeARMTargetMC -ffunction-sectio= ns -fdata-sections -gline-tables-only -Wno-for > > mat-zero-length -Qunused-arguments -I/usr/obj/usr/freebsd-src/arm.armv7= /tmp/legacy/usr/include -fno-exceptions -fno-rtti -std=3Dc++14 -stdlib=3Dli= bc++ -Wno-c++11-extensions -Wl,--gc-sections -static -L/usr/obj/usr/freeb= sd-src/arm.armv7/tmp/legacy/usr/lib -o clang.full cc1_main.o cc1as_main.o = cc1gen_reproducer_main.o driver.o /usr/obj/usr/freebsd-src/arm.armv7/tmp/ob= j-tools/lib/clang/libclang/libclang.a /usr/obj/usr/freebsd-src/arm.armv7/tm= p/obj-tools/lib/clang/libllvm/libllvm.a -L/usr/obj/usr/freebsd-src/arm.armv= 7/tmp/obj-tools/lib/libz -lz -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-t= ools/lib/libexecinfo -lexecinfo -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/ob= j-tools/lib/libelf -lelf -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools= /lib/ncurses/ncursesw -lncursesw -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/o= bj-tools/lib/libthr -lpthread -legacy > > ld: error: failed to open clang.full: Cannot allocate memory > > c++: error: linker command failed with exit code 1 (use -v to see invoc= ation) > > *** [clang.full] Error code 1 > >=20 > > make[4]: stopped in /usr/freebsd-src/usr.bin/clang/clang > > 1 error > >=20 > > The failure occurs many minutes after the log entry appears.=20 > > No errors on the console, swap tops near half a gig. The=20 > > stable/12 sources being compiled were obtained via git. The=20 > > -current sources used to compile the running system were=20 > > obtained using svnlite.=20 > >=20 > > I don't recall seeing this much swap used before by armv7,=20 > > but that's the only notable oddity. =20 > >=20 > > The stable/12 sources have been updated every day for the=20 > > past few in hopes the trouble might go away, the -current=20 > > sources seem to be updated as far as svnlite goes.=20 > >=20 > > Any suggestions appreciated, thanks for reading > >=20 >=20 > I presume the historical system tuning for small memory systems > from our past investigations and help that we got. >=20 Correct. /boot/loader.conf contains=20 vm.pageout_oom_seq=3D"4096" vm.pfault_oom_attempts=3D"-1" > You do not mention -j4 vs. -j1 or the like. I'm assuming -j1 has > been tried. >=20 j values of 4, 2 and 1 have been tried, to no effect > You have not said if you have adjusted the likes of: > kern.maxswapzone or VM_SWZONE_SIZE_MAX or if you get the boot > notices about the likes of: >=20 > warning: total configured swap (??? pages) exceeds maximum recommended am= ount (??? pages). >=20 > Mistuning these could lead to RAM fragmentation issues as I > understand. > Dmesg reports=20 warning: total configured swap (579552 pages) exceeds maximum recomm ended amount (467936 pages). warning: increase kern.maxswzone or reduce amount of swap. but that hasn't caused recognizable trouble up to now. Adding even more swap doesn't make things any worse.=20 =20 > Since the allocation failure is reported by ld while it tries to > produce clang.full , I wonder if you have used: >=20 > LDFLAGS.lld+=3D -Wl,--no-threads >=20 > in, say, /etc/make.conf ? If not, the llvm ld will try to use > all 4 RPi2 v1.1 cores (via threads, not via processes) and, from > what I've seen, will end up trying to use more RAM than when it > runs single-threaded (main thread only). Of course, links then > take longer to complete. (LLVM ld actually ends up up with 4+2 > threads, counting the processes's main thread as well.) >=20 > Except on the machines with plenty of RAM (by a large margin for > the hardware-thread count), I use --no-threads for contexts where > I can readily control such. >=20 > (Ports do not use LDFLAGS.lld and gnu's linker does not tolerate > --no-threads as a command line option. So "readily" is limited to > buildkernel and buildworld .) > I didn't know about LDFLAGS, but a re-try with=20 -j1 and LDFLAGS.lld+=3D -Wl,--threads=3D1 (apparently the syntax changed) in /etc/make.conf=20 promptly reproduced the error.=20 At this point the host has two source directories: /usr/src, obtained using svnlite, holding -current and owned by root, along with=20 /usr/freebsd-src, obtained using git, holding -stable/12=20 owned by bob (member of wheel) where the error is occuring. Could this be either a permissions problem or a conflict among intermediate files left over between make attempts? Each source directory is matched by its own object directory, so that seems unlikely. There have been no warnings on the=20 console related to memory or swap, only the cryptic little=20 "cannot allocate memory" in the buildworld log. It's especially puzzling that the problem is evident only when compiling stable/12 as a regular user, but not when compiling -current as root. I'm trying a sanity check now by deleting clang.full from /usr/obj/usr/src (two copies) and restarting buildword with -j4 and no LDFLAGS to verify the error isn't present in -current. That might take a day or two.=20 Many thanks for readying and replying! bob prohaska