From owner-freebsd-arm@freebsd.org Mon Dec 28 06:56:25 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A0B54D2DAE for ; Mon, 28 Dec 2020 06:56:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-8.consmr.mail.gq1.yahoo.com (sonic316-8.consmr.mail.gq1.yahoo.com [98.137.69.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4D47cl3hQ4z4mXh for ; Mon, 28 Dec 2020 06:56:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1609138581; bh=6MdPa1XKxq+hq15nQINcVSz5CVUcRxuUxdc1N0fNKzX=; h=Subject:From:Date:To:From:Subject; b=sqaboXr2tFVBblTGGyAjc7leO50yXdvFncUU/ae5WLVzsTQ6f+4K4vVV+wWA2RHmZMIxEMVftMVCRywXCJIQ+dUN2f0jMGn07Y/9j1Lhz28+L/q2jkmFHBdBNc0Q5c1qNGugdEpC5qthWaAdRcKXgIrd03pcpa67CeJ1MgaoSmtMcwJlWwKJiRq6KroOFGO7bMsCDp8GtZ1jom2isRaJ1tJH7zYFGuUSp0sCgq9NUJECUPcFK24xKFBjQCmkHVqoRL6vw2hwSORYTh8u3zv/OTmTIFWlUBzWVau91Nu03ZifpWdQTBbNAM8eF7PgiyXSHdC3JDk0xGkytHS6LtdRZg== X-YMail-OSG: IXsf8S8VM1mkayLyeklR.VbqGwy9tAnTUDZFtOJgNzKfCdCk2Zj1TdVysk9IRCU hufZmxBINeJvhOYiEM4YNfPASdVYD6kHQa_fjybt.tYWXtno2xAaakHt0F2pDoWp.F5x.iKSG8ds Iuh7K5F7zPAxmsz9y9jaEXokuIkV5CNZ2fhT_Hucv8PS8s25eQtqMZb10PIug5AGglc4k07amAdZ PeH0T4Vds7YUM5hgLQfkUZ3tGGNLqi2pp6ALENzxXLvudNOv5l3G0a3wU1fXighIPataOAuRaQiO J6_hOmZKkJVGYvFkw0jOADT8u8MoD5cH1pq2h19q84QvlZZEka04o.tpBV_IUbp7JJlLe9xnGpaV .KkApiS9PEfasBlfi7VNrjKDjtXaWwaHPm88q1hemdBye1mNdrk9_64yqGJpaih0RsRL.rZeXkFV aIvvUCrgFsETNRZhvbmEZ6ZGn89CEMvuwB6ukHRmNQnYTbtBTpGqhJAagErrD2LJQtoOw4w.xMAC woDe_0gbF.XXaHGCT8XEtC_c7Ko4ed1XHTtOdXMZc1lQ9vnZT5maWYakAYnGPFbr.EbYUtRjuuGG LFVspsSBCWVQPCbiBYlqDGfTVOiF0vLF6P03VtFrXp0FJtGDBNvNlMDNAxKAlWh3ePJiQ7czvxSZ T0HQNb6lueDv9iQ2sha0JcZ50mhg_w6OuB6vhcOnhgC8sny19HIvT4V7bnCNcBP9OC7w4sonSjN8 cS4GtttC9F4EmimpcGIJBzpbuJEknY4fmjTn1cXomCuLpCzbvPOmxUOgsXunu9QHJ_fnmgP5LcnN Gy9_IzrRQZ_qgB5IuzvwMhhNr_ml.irW6UnWupi_2L.pnY7dsVQObZx1r_nV5bn1wlYDwqW0cYtF l31RUErHTOhiop9YKL.SsEy9YGimiX3PCZSEFYWYNAJNkP9Q4dIRyqKhj8c55Tfnsr2X9hDaJ3Nb nJWRe5HnPx.vz5zlZRHCanaafBsMiaEdzxU10SSyztWhIA1Bty6n5xF6iHaaghzsIRpZvnQaOBPz qKn.bYmKdhYq1avVCwcPxdTR0fpxXhbsXYhKUiptKRmp8327o8VQzFK82ydmn3BQsDUQwgDFj6jl ThPRquOW.8SUta8P4rV7ttjX2B0wz6IqNXXre2S7WYbHViMCFAfwxI.jm9_PhP6sQVudUUiI9K8b iyslyN3.tH6i6D_c8FrCHA1ggLh4Ky.Lkw02bVaOJ0kuUG3Vw_gre0INUNvoKUlTgsOSK0PybQpQ Yoeatd0juNc_TqO7xOKLFO6hY6M4laQhBRk1_8GrhmX4HY3GL9x3oZuQ21S4VvT.QmNCL8P4E2L7 D9Br99IyX36xZP35J7el6ZFjWKbGZmIqnhlu3L68rbxi6WgDfn6pzNTBjQ6hqlObAGd_.P2QweOf 4sSHG5AbP.82PRPSRz3dimXC.34oWlfsS6ozpmlTmBNuLJ06qf3LHjGh0CxgwSjt_kJ7ooTBTx.I njlCqs7B0tvMnY5TvjT7bYiKZ2UpOCQpUlc6HVjIwYH7HXa..1V1IXOhkqfxlgBc6CQFW8loPzH5 H8uqu0Brg2EwP9lPv1jy2Uqb_QTk5ThvYAxkfS1_tR61XKFVxZnenzVX9wzOBDhxHX9ikZhDciEE hVHQBRIdWfTytdFk7H_brJ7IxEADOZNEqkq52WHA2xlsGCRN1OIkzLHbtpmfOoOpMaDE0V6viXax L36C9qMjFIMsTSvwkJ3C4HKPfaof0M2gMpgU0FeVHopquH5KQ7Jik9BjdU5pxauDABovNuho6CFv oqIE2JkRbPKi_b.u3cCl0vOa6XLS3h.eO8bPlCXbzNZM513BtV8WlG1UFGM_ePFQ1iCUWHBbG4qx oER4HMQ5xd_LlELkdlBNORwbDKWum24CrrnAb8ovp4sa36Dig8dpW9XNDHozrYxQ4s_Aslyolc.y QEhZ.zjMwl27cxnm3lhJyeWIq_yu28LT82GE3u3u4leNU9ZV_63BrFjjE5Op1Bz4Dy6RC_bfEGJy tVN2Lp55xBERhZn30NzsdlMXS3iCu_xrvKrX0UsI9NBSTLZDMBnSE0AO7fZoDIDMSFxIhFewXIFU sEmCOwMehiqz9KMYEt1Jsk8JeTzvlt2qQfReRfemPUtQTwWBUl.FyBgoWslIKc6KqZlFcjUEmVlx 3LBzsCgIyTfYjGJdM8gdW1wMYAYwBgNiEz8UeMPmlGCyy5Ab4I4FYSPzA_PdCz8jMQdOfJ8SbgwN BOoM9yr9VRmgVrh4cpQuklUV2AGPIk_Ywv.KfVMytD3eazXUzhPLAMwTSsRRzrTgPmZUyT_mmF8X JWSEDbgKUARStVB7UpDynwGvmTOsWA2lvYGCI4WrfO7Wl95mNkA0tVgZBnkwDIz_wJ3bVBEFPLTq K3Er6SNUZoiWK3V5n4L4RBQ6Cslj6ReXDazffWK71UcCk1B0aon47udHSdA0ckShwo82c138yXhd lzDiPEkzEwX5N4y8GsHfiqUNiR569xZIBBfJUcKt7eOfai94AQav1yE7H.CWnf57HZdbjhzyU7pA B4NNT448f3_vzg.rLfKk8_pgd2LM- Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Mon, 28 Dec 2020 06:56:21 +0000 Received: by smtp423.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID d794e8c26863a234954cb9a7dffc7c12; Mon, 28 Dec 2020 06:56:15 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.40.0.2.32\)) Subject: Re: Migrating from -current to stable/12 on RPI2B (ARMv7) From: Mark Millard In-Reply-To: Date: Sun, 27 Dec 2020 22:56:13 -0800 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: References: <20201228044840.GA28380@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3654.40.0.2.32) X-Rspamd-Queue-Id: 4D47cl3hQ4z4mXh X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.69.32:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.69.32:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.32:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.32:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2020 06:56:25 -0000 On 2020-Dec-27, at 22:10, Mark Millard wrote: > On 2020-Dec-27, at 20:48, bob prohaska wrote: >=20 >> Having a bit of trouble migrating to stable/12 from 13.0-CURRENT=20 >> FreeBSD 13.0-CURRENT #3 r368820: Sat Dec 26 19:01:50 PST 2020 =20 >> bob@www.zefox.com:/usr/obj/usr/src/arm.armv7/sys/GENERIC-MMCCAM arm >>=20 >> This is on a Pi2B v1.1 The MMCCAM kernel option was added for sake=20 >> of the experiment and hasn't, up to now, been accompanied by any >> visible problems. World and kernel have been built many times, no >> problems.=20 >>=20 >> Buildworld keeps stopping at >> --- clang.full --- >> c++ -O -pipe -fno-common -mlong-calls = -I/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libclang = -I/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libllvm = -I/usr/freebsd-src/contrib/llvm-project/clang/include = -I/usr/freebsd-src/lib/clang/include = -I/usr/freebsd-src/contrib/llvm-project/llvm/include = -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS = -DHAVE_VCS_VERSION_INC -DNDEBUG = -DLLVM_DEFAULT_TARGET_TRIPLE=3D\"armv7-unknown-freebsd12.2-gnueabihf\" = -DLLVM_HOST_TRIPLE=3D\"armv7-unknown-freebsd12.2\" = -DDEFAULT_SYSROOT=3D\"/usr/obj/usr/freebsd-src/arm.armv7/tmp\" = -DLLVM_TARGET_ENABLE_ARM = -DLLVM_NATIVE_ASMPARSER=3DLLVMInitializeARMAsmParser = -DLLVM_NATIVE_ASMPRINTER=3DLLVMInitializeARMAsmPrinter = -DLLVM_NATIVE_DISASSEMBLER=3DLLVMInitializeARMDisassembler = -DLLVM_NATIVE_TARGET=3DLLVMInitializeARMTarget = -DLLVM_NATIVE_TARGETINFO=3DLLVMInitializeARMTargetInfo = -DLLVM_NATIVE_TARGETMC=3DLLVMInitializeARMTargetMC -ffunction-sections = -fdata-sections -gline-tables-only -Wno-for >> mat-zero-length -Qunused-arguments = -I/usr/obj/usr/freebsd-src/arm.armv7/tmp/legacy/usr/include = -fno-exceptions -fno-rtti -std=3Dc++14 -stdlib=3Dlibc++ = -Wno-c++11-extensions -Wl,--gc-sections -static = -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/legacy/usr/lib -o clang.full = cc1_main.o cc1as_main.o cc1gen_reproducer_main.o driver.o = /usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libclang/libcla= ng.a = /usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libllvm/libllvm= .a -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libz -lz = -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libexecinfo = -lexecinfo -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libelf = -lelf = -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/ncurses/ncursesw = -lncursesw -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libthr = -lpthread -legacy >> ld: error: failed to open clang.full: Cannot allocate memory >> c++: error: linker command failed with exit code 1 (use -v to see = invocation) >> *** [clang.full] Error code 1 >>=20 >> make[4]: stopped in /usr/freebsd-src/usr.bin/clang/clang >> 1 error >>=20 >> The failure occurs many minutes after the log entry appears.=20 >> No errors on the console, swap tops near half a gig. The=20 >> stable/12 sources being compiled were obtained via git. The=20 >> -current sources used to compile the running system were=20 >> obtained using svnlite.=20 >>=20 >> I don't recall seeing this much swap used before by armv7,=20 >> but that's the only notable oddity. =20 >>=20 >> The stable/12 sources have been updated every day for the=20 >> past few in hopes the trouble might go away, the -current=20 >> sources seem to be updated as far as svnlite goes.=20 >>=20 >> Any suggestions appreciated, thanks for reading >>=20 >=20 > I presume the historical system tuning for small memory systems > from our past investigations and help that we got. >=20 > You do not mention -j4 vs. -j1 or the like. I'm assuming -j1 has > been tried. >=20 > You have not said if you have adjusted the likes of: > kern.maxswapzone or VM_SWZONE_SIZE_MAX or if you get the boot > notices about the likes of: >=20 > warning: total configured swap (??? pages) exceeds maximum recommended = amount (??? pages). >=20 > Mistuning these could lead to RAM fragmentation issues as I > understand. >=20 > Since the allocation failure is reported by ld while it tries to > produce clang.full , I wonder if you have used: >=20 > LDFLAGS.lld+=3D -Wl,--no-threads >=20 > in, say, /etc/make.conf ? If not, the llvm ld will try to use > all 4 RPi2 v1.1 cores (via threads, not via processes) and, from > what I've seen, will end up trying to use more RAM than when it > runs single-threaded (main thread only). Of course, links then > take longer to complete. (LLVM ld actually ends up up with 4+2 > threads, counting the processes's main thread as well.) >=20 > Except on the machines with plenty of RAM (by a large margin for > the hardware-thread count), I use --no-threads for contexts where > I can readily control such. >=20 > (Ports do not use LDFLAGS.lld and gnu's linker does not tolerate > --no-threads as a command line option. So "readily" is limited to > buildkernel and buildworld .) Another type of thing that you have not mentioned: non-debug vs. debug status of various things (debug usually meaning more RAM ends up in use during builds). But I've not experimented to know if some of these could make a nontrivial difference in the RAM use. Is the host 13 built with debug information and debug code generally? kernel? world? Is its llvm toolchain build with debug asserts and the like? Is the clang.full being built one that was built with debug information? Debug asserts? Similar types of questions for the stable/12 "cross" toolchain (older llvm vintage?) if such is built and used. In stable/12 there are (for /etc/src.conf use): WITHOUT_ASSERT_DEBUG Set to compile programs and libraries without the assert(3) checks. and: WITH_LLVM_ASSERTIONS Set to enable debugging assertions in LLVM. and: WITHOUT_MALLOC_PRODUCTION Set to enable assertions and statistics gathering in = malloc(3). It also defaults the A and J runtime options to on. So using WITHOUT_ASSERT_DEBUG=3D but not using the other two would be appropriate for having less RAM use. But, be warned, 13 is still at a stage where it has reversed two of the orientations (default vs. what it can be changed to): WITHOUT_LLVM_ASSERTIONS Set to disable debugging assertions in LLVM. WITH_MALLOC_PRODUCTION Set to disable assertions and statistics gathering in = malloc(3). It also defaults the A and J runtime options to off. So care in handling src.conf is required if you have been explicit about those last two in past build activity. The combination for minimizing RAM use from these is: WITHOUT_ASSERT_DEBUG=3D WITHOUT_LLVM_ASSERTIONS=3D WITH_MALLOC_PRODUCTION=3D If I understand right, both stable/12 and 13 will tolerate listing those 3 in /etc/src.conf . (Despite the src.conf man pages not saying so.) There are, of course, other tradeoffs involved in minimizing RAM use (host live use generally and target output-files related) for what these options control. For 13's kernel options: what of the status of (shown disabled) . . . nooptions DEADLKRES # the deadlock resolver nooptions INVARIANTS # calls of extra sanity checking nooptions INVARIANT_SUPPORT # Extra sanity checks of = internal structures, required by INVARIANTS nooptions WITNESS # Enable checks to detect = deadlocks and cycles nooptions WITNESS_SKIPSPIN # Don't run witness on spinlocks = for speed nooptions DIAGNOSTIC nooptions MALLOC_DEBUG_MAXZONES # Separate malloc(9) zones and possibly the status of (shown not enabled): #makeoptions DEBUG=3D-g # Build kernel with gdb(1) = debug symbols and ( in, say, /etc/src.conf ): WITH_DEBUG_FILES=3D (if the DEBUG=3D-g was used in makeoptions). (Avoiding debug information being in the loaded kernel files helps avoid as much RAM use, for example.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)