From nobody Mon Oct 31 04:00:27 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N0zvn1MnQz4gWWZ for ; Mon, 31 Oct 2022 04:00:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4N0zvm1HGdz3fxL for ; Mon, 31 Oct 2022 04:00:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1667188833; bh=gO8i94fL/mZOFPBAHsIX5/VS+v2UJDADNVxb6T9ku5I=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=TBYbEUnnYBP0LGKpCeNbymKjY+pQeVLy/hGYBq5PzPgvS4tdgae0l5ZLlqSOj+LxKr+A9YU0SUiCaAMfvdJUOX06AEM0RDyFr6cpD2whv6gRX2J+O+QMNVLkUvEShzLKGeCrIkWI8cJaShreBeQURKPK5zp7F11F2Z2z7VX+fCaR/+RNh3xh3OioTCCQAyit3SWwUQgo4SFhSs18Be13UNe4kpQf/k776NJoCwPFQWvLkxJXON2d8uica2fFegAN2BI1Dzh5o8aopGMKQMPSdfwe/yQu95KX6V+Z7YuzJcah6fTq9FohgKwVggdt6TxOGeJGn4VHNjQQ9x6TWxO6mQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1667188833; bh=w5icg3IJWlJv+w/yU4ltfFDXHI+eM7QHGzyetIxcHAt=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=nMI7aDGB9JmoO93EnQvt3Cf6bcXDZdLdJ2D08+qh7eco7Sv3idMkHq1oy0VCRUh19HVASVO3XskHf2MoY8ZxUo7KRLKokGVGnavfvMQYQTZG8mq8bPliwDrHyeg2PKwIVeyQTzmJez+sv4/npymeshDkJbS2VPsqQHVK5dSgQfViyIPAY2Ieekk5nmx/Pxri/BxfCRlla6Rv4MG9UohbkCRg5sZqtwfCqEdkr+im0iHmJFq8dAAD88wDMAQ6SJ89lfYVGpoY9jqrV1iKHM+7czVkGECXCXeQC7drb9CAkoaSm1J2ybAIXbhJ9VFOnmuTmqOCi1bNdYn3fc2DN6JRCA== X-YMail-OSG: ZXHUv0IVM1kyT8Fkd3pBZ98kmjvCvU2_vckBeh98RRHo_61yFJfdfwsm6Z9ut_c owA0xnTNMn5NkrFbMh8ETMA3QB3HSYtMdj12RKqHIOFukKGP2ZpCQvt1JHfkctRCTOCwvCXXiLgi q71U9pXXoHeV8guP5u5Tt1vGH..mzFMhlgRshPbbJsWMESquzYO_UIV7gq8oFhHrlzHNkyARmmRS IwcOixqwxjVaUA9DhkBY78li58FFApv3vym5E9mh1tfZUdfOq4y4UiZIVZvBYdVZE3F6_QlGHMHR b6hfcvpj_TEKp_kfmqztVqR4s4tyNECvlFIMXqB.Ajd1fH1K6AppE1Qk4ptg2RzM_uBsuj6ubId_ 4ZwFGPoSpHvaFu99HuavCKIYw40L583Wu9FS1Ry89qW_EorHgAVr4sdBEKhp7pgDEk2tPydLwKbg pkTvxYvOWFd9n3Im73CGjzRgQFPTsV3yCngqoi0EXrupq3lcZu7foaQZj_9RItORP7EYW2P4hN5H pyhMt48CEKACD_b_SH5UzL5C7wBY.uAsShfsUPGrBrllVuFpg_7UpwfOjV4N01AL_sr_Jng0uh2A j3BEJJo1gmcn8CMH9G2G4repsASkCTkaDhsHRWzWwIUBEfse2eE32hSJA5JBAoPkRv2mOVbSh3MG pkX9NUZEJVdFq_FSHlgZiW5PW8mG.WgJZDm97CQdlORYjOjcm1MZs35tnryxzx3A1mlXV35Vmtbz l.cikA0seibnI5Hc5OcR6C0RFS76OlXLd0sg2PkWiAXDxhMElF9uUZeACOPwP1XPAkmSqu3gYfqy fy3p0Wf6ImQXuF4nhGS3QB6Xl0VmfoOo2bGiFmjgcuRIgwpJRhZOhXGkb3AlM9B4us96v1kEBje1 BDGGOjoOHuo9FCj1kZDgHo6bN_2BzsqJDVchbNWUkSLx4QoiksRbWNR36v4mkzZqC99Xosb0Gy6b I8KO7lRkL5cUyrDhr.Wn_sUBVT.tYkfdVJ3syWIsYyYj2cTLyZydOdIrgvjNqGGUiAgd03d3huwt XMumSHizzBkCgvYqAM1OxF8odSuC1dg1ZEv_Zr.qzSHsmQ4OkC_AhdBkocT03g9Ee0ewzZi55BH2 mySnWEhMy_ADFNPirO5SZ.YcZNTHfcGXhpbErUxOoFrN3.Hxyxglz47o3ZkrtqNKt6_HrRwDbwti b6UOdkeKL4KoQ0bcMctIZM2iIxeJoNRtEDPRdOjv1wGb6o7XqWVx1IutzA0uJRw297kZSWX6.82i uh1snz2JK0GAAH7W5uN2W1TUH24QfGzOhJfCcwWMqokUuRcA_mK6PX.fzZvJY_A6zvNxbP7XOlh6 QPg00hk_Bxwh5.QOzuNpNFSXSkpMaautaGGviuElaOyrR3YrdmRVUixd1NrvMo10jtvDhbNpN1Rq 7mE0RWk7QczEaY0G2Bz_7iXJWmCnG8cZhDXCEOEHMWK7Q5BB3AdpIVi9b4U2B0b0lGZTtTUjSSZz C_qRRHL9PvWqr9P.Xe.OdKXfjWD8gVLmLagRHVbNx8Rgf_XRrOa82fHNZ2BiWem584rtALd1uBuj c7GKZ19o_aapG8hON_07DBSmHsREDaDPVGlta6A9Lj0HMv3VY.epKtvLHqkRrE2UPouJ5bp8odbg AC_7FqbiYhENL6K7CIVF5mFX57ffuCf5owdPm6VaNHydxZQk1QKxAb3KqjA6w3LVmKlnJR2lN.B7 HruIW2iJDSA_5Wxgp.NuAR4ioSUZqRXUcZdqEP9qPXnwFHIp_YCSseTt5KG7zow_OyrPg5CGKlPX rCQYK2PNvm4LoTGqXJ5N0asGwv.GowK8.qBDVZD5mEnuCat1X6FdEE6L5afRhGXDaAUaLbZuZAm4 JU2Iib5xHIQg1YnfcFaEgnXn_0eV2Wo63bs6T1memfpnG2wC0whCoxLTb22O70ps.ic1I2Anzrq9 Dra2Sgmcji_Z7Ni5PzsnJ_enHQcOZHJ4O.OIcF82QM0lxPS4IRZQEU1KzKaEDGaUT8bzPd7nLc2J xYKyGuP3M7Pn37Nx3S_HpQlBn93xP3DgF52V9yUtx7TXaWBFkSUeeH3WWUyFCtK5DHD7J48qplNV wETC8_mipmUa1F18cepvQh2GiqePtTNHRY0tbrVke5B7Y.nPtWnVVOpdJhmzhKjZfIhzHI8.bHgb M22lXZkguZRsOH9E06nd8hNEC3yFhnZqXc4rmNMGTc2lwnSBq16ebU9WqouZwkOd6YNt92HXsmj7 WsTit6IGQ5LvGKwO1SO4CkXnAYjXsyUvKkq5ceJLcf3FqXzDH2JyIGKGeKo4qq_K.411yp3ZcVC6 QM_o- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Mon, 31 Oct 2022 04:00:33 +0000 Received: by hermes--production-ne1-c47ffd5f5-cr29n (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID fea0a44a9000fcdcbb4fde1f4cb12275; Mon, 31 Oct 2022 04:00:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: 14.0-CURRENT failed to reclaim memory error in RPi 3B build From: Mark Millard In-Reply-To: Date: Sun, 30 Oct 2022 21:00:27 -0700 Cc: freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: References: <84376CC9-B991-4BF3-AF5F-0AA09CB28339.ref@yahoo.com> <84376CC9-B991-4BF3-AF5F-0AA09CB28339@yahoo.com> To: Archimedes Gaviola X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Rspamd-Queue-Id: 4N0zvm1HGdz3fxL X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=TBYbEUnn; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.84:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N On 2022-Oct-30, at 19:47, Archimedes Gaviola = wrote: > On Mon, Oct 31, 2022 at 1:29 AM Mark Millard = wrote: > Archimedes Gaviola wrote on > Date: Sun, 30 Oct 2022 13:41:52 UTC : >=20 > > I am building a kernel and world in 14.0-CURRENT > > = https://download.freebsd.org/ftp/snapshots/arm64/aarch64/ISO-IMAGES/14.0/F= reeBSD-14.0-CURRENT-arm64-aarch64-RPI-20221027-769b884e2e2-258837.img.xz > > with Raspberry Pi 3B (ARM kernel config file and in default system > > configurations) and compilation breaks due to "failed to reclaim = memory" > > error as found in the dmesg. > >=20 > > pid 91224 (llvm-tblgen), jid 0, uid 0, was killed: failed to reclaim = memory > > pid 91131 (make), jid 0, uid 0, was killed: failed to reclaim memory > >=20 > > Here's the set of the build commands I invoked. > >=20 > > root@generic# cd /usr/src ; make KERNCONF=3DARM TARGET_ARCH=3Daarch64 > > buildkernel buildworld installkernel installworld distribution > > DESTDIR=3D/home/freebsd/rpi3b > >=20 > > . . . > >=20 > > Any thoughts? As I don't have any idea about VM pageout. >=20 > Hi Mark, > =20 >=20 > Multiple configuration things from what I use: >=20 > I use a swap partition (not a swap file!) to give the system > someplace to put copies of inactive memory pages (paging): >=20 > # swapinfo > Device 1K-blocks Used Avail Capacity > /dev/gpt/Rock64swp2 3670016 0 3670016 0% >=20 > Oh I see, there's no swap partition in the default installation. >=20 > root@generic:~ # swapinfo > Device 1K-blocks Used Avail Capacity >=20 > root@generic:~ # top -S > last pid: 92429; load averages: 0.00, 0.00, 0.00 = up 1+12:49:11 23:41:10 > 52 processes: 2 running, 48 sleeping, 2 waiting > CPU: 0.0% user, 0.0% nice, 0.6% system, 0.0% interrupt, 99.4% idle > Mem: 2000K Active, 626M Inact, 223M Wired, 97M Buf, 20M Free >=20 > Let me try to create a swap partition. Let me mount a spare USB flash = drive for swap as during the installation all my microSD card storage = was allocated in the root filesystem with growfs. =20 > =20 >=20 > where gpart show -p lists it as (a gpt context, not MBR): >=20 > 534528 7340032 da0p2 freebsd-swap (3.5G) >=20 > and gpart show -pl lists it as: >=20 > 534528 7340032 da0p2 Rock64swp2 (3.5G) >=20 > Okay noted on GPT not MBR method with gpart. I did not happen to have a MBR example around. So I could only show GPT. The note was more to avoid confusion than anything, since the two are not equivalent for how they work. > By the way, what's the proper allocation size of swap in FreeBSD? FreeBSD has a waring that it produces indicating possible mistuning when you potentially have too much. An example is: warning: total configured swap (2097152 pages) exceeds maximum = recommended amount (916632 pages). warning: increase kern.maxswzone or reduce amount of swap. The numbers are dependent on the amount of RAM present and other details. My understanding is that increasing kern.maxswzone has tradeoffs. I avoid getting the message because I do not understand the tradeoffs or how to manage the tradeoffs or even how to identify an instance of hitting such a tradeoff. For aarch64 I've been about to have swap of about 3.4 to 3.5 or so times the amount of RAM without getting the warnings. That is why 3.5G in my RPi3B example. (So RAM+SWAP approx.=3D 4.5*RAM.) (armv7 only allows more like 1.8 times the RAM before getting the warning.) I avoid even getting too close to the warning as there seems to be some build-to-build variability in what fits vs. not. This avoids having to frequently adjust the size. Going from the other side, how much RAM+SWAP will your activities use? To avoid accurately figuring out such, you may just want to have near the 3.4 to 3.5 times RAM. (There have been times when clang had memory use oddities that required more than normal for a time, for example.) > This RPi 3B has 1GB of RAM (~947 MB), do I need to set twice the = capacity of this physical RAM? Ultimately your choice. How much parallel activity you want to attempt likely contributes. If you build ports, you might do so in a way that uses more RAM+SWAP than system builds do, for example. > (Note: swap file usage is subject to deadlock conditions > avoided by use of swap partitions.) >=20 > This is noted. > =20 >=20 > I use a serial console & ssh session only context to avoid > having sizable competition for RAM. >=20 > I avoid using tmpfs because it competes for RAM use. >=20 > I use the likes of ( in, say, /boot/loader/conf ): >=20 > # > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 >=20 > This delays potential "killed: failed to reclaim memory" kills, > possibly long enough to reach a state where sufficient memory is > reclaimed. >=20 > Alright this is well noted too. There is tuning related to "a thread waited too long to allocate a page" that happens because of paging I/O characteristics. But but I've not hit that type of error. I'll also note that the "out of swap space" case is a misnomer in that it is one or two of 2 internal data structures that is out of space, not necessarily the swap space on the media. Again, I've not ever hit that type of error. I'm not aware of tuning for this case. > I'll note that the status "killed: failed to reclaim memory" does > not require that swap be used much at all. Sustained low free RAM > from just one process that always stays runnable and has a > sufficiently large active set of pages can be sufficient to end up > with such kills. Having swap allows for inactive pages to get out > of the way, which can help. >=20 > I use the likes of ( in, say, /etc/ssyctl.conf ): >=20 > # > # Together this pair avoids swapping out the process kernel stacks. > # This avoids processes for interacting with the system from being > # hung-up. > vm.swap_enabled=3D0 > vm.swap_idle_enabled=3D0 >=20 > This allows paging to the swap space but disallows moving > kernel thread stacks to the swap space. Otherwise the > processes used to interact with the RPi3 can become > non-runnable, preventing such interactions. >=20 > Okay this too is well noted. > =20 >=20 > I have NVMe or SSD based USB media, not microsd cards nor > spinning rust. (I use just bootcode.bin and timeout files > on microsd media for the RPi3B. Even the rest of the RPi* > firmware is on the USB media, as well as u-boot.bin .) This may contribute to why I've never gotten a "a thread waited too long to allocate a page" on any system. (Some systems, while bootable via USB3 media I have, also have have even faster internal media that is normally used.) > My usage of such a configuration struture for building > software (world, kernel, ports) applies to all the > systems I do such with, including ones with a lot more > resources, including a lot more RAM. >=20 > Thanks for these inputs, noted on these things! I haven't tried NVMe = and SSD media in my RPi 3B. So, they are far more superior as compared = to microSD cards when it comes to building software? My understanding is that microsd card media is fairly generally not as good for such contexts: slower, fails sooner, etc. I happen to boot multiple types of machines from the same media so I use USB3 media that is compatible with USB2 use, a single such USB3 device not needing a powered hub for use on the likes of an RPi3B. (Lots of USB3 media around would require external power for USB2 or an RPi3B use.) I need a powered hub for 2 or more such media on a RPi3B. =3D=3D=3D Mark Millard marklmi at yahoo.com