From nobody Sun May 1 21:53:32 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6E7B11AB0418 for ; Sun, 1 May 2022 21:53:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-55.consmr.mail.gq1.yahoo.com (sonic308-55.consmr.mail.gq1.yahoo.com [98.137.68.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ks0NT0yMvz4cWb for ; Sun, 1 May 2022 21:53:45 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1651442017; bh=2gChDu23QKAmX5medQOFevaN8vzJWF0pgdH7yHVjB9A=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=kcrYaEZy+PVbU4+B2QilqukkFPRAx27qiuFFXhFux9t+mjJGnVu6sJ8xX87/H049vxfzgA+k5r1GkMW8HqFpOE+rJ/HV48QWGciSNIZ7i7Av+YN1J5x1IQWpopIJkehDJEn2qrdtC+vGKksrGox4p2yEK6D3Lpp5k/s1LSXRqSPNlbewjEs3i/ApMy+6xZ3uJ+R/kCByLIy5dERiLVzW0ljdp0s0UNzXJjdh0D4wwwBV3uKf1FIFr9q5iI57eTssTT8xEeSveiNHLqC9/xhovIB0T+evqofrF1/DTgdv+fzBSY4N6yPgQm9sNrztv8AoHEDC3ze/QH8TlCmWWyAafQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1651442017; bh=AvxxmfeHWM865Yiact5XmzLDnEcl/rk3+TverUkl6iu=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=ZyHCMNpwdedG/UCWbu//anEmVO0dsjanFykF8m19IA6VFTlEf1C3rmm2NRfH9G47T2PyhwzaVU8Ufk2o+tWfN18iPr1T7242IcawKe+wpfWBzb0CxpsbrBbOhVq1ch4QBZD9/28us8h5hvkAP/TkYwZs0zsf5mHyLfpvji97LdHE/Q39VQq1Hi0QWoQ3h2VKKBjsqwlZeH0BKg0fVSzS1tyHs6D69P3Jcc3X/PweDS1zUnEH/zAsqlmYlSzb3gQLj/s9A36VVT1xtdPqpk/Yq+/XYY5G1ok0RLmW+Drf8BYfpmC/EuHVIBuDOCynMyQd/71y0IGbhRiEWWbsqar/CQ== X-YMail-OSG: _6ugAxgVM1nijpPixr09OdoCQFL.gf.Mt0FAYbNMY9PNZn.D.KCdSAM7wq4X8kF sDKtgNoY5NL.VjH9wpXcXiLwg8q5XoIi4ozFDBFVqWUTgkdwM5OxinhL5VvKDtLBmVndtufGFc68 CYUnr_Q_ctOtDdd.M1KbQyC17TR51YSV22lRujh9IDxBNJPtZehnIwW21t_DfHEHdhMCzv_nvu7k ZIu8_P3xHKPFZdbjc0zCTWYwxkhOmz0wYV_.tLuuwGSUbucJbMF.5RsIYQ1YIHcCyLy0Sazz_hO_ MU1E5j7mcdMp37CPkPom5.knRmQjaQQz29szVNeFL.ofoEzegVbaoY9MgPMBV.surjNTwJcPH_Qv zpRYZKiJn4wmV4m9j92icagZYMiMrTLaCF9ozXgRMCUKfQmz_SdvbFoy.5pnUbNg9IbxP9FeFpeo yc0ti9XMP4LALW0TtR.TNuJeKyLdLtBk6Bs.Xc86nv3Qr5nFXIeuFQOBAjyF24RpS44mIPGHqLYD szltCTbygzn1pFHLpUF4IPv.tgU2PCn2ebbKhSQbKgHixjFS.JslYI7xWd3L3EDhv0bywKZs60L0 q_HNuOQ1GHtaBPZ2yMxOwxZLfIOCpT1jCTGIqZxbdbpgGM97pqF8_CJZCLCbpU74ncdQ3TYCDItu HK31ZrCfI8IQWs0VY19IEG0agzt0fLOdGQNsrgTQoZVSJJquKF9KdbBXBBVR2l9xFx_0U6oXiFGn cz4kbk3TGEPilK0Hx48hLbxUVuRoLh8rySyMSJPZG.HP4gMofStI5SAHyBxsW564qqLwf6_UA.6c jnVrY0rVQ9Fhb9dZrfP09eBLdDjCCeJ.91BFSKYBaKUBs2QBNaV3pVBa6W1iGxeWeyi97_GCXckU bK3mEWQU3b6Eq3sVmId_b4tCd7LtVK86zLlG1vfkheHb4XMo7Db2w8B5RJ1Mo_yPD9rRxcTUBCVD sesXJQucL0hs24Kd7evk4t9.0GyiNqMd8t_CgKPX1QD9l0aeBPUtaCLeAmdiG0Of7sIByIKKyuZF fhHvHKUcje1Coo9wy7EJ.9IPomB.I0S3cfWFhDCoddfpYtzgPI4cHJ1fpKLlChNVQZiFTCtR._cS f9C9S5co9bBpYhfvxbMFyQ2DzW7fjMzgeOGto.Dgs5SYcNCeaE4DO8d6tYPDKxlwpeTyIwgXC4eI Lj1OAXhe3636MWAO4.xEms933_ckc80xZdH4maGzGBJp21EJdZF2MTactgvuO442SaGxZy.MreYw MWzPVQLinkVvPwIbXGmaAJiRUuVkjJbmYs36pCe3m9KzfVIzc3Od3YLVysZDxjfupxj4sMosPeKz kEBTRmKzEFQFb4.4W2Kl2bIPXwNTzx7KNKkmcXUWVr0.KPQ0wuyKEPueHLiaDluj.Hz4lJII43Sc wJJgXlCMQU7KwfqlZWtkBUE7QX3LRLGA6qHwf5Df8xDV8iITa6ZoBLuU6m6UznDFRVFQ7UFTZ2V. ayao1j9rCaSCZUN_.7HH.hX8BqpTQety4SGH8Oo3LWhE4peeN2kkwjRRWQUmr_dKcrJ.8bbeMcyi Dz6KZj4OguLrUjESfIdi0K5MkGgz7fvbbs54pcqpJZKZiAuQTZXphWChDDXsUnwGbTYnyHs_tnhq aWM89tzwuVf1zRyAYt8UsMlOwimre6Nc1zjLH5ecox_Uye_rqk5m_C78RaJB1JTZcHuhjM.SHtJL skvXvnq7C2cu6zDvYhXKDJ8205wuF1Z5rP2DT2ncIQhFeFPRh49NjzMjjaFvEc5_cpfh09VuXwhC Oya7dF.Z25qaFwpytYe38QWlrJPfN3XAbwAJP2z0JWkVcTBO3zHY4vDhj4CaMPXCCgxY6nix74RM zQoMso_nwCmHBoH3mxIAdgiA4VNWIndLuhL42eGX6ZtTu0HITxHhlowyb1IUQHwp8.eFuA2DIvTK HWADjk.9tJT_BR_geEX4TXCxTWmAUEuco8f0LSHta9huNqbWhltFGmdkewl4MNzZ5Xl_PbHjzpOl E.naiW47CqY0.40ree6F6hmVeqx_m3x2M5q1ta3D0AbwpOHMua8nipauh65f.lYEdY5Hdy_qwXcb u38GpqJ5ZXkvmjK3srSwfhjBx5lrnhaJT89zJP5Algb94g.Rxnq25PfZOHQespY._XRpZmkR7l8Y .XbnsVAMfOirDKMl6b84UFo_6S4GY.P5Gyz9LEFnHIj4eKH.kN2Np5Oa9_PK6QLuw9KEuIolh.I9 Nt_1UCFBUNRkmlN20e6dVLPxOMGTNRzfB1JeC6zU97U5nTSwqIgiMtHeU1EIUc5hdUBfdE9Trxh9 w33tIsRE1uT0w9w-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Sun, 1 May 2022 21:53:37 +0000 Received: by hermes--canary-production-ne1-75b69fcf97-8xk25 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 62025c63a2cfcb187d5fa331aebad27a; Sun, 01 May 2022 21:53:34 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Cross-compile worked, cross-install not so much ... Message-Id: Date: Sun, 1 May 2022 14:53:32 -0700 To: pmh@hausen.com, freebsd-current X-Mailer: Apple Mail (2.3654.120.0.1.13) References: X-Rspamd-Queue-Id: 4Ks0NT0yMvz4cWb X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=kcrYaEZy; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.31:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MLMMJ_DEST(0.00)[freebsd-current]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-ThisMailContainsUnwantedMimeParts: N Patrick M. Hausen wrote on Date: Sun, 1 May 2022 17:29:27 +0200 : > > Am 26.04.2022 um 17:47 schrieb bob prohaska : > > If the result is unsatisfactory, self-hosting isn't impossible. I've been > > doing it for a few years now, albeit with much help from the list. On a > > Pi3 running aarch64 memory and swap are a constraint. I'd suggest 4 GB > > of swap and -j2 or -j3, perhaps increasing to -j4 as you see how things > > go. If you can split the swap across devices it helps some. Useful > > /boot/loader.conf tweaks include > > > > vm.pageout_oom_seq="4096" > > vm.pfault_oom_attempts="120" > > vm.pfault_oom_wait="20" > > > > Mark Millard made me aware of these parameters over the list. And going back farther: Mark Johnston made us both aware of vm.pageout_oom_seq and its use back when you were having problems with the system killing processes. He had provided some investigative patches that we used as well. THat is part of how Mark J. determiened that vm.pageout_oom_seq use was appropraite. Konstantin Belousov als corrected my mistaken mental model relative to FreeBSD swapping/paging back in that time frame --and that feeds into what vm.pageout_oom_seq controls. In recent enough 13.1-??? , stable/13 , and main : Console messages with: was killed: failed to reclaim memory are tied to what vm.pageout_oom_seq controls: the number of tries to reclaim the targeted amount of free RAM before initiating kills to do so. Console messages with: was killed: a thread waited too long to allocate a page are tied to the combination of vm.pfault_oom_attempts and vm.pfault_oom_wait used, which, together result in an overall time frame (multiply the two) before such kills start. There can be messages with: was killed: out of swap space which is somewhat of a misnomer: the out-of-space is actually in one or both of a couple of related kernel data structures for managing the swap space, not the swap partition content itself. As near as I can tell, this type of failure is rare. But I'll note that in FreeBSD versions before the messages were added for "failed to reclaim memory" and "waited too long to allocate a page", the messaging always said words about "out of swap space" for all 3 types of contexts: rather misleading. > without any additional tuning but with an SSD connected via USB > and 4GB swap on that I was able to compile with -j4 and a mostly CPU > bound system. > > -------------------------------------------------------------- > >>> World build completed on Thu Apr 28 10:30:53 CEST 2022 > >>> World built in 155832 seconds, ncpu: 4, make -j4 > -------------------------------------------------------------- > -------------------------------------------------------------- > >>> Kernel build for GENERIC completed on Thu Apr 28 13:11:37 CEST 2022 > >>> Kernel(s) GENERIC built in 9643 seconds, ncpu: 4, make -j4 > -------------------------------------------------------------- So: a little under 46 hours, if I calculated right. I'll note that in some past experiments on some types of RPi*'s, using -j3 actually took less overall time than -j4 when deliberately repeating from-scratch builds. The differences were not all that large as I remember. But, if -j3 and -j4 end up with even similar time frames, then -j3 has the additional advantage in limited-RAM contexts of not being as likely to have resource problems. So -j3 could be appropriate. I do not know if -j3 would work out better for you or not. I'm just noting that it may be worth experimenting with. > Thanks everyone for your valuable hints. Guess I will subscribe to > -arm, since there are some more rough edges compared to "just put a > Debian or Ubuntu image on it". > > And then I wonder what workload I can put on a seven-node FreeBSD > cluster, since it won't be k8s, obviously. Let's start with Ceph, I guess. === Mark Millard marklmi at yahoo.com