From nobody Mon May 15 03:12:23 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKPZ03CPVz4B2fr for ; Mon, 15 May 2023 03:12:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKPYz74b9z4KXy for ; Mon, 15 May 2023 03:12:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684120358; bh=CWN9ovSmaqM9MGi0EyYFhRF8s/X7bIhEBBI5UaujYdo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=j766NwtnjOZibyn0C/1rQ5ELhiKktWeodwRSk9JeC1QPJ6Z9MKfDY8HG8vqeWpRWUgMJ46uzW3Z+0Q6/ByvKSGTz46qm+iwg5HL9yMuWFad7Gu3m0hSp3pjDdOCRNiUOaoUbQL5ez3HYD9u2Ow0BHP5/BRVMMILAucSbb9zO6L76gGi0RMq9MUaWMHhMqBkTX5nXotvEeHL9F3iPT3zBnqQjxI2pjYFM3RbgT6SvX/JGfxB7OGjgF5qECo0x0dWt1XiU32+/ujQae+evWXRwXDR3LE8j4x+7mmCEYx+cpF9IfFpVdeRcSmP5j/zaiWuyVRD7JEnHJVov6tFxwiU/PQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684120358; bh=NMvwyKkeh7p6Cig2/laB65rz1QyA4WCRDMsXpwOttvX=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=hyIS56nzYDtDD1S+Ub6lErR6iXo5zxqd6iwLYu8n4EFwoc29iJJVZY9uSFcx/PPeEGqaJwpfT7WNkiMRQXuUlvGibh8k/fyq/TFlJQvo9ZKN0nLW+t7hGZOBBiqDKkO9jEU5TPD1IBkjvTRvoivBNNj5xBBDKpomd7PGEabX9L52XSixj5IaImTSY9riDYcy7oWMIqPq37STdfpZAfDrBq7Qxv8+rhgETpV+mI02txFdASrBSQk8eYQqvWpO5QJ0Rwy81ELyfc/eKl/pRSdt2X5fhszUphkuDuI7WcsKx2VoJSp8SKqbF+B7MTfbCCL3/q8TSqNf3dPq7qNo2xj3Zw== X-YMail-OSG: sYap_UcVM1n8qzvrn1tLD8LiLLUPzUIQ0PQBOiVvHU3SKXv19y7wc2NcrMDCugT Sv4nhdcBzJTnEuP_sroxHDd4NIyXRAhL6HH5Mi5g7ITJLqv_ujhDZf_YWqiWLuJRADJo8vD4WQo9 oz1hOVTnefC2.jxhbOn7DKQFbN1e6Gb9RT7FU071jiDD9I_KW7ewzqAVs9ejm_hoSOf95sCt0a0Z a44pIlm5xt3ttYuBraes462D6tHP8.wSYtX5icYLw6AuQMnfg_BPQaqy2PcIgGkgnMver20PWEWn ZUJzni4zOMZR_LC39YFGgSZvdL4dbeLqXH3qK3vi1RNeOjnNouuif3reOYPtP8bn1mvSij5s2tNS W6Ko1IZjqzHsOnkJLjECCKcxf_AYj7I4aJEzUFuOHz.U4Cw4UD3_EriPIYtyyn64FxZvVt9Ay_bv zdtsR8AYG88AUyOFboztfAIpn3XTPoMMq52ASlfVdkE04O5XA00m1m74WeaPNOPHWh6hBggTdXNb zTbo5HujXtEFpxvJeGX4Dj0P4irQ6d_iWpt87kBsXb9eExS.c7xg0Im1tBkxGE5RMrmdAagYv.Az Cbt.QRLydehSVPgUxVCoe5ARbqM7NM6YJ5Lhn73naTH0jnVkDigzVtvYvwrhcYtsr8t6KJ7EPhl_ Csri5M.b9WRfPIDKZAvBo449P7UT_J.7zz8tjCt_H2rnREc9TtMj6Zlj8CKIHSo2ipFkKEyzf5WB n4tE6InABrVYTI_Nj1mLTtgEWUN6G3lyDugQZIvyQGE6JvpDt5cQcRqzPG3QyELXj0rL0zF0M7NB .hLWYP8O0o3FGU4AIHBduALR6XsQ6nrV_QU3WVtNX8gKikjbLQL5xqnXDFmcJoo7I0QFXrJJIz.z nW7Xlkzdox_PDTSe57dq1W1J62N_Z.5dYEXlIO0_MrhAjR9n.mAcOf8ByFPfanBZ7VxjjthRnOF4 U7TbRle0Eg9bHpxwi6lJGEDfKAGvnYXC7tnZGNHUk3u4_ZTrZBa1so0K2lJlV6UmnkpbdLZ3vbPq Ds8puCmGj_XjY7.QG.F_99Rp29smqITXdV6sOxHLcC6I4K4xcy7xQmJ87tO274nbXPP101XzTJDt v7x.9H_r1a_RylSYjvDrH2Qo8VunTei.2KQ2MPB5o_xw16Wlt963pWYV7OVI564sAx9SDNDG4FmL QA1PqjzDEZGOW0KnKy6Vwhgn_1u.cqF0ZCkSsgPVsXPX04fW_OT0ibI5s37Tq5hdjlTndSwIKTje LinMEzNbex4Wh5K3PaXJ4NMgRny_SrgIFet_SP38FgD2BbIntjtnJd19C_o8Jajon6KoBleVQ8LL d7TYjAfnicFLu.2f3f4u11ZiCsh5fGXoXPLXHiTRdmHNzQoDsDI82lf_nkJuC8hZF4ExCF.HAccb OH1V_mwkstFos_5fpRnEhI.PAij5xBRMhPtxFEBbW8AtVLRTEYKhAq_oqRpRnuZA5UfTgeT75Fy3 DoDPCa2drg2perxAoZv8Ct0YiDBkSakdMXK0qMM88_1Ebm52ARQSOAUWXay8FSgCnsw9Bw4eAS.c yn4jbPQMf8sj8a6Ju5NwB.MSItiu09oNol93M7iMstVibuf5UOzUYxSTk.5zBEGl8R6V2KTE9cx_ ZvfxX5CBDJTeP2su1bHlOOILK0eSGhSaz0r5lb77y5iVzc8lG5.j3_o0ZHMFVKNYYv9SQi4Q.8qE NGcQNFApmCGoISZPZIIzNQG_8fbepR5Q2PgOIXN9r4HAB9mnWu1ycNYBWlduQJMG8QEQeNrhchzl VsxT87_45UIXLi71e_6_qi1fEgM_HDRfNy4tFB9TgPliY6V..z3vo9_lQ2HCA5V8UtfQhlXBbMRP xqxmtrkKyibMr2VWCJP4WvyuOMtOyKYfqaAWP6wXRcPt5.1d.eJe5ucYvYM1HvJc6LeDZ5oXx459 5LCuNM12ro0EbR1_XF8uPYsNOHqfqC2IS6tSTb.qInxLplx1hi0nUdVmViiXbbOfzTa7avTnblb1 AdZVbhDnxenTfkE9P_jsjwADiD1v7AAhM1AlGBSvkRcsJRms6Xwi4hAU_K1N5WMiBt3bVidGGl.q trhpop8A.uSnrtgMzMYbQtS07.4VsS.Ml07l6vtZ5S8thqr0TZ9UnEXsC18uoG10UCyyDSE3Ol5i wmv8XsPFip16xB8c5U22IF1NNi8Nohck1SqNbwjJ8MbaH2t1W6tK1GRob9H63ND6iZB8Q4XJDAjS 6IUaS2TOBpFwZ.ImYioVfFdag9.rzCeYLFOpO7zGWNVoZUQMLxbg.xNY5JirQAIPeokQNMM.RHfb g8w-- X-Sonic-MF: X-Sonic-ID: 24c8c4b8-d6dd-4b22-8eed-f3dcb4b601b9 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Mon, 15 May 2023 03:12:38 +0000 Received: by hermes--production-bf1-54475bbfff-xzdff (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 665d70e0029b1916ece20d4fe77961f0; Mon, 15 May 2023 03:12:36 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Armv7 (rpi2) getting stuck in buildworld for -current From: Mark Millard In-Reply-To: Date: Sun, 14 May 2023 20:12:23 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <7758885B-3115-47F0-A453-1C010313D4B8@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4QKPYz74b9z4KXy X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On May 14, 2023, at 16:58, bob prohaska wrote: > On Sun, May 14, 2023 at 12:31:29PM -0700, Mark Millard wrote: >>=20 >>=20 >> In my environment, I use /etc/sysctl.conf , which >> is a place appropriate for non-tunable but writable >> sysctl values: >>=20 >> # grep vm.swap_ /etc/sysctl.conf=20 >> vm.swap_enabled=3D0 >> vm.swap_idle_enabled=3D0 >>=20 >> I suggest moving the assignments to /etc/sysctl.conf . >> I expect that this will get rid of your problem once >> you reboot with them in a right place. (You can also >> interactively set them via sysctl use.) >>=20 >=20 > At some point in the past I did that and failed to clean > up /boot/loader.conf . >=20 >> I suggest avoiding confusions by not having copies of >> those 2 lines in /boot/loader.conf (where they will >> not work). >>=20 > I elected to comment the incorrect lines out with a note > indicating why. If I got confused once it may happen again. >=20 > IIRC the lines were added because ssh connections tend to > drop when the system gets busy. That's still happening, so > they're not the cure, or at least not the whole cure. >=20 >>> A running diary of experiments is at >>> http://www.zefox.net/~fbsd/rpi2/crashes/20230514/armv7hang >>=20 >> There you report reducing the swap space partition size. >> Were you getting the message about the swap possibly being >> mistuned prior to that? >>=20 >> For 1 GiByte of RAM 3647M looks to me to likely be a little >> below where that message about mistuning shows up. If you >> were not getting the message, the size should have been >> fine. >>=20 >=20 > The last "too much swap" message I can find was: > warning: total configured swap (1048576 pages) exceeds maximum = recommended amount (922200 pages). > Space was reserved for 4GB of swap, suggesting that only about 1.6 GB = is recommended > if I did the arithmetic right. Resizing the swap partition is easy and = 1 GB should > have been more than enough, but the machine stalled again with 30-odd = MB in use. My screwup: about 3.6*RAM_SIZE is for aarch64, not armv7. armv7 is more like 1.7*RAM_SIZE. For armv7 I've used: # gpart show -pl . . . 534528 3563520 da0p2 BPIM3swap (1.7G) # For 1 GiByte of RAM = RAM . . . 4311040 6291456 da0p3 BPIM3swp2 (3.0G) # For 2 GiByte of RAM = RAM Going in another direction: Note that when top displays something it is showing a point in the past by the time you get to see it. "32M Used" need not be even approximately true at the point of failure. And your first top output shows "358M Used", indicating that it staying small like 32M is not likely over the whole build. > In the distant past armv7 seemed to use little or no swap with a=20 > -j4 buildworld, Not just armv7. > now it seems to require at least some when building=20 > llvm. So far having too much swap hasn't caused visible problems,=20 > but that may have been an artifact of it not being used.=20 >=20 >> In other words, I expect it is appropriate to put back >> the original size (or some approximation of it that >> avoids the message about possibly being mistuned). So much for that claim. Sorry. >> Everything that you reported looks to me to be consistent >> with some kernel stacks having been swapped out for some >> processes/threads that would otherwise be involved in >> interactive I/O activity. >>=20 >=20 > For the moment I've updated /usr/src, set buildworld to -j4 and > am expecting it to hang sometime overnight if the problem is=20 > repeatable. As I write this swap use is pushing 600MB Like the "358M Used", there is plenty of evidence around that expecting a -j4 build to use little swap space for 1 GiByte of RAM is not reasonable for FreeBSD and its use of LLVM (even on/for armv7, as well as the other architectures), going back a fair ways: the status is not a recent change. I'm unsure if you have well avoided having any tmpfs based space or the like that would compete for RAM and use some of the RAM+SWAP. In the low RAM environments, I avoid such competition and use UFS to exclusion. I'll note that causing swap space thrashing can make builds take longer. "Thrashing" is not directly the space used but the frequency/backlog of swap space I/O. I always avoided configurations that thrashed for notable periods of time, via using -j given that I'd already avoied RAM+SWAP competition. But thrashing is also tied to the likes of spinning rust vs. various, for example, NVMe USB media. It is probably generally easier to make spinning rust thrash for notable periods. I'd also avoided spinning rust. > with ~60% > idle time, which is far more than I recall seeing for armv7 in > the past. It's still running, and the scheduler does seem to > find threads to favor. >=20 > The behavior starts to resemble aarch64 on a Pi3 but less extreme. >=20 > For some reason the ssh session controlling buildworld > tends to live longer than an ssh session running a tip connection > to an adjacent Pi's serial console. Since the problem of dropped > ssh connections hasn't been cured by use of > vm.swap_enabled=3D0 > vm.swap_idle_enabled=3D0=20 > perhaps it's best to remove them, for sake of simplicity.=20 >=20 No. Removing them would just mean there would be more ways for you to lose interactive control, including over a serial console without ssh involved if you had such at the time, not just over ssh sessions. I never claimed there was only one cause of control loss. I have claimed that these lines have been used by various folks to avoid one mode of failure. (Some times one is lucky enough to have one access path fail but another still working, such that one can inspect to find out the cause for the failure path was. Such has shown examples of kernel stacks swapped out. Such folks that added the lines cut down the frequency and conditions would lead to lack of access/control.) Separately . . . Your online file report says: QUOTE The disk activity light pulsed steadily, the the time display in top stopped updating and the = system was unresponsive to the enter-tilda-control-B debugger escape.=20 END QUOTE The disk activity light suggests that the system was still doing the build and what you lost was just interactive control and interactive monitoring. If you could tolerate waiting for it without access beyond the activity light, you might have ended up with a completed build. I'll also remind that having one or more logs with an overall high frequency of updates being written to the media adds to the I/O issues. =3D=3D=3D Mark Millard marklmi at yahoo.com