From owner-freebsd-arm@freebsd.org Thu Mar 26 23:25:28 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 35C6F264775 for ; Thu, 26 Mar 2020 23:25:28 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48pLgX6JQYz4CvB for ; Thu, 26 Mar 2020 23:25:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: bf4m7QcVM1nsSwiCiAXmII897i9fhaTM5XNG6RmG3zbKvw1QVR9i7t3BnQEb6op Qf5aVOleBoh5k_I9xVYYKpU6wAwPg0na8j9efst97bgITNX95nf5mVrqLFGgjVz6cYcP6mMLu6D_ jZR_NSlfQ2.2PWFA6k8LlZOaLA4WQ1AT_EB_rpNAdJB9gMJ3_55xVLZdV8ee5Ie2ld2qSieow4u8 NRgwoQScajTQHrSadPJAYCd4ZsS85q1RtwNRAbv2arvBMIK_Wq8hKuj0iGCQN1cdi1x3gxxbnHKT 99BfgdjHpdaFkhBW0kru5dCgbsrfQErdqiC0uphC40FX_FsDVMa8LO6_d8TpedjAnL3ywWCJImg1 0ZVnJDY5h4eixfSDq4qfhaoDTDuNYqKL1eaC4R8LoupHg9NRNAl5hbMn0tmgJQiuaf9U7a7oZ0P_ 3CadpbJvdo7EteuHUSWr2Lbz_TUqFMZesVGhwW5ADskya6JXLbTou_ad1.5uKO44PDNBpYjtzT0R 3juh43gf9WyoyZqTSp0_GNICidWWw4wm15SSrcrnrbpm.tVoK.IluMlE3zEMj8flZ9TfRoPkAuM7 HAhWBmmRzOPj3sgzEUMhnO24KMNfPqn25UlmxWMFza6HM9FeU.mlom_o7fzqkMU4314Xb50SgIxs 5sru0Tova8inFvDvvplJgB2p4xVHrEZz08ubhXbPzUnIBcQeo75dKWv4YJZiq46fq7tgy1Unx42V 36k0FFx4q3h_pDwoDh0V9lxmirwL__5_w2ChuJmgBeH.7ImFEUoYPJ4ayNZZpLQtPNM_FWCPtaCO aEGkBI15XpmR1KZ5zNSc7VuGio3uijAIsVs9lX5F8FZo1ZAZxa32RzZth8SswAY_3pQUN.EPfdm7 P25BfJ.4HX8cr2pXIO4ngveXCOWJz8VEw7LuAYp7gIr3JBtv95_hYmC3xIQ3ZbgfclWlmdzbDCaT NJ7AeR9tuQjCGh8Cc.164eMSpm.O6UDrMPKYjobr11FTDPcwzwgTuhlLSmKAWA6UAejsHeQBN5lF QC1WNEiE9LdjkA6bbxa4AKLAMT8CJhJsvLx48dgcLRtSgVMSYjz3J9X6J538kwHr.iARyAvgFu2f .IdttVMA69s.s6.Sk_1BUPI.3_wq3TEzScULhtHnSvdexxoknGgGJl1LdQzqhMr_by2K8P..TGr1 qzVqxUAwga0cFQth6EuA5t8rBJYj_fgquBAe.u1EJBOXI3kL7hB7_hNrJguT.0RkEqUIz4tjzqmG oouSje0NiwbQaSlI3RsCfVcQcBmr6yqBUj5ODEJH7GPIG3yGNzb4Sf6M_wVvpSYhy8XFIca38KBp 3MCUbXb0RM44lqgt4ezt5aAJhiVDCGkBPP1.4xuqcWr.fjxV5S6bRDpdwAKjp0Elqc7TUI3RO3Oa Ec1Lxcxf5DpzXAH.aAhU- Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Thu, 26 Mar 2020 23:25:02 +0000 Received: by smtp412.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID c6d89609fe3116e518e2388efb074be5; Thu, 26 Mar 2020 23:24:59 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Belated out of swap kill on rpi3 at r359216 From: Mark Millard In-Reply-To: <20200326220649.GA99824@www.zefox.net> Date: Thu, 26 Mar 2020 16:24:57 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <0A8CF8D1-8D0F-40E2-A10D-EB44BEEAB557@yahoo.com> References: <20200324155753.GA91922@www.zefox.net> <83E41A13-6C24-4B56-A837-779044038FBC@yahoo.com> <20200324185518.GA92311@www.zefox.net> <75CE3C07-8A0A-4D32-84C6-24BEA967447E@yahoo.com> <20200324224658.GA92726@www.zefox.net> <764D5A86-6A42-44E0-A706-F1C49BB198DA@yahoo.com> <20200325015633.GA93057@www.zefox.net> <0FF6BC4C-296F-49F3-8FB8-AA87A49349E2@yahoo.com> <20200326220649.GA99824@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 48pLgX6JQYz4CvB X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.46 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.961,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (0.87), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[206.68.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[206.68.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 23:25:28 -0000 On 2020-Mar-26, at 15:06, bob prohaska wrote: > Just to wrap up, I tend to agree that > delays writing to the microSD filesystem were > blocking swap traffic and causing OOM kills. > Turning off OOM allowed the OS build/install > to complete successfully. There are still 2 types of "uma zone exhausted" issues that can lead to OOM activity, as well as vm.pageout_oom_seq being more of a "needs to be sufficiently large" than a direct-disable of the OOM handling of sustained-low-free-RAM periods. (And, only trying examples establishes what figures are sufficient for specific activities.) > Why this behavior started recently is less clear. > The card was placed in service in October of 2018 and > has been in strenuous use since then. The first hints > of trouble occurred in late 2019. Perhaps this phenomenon > is a useful warning of impending wearout. I'm not sure of the relative timing, but vm.pfault_oom_attempts and vm.pfault_oom_wait are fairly new and were (are still?) specific to head at 13. I expect they were in place for a while before we learned of them and what to do with them ( and so started using vm.pfault_oom_attempts=-1 ). This is sort of like vm.pageout_oom_seq being around for a long time before I learned of it and the background to understand using it: Mark J.'s earlier material from an earlier round of finding why OOM activity was happening on small arm boards. > The gstat logs are at > http://www.zefox.net/~fbsd/rpi3/swaptests/r359216/ > in case anybody's curious. > One of the unfortunate things is that having the logging of gstat and such also go to the same media (or even over a common channel but different media) also adds to the competing I/O load for paging/swapping. Also, if I remember right, all USB ports on the RPi3 share a common channel in the path (internal USB hub), limiting the utility of using multiple USB drives for helping manage these issues. It might be that one USB drive and one microsd card are as far as one can go for independent channels (ignoring WiFi and such). Microsd cards have issues of their own when involved, however. Side note: I've got access to the old RPi3 because the Pine64+2GB is having problems with I/O failures to longterm media (microsd cards, USB media directly attached, USB media via a powered hub). It might go a week without failure but when it does it tends to be a large sequence of failures. Various separate media all got such a problem eventually. I'm trying to see if the RPi3 also gets such issues with some of the same media the Pine64+2GB did. Anyway, I may, for a time, have one context that is more like yours than is normal for me. As stands, the RPi3 is doing a from-scratch buildworld buildkernel . (Reconstructing the head -r358966 that it is already running.) It is not splitting the I/O load but is using a USB SSD (via a powered hub), not the microsd card. No extra logging. vm.pfault_oom_attempts=-1 and vm.pageout_oom_seq=120 for this attempt. 3072 MiBytes of page/swap space. It is a -j4 build attempt. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)