From nobody Sat Apr 23 01:46:56 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1B6041A83DC9 for ; Sat, 23 Apr 2022 01:47:11 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-25.consmr.mail.gq1.yahoo.com (sonic311-25.consmr.mail.gq1.yahoo.com [98.137.65.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KlYzx277fz3rL6 for ; Sat, 23 Apr 2022 01:47:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650678422; bh=BOuDlGBOdTo+iX3nq1ikGs6jHkjqF7p1aMbqWgng6uo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=aFPMnPnbLDNFkmwkpqvfwgz9otpQygrIRoqsydtAprawrpczQj9Khyvq3PhkXvhWw78fSmhYDNX2Mf34b6F32P7SvVHg5fKaD0mUz8g0nyLIQ4OHMCBNdh54ztRDxjQzQpsk0c8VHeUgyzOowR6utyzGDMHiakrkQZBwSfBaiyWLYtdB+39a1zmKdI0L4PEiDmUHQs5cqACXRjgbtx+gBpOqiJonbXZV42+t4qXPVSk+pzMyPY7ZElBrMBnT8Kj6SRrFwXyvmDlyb+p1Cja3O+MOT0mTsABIKXj/dtl8/l3yvSSpdOu3QgI+biuVkyI2dgQiSps5IN52ahFEfJOyHQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650678422; bh=+MNW7omTO82MqqPcz/frMTS3YAHvxpkFqb4+nN0CML7=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=PPgsdfnHrHwZpCbYkIOhA+Eku9vEEB8dylN6RnNYUX4/648b9UH7eALqsDM4bnKw2q+nLxY/hD5/ewYP5aev1Z6mveS2R5Fd/lmeLQw8aCOJeQoDz5DJK3+PIuSGAjdRO7R450QydAskDk1Nv8D4q4gVjPh7qHahdOoVM0tyKUWQjn0yDJbMkNIGtDDGxLaAcfry1fuqR6DvAmJfoIDiUiSowOmUgqb06zzh591lm2HYMOjLltraZTroASVsSYESqmpo3Qw6zrKxjOi+eSkuv0sQfa8hRoHOccm82VQoxSX2aG1gxfiC7u/smDT7zHCVlgvS2+zA+m6xis0Y+IZ2+g== X-YMail-OSG: iuV.Q5oVM1mxBbgdG_etjxJfNZ1lU0mw4t5CffrGqCArd08t.lBDnkBKSZQ5j_Y 6MOdlRRVN8VcEI8pNS01CJ8Xjq_DvlmnPxqMV64zbObahNhN3cs0xMawKtkXvZcc62qMcW27WaVK qUSaBLdAijj.lCSYOXK11DySgOFGpTLpwt8eoTfEpNIAMcXKmjjw65ALse3t7Gr4SlmPGcDtonIP M.DeWnoBnFzwMsXw7Mo3B4z_Roig_4Hiy2dZOxT4TYmAbCnE4MAGxv.FIrQu5HsjJJJkNPWAoXoF L0MfGPU5zwCoQ9q06hCLqamafDDYbnXpNhvEDdDU6EScKTEjLuwKlKiMv_ei2xyRXZfMzL3fcu7J FgsJ94X1Bd7zGv.iMEfaxQyOO37WrR.MaWdVMWcWHOl0MHbK4sQlbmstEP9LywJkTJHVK6eHzLc5 PRf6MheWeozJOYo6lhWMl1.FpX8cLkK9crLvZo6QwToTERMPktnckD97epfkOJKCKnhQzJu8eXl6 kzinHvujDRYUfmApkoB98SeqWUtYKz_hpwMAyc91lfzqelt.OyblRcNurcEsxpQSsCh.Gdt_w8sK pbKhao8nFJRG17w6QmLOmzML9oLYWN48W5RTkN6JYX_2P7sQH5HS2iNUO4XfhBWmXPCa4XiN4_Lf AUbyfwYA7OyuV4nNH1uG1KgEw6beEthJ6rlBZHPkbwYPtJ3YwRw_kitHlMVRGv7wkyRJqzSeQrzk Pdb5N6_C1jV37VdvbcJPv5FfqoPFxCK7IBg3xl7CHSQy7Fn3lC069.eRWrUxNLnAlSy1kpqQ7vuM WqRKmd_NFM9YaTvZhfp5pSSxYKO5YYHRtsdlynFEWmqWo_Ga88c.wGXIWJcrGKwc94EBS.m76BSu vBrYdbTfLl9qgkaAw4n2sQ.jZZgTbxrSbgovIicrj4omOlfjOCEnX6wDLcuTNU.6FHDJAHMvClBJ t0hkF78GA7sJ3GKgkiTjCHX_zJ2Q34MfR_24cNp4zmLUFyXVBry4D2dPUJ57.OLbhwDfuyHZ8tj6 oRVPBvu1QF6R3wDdAGOJ6W8iwr6thL2oFikpChDj2H_wAfWJwWn1X0KOFYkkENH8Yf7abC09QREP qiJhj8NJreoZkkLgi_hsS5jzaFXCvX4dbOMfcyoG9a3IbRKrpfZyu9f0NcqNOAwYTDfroCc5XK4X 9iN7briEPy0.DM4R92b_CFQ_KKfldGqbLFFggRVQ_1gkxRCVe1JgtWBQ9j9tf1FWfvztfQ9EBAse hwGX4gkQRGFNRjYzuI.skyBsGGu1Ko7dRX4wY5.CpTN2UAWo1KFYcY0hxSU5gH42935stZ4t2wJT n4uG4zO6i8ypWnrTU44d8RjM5Ysp.H4IKCA35Pt5ss83MSvVKQsiCabbcVulbSrKfK4euyyDmlgu _xq8hugilBCmiRK3en3oQE0pUioTpZx__H0XIy07l2HyMkaNYMzZhvABuR76KaOJxyfVKXZ._XFc LgdeXrJQwKMG.ggCkNpGt0Bg2J9QQcrLyl_94qwZw0XFy67rekull0_mijcjUloEBaWqLnJUdZ4F NR7540LV4W82Rls6Su8nriR7GZdUVXAFP1FH5d74XtM6vyutzMAuYy3CJHl8DZSHO8kuIw1Felcm 1ZTMBEFNdDvcnHDjTOGbHJTZNiYulzqoQP8ySQMXE0Q8YJ_uXfInNm2DBvYCX_SS0hboy4LHjUDA oyQbulcUNNcHisVm2g1v9HAUna3uysdN1E5zkOPtV7vqa1tYMz53lZVpOyJOd8UBlgwt8NLBk89Q vMgVtbJVC4EF8_.aFR0QR8hUnuzfiXrvoBDa.jNmAhdGv9Ej9Ba31vlTlxD82bnr8O5WVSVWv_gw xrDXczKgMuuV5Ldj.RVor_1eBiSNBHy4Oy60Txamws3FYgpKtdvRq7nKXxmbAzf7MjYnHK9s_yBN VI1n5cBzIuZp7mBNH3s9GfVaYEZusHUAGxwMCvFH4SqUqS0tnYPeQsOwDE611azrX4Txeelql8bv fGP.X7MKL4FFV1meAH9R07s3fvyMku8H1r54KOw4Zntr6pkN1EziBaMnNz3dm1TiNUwLygAMXzBU FtejCdqTqv32p0QfbzZlY3qHRBhUwbG40S2my1PAbT9WSlBDRNpkbF7scmzDSO25FoGHklDC8XX4 YSED4MqOv0GUllK_R.jXFkkiVoeepBGYr4_cqo.CxJ4QelbRDpZm0FyMeNBo0HzCfnda3Geltycz YycWE2gPxxjkfXJ351qlSQOwi4jnOzRPYEQdyFgOW_hc3tKFJwdlE4E3QAzA9eeRJLWaHRiJsk1w QT1pnPRHeW7qVDQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Sat, 23 Apr 2022 01:47:02 +0000 Received: by hermes--canary-production-gq1-6b7487d8dd-zffkr (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 45ebfc647c1efabf60596bee80269486; Sat, 23 Apr 2022 01:46:57 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Chasing OOM Issues - good sysctl metrics to use? From: Mark Millard In-Reply-To: Date: Fri, 22 Apr 2022 18:46:56 -0700 Cc: freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: <94B2E2FD-2371-4FEA-8E01-F37103F63CC0@yahoo.com> References: <83A713B9-A973-4C97-ACD6-830DF6A50B76.ref@yahoo.com> <83A713B9-A973-4C97-ACD6-830DF6A50B76@yahoo.com> To: Pete Wright X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4KlYzx277fz3rL6 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=aFPMnPnb; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-0.62 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.85)[0.852]; NEURAL_HAM_LONG(-0.97)[-0.971]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.206:from]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Apr-22, at 16:42, Pete Wright wrote: > On 4/21/22 21:18, Mark Millard wrote: >>=20 >> Messages in the console out would be appropriate >> to report. Messages might also be available via >> the following at appropriate times: >=20 > that is what is frustrating. i will get notification that the = processes are killed: > Apr 22 09:55:15 topanga kernel: pid 76242 (chrome), jid 0, uid 1001, = was killed: failed to reclaim memory > Apr 22 09:55:19 topanga kernel: pid 76288 (chrome), jid 0, uid 1001, = was killed: failed to reclaim memory > Apr 22 09:55:20 topanga kernel: pid 76259 (firefox), jid 0, uid 1001, = was killed: failed to reclaim memory > Apr 22 09:55:22 topanga kernel: pid 76252 (firefox), jid 0, uid 1001, = was killed: failed to reclaim memory > Apr 22 09:55:23 topanga kernel: pid 76267 (firefox), jid 0, uid 1001, = was killed: failed to reclaim memory > Apr 22 09:55:24 topanga kernel: pid 76234 (chrome), jid 0, uid 1001, = was killed: failed to reclaim memory > Apr 22 09:55:26 topanga kernel: pid 76275 (firefox), jid 0, uid 1001, = was killed: failed to reclaim memory Those messages are not reporting being out of swap as such. They are reporting sustained low free RAM despite a number of less drastic attempts to gain back free RAM (to above some threshold). FreeBSD does not swap out the kernel stacks for processes that stay in a runnable state: it just continues to page. Thus just one large process that has a huge working set of active pages can lead to OOM kills in a context were no other set of processes would be enough to gain the free RAM required. Such contexts are not really a swap issue. Based on there being only 1 "killed:" reason, I have a suggestion that should allow delaying such kills for a long time. That in turn may help with investigating without actually suffering the kills during the activity: more time with low free RAM to observe. Increase: # sysctl -d vm.pageout_oom_seq vm.pageout_oom_seq: back-to-back calls to oom detector to start OOM The default value was 12, last I checked. My /boot/loader.conf contains the following relative to that and another type of kill context (just comments currently for that other type): # # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=3D120 # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: #vm.pfault_oom_attempts=3D-1 # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes (showing defaults at the time): #vm.pfault_oom_attempts=3D 3 #vm.pfault_oom_wait=3D 10 # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) There is no value of vm.pageout_oom_seq that disables the mechanism. But you can set large values, like I did --or even larger-- to wait for more attempts to free some RAM before the kills. Some notes about that follow. The 120 I use allows even low end arm Small Board Computers to manage buildworld buildkernel without such kills. The buildworld buildkernel completion is sufficient that the low-free-RAM status is no longer true and the OOM attempts stop --so the count goes back to 0. But those are large but finite activities. If you want to leave something running for days, weeks, months, or whatever that produces the sustained low free RAM conditions, the problem will eventually happen. Ultimately one may have to exit and restart such processes once and a while, exiting enough of them to give a little time with sufficient free RAM. > the system in this case had killed both firefox and chrome while i was = afk. i logged back in and started them up to do more more, then the = next logline is from this morning when i had to force power off/on the = system as they keyboard and network were both unresponsive: >=20 > Apr 22 09:58:20 topanga syslogd: kernel boot file is = /boot/kernel/kernel >=20 >> Do you have any swap partitions set up and in use? The >> details could be relevant. Do you have swap set up >> some other way than via swap partition use? No swap? > yes i have a 2GB of swap that resides on a nvme device. I assume a partition style. Otherwise there are other issues involved --that likely should be avoided by switching to partition style. >> ZFS (so with ARC)? UFS? Both? >=20 > i am using ZFS and am setting my vfs.zfs.arc.max to 10G. i have also = experienced this crash with that set to the default unlimited value as = well. I use ZFS on systems with at least 8 GiBytes of RAM, but I've never tuned ZFS. So I'm not much help for that side of things. For systems with under 8 GiBytes of RAM, I uses UFS unless doing an odd experiment. >> The first block of lines from a top display could be >> relevant, particularly when it is clearly progressing >> towards having the problem. (After the problem is too >> late.) (I just picked top as a way to get a bunch of >> the information all together automatically.) >=20 > since the initial OOM events happen when i am AFK it is difficult to = get relevant stats out of top. If you use vm.pageout_oom_seq=3D120 (or more) and check once and a while, I expect you would end up seeing the activity in top without suffering a kill in short order. Once noticed, you could start your investigation, including via top if desired. > this is why i've started collecting more detailed metrics in = prometheus. my hope is i'll be able to do a better job observing how my = system is behaving over time, in the run up to the OOM event as well as = right before and after. there are heaps of metrics collected though so = hoping someone can point me in the right direction :) I'm hoping that vm.pageout_oom_seq=3D120 (or more) makes it so you do not have to have identified everything up front and can explore easier. Note that vm.pageout_oom_seq is both a loader tunable and a writeable runtime tunable: # sysctl -T vm.pageout_oom_seq vm.pageout_oom_seq: 120 amd64_ZFS amd64 1400053 1400053 # sysctl -W vm.pageout_oom_seq vm.pageout_oom_seq: 120 So you can use it to extend the time when the machine is already running. =3D=3D=3D Mark Millard marklmi at yahoo.com