From nobody Sat Apr 23 19:31:35 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9C9121A807CC for ; Sat, 23 Apr 2022 19:31:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-20.consmr.mail.gq1.yahoo.com (sonic305-20.consmr.mail.gq1.yahoo.com [98.137.64.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Km1cM5236z4VqY for ; Sat, 23 Apr 2022 19:31:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650742299; bh=ctTCAgqF//+AAiw+nMkXGb8dgYHGU1dC9DaxM/IyEr4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=ThQZkwjzNGD7PoXE58o9sYBVJwEXi793x26dVRpFriOvkwY5QkrFu5+7u4/6izoBmIR+y58FUQPrTT8Gf6onPWbWgxF0VtuMqHZdwEoPZYyRoB/Vcl8ev8KEXdLCJxz4grR3HSMTb2KJVuQ9yfgjvKHKMxUYZ31amJSr9tJJB5JjPiwxubUCCD5qssZvi8aK9p0Oi3HtkC2Sj7IC3JgrHhpdXsnYTiDC6JkN01U+m7qB597HIsbeteyX4TWKmK7GTSKAwuvDj1FIZ0NVaMaIHq0JX9tG+SQTdBOAYbrz4Y02Z2BGVJro/EpjGkRWiVs+3c+IcNRBJ/z0ciYGOnyslg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650742299; bh=4wV3QeTvOF4ZIH92wMuywjr6k9HKMF5I0tTDSpCpEol=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=NPciqJL4ACPxA7kk4iFV3QOMo4YSY8zQhNpjUOBlY58HR3s1vMdSKJ1PJekVpOiu/bEH/ElbXvs68Y5f6kWAwDD1Bcn3VrgvEtn/36BR3P4hf+AF7yaOSoooPHAlKVHDQG3oCRjmMG0+LQ9A02jdD2yFufsM3HFADB8AekXyMEN2ZH5QWrZ9ReNEmtNF5iqQj5kNeQ1gkR9lY6zRQOPmf9ZNdWu5J09flX6f1RYbGp5414VpI8Q8WZfKRmBRXTyyHyLjlhGlWPVhMyZQr53bPQgknZOsYIs09bUF0P1aeeaCen4Q7byv+KRYvk3WOWa/iPLVOtoFUUKSt9E18VVr4w== X-YMail-OSG: WHLcV2gVM1lcVtDkEqyW_Ulmc1C.pNNJ_H90jMfnS0_6ByM5Yozy4Utst6GQLuj HMu5PSw3yrwZLHXWHRbZZWX3JVkVJhyqG0zmwP5QNvabA4MF2f77kYdFTi.0HhxjqToxUOpJv.Lc bCy_HOrbfO4PBCjs3QhT8_GR34riWz1_PZtLkqJeV2e1eFuaT1q9JVacxnnGDE9SBZBBwiYIzlTE 9ba97M1I1KGm2xl.Ec6o7xXQaJfJaLX1x3um7WYEYki77rNPxxJbfMGsRG4QntvDpFeJBs2lNYlp ds.rYNC3sVtwkAYy81a1F46p9XDhtNafwZwD6CI7uCacYZ7Ss7T.C6r.Yo7Cf4F.jGdhHtPl03FH 8P_mMklfAxFR4hINHSizkjgEY7gy1q9e9TWkP9Pto4lt26yybvDSc.F1vyLmJvKfzGrU6_hqRplK fZvdtfyHFRE7W6e0bac_.ntH6Te4VJnrKauvSrfT06dx5iXrq1wU7W_HM77P8TXuVEzrKGogfbHI cTzMV4pTKz2ukeqeLy8lhMIwaJRPZJdFw13C8hQ.n1HY1Goll.f_IYTSx427pws0dVPNsei8CRJv Q.AR_ON13dB7DQyo_pz1oyQGjS0SwtcQ8J9xRcXaFrnBZrXZEgu7i_b0eKCs6UPijoNJhvjMqodj qqI1P6YC9H.qyP71903.o299rTj62VbLMMvN.KpjrGSFMp.mVLW2I7YZJSQ4uht3Z9bj1jdTyUBz CVOERDfilxQgtkp0v1_g4XDLXwopN472piZXw8x3m_vbILGGwORwkxUeg0sd_DZnwfJDpxbmqog0 3tgrsR3o10IZCYuftPJZdDIp5GBTZpLpil2EA.xHOoZBPRfiU5of1l6Y0Yci6VqmOidSokm5nvgE lDckLJcLhza77fnC3Nw4LjdfWTdIKAv6xZxf8xJtLCxomUgJTfi.ddGYOeZYGw2VavogCZ0gV5hs G9AR4dVsEiIK7QQ2oKRVv9Q1em.yqaRlT1maxFyo9qWPJfhPvt7Y6GVpJ7qOAJ1DEQ0f9OE4U5pQ CVl8M.kZc0ay9nAtXoihJxWjfE6J9tA4kRpPos.Kmtde01D7KaKzLA2lfUFe3EtpXOuqaeBx6ZaV 8fidSHPqdE3jW7EELsQhyChAz_oy8y5F5d7_6RYl4BVNZAIQYr12N0EPgXf1p4elkYDI2cHenLHs xjEUncth1dmplEbkyxbko5TZwvI4GbDml5Mdg_laBi8MmQb7Z3odr6aCCrCyrF677jfJB2s4sly0 1McCnDW63WUE7.clFQwIRdRBl4rp9xY3RfG2oWpEZxv30OuwqDtaXAT2dTKyaN3TIikonZHVoZKG fWYqXSGrNyR1GfYYmbvo61uYzGvlZqFOb79YmpuFzuExS4ywN3dqku9bDOsJOl._1XO6lVqrREA9 Ghfz6uNNBdzhk1Hup6RCEylbnix16uvx9vpwLbBhp6fiwCUflVyVP2h6c6YwIZO4zRWQuzFdckEX rFQfbpTXIOghA4TCCndOYaD2hUL4lks_MTek5BW3xxvC3axZU8NqzN798Hac02NM6hMLygXY.bvV _7LrM9jxxZV6_Ixu5uJnIv8Q9YMUPNTnmyCpvpXFx4Q7QF4SCyN3vVL8lywSPYJC6VD.LqBjRG_8 g8kVjvBpvY4uOJEiZWVywuBsVgxsbo3y1Urm4._IioTp_zCmXlrUIihWnZiPRMBrVoTDlfXSBxSG 5._a7XYWAh7KXyeUuO2EzIhLH0F0wXFsmn7xKDs7BZ2VsB4WX_gImgVIqe5.FTxFGc_QXfXgd0Il uGaAsDZ.GQC9KlmzdYptf.MGNL39R.wv5FhikuuVSUbAgILzbdR3yAJ.KkygrrV3AUCOKr7dB208 hTcq09USPBIvY_zwEj8HR9lROHpVesSG52aPHsEgl6G3yrQKb._S5DzhWAAW13EU2ERvV4hZfqVU _ZHUy_s3Z2Kl_A6tki5qPhBAZ_501VN9eRg7mhfYO67zvJvI1hYvAupn9CR7Y2VQ9wGzkNgrhpPK j6CS7mJ3eE.GSloASFjdok79v48.qGbrvz.SwCEFimwRUvmAzTT_b1CFrcfn9gCSSRgIlbSKdU6s zYAp_TpiTNzR713rr9kSAB.LepyRHIYxTrzhLi_IZXYYCoCH6qjAjha_yOzvP.K2RebwcFQlcMDc 2sjavcNKgSCkaL9LGuNLmYvuczR2EIcoHJILDGM8pCHdAmOuWw2IqlVw.igmCj0d_ed5w9gXpfIX hR4wdPOKtLPGQy8DqXjZc4hp7ic7V0v4FG62uT2l2UEuzScomse9QN92GAH1zLEMVuPUNueVPd1F 128uf0jxz X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Sat, 23 Apr 2022 19:31:39 +0000 Received: by hermes--canary-production-ne1-6855c48695-pbbz7 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 43e328663df93d23343e95fc23d6b35c; Sat, 23 Apr 2022 19:31:37 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Chasing OOM Issues - good sysctl metrics to use? From: Mark Millard In-Reply-To: <0fcb5a4a-5517-e57b-2b69-4f3b3b10589a@nomadlogic.org> Date: Sat, 23 Apr 2022 12:31:35 -0700 Cc: freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: References: <83A713B9-A973-4C97-ACD6-830DF6A50B76.ref@yahoo.com> <83A713B9-A973-4C97-ACD6-830DF6A50B76@yahoo.com> <94B2E2FD-2371-4FEA-8E01-F37103F63CC0@yahoo.com> <0fcb5a4a-5517-e57b-2b69-4f3b3b10589a@nomadlogic.org> To: Pete Wright X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Km1cM5236z4VqY X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ThQZkwjz; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [1.86 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-0.64)[-0.643]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(1.00)[1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.83:from]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Apr-23, at 10:26, Pete Wright wrote: > On 4/22/22 18:46, Mark Millard wrote: >> On 2022-Apr-22, at 16:42, Pete Wright wrote: >>=20 >>> On 4/21/22 21:18, Mark Millard wrote: >>>> Messages in the console out would be appropriate >>>> to report. Messages might also be available via >>>> the following at appropriate times: >>> that is what is frustrating. i will get notification that the = processes are killed: >>> Apr 22 09:55:15 topanga kernel: pid 76242 (chrome), jid 0, uid 1001, = was killed: failed to reclaim memory >>> Apr 22 09:55:19 topanga kernel: pid 76288 (chrome), jid 0, uid 1001, = was killed: failed to reclaim memory >>> Apr 22 09:55:20 topanga kernel: pid 76259 (firefox), jid 0, uid = 1001, was killed: failed to reclaim memory >>> Apr 22 09:55:22 topanga kernel: pid 76252 (firefox), jid 0, uid = 1001, was killed: failed to reclaim memory >>> Apr 22 09:55:23 topanga kernel: pid 76267 (firefox), jid 0, uid = 1001, was killed: failed to reclaim memory >>> Apr 22 09:55:24 topanga kernel: pid 76234 (chrome), jid 0, uid 1001, = was killed: failed to reclaim memory >>> Apr 22 09:55:26 topanga kernel: pid 76275 (firefox), jid 0, uid = 1001, was killed: failed to reclaim memory >> Those messages are not reporting being out of swap >> as such. They are reporting sustained low free RAM >> despite a number of less drastic attempts to gain >> back free RAM (to above some threshold). >>=20 >> FreeBSD does not swap out the kernel stacks for >> processes that stay in a runnable state: it just >> continues to page. Thus just one large process >> that has a huge working set of active pages can >> lead to OOM kills in a context were no other set >> of processes would be enough to gain the free >> RAM required. Such contexts are not really a >> swap issue. >=20 > Thank you for this clarification/explanation - that totally makes = sense! >=20 >>=20 >> Based on there being only 1 "killed:" reason, >> I have a suggestion that should allow delaying >> such kills for a long time. That in turn may >> help with investigating without actually >> suffering the kills during the activity: more >> time with low free RAM to observe. >=20 > Great idea thank-you! and thanks for the example settings and = descriptions as well. >> But those are large but finite activities. If >> you want to leave something running for days, >> weeks, months, or whatever that produces the >> sustained low free RAM conditions, the problem >> will eventually happen. Ultimately one may have >> to exit and restart such processes once and a >> while, exiting enough of them to give a little >> time with sufficient free RAM. > perfect - since this is a workstation my run-time for these processes = is probably a week as i update my system and pkgs over the weekend, then = dog food current during the work week. >=20 >>> yes i have a 2GB of swap that resides on a nvme device. >> I assume a partition style. Otherwise there are other >> issues involved --that likely should be avoided by >> switching to partition style. >=20 > so i kinda lied - initially i had just a 2G swap, but i added a second = 20G swap a while ago to have enough space to capture some cores while = testing drm-kmod work. based on this comment i am going to only use the = 20G file backed swap and see how that goes. >=20 > this is my fstab entry currently for the file backed swap: > md99 none swap sw,file=3D/root/swap1,late 0 0 I think you may have taken my suggestion backwards . . . Unfortunately, vnode (file) based swap space should be *avoided* and partitions are what should be used in order to avoid deadlocks: On 2017-Feb-13, at 7:20 PM, Konstantin Belousov = wrote on the freebsd-arm list: QUOTE swapfile write requires the write request to come through the filesystem write path, which might require the filesystem to allocate more memory and read some data. E.g. it is known that any ZFS write request allocates memory, and that write request on large UFS file might require allocating and reading an indirect block buffer to find the block number of the written block, if the indirect block was not yet read. As result, swapfile swapping is more prone to the trivial and = unavoidable deadlocks where the pagedaemon thread, which produces free memory, needs more free memory to make a progress. Swap write on the raw partition = over simple partitioning scheme directly over HBA are usually safe, while = e.g. zfs over geli over umass is the worst construction. END QUOTE The developers handbook has a section debugging deadlocks that he referenced in a response to another report (on freebsd-hackers). = https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/#kerneld= ebug-deadlocks >>>> ZFS (so with ARC)? UFS? Both? >>> i am using ZFS and am setting my vfs.zfs.arc.max to 10G. i have = also experienced this crash with that set to the default unlimited value = as well. >> I use ZFS on systems with at least 8 GiBytes of RAM, >> but I've never tuned ZFS. So I'm not much help for >> that side of things. >=20 > since we started this thread I've gone ahead and removed the = zfs.arc.max setting since its cruft at this point. i initially added it = to test a configuration i deployed to a sever hosting a bunch of VMs. >=20 >> I'm hoping that vm.pageout_oom_seq=3D120 (or more) makes it >> so you do not have to have identified everything up front >> and can explore easier. >>=20 >>=20 >> Note that vm.pageout_oom_seq is both a loader tunable >> and a writeable runtime tunable: >>=20 >> # sysctl -T vm.pageout_oom_seq >> vm.pageout_oom_seq: 120 >> amd64_ZFS amd64 1400053 1400053 # sysctl -W vm.pageout_oom_seq >> vm.pageout_oom_seq: 120 >>=20 >> So you can use it to extend the time when the >> machine is already running. >=20 > fantastic. thanks again for taking your time and sharing your = knowledge and experience with me Mark! >=20 > these types of journeys are why i run current on my daily driver, it = really helps me better understand the OS so that i can be a better admin = on the "real" servers i run for work. its also just fun to learn stuff = too heh. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com