From owner-freebsd-current@freebsd.org Tue Jan 28 08:49:48 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8404C1F9EDA for ; Tue, 28 Jan 2020 08:49:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-23.consmr.mail.gq1.yahoo.com (sonic303-23.consmr.mail.gq1.yahoo.com [98.137.64.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 486L0C1MH1z4rvX for ; Tue, 28 Jan 2020 08:49:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: LtQoIUsVM1npwviTre99yxqBH2Hoez9avEXZrZ4UTpr7d8kqGVXND2seg9ck.ue RhlJ8rtTB4RexAhsTTJp8ads6LieO2xWdDT9WO3N9d3BGLlWvC0lFNrEA5brXnvOSy6bjMgOmLM4 UJxN5AiH6os6f2HF.paKzo4nnvQuAJbZaPl1RsRLuURkoChsObc8uXsAAOWlS2iuQ30L2_zpLME. 2lLud.SPL4y41HzJuMu15zDRQLPY_FkbP86l6iaulvPZbVPAFbRU_jEbCZA_Y4tjn7DZMRWwFL9O o2gxr4qIy7f2TzBbM_87Cfb7vHfto1ckgeAnpz2NopgF9X14n3zgLnPEWKaOvU7pb9God_njbkPD 2KSoM4dq6wPRfWmjwFKu9BRM40rJ.oss0_wAwxClhkqBteDY9npcERG2MqX4tpMdLx5AGNzferOf hn.WAhwVjOEE9vq4Fz18nfoR.2h4aKNvWkxh1HBRQ9O_.MSU4xXVeGAqp0nhTBi3cCoP72VztON5 bIZpw0rkZpGPp3HOV138Q7PwS8ApP3gkQ3wgeJnVSuHigYNNDpQOnobzz4FEJAC3EYDdg4w.bxjI aOIvOz1QbK_LO8TLqkT459imluNPMyuaI3iyIzDJ3RfSKgNPVpy6iREA3K8j4QiiGVWSIlke4R0m K9iS5zJ9PMz1PrZlA2JTpQRHmLE8Ab3f._IXI3EXOOLL5Dejn6Qd21mXRLRywYlxiE.BBjxT4EHo SihS6Xgk6rZZ6OGAgycVVcheTqClfGf4o6p__Pg5binrmyo5lgc1aZR1UxaoMC..ct06IphqvIFK F80tF49iKgitrwO3wky7RLWO6ykpIefaGf7jl3HmL51jP84EUzpJ.WBk9No.gasqPlNCELHlaAMo DPtmz8MRKUklKNOunNU8e7eSCOfvm6fRCSwlbH1rBfPS46kuAO5WTqaiAlumZjsZkG_.k46Lhtbt .zWphv5oieTvwFwK.860JAteHmDVTxOpLeQ34haNbGoGJDvJPXJl5dYVjsqVuda684CsWTJXCbxr T57POjPbM0itFS3XRkAIHOmBx1jxO2_bwkLkSq9BrgJCgM11q9kmhzB4qACu.56inWxcb43w7z5P wU07iWekcNcbpiPL92h9xvRNrO7Ibb6iIP9j7KdofeE8KFMW1EIm1cwHM7KUujhoyyCUUkGBCw7u ZAzYeHXGUe_Bp3urwy0gQJ.QdoGBSkvQ0YgD6XMxDzmT.LujMPqOC3_1jLkYYoV4yrf3KMEuf3Fi pkbIoFo8vum6EK6SWRKUl7fTsvg.nuFg48CWn_KGf9VRsYBjt_Zw58KmAP0mNnj0.qf3su58GM1Q VfrDEAGNkQBBdbExSXHrULNsmnElni2HuecRXFOts9WnkbMWkxPQqUARpWrJW2286wO5raLLpaSK FU4N5jm1K.tHkMjg2aAYzC3SnGwzIuDnCkNkdp6w- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Tue, 28 Jan 2020 08:49:45 +0000 Received: by smtp414.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 55dfa43afc642adea1c2113df8510244; Tue, 28 Jan 2020 08:49:42 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: OOMA kill with vm.pfault_oom_attempts="-1" on RPi3 at r357147 From: Mark Millard In-Reply-To: <18150258-6210-451E-A5B9-528129A05974@yahoo.com> Date: Tue, 28 Jan 2020 00:49:41 -0800 Cc: freebsd-arm@freebsd.org, freebsd-current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <9BF68EF1-F83A-473B-9A7B-B3956D6A5EFD@yahoo.com> References: <20200127190709.GA11328@www.zefox.net> <20200128035317.GA12644@www.zefox.net> <18150258-6210-451E-A5B9-528129A05974@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 486L0C1MH1z4rvX X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.39 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.948,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.94)[-0.939,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (0.42), ipnet: 98.137.64.0/21(0.84), asn: 36647(0.67), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[204.64.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[204.64.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jan 2020 08:49:48 -0000 On 2020-Jan-27, at 21:29, Mark Millard wrote: > On 2020-Jan-27, at 19:53, bob prohaska wrote: >=20 >> On Mon, Jan 27, 2020 at 06:22:20PM -0800, Mark Millard wrote: >>>=20 >>> So far as I know, in the past progress was only made when someone >>> already knowledgable got involved in isolating what was happening >>> and how to control it. >>>=20 >> Indeed. One can only hope said knowledgeables are reading.... >=20 > May be I can suggest something that might kick-start > evidence gathering a little bit: add 4 unconditional > printf's to the kernel code, each just before one of > the vm_pageout_oom(. . .) calls. Have the message > uniquely identify which of the 4 it is before. >=20 > . . . Below is a stab at implementing the suggestion. A couple of the printf's are basically what Mark Johnston supplied long ago. (Other code from what he supplied back then did not survive updates made to FreeBSD.) One of his printf's is not tied to indicating vm_pageout_oom use. (Sent this way some whitespace might not be preserved.) # svnlite diff /usr/src/sys/vm/=20 Index: /usr/src/sys/vm/swap_pager.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/swap_pager.c (revision 356426) +++ /usr/src/sys/vm/swap_pager.c (working copy) @@ -2021,6 +2021,7 @@ 0, 1)) printf("swap blk zone exhausted, = " "increase = kern.maxswzone\n"); + printf("swp_pager_meta_build: swap blk = uma zone exhausted\n"); vm_pageout_oom(VM_OOM_SWAPZ); pause("swzonxb", 10); } else @@ -2051,6 +2052,7 @@ 0, 1)) printf("swap pctrie zone = exhausted, " "increase = kern.maxswzone\n"); + printf("swp_pager_meta_build: swap = pctrie uma zone exhausted\n"); vm_pageout_oom(VM_OOM_SWAPZ); pause("swzonxp", 10); } else Index: /usr/src/sys/vm/vm_fault.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/vm_fault.c (revision 356426) +++ /usr/src/sys/vm/vm_fault.c (working copy) @@ -943,9 +943,9 @@ vm_pfault_oom_wait * hz); goto RetryFault_oom; } - if (bootverbose) + // HAVE PRINTF BE UNCONDITIONAL FOR = TESTING: if (bootverbose) printf( - "proc %d (%s) failed to alloc page on fault, starting OOM\n", + "vm_fault: proc %d (%s) failed to alloc page on fault, starting = OOM\n", curproc->p_pid, = curproc->p_comm); vm_pageout_oom(VM_OOM_MEM_PF); goto RetryFault; Index: /usr/src/sys/vm/vm_page.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/vm_page.c (revision 356426) +++ /usr/src/sys/vm/vm_page.c (working copy) @@ -3139,6 +3139,7 @@ * race-free vm_wait_domain(). */ if (curproc =3D=3D pageproc) { + printf("thread %d waiting for memory\n", = curthread->td_tid); mtx_lock(&vm_domainset_lock); vm_pageproc_waiters++; msleep(&vm_pageproc_waiters, &vm_domainset_lock, PVM | = PDROP, Index: /usr/src/sys/vm/vm_pageout.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/vm/vm_pageout.c (revision 356426) +++ /usr/src/sys/vm/vm_pageout.c (working copy) @@ -1741,6 +1741,8 @@ * start OOM. Initiate the selection and signaling of the * victim. */ + printf("vm_pageout_mightbe_oom: kill context: v_free_count: %u, = v_inactive_count: %u\n", + vmd->vmd_free_count, = vmd->vmd_pagequeues[PQ_INACTIVE].pq_cnt); vm_pageout_oom(VM_OOM_MEM); =20 /* @@ -1933,10 +1935,24 @@ } sx_sunlock(&allproc_lock); if (bigproc !=3D NULL) { + char *reason_text; + switch (shortage) { + case VM_OOM_MEM_PF: + reason_text=3D "fault's page allocation failed"; + break; + case VM_OOM_MEM: + reason_text=3D "free RAM stayed below = threshold"; + break; + case VM_OOM_SWAPZ: + reason_text=3D "swblk or swpctrie zone = exhausted"; + break; + default: + reason_text=3D "Unknown Reason"; + } if (vm_panic_on_oom !=3D 0) - panic("out of swap space"); + panic("%s",reason_text); PROC_LOCK(bigproc); - killproc(bigproc, "out of swap space"); + killproc(bigproc, reason_text); sched_nice(bigproc, PRIO_MIN); _PRELE(bigproc); PROC_UNLOCK(bigproc); =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)