From owner-freebsd-current@freebsd.org Sat Jan 25 20:02:14 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4AD231F92A2 for ; Sat, 25 Jan 2020 20:02:14 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 484n3S6S14z429r for ; Sat, 25 Jan 2020 20:02:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: c7r8s3cVM1lK_qiYM9pKhxrd5ytgW45LAAu3QzDJQgkIO9MA0DdtKi5SYcVJ7ij WHtRAoDx4ys_okoymiHin2XoEis1bkQe4zY3OkhItIoEQa2BbbA.H4WkRQnCkj6pMyFnSY8n7Urn ZW8smVLB1gOWAFSTuqwGybep3RquuDHOQzdkXGbQ7NU1MNNJ4aRsMcv6wZGHPjqz84Hgs5x1L3t9 yeJsvVehcDQ2s8t48m8BiCCK7Hm5Z3wN23DFV65_Z0pib_2Ltn34Lx6ORHqdohBKyGxeO2F6Q2gZ wQunvD3zxv4VgAwTU4lKbvFwioApyEITfsbTsjzsJ5rrYUTVDKpIPHq1sN849uS3wbdlmcNZnljK F75rsJXLYL6MWnQ8Zhc1MEJ8HoO2wU8UxaYe0LzerKACLxV8j6_iRipd4T6uSUmXErKdVW3ZOYBK fn6G5slVxBrernLzK_rI1Nob630SgVUXij2vDElsMjWekvOb_kLt9MMwNX5a55weEME.Vyec4qg6 QDddHbcSiBp7EqBt8eBPpOXDNrBISAolmAwfbSdZz0a6zsP0oU6RRyuJmpDJxoJ0iqLAVg1qrpJ1 f_lEpQ0Gxxs7Ui.5GEjSttSujRhngc2k95xrS7T86S5b61TgkyVlyZPuUFnq5yI1H70deo08_GdU eAxojfAZWA_rpLkt.m4deN0V.TZLXYne9nt_Np6zohMwrT2CY1b.0JuiJ5fMnoBHxhmSHatskcuT vZ5hziE6U0ru3MlrAoeSljHR03c42Z7gt6mqaDzeCty69e3TDscEEJGlWkOKHcvpK4_dGOGcGumD tG6cM6qLhjzJjRN87RjqEGe57WySWuB0XfZPGi2oGF8ytdt1CrDUBdd21xVimLisqsHpFjo5eyLx .h8v6xdrpaGoUyq96uxm1DUXf97dEOAa9isp5fU2nP33pTPSgwLuuaBhg_xFbgQAL3irhqd43koG 3F4mor0zs8FHmvYqIAj.Ln7XxlMREifIx3JtXhPQf8cHKOVc4NezJ1pofBpX.lGcHOTIhUV9qd9W DajXO6cQ6OALWJ.bllHuIT09s3UG4Dap5lU.RqpE2KRGU7cVtVvvZORmUSfk08D8SvSp4vuoiOS. _41L4XW5ry2noPbyZ1S.fJYnol4WAp_MvfckXhlh7w4h0WoIPTOT.E.TGeUw5chR5bDsCfKTHzXJ rYSsjuIfNsPfsEuXxiJLTLgFgas8HJV1XdgQ_s9delswNUKSG8LW6f8okctJLfeeNAqG2rF58Sdj nxPRx2oDMef2Wgvn56c33wcckczjUrr0hwLzz07ZuB8.UcEJ4wONHdFaoiMRkO2xqEszBFtGVI6H 8dLb6ujTMYC3DkziVV2WfotpVqmRzjKfZDMF05ZiXegq8gCTNSnyecKbrMZR0nPIFFsdqKYVsw00 UViB.ER3WZCFvAO4r2UnTCc1CTQ95_DVcZFk- Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Sat, 25 Jan 2020 20:02:11 +0000 Received: by smtp426.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID f19ba718b2f6063783793984d34d18f2; Sat, 25 Jan 2020 20:02:08 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: After update to r357104 build of poudriere jail fails with 'out of swap space' Message-Id: Date: Sat, 25 Jan 2020 12:02:07 -0800 To: yasu@utahime.org, FreeBSD Current X-Mailer: Apple Mail (2.3608.40.2.2.4) References: X-Rspamd-Queue-Id: 484n3S6S14z429r X-Spamd-Bar: / X-Spamd-Result: default: False [0.80 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_SPAM_MEDIUM(0.76)[0.759,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[147.65.137.98.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_LONG(0.54)[0.543,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (5.60), ipnet: 98.137.64.0/21(0.84), asn: 36647(0.67), country: US(-0.05)]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jan 2020 20:02:14 -0000 Yasuhiro KIMURA yasu at utahime.org wrote on Sat Jan 25 14:45:13 UTC 2020 : > I use VirtualBox to run 13-CURRENT. Host is 64bit Windows 10 1909 and > spec of VM is as following. >=20 > * 4 CPU > * 8GB memory > * 100GB disk > - 92GB ZFS pool (zroot) > - 8GB swap >=20 > Today I updated this VM to r357104. And after that I tried to update > poudriere jail with `poudriere jail -u -j jailname -b`. But it failed > at install stage. After the failure I found following message is > written to syslog. >=20 > Jan 25 19:18:25 rolling-vm-freebsd1 kernel: pid 7963 (strip), jid 0, = uid 0, was killed: out of swap space This message text's detailed wording is a misnomer. Do you also have any messages of the form: . . . sentinel kernel: swap_pager_getswapspace(32): failed If yes: you really were out of swap space. If no: you were not out of swap space, or at least it is highly unlikely that you were. FreeBSD kills processes for multiple potential reasons. For example: a) Still low on free RAM after a number of tries to increase it above a = threshold. b) Slow paging I/O. c) . . . (I do not know the full list) . . . Unfortunately, FreeBSD is not explicit about the category of problem that leads to the kill activity that happens. You might learn more by watching how things are going via top or some such program or other way of monitoring. Below are some notes about specific tunables that might or might not be of help. (There may be more tunables that can help that I do not know about.) For (a) there is a way to test if it is the issue by adding to the number of tries before it gives up and starts killing things. That will either: 1) let it get more done before kills start 2) let it complete before the count is reached 3) make no significant difference (3) would imply that (b) or (c) are involved instead. (1) might be handled by having it do even more tries. For delaying how long free RAM staying low is tolerated, one can increase vm.pageout_oom_seq from 12 to larger. The management of slow paging I've less experience with but do have some notes about below. Examples follow that I use in contexts with sufficient RAM that I do not have to worry about out of swap/page space. These I've set in /etc/sysctl.conf . (Of course, I'm not trying to deliberately run out of RAM.) # # Delay when persisstent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=3D120 I'll note that figures like 1024 or 1200 or even more are possible. This is controlling how many tries at regaining sufficient free RAM that that level would be tolerated long-term. After that it starts Out Of Memory kills to get some free RAM. No figure is designed to make the delay unbounded. There may be large enough figures to effectively be bounded beyond any reasonable time to wait. As for paging I/O (this is specific to 13, or was last I checked): # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=3D-1 (Note: In my context "plunty" really means sufficient RAM that paging is rare. But others have reported on using the -1 in contexts where paging was heavy at times and OOM kills had been happening that were eliminated by the assignment.) I've no experience with the below alternative to that -1 use: # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes: #vm.pfault_oom_attempts=3D ??? #vm.pfault_oom_wait=3D ??? # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) I'm not claiming that these 3 vm.???_oom_??? figures are always sufficient. Nor am I claiming that tunables are always available that would be sufficient. Nor that it is easy to find the ones that do exist that might help for specific OOM kill issues. I have seen reports of OOM kills for other reasons when both vm.pageout_oom_seq and vm.pfault_oom_attempts=3D-1 were in use. As I understand, FreeBSD did not report what kind of condition lead to the decision to do an OOM kill. So the above notes may or may-not help you. > To make sure I shutdown both VM and host, restarted them and tried > update of jail again. Then the problem was reproduced. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)