From owner-freebsd-current@freebsd.org Sat Sep 14 23:56:35 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D0484D2F9C for ; Sat, 14 Sep 2019 23:56:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46W8YH0FL4z4d08 for ; Sat, 14 Sep 2019 23:56:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: Pk7Lg44VM1mhl5nqPWTt24AQrCpWJbiabHpqarYh2tP6kwDS0Kbl7NPQ3F2g_83 buQL674.ZLsE3aVxIkVpf3Hbh8thPwlB0crzaKK6Acq0FW9OKZku4PASJAzY5D7b9rW82rSfdp6Z r003hYSvJf0yTJvR3KtPZog5WvhvNDoZ8lPl3I1PiUNCNkUNbVhzSCEvpZxRAArDAc7GcdvyrE04 HUzLpC6McaZ7RUxnR52XdUgbW4kpoMKZv2cipE7iigXXbA2PhmIx0LpAqNJGIusFLqc8SibRwaOj UKhXlTpa9fvE0vfEVf9EkTEgXgum_i3b_Ffne_tbNoyIyd0XL7dfxNPhHwFzKdHcf9HUsT3.znax bZB3f1ZpIg18nJ2PUuMwQP57OYVOwQUXiMF6VgTN8dRNEGONvaLfUxRSb6mKK9_rplvoicnYs.y0 HEqJmQodpSQ2jaUmh5iG6RKQGBJtKZV7vnaSXe1YhTrkrhOW8MUNCILuF30g0Y2C0vaiDRervXm. j_co3UStpvRz7FCeI.VpagI5FGIEu13Qvc5A539C6Vz9JtWkHjR9QK5PsiqXJcTudG.6sRXS6fvf aeaLl5pFPQRPUQ7yuEl7f2SCrCpn2kkPcbUfMEjbB7Wz42YPkXc6dMd8jVDUxHeOlPWosLCvuwEZ 9BaP9Hef_ZytbG64ZUWUfJUoX_SwGFaOcgoXATjAL37LwDiI13To3HT9cEvzeae4jlUZ3NSzVhId sBxffEIyJKH59B9Y.3EzfQd8wpjOqRgNQFGRWocSyGWeIVMeszXGvSE9d.14TFv0B0.gYRiDd2sI YNw5j3v8.YIZNL_M2cFNpjk33u5IfRXQsk6_Xx8S4Gskt4tFKkj_5kCYFHqcQg4Jc4U.3WaufBE4 MoZFBjLwhkjpX0dfCCU8Vs93Cndqb_B0qQurQl5PE.ZziL7wKoy_TjgXvambwjhxSdjJwsYWoWF2 _LZEqmbiNSRDZQl0cUetJReGXTTPT4yNLpawurAA571Xyq80sEfWEhlOs_SfBAoJxg8cRdWe6wZM UETvUqcwbGtKDgbLPcCAWpUXyJCMvpEaSkCHsz4tAduq1nehT0AL6oJHJLLGmr91V8_Gn2IqaHhW nOUr46onZQciFTvRfKSghDacJ0ivQ1OVsOnkupY0xbco0kcXAniFjpvv8Zz63.g7pALAIhk_4ewj ZZjk_PU3o_qoY0oD2dg3.tNV_q2j0fq0uEwgIizaiZj_yg.f_SysMKYRWUhtPt90kM5aYrRBd9VF TygXb70O0e.MlZjXBZ9tDHu8JSEGn5zbJQhto5UJHJjV_LgCk25CoD84XplnuRLEmU08myf4t._P viWtftmDZeRQA.w8XoYZuAfIwiP2AKBZM.9A5EQ-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sat, 14 Sep 2019 23:56:32 +0000 Received: by smtp401.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID b0136dac9959d9f52e5dda39ac632019; Sat, 14 Sep 2019 23:56:28 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: spurious out of swap kills Message-Id: <8269B1A1-48DE-4997-B1DD-0E1F69EC9A52@yahoo.com> Date: Sat, 14 Sep 2019 16:56:28 -0700 To: "truckman@freebsd.org" , FreeBSD Current X-Mailer: Apple Mail (2.3445.104.11) References: <8269B1A1-48DE-4997-B1DD-0E1F69EC9A52.ref@yahoo.com> X-Rspamd-Queue-Id: 46W8YH0FL4z4d08 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.47 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.979,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.996,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (1.22), ipnet: 98.137.64.0/21(0.93), asn: 36647(0.74), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[148.69.137.98.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Sep 2019 23:56:35 -0000 Konstantin Belousov kostikbel at gmail.com wrote on Fri Sep 13 05:53:41 UTC 2019 : > Basically, page fault handler waits for vm.pfault_oom_wait * > vm.pfault_oom_attempts for a page allocation before killing the = process. > Default is 30 secs, and if you cannot get a page for 30 secs, there is > something very wrong with the machine. The following was not for something like a Ryzen, but for an armv7 board using a USB device for the file system and swap/paging partition. Still it may be a suggestive example of writing out a large amount of laundry. There was an exchange I had with Warner L. that implied easily having long waits in the queue when trying to write out the laundry (or other such) in low end contexts. I extract some of it below. dT: 1.006s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name 56 312 0 0 0.0 312 19985 142.6 0 0 = 0.0 99.6| da0 Note: L(q) could be a lot bigger than 56 but I work with the example figures that I used at the time and that Warner commented on. The 142.6 ms/w includes time waiting in the queue and was vastly more stable than the L(q) figures. Warner wrote, in part: QUOTE 142.6ms/write is the average of the time that the operations that = completed during the polling interval took to complete. There's no = estimating here. So, at 6 or 7 per second for the operation to complete, coupled with a = parallel factor of 1 (typical for low end junk flash), we wind up with = 56 operations in the queue taking 8-10s to complete. END QUOTE Things went on from there but part of it was based on a reporting patch that Mark Johnston had provided. Me: It appears to me that, compared to a observed capacity of roughly around 20 MiBytes/sec for writes, large amounts of bytes are being queued up to be written in a short time, for which it just takes a while for the backlog to be finished. Warner: Yes. That matches my expectation as well. In other devices, I've found = that I needed to rate-limit things to more like 50-75% of the max value = to keep variance in performance low. It's the whole reason I wrote the = CAM I/O scheduler. Me: The following is from multiple such runs, several manually stopped but some killed because of sustained low free memory. I had left vm.pageout_oom_seq=3D12 in place for this, making the kills easier to get than the 120 figure would. It does not take very long generally for some sort of message to show up. (Added Note: 65s and 39s were at the large end of what I reported at the time.) . . . swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164064, size: = 12288 waited 65s for async swap write waited 65s for swap buffer waited 65s for async swap write waited 65s for async swap write waited 65s for async swap write v_free_count: 955, v_inactive_count: 1 Aug 20 06:11:49 pine64 kernel: pid 1047 (stress), uid 0, was killed: out = of swap space waited 5s for async swap write waited 5s for swap buffer waited 5s for async swap write waited 5s for async swap write waited 5s for async swap write swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314021, size: = 12288 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314084, size: = 32768 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314856, size: = 32768 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314638, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312518, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312416, size: = 16384 waited 39s for async swap write waited 39s for swap buffer waited 39s for async swap write waited 39s for async swap write waited 39s for async swap write swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314802, size: = 24576 . . . Warner: These numbers are consistent with the theory that the swap device = becomes overwhelmed, spiking latency and causing crappy down-stream = effects. You can use the I/O scheduler to limit the write rates at the = low end. You might also be able to schedule a lower write queue depth at = the top end as well, but I've not seen good ways to do that. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)