From owner-freebsd-arm@freebsd.org Mon Aug 20 14:00:57 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B45EE106E938 for ; Mon, 20 Aug 2018 14:00:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-11.consmr.mail.bf2.yahoo.com (sonic304-11.consmr.mail.bf2.yahoo.com [74.6.128.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 57B368E2CE for ; Mon, 20 Aug 2018 14:00:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: urqYPiYVM1kolQrSpgtaXw8ymhauOQzyMCXtUGuDxe_O92H.RUqZdzXlGlpvfmS NwY3UAnfMjgEJOFAYUmrdxezAr53H0OVdLXv3D9L2kp.UbPDAeXmqhwPA29qHNhun_xp9m0HV0ah MIl9l4hpu_O4nmPMjCtXVae.Xn2knJUZTVCFPpg_gcKSGIbgECVQn3d5mTUxyHYSMkZqB.1ElZqw NE3FLPHqwPFQjAaMg3aj_ugajD32m6rEvBzgZmSNdpg0IjC5GvlqoTvmPgyB1hkQKt.opHvw19Q8 m7gnyuu3FzOuNY8DHN3ckV5atORSRserm7.IvLehxZr0X89BA770ipQN3yq3MnJRV0lUVX.VWMo0 fNr_fUldZBnaidy7u5yngCiilb4jwPcNCQ9X5mxFN4w.evddQTY9v4H0kqQ5wR1LwAExUrCcr9fL QC1rTd2QhXyFUMTtQXv39OIkydyzZYFXnaizvj6UAmsRtTXwr72A4_En_y0uzEGgJYkDxwwgZId8 mLUgyG9c5_TlejCT_sM7HVyY92jFPHtXKiFL8KvVGp64j3SWj5jkWxdCDxwcg93Vc9qPyB871C6i IyhQ5Vn8vTMLh3_rm5Q3F10kIl6azrK9BxF6Gl_kYbJtWgHMbuxAJs.RgnEq3_1n9ZfPXKkclxX6 IHnHRToiGtGSa8CYpIeBHvoKr7KVy_EzYy7if7WqHF6T9HyLEPweaHcR8sbuzn2VaCXzRFG7EWTe cgjOLK4uJw7uFIjdSU5CmhbqUFKvtpqUAu.DcY7OwGhjd7jl9jsBQv_EZ0_NZUVoqRBjupKrJXRO XR854EgcwPaOpKtAw.iro.hSqGwlryJAeGtIMu9zpuHoQ4zUqWvh9wgupT_uHEKk4q4_KkN6RS7t PNdX7Pk4nHl59flPihQHV6iVpIZabLNsJmyF8hjL4EPO02Fx4TTjSSr6oFj97Ihht1Ufb7frhw8q dsC4.KZJTfUzTjhmfv2Ylq879Mtiesd7M38NfHl3S4TVNpPTethZfxW5wb7PN58trhFcqmC6MLrL 1AUzNmeOaOm8X Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.bf2.yahoo.com with HTTP; Mon, 20 Aug 2018 14:00:51 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp408.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 644fa8ea4d4f672076193c7f579efec3; Mon, 20 Aug 2018 14:00:51 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] From: Mark Millard In-Reply-To: <9EA5D75D-A03F-4B25-B65E-03E93DE30130@yahoo.com> Date: Mon, 20 Aug 2018 07:00:48 -0700 Cc: Mark Johnston , John Kennedy , freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: References: <20180808153800.GF26133@www.zefox.net> <20180808204841.GA19379@raichu> <2DC1A479-92A0-48E6-9245-3FF5CFD89DEF@yahoo.com> <20180809033735.GJ30738@phouka1.phouka.net> <20180809175802.GA32974@www.zefox.net> <20180812173248.GA81324@phouka1.phouka.net> <20180812224021.GA46372@www.zefox.net> <20180813021226.GA46750@www.zefox.net> <0D8B9A29-DD95-4FA3-8F7D-4B85A3BB54D7@yahoo.com> <20180815221728.GA59074@www.zefox.net> <9EA5D75D-A03F-4B25-B65E-03E93DE30130@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 14:00:57 -0000 There is a way to explore Mark Johnston's swap information reports (from his patches for such). Taking a Pine64+ 2GB as an example (4 cores with 1 HW-thread per core, 2 GiBytes of RAM, USB device for root file system and swap partition): In one login: # nice -20 gstat -pd In another login: # stress -d 2 -m 4 --vm-keep --vm-bytes 536870912 That "4" and "536870912" total to the 2 GiBytes so swapping is induced for the context in question. (Scale --vm-bytes appropriately to context.) [stress is from sysutils/stress .] gstat tends to show things such as: dT: 1.006s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps = ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 = 0.0 0.0| mmcsd0 56 312 0 0 0.0 312 19985 142.6 0 0 = 0.0 99.6| da0 where the ms/w and kBps are fairly stable but the Length of the queue length is widely variable. For the above it makes the likes of 56 writes queued * 142.6 ms/write (mean) [as an estimate] a rather large total time for the last of the queued writes to complete. (If I understand how to interpret the above.) It appears to me that, compared to a observed capacity of roughly around 20 MiBytes/sec for writes, large amounts of bytes are being queued up to be written in a short time, for which it just takes a while for the backlog to be finished. The following is from multiple such runs, several manually stopped but some killed because of sustained low free memory. I had left vm.pageout_oom_seq=3D12 in place for this, making the kills easier to get than the 120 figure would. It does not take very long generally for some sort of message to show up. waited 9s for async swap write waited 9s for swap buffer waited 9s for async swap write waited 9s for async swap write waited 9s for async swap write v_free_count: 1357, v_inactive_count: 1 Aug 20 06:04:27 pine64 kernel: pid 1010 (stress), uid 0, was killed: out = of swap space waited 5s for async swap write waited 5s for swap buffer waited 5s for async swap write waited 5s for async swap write waited 5s for async swap write waited 13s for async swap write waited 12s for swap buffer waited 13s for async swap write waited 12s for async swap write waited 12s for async swap write swap_pager: indefinite wait buffer: bufobj: 0, blkno: 161177, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164766, size: = 65536 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164064, size: = 12288 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 161630, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 161177, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164766, size: = 65536 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 161630, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164064, size: = 12288 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 161177, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164766, size: = 65536 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 161630, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 164064, size: = 12288 waited 65s for async swap write waited 65s for swap buffer waited 65s for async swap write waited 65s for async swap write waited 65s for async swap write v_free_count: 955, v_inactive_count: 1 Aug 20 06:11:49 pine64 kernel: pid 1047 (stress), uid 0, was killed: out = of swap space waited 5s for async swap write waited 5s for swap buffer waited 5s for async swap write waited 5s for async swap write waited 5s for async swap write swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314021, size: = 12288 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314084, size: = 32768 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314856, size: = 32768 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314638, size: = 131072 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312518, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312416, size: = 16384 waited 39s for async swap write waited 39s for swap buffer waited 39s for async swap write waited 39s for async swap write waited 39s for async swap write swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314802, size: = 24576 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 314934, size: = 40960 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 315617, size: = 32768 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312484, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 328616, size: = 131072 waited 31s for async swap write waited 31s for swap buffer waited 31s for async swap write waited 31s for async swap write waited 31s for async swap write waited 5s for async swap write waited 5s for swap buffer waited 5s for async swap write waited 5s for async swap write waited 5s for async swap write =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)