From owner-freebsd-arm@freebsd.org Thu Aug 2 04:27:40 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 491981056118 for ; Thu, 2 Aug 2018 04:27:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AEBF8719A4 for ; Thu, 2 Aug 2018 04:27:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 8Na0_XoVM1lXGiB5LaizAvMibPrcHxdIx4A5R7abdBRG2SQtJpviYQgALgKFkRA s7isd6j2VLupygG3wtxm87mL_C2Esr9.aix4HUvzFM.pf4NnnUm4VK3IEfvyZMsgXicflkhTi2Yf ZWHsl0iyaALos9fgLX_MaKKYrJjRJ8o6fq_08.OOWEnQh63QYGuFxJHJJrpCjLChWTbYzVataq4E XQmGkiAgGpPhAShrfdG.10dGVk2Dmlyl86M66AafgflAD5QYDykXAirbrF.DjFri0mkHS8hAZAdZ PovOAwAbaBSUhmwgBb6xQ0GRp0CSMg1BqG2x0L7f0ZXzcIpuAEedkJFlevicRFHqpKqs4Sbum9Eq ho.hog7u0NRDvqJrbhWFNiihtfrCOiWuY4sMlh5XMmPW2VzDlF44A3Vl83XEMsBggUTMfioljQew rRSmjvvNqLmmW.xChNY3Y3nfUzb6LqDmKkpc9lHmvsxLoHdVU2kcmitLmOgGVGj1NSV68J5BLGom 40Wai9a3Sa56A68lVgHENGhZfsXprMgkMIeVZ_XlWRiNt9q2JLMSYm5mVqVKBONK2d7CDuJ8xh7. kscnwS2uShFR2iXbNJaN.D_N2pg0KsV30410aj4xEwM338JxXtQOOLkwpbB1ltlrKUxuetZRIuYY g.1b3vlo8FcHAkq.geArnGNdTynLMBgH_3JIjKM7wiO.cf7x_OqSRhhtREV7XjPPBFD3NEdirDZY ZK5F9qk_9vphwgkPIN1wXWkAb7jJbEIBtmg2O0_Ir1oypxBgRl_ehgDd_2UnmP_svfjsfYVmiamd VJFioxnb2k7PyVUuuLypN1qMpooaXDVrQmhsgDKATNMpSiQADQywXeI.FBg0bIU8iKdaEL_5gOY7 LQya4lW8V6j50dDOWE4euJ0tEK2SwdWWCViSt3oPW7ZfzhjIlu0puKUFGfPjgQaz_s9sHfwYEXDX Hne5K8BU7WhpDIRFXmqi65FD02tUV088s5E_qrEmGL2KNEggXrnqznMAZ9PSUOQdI7QOVmIo8rzN B00sjz8UunAIHRkLhZw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Thu, 2 Aug 2018 04:27:32 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp421.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 812d4bb9e8fa533c39ce787735e349a8; Thu, 02 Aug 2018 04:27:32 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] From: Mark Millard In-Reply-To: <20180802015135.GC99523@www.zefox.net> Date: Wed, 1 Aug 2018 21:27:31 -0700 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20180731153531.GA94742@www.zefox.net> <201807311602.w6VG2xcN072497@pdx.rh.CN85.dnsmgr.net> <20180731191016.GD94742@www.zefox.net> <23793AAA-A339-4DEC-981F-21C7CC4FE440@yahoo.com> <20180731231912.GF94742@www.zefox.net> <2222ABBD-E689-4C3B-A7D3-50AECCC5E7B2@yahoo.com> <20180801034511.GA96616@www.zefox.net> <201808010405.w7145RS6086730@donotpassgo.dyslexicfish.net> <6BFE7B77-A0E2-4FAF-9C68-81951D2F6627@yahoo.com> <20180802002841.GB99523@www.zefox.net> <20180802015135.GC99523@www.zefox.net> To: bob prohaska , Mark Johnston X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 04:27:40 -0000 [I have a top-posted introduction here in reply to a message listed at the bottom.] Bob P. meet Mark J. Mark J. meet Bob P. I'm hopinh you can help Bob P. use a patch that you once published on the lists. This was from: = https://lists.freebsd.org/pipermail/freebsd-current/2018-June/069835.html Bob P. has been having problems with an rpi3 based buildworld ending up with "was killed: out of swap space" but when the swap partitions do not seem to be heavily used (seen via swapinfo or watching top). [I will avoid the long, complicated history of investigations here and also any past hypothesis about contributing causes.] Bob P. recently introduced Mark J.'s patch to report the likes of: v_free_count: 5439, v_inactive_count: 1 . The issue happens during time periods were simple means of observation suggest that their is lots of swap space available. Sorting by time (a looping script was running, logging output, including swapinfo output): Wed Aug 1 18:08:10 PDT 2018 Device 1K-blocks Used Avail Capacity /dev/da0b 1048576 28248 1020328 3% /dev/mmcsd0s3b 1048576 28256 1020320 3% Total 2097152 56504 2040648 3% Aug 1 18:08:13 www kernel: v_free_count: 5439, v_inactive_count: 1 Wed Aug 1 18:08:21 PDT 2018 Device 1K-blocks Used Avail Capacity /dev/da0b 1048576 31768 1016808 3% /dev/mmcsd0s3b 1048576 31640 1016936 3% Total 2097152 63408 2033744 3% Aug 1 18:08:25 www kernel: pid 93301 (c++), uid 0, was killed: out of = swap space Wed Aug 1 18:08:35 PDT 2018 Device 1K-blocks Used Avail Capacity /dev/da0b 1048576 24840 1023736 2% /dev/mmcsd0s3b 1048576 25404 1023172 2% Total 2097152 50244 2046908 2% The above is a clean up of the output which had more and had repeated information from the tail of a log until it gets new messages. The original script was something like: #!/bin/sh while true do gstat -abd -I 10s ; date ; swapinfo ; tail -n 2 /var/log/messages=20 done I ran out of ability indicate what more to investigate. For example I'm unsure of when to do the "sysctl vm" that you [Mark J.] have suggested back in June. It is unlikely that Bob P. will happen to be there when buildworld has the kill(s) occur. When it happens in the build sequence is not stable from one try to the next. This introduction is a reply to the following. On 2018-Aug-1, at 6:51 PM, bob prohaska wrote: > The patch to report OOMA information did its job, very tersely. The = console reported > v_free_count: 5439, v_inactive_count: 1 > Aug 1 18:08:25 www kernel: pid 93301 (c++), uid 0, was killed: out of = swap space >=20 > The entire buildworld.log and gstat output are at > http://www.zefox.net/~fbsd/rpi3/swaptests/r336877M/ >=20 > It appears that at 18:08:21 a write to the USB swap device took 530.5 = ms,=20 > next top was killed and ten seconds later c++ was killed, _after_ da0b > was no longer busy. >=20 > This buildworld stopped a quite a bit earlier than usual; most of the = time > the buildworld.log file is close to 20 MB at the time OOMA acts. In = this case > it was around 13 MB. Not clear if that's of significance. >=20 > If somebody would indicate whether this result is informative, and any = possible > improvements to the test, I'd be most grateful.=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)