From owner-freebsd-current@freebsd.org Sun Apr 1 02:31:24 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6165F73ECF for ; Sun, 1 Apr 2018 02:31:23 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic317-28.consmr.mail.bf2.yahoo.com (sonic317-28.consmr.mail.bf2.yahoo.com [74.6.129.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8D899839B1 for ; Sun, 1 Apr 2018 02:31:23 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: jRZ_IbEVM1mDE7zogIS1ClFi8g78erpD2dFpwa6gP8Uznf4RplOCtsONZZwJaEK jQ0p6_CTTc.Pz4i_WmY26YiinQsQlfn9Hy0GOwNoW9KEXblTjY_tl4vXilcqNFmc0b0OVY0XF4.d ESuv2fb2Dne6zX.PhheYoW4Duayb6Ab6m7nNPfpzfN83idSyugj4tSHxcxcEdacA0LrKpHy6Ab0Q Tx6lTbqUycKf6kVFGs184wvUz8OJ3Zvr4KWKM1HS66IsHqGw0k1vIhsH0piEDR79yxK7PMVQrXw3 hdiKwkicI.5xXFmPCwlb4onj5fFreo5WfoNLcx8KRhPu4CGPoD1T0D81iFyBrhKLY14WriANYtKF Qf9WnbdPmlR_J6XeBHVCphXC8VEo.xACJze8iM7MOVwwjDuTFsjM3t9o3oI9cQjeiBmcGUZopfNf ommP1RbEJOn9Ih9l0jaB5YpKRK1tW4qnaTc_wdsm8.ogHmkAi3.mt.LB3fELNHW6pOnH3DtMH04G _aVcDNI.wFTtWxzANpPf9tQd36ogNFLDL9OT6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.bf2.yahoo.com with HTTP; Sun, 1 Apr 2018 02:31:17 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp417.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 7b8b6710b224f1a999599491619c0e43; Sun, 01 Apr 2018 02:31:15 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT From: Mark Millard In-Reply-To: Date: Sat, 31 Mar 2018 19:31:13 -0700 Cc: freebsd-current@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: References: <20180317103915.081ca2dd@thor.intern.walstatt.dynvpn.de> To: Andriy Gapon X-Mailer: Apple Mail (2.3445.6.18) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Apr 2018 02:31:24 -0000 On 2018-Mar-17, at 11:26 AM, Andriy Gapon wrote: > On 17/03/2018 18:51, Mark Millard wrote: >> I'll note that top was a -w that reports: >> >> -w Display approximate swap usage for each process. > > As far as I can tell, this option is quite broken. > The "approximate swap usage" it reports is nowhere like it. I have a hypothesis for part of what top is counting in the process/thread SWAP column that might not be what one would expect. It appears to me that vnode-backed pages are being re-classfied sometimes for inactive processes, and this classification leads to top classifying the pages as not-resident but swapped (in that a "VN PAGER in" would be required, in systat -vmstat terms). Supporting details, if you care, otherwise skip the below: The hypothesis is from observing various hours of over 20 hours of poudriere-devel "bulk -a" activity, for which, at that point: vm.stats.vm.v_swappgsout: 0 vm.stats.vm.v_swappgsin: 0 vm.stats.vm.v_swapout: 0 vm.stats.vm.v_swapin: 0 vm.stats.vm.v_vnodepgsout: 6996 vm.stats.vm.v_vnodepgsin: 32641833 vm.stats.vm.v_vnodeout: 1030 vm.stats.vm.v_vnodein: 4305027 Sometimes top showed lots of wait/select/pause and such with positive SWAP and (mostly) 0K RES. Most of the processes were in the "wait" STATE. At other times, more like between 1 and 2 dozen had positive SWAP. There would be sudden large jumps in the number of such processes. Then over time it would decrease as the processes quit waiting (children process trees finished). The large jumps were not tied to Free becoming small or anything else obvious from what I was looking at. But the Free figure would increase at that time. For example, I recently saw such a large jump that was associated with Free increasing from "90G" as shown in top. (Much of the time there were between, say, 170 and 300 sleeping processes.) The context was under Hyper-V with 29 logical processors assigned to FreeBSD on a machine with 16 cores/32 threads and 114 GiBytes of RAM assigned to FreeBSD (of 128 GiBytes) and 256 GiBytes of swap-partition set up. PARALLEL_JOBS allowed the 29 and ALLOW_MAKE_JOBS was in use (allowing a potential for 29*29=841 or so running processes via poudriere bulk). For reference: at 25 hours-in [idle] had 148.3H (around 20% of the 29 threads * 25 H/thread) and [bufdaemon] had 48.9H (around 6.7%). [kernel] showed around 13.6H (817 min converted) and [pagedaemon] showed around 1.7H (101 min converted). Other processes had less TIME than any of these. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)