From owner-freebsd-hackers@freebsd.org Wed Jun 3 21:33:49 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 434382F3BF5 for ; Wed, 3 Jun 2020 21:33:49 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [IPv6:2001:470:1:474::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49chx85SYxz3WYg for ; Wed, 3 Jun 2020 21:33:48 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 23AD0D48C; Wed, 3 Jun 2020 21:33:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 23AD0D48C Subject: Re: Constant load of 1 on a recent 12-STABLE To: Gordon Bergling , freebsd-hackers@freebsd.org References: <20200603101607.GA80381@lion.0xfce3.net> <20200603202929.GA65032@lion.0xfce3.net> From: Allan Jude Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Message-ID: <8b1498ea-e343-506e-79c7-c25b594808f0@freebsd.org> Date: Wed, 3 Jun 2020 17:33:37 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.8.1 MIME-Version: 1.0 In-Reply-To: <20200603202929.GA65032@lion.0xfce3.net> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JJy5NuC0GMkBaaeOypx0dOeSnExTkv0aX" X-Rspamd-Queue-Id: 49chx85SYxz3WYg X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2020 21:33:49 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --JJy5NuC0GMkBaaeOypx0dOeSnExTkv0aX Content-Type: multipart/mixed; boundary="o0hL8YY1wPr7H8G1B2kjIvX65vRNX7Ins"; protected-headers="v1" From: Allan Jude To: Gordon Bergling , freebsd-hackers@freebsd.org Message-ID: <8b1498ea-e343-506e-79c7-c25b594808f0@freebsd.org> Subject: Re: Constant load of 1 on a recent 12-STABLE References: <20200603101607.GA80381@lion.0xfce3.net> <20200603202929.GA65032@lion.0xfce3.net> In-Reply-To: <20200603202929.GA65032@lion.0xfce3.net> --o0hL8YY1wPr7H8G1B2kjIvX65vRNX7Ins Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2020-06-03 16:29, Gordon Bergling wrote: > Hi Allan, >=20 > On Wed, Jun 03, 2020 at 03:13:47PM -0400, Allan Jude wrote: >> On 2020-06-03 06:16, Gordon Bergling via freebsd-hackers wrote: >>> since a while I am seeing a constant load of 1.00 on 12-STABLE, >>> but all CPUs are shown as 100% idle in top. >>> >>> Has anyone an idea what could caused this? >>> >>> The load seems to be somewhat real, since the buildtimes on this >>> machine for -CURRENT increased from about 2 hours to 3 hours. >>> >>> This a virtualized system running on Hyper-V, if that matters. >>> >>> Any hints are more then appreciated. >>> >>> Kind regards, >>> >>> Gordon >> >> Try running 'top -SP' and see if that shows a specific CPU being busy,= >> or a specific process using CPU time >=20 > Below is the output of 'top -SP'. The only relevant process / thread th= at is > relatively constant consumes CPU time seams to be 'zfskern'. >=20 > -----------------------------------------------------------------------= ------ > last pid: 68549; load averages: 1.10, 1.19, 1.16 up 0+14:59:45 22:= 17:24 > 67 processes: 2 running, 64 sleeping, 1 waiting > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idl= e > CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idl= e > CPU 2: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idl= e > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idl= e > Mem: 108M Active, 4160M Inact, 33M Laundry, 3196M Wired, 444M Free > ARC: 1858M Total, 855M MFU, 138M MRU, 96K Anon, 24M Header, 840M Other > 461M Compressed, 1039M Uncompressed, 2.25:1 Ratio > Swap: 2048M Total, 2048M Free >=20 > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND > 11 root 4 155 ki31 0B 64K RUN 0 47.3H 386.10% = idle > 8 root 65 -8 - 0B 1040K t->zth 0 115:39 12.61% = zfskern > -----------------------------------------------------------------------= -------- >=20 > The only key performance indicator that is relatively high IMHO, for a = > non-busy system, are the context switches, that vmstat has reported. >=20 > -----------------------------------------------------------------------= -------- > procs memory page disks faults cp= u > r b w avm fre flt re pi po fr sr da0 da1 in sy cs = us sy id > 0 0 0 514G 444M 7877 2 7 0 9595 171 0 0 0 4347 43322 = 17 2 81 > 0 0 0 514G 444M 1 0 0 0 0 44 0 0 0 121 40876 = 0 0 100 > 0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 133 42520 = 0 0 100 > 0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 120 43830 = 0 0 100 > 0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 132 42917 = 0 0 100 > -----------------------------------------------------------------------= --------- >=20 > Any other ideas what could generate that load? >=20 > Best regards, >=20 > Gordon >=20 I agree that load average looks out of place here when you look at the % cpu idle, but I wonder if it is caused by a lot of short lived processes or threads. How quickly is the 'last pid' number going up? You might also look at `zpool iostat 1` or `gstat -p` to see how busy your disks are --=20 Allan Jude --o0hL8YY1wPr7H8G1B2kjIvX65vRNX7Ins-- --JJy5NuC0GMkBaaeOypx0dOeSnExTkv0aX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJe2Bc1AAoJEBmVNT4SmAt+/O0P/2gZobuWgH1swmDrj9+i2Sb0 4dgCEjRitiUP2F2z2IJ8WcUbU4zMuCHIjY/6lOG+AczSfWp50kBP/Q9/lpmKo6F0 PrHNZTN5ODkBkJ+9Rv0bzkBFD/k0mp6y0lUdZ7GxVqvZBsw46XpVkDPY4HFdFpEQ wgjgkPXwgI6j2Y7YONKttHTUue9VpWlfEVatJMt58ebTnzDtsQkji6voTmsilVNu oCK4XEinf4q9aYil8FvA3ywg01ckK/QwSM3sIknEa5TDvwAlba+uKAl44IwyjXiU 4E8BHfoO3YF3iS42bAOWRMHxQJDkQhrrs+kxYivirEyXtkhYehSMi47IEUKTTecx i9+CUDOmgA1+S52l/4tOamzI6/Yc9gV8Qnua31hEQbbPjOrOqLuciCBRITvYSkBa EZuMXwt3DCf2UShoMhKQsPiI+33D0r1/W/K/EQH11OAY4Fz+VSDrs0Smfx1+3X0C LsM4Sl14H7cWbrKA4v+mqooIjN5yZ/bY35fPR2k36kXF762Sc6p1NvPo3CEXN4NA A2/33b3O3z1qjeABn2j3JY6FwdI++tbiroCk6NUBXco+I4w1cTPPKbzxrLf1yNyQ ohVm+zPMJWfIQJDNoMmHceZcFSNCEtrQ7MKFEpcOqiiz+lGqBZcHerxIOsHtYe9C qw4YxOWlFhrCmhiTgpLh =XPqY -----END PGP SIGNATURE----- --JJy5NuC0GMkBaaeOypx0dOeSnExTkv0aX--