From owner-freebsd-current@freebsd.org Sun Mar 28 14:39:44 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8DCFD57B6CD for ; Sun, 28 Mar 2021 14:39:44 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F7ddr3ccGz4hQb; Sun, 28 Mar 2021 14:39:44 +0000 (UTC) (envelope-from se@freebsd.org) Received: from Stefans-MBP-449.fritz.box (p200300cd5f31b2004c17410e7f93834e.dip0.t-ipconnect.de [IPv6:2003:cd:5f31:b200:4c17:410e:7f93:834e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 228A3F107; Sun, 28 Mar 2021 14:39:44 +0000 (UTC) (envelope-from se@freebsd.org) From: Stefan Esser To: FreeBSD CURRENT Subject: Strange behavior after running under high load Message-ID: <58bea0f0-5c3d-4263-ebee-f939a7e169e9@freebsd.org> Date: Sun, 28 Mar 2021 16:39:41 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="8NgVqi8xO790SvAs7GtbpRIhcJkdVL7kf" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Mar 2021 14:39:44 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --8NgVqi8xO790SvAs7GtbpRIhcJkdVL7kf Content-Type: multipart/mixed; boundary="ElzUDxtRwMaKrPwRmcUzluqPLMh2LVnJA"; protected-headers="v1" From: Stefan Esser To: FreeBSD CURRENT Message-ID: <58bea0f0-5c3d-4263-ebee-f939a7e169e9@freebsd.org> Subject: Strange behavior after running under high load --ElzUDxtRwMaKrPwRmcUzluqPLMh2LVnJA Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable After a period of high load, my now idle system needs 4 to 10 seconds to run any trivial command - even after 20 minutes of no load ... I have run some Monte-Carlo simulations for a few hours, with initially 3= 5=20 processes running in parallel for some 10 seconds each. The load decreased over time since some parameter sets were faster to pro= cess. All in all 63000 processes ran within some 3 hours. When the system became idle, interactive performance was very bad. Runnin= g any trivial command (e.g. uptime) takes some 5 to 10 seconds. Since I hav= e to have this system working, I plan to reboot it later today, but will ke= ep it in this state for some more time to see whether this state persists or= whether the system recovers from it. Any ideas what might cause such a system state??? The system has a Ryzen 5 3600 CPU (6 core/12 threads) and 32 GB or RAM. The following are a few commands that I have tried on this now practicall= y idle system: $ time vmstat -n 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr nv0 in sy cs us sy = id 2 0 0 26G 922M 1.2K 1 4 0 1.4K 239 0 482 7.2K 934 11 1 = 88 real 0m9,357s user 0m0,001s sys 0m0,018 ---- wait 1 minute ---- $ time vmstat -n 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr nv0 in sy cs us sy = id 1 0 0 26G 925M 1.2K 1 4 0 1.4K 239 0 482 7.2K 933 11 1 = 88 real 0m9,821s user 0m0,003s sys 0m0,389s $ systat -vm 4 users Load 0.10 0.72 3.57 Mar 28 16:15 Mem usage: 97%Phy 55%Kmem VN PAGER SWAP=20 PAGER Mem: REAL VIRTUAL in out in =20 out Tot Share Tot Share Free count Act 2387M 460K 26481M 460K 923M pages All 2605M 218M 27105M 572M ioflt Interru= pts Proc: cow 132 to= tal r p d s w Csw Trp Sys Int Sof Flt 52 zfod 96 h= pet0:t0 316 356 39 225 132 21 53 ozfod nvme0:a= dmi %ozfod nvme0:i= o0 0.1%Sys 0.0%Intr 0.0%User 0.0%Nice 99.9%Idle daefr nvme0:i= o1 | | | | | | | | | | | prcfr nvme0:io= 2 totfr nvme0:i= o3 dtbuf react nvme0:i= o4 Namei Name-cache Dir-cache 620370 maxvn pdwak nvme0:io= 5 Calls hits % hits % 627486 numvn 168 pdpgs 27 x= hci0 66 18 14 78 65 frevn intrn ahci0 6= 7 17539M wire xhci1 68= Disks nvd0 ada0 ada1 ada2 ada3 ada4 cd0 430M act 9 re= 0 69 KB/t 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12696M inact hdac0 76= tps 0 0 0 0 0 0 0 54276K laund vgapci0 = 78 MB/s 0.00 0.00 0.00 0.00 0.00 0.00 0.00 923M free %busy 0 0 0 0 0 0 0 0 buf ---- 5 minutes later ---- $ time vmstat -n 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr nv0 in sy cs us sy i= d 1 0 0 26G 922M 1.2K 1 4 0 1.4K 239 0 481 7.2K 931 11 1 8= 8 real 0m4,270s user 0m0,000s sys 0m0,019s $ time uptime 16:20 up 23:23, 4 users, load averages: 0,17 0,39 2,68 real 0m10,840s user 0m0,001s sys 0m0,374s $ time uptime 16:37 up 23:40, 4 users, load averages: 0,29 0,27 0,96 real 0m9,273s user 0m0,000s sys 0m0,020s --ElzUDxtRwMaKrPwRmcUzluqPLMh2LVnJA-- --8NgVqi8xO790SvAs7GtbpRIhcJkdVL7kf Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAmBglS0FAwAAAAAACgkQR+u171r99USX Pgf/Yn0RCxuJ5flyciL/EneUYMQPULUiCitPn7nYysbZyWKXa3AsejiFo4+qkSMbzBAvccZAVIhZ 7OuOfDdwoIjgqA5Z06U83vRdCsUUpRvvTinP8Ifzl2VJSQFJY/biFXmzwRqsvifoubguMHlyJt/O 1calEHYYIZj6i/vuymsqZs0olVxg6w+NTN8/6Xo7EVJizsjxLTkWRh9M7xX4p0ugfdaVI7L3ZDKb 40JSPOv6reVQi59O/6E3zOW55lqB2kr4knPe+wPFzZv3GhosHEMqVZf0PAlBp32ep4HYyWJ22aLJ PUbaV6ZiWuK64jQONsUcHqS/chULWDegOE4/qJqVqQ== =3yYF -----END PGP SIGNATURE----- --8NgVqi8xO790SvAs7GtbpRIhcJkdVL7kf--