From owner-freebsd-stable@freebsd.org Tue Mar 23 07:56:17 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E472657CD5F for ; Tue, 23 Mar 2021 07:56:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-8.consmr.mail.gq1.yahoo.com (sonic315-8.consmr.mail.gq1.yahoo.com [98.137.65.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F4Nwd4PR1z4Ssj for ; Tue, 23 Mar 2021 07:56:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1616486174; bh=moJ8WXPb9sbF1Dx4fmAXJb4P//9ecI4hRRUuznOzAh3=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=OMduTXucQssz0CK5jddP9wJ0ee0/RRLOTQ7UmBct3J1ygpSHpukY1Eoi9jj/C0sBs/m5Wdqf4oCvDXyVidBt6k6qXx1wkJCuOw7n7xS2wlIVm/RzBk9kYwHPPlFXlg98dO6teuBLxV3vhQJAGnTNBR7uLNB2SXUq1dKdayg8HpRM6UTeiqPH3cJrdX3aN6CiABilvFLe7Mcusl/YpgavNHb/vhH3OEBHP48mtMPaxrfqRrGQGO6zJncgpnpFDi55gbFoIUDiCnY6uNPKplIqwutJL/mgOUw7+6r7+PpYo2en5TYTqWbFoBHQ6UAFld4bA+ebdQxC3B2Z+HpdDcrdhw== X-YMail-OSG: wzuS2G4VM1k9F5f60o1e1HFwKVCMsQAxb9e96VWZulNPQWShOmoqgQs4ieG5sB7 4s5UGjWCuOkJHKstxK2OtVGPNwQpv1KX7ZNdbo41GM4jHzFPDz..Yi1APiIIIvw23wBmLpYVpEFM SmzGZIs.I1l238P8q3_z5QNtBFirbDAhjuNpFCaKO8Z8oLQh0fMv0OMzFSUouQAsYNCFaEzifuVt .JtRSWJ49UXtYVfSbzJJ4wRZfQTD.0L.92ay1uxkfu4xv05OWCmSFlW6aKNwvirUWJAYzwlS9K1z b0ajfNieYE9yjB6RHuxj3k57vvG7FSPzbDS1Uk5bzQX.jCxD4uoBuHRUv9DsI.MGfwT4q5ylWeSm 6ILfeJ2mcjyOyoKSBuqQE0YhauIHZBEPdOQepfNyANvshSusp_EBKwSgerbwDZWkXIWas25tTDxd kuCWFdduY_g5QrTGC3gZMeXUWlheI9txn__oPmNOvoZSMnSeXTZqMcaRpucrOHgYbCSJX8ih7za7 HGszsYkTI.AdRbOvZFpt9i2S5FGvfkvu_lIaPFS4rj03.Z9SyvV4Wd_wGaAEaPDhnQsgvsH6Fyoz uH4jYAAgD3OI3z6mb_AAyLZqinlT0XAxfnAWjcs.kBXMZ81FZSz0kzgZwoyB7uAf737fb0_1WfdY 374TAcow2iF_hbPMJ.pOKBAq2l2Mk_odCdHIHR4tEcaT_qmp1LJwXongzcOJJf9vs_V51g.0vMLR 10OKNYzi2752GZg2NgLUj2zc11ulyQDH99k8sM1DZ2ZLavxTv.jBOOqzbMG7E1KmJUwNs0FJwzU. EG8IRr6jtnY4NyJSjxIzXy4PFNYVEqIIZPtlVIFv8v1WBgIymdHsEvEOixmuM5_.DtQUf4yIJlL8 RFrVHvIFevnM9HjsRfIu9SGqgtjQQOoB.5Ga1PSBQ7rUJTPF1GKYhlO6Ux8wx8ZAfMbjPqFvpybX mp8wjKFcMzFGxy8jrrInXUB3A6OUV6NtqIJBwwYCZUSti7CGs5UntEHis3nZSpv_iiBhgvpodlw8 LWcRq_Iz1f5Vn26pKpzDI3RxPReThkOuTi3YUj0GsZu71jugl.g41r1ylpoYZtVlfPhbqO6aIeHG Hpb33WysJmZjVjgZf9qMWrotQhMjW701IeE1xiFGJTLEeCAqy7mTJEjwOLGZkzeKGFjOUtVM5Sw2 x6x_4h3u0xHCmR0FbwGblK0a8dMgSaTL_qcOY4RsAX98z82Z5rnJYuPrypDGPqxjHouWMrV0gefr dm4l6gOpUNR4qUnl6aq8TI.Dk2uD2_N1OOJkaNvMra7uEbeIJ9a4rc6wfvMpR6R9cY5SWKz3CPlb dSQh4yvxsy9wMvlhr9YYFk44rggqxMosVKNP7E4ZxYw0q8dmGtqErfztjHlsQXiFkQMKBnqk2G7v VRB3LYI4ybrH5EFm4h21W8GFOUwHSC4kT9SXY1.A_cxHPauEjiP2QbFTpBytoWbe7PmwPIF.Yl96 OYIxyLjJ8MHhKt32DQ6KW0EqnYdvQzgRx9dJ.htneJaDdodrXaje4D41W.qoW07NXfR_9N8i677L eclJH_s2PUwo.Y6t8jZIZnaVogERoDd6t6JAuIFwJKpkEqlqn1Eu.oSrlrJQy3oBd2I_c_KivmSv su1KwJn5zdO5n6HlLWiYzsWhv.6ghcV3H7II5pwyjFwTqTz2hvNwV8fc5zB8wjFwUS2phNr1iRRF HkYw325YUzze6H9oQ17HctDsZTDUY63R5UT_ge2_XjuzD2rg59AykNq7CE15TF6dMtTEvX1pLILm FbOhdNcGbaSYxUsMuFQAhxhx60yE5W27n90Vv05ymeANBLGWKrmP5RdQcqxGc50BHfpEytOFtAR9 BDSsuAI6M9vAB275Pc_gifOWgyS6KzUqD1tsyXOM52szW3lSddOyruAvQeD5RMHiR13rQFaPgW0H K25v9CX0fJ45lsiaMrrZuS8hfKCq7kmsWIsXO8px.vRxcpw5_blpJgVdxcwQJeE85Ojbd2Hwix5y 4htr4aUlbYFgCw2NnXbk8jn2pi0jBMjlej9BKbIAjg.Cv7Sx4g0oP6VYvl72_9ePUIEfOHiZ2HvU P0j_7jLv6.BkbnlzeMBs3rUB9UDzuZrdMgMXX8JTBMiHK9dt6yAIbj85b2G2PO14xd4gbhMqJfph lapnFE01UYTTpi6i9Xeruqu73o8qVYQOvwMwqGJNeLDKOhsR9nupfoRW5BZBFhJzXpk0vz5l6L41 FwGS2Zi1JtVyl7g09q_uzOs6CmyneAmr7rqj5SAd.gyC6g0Jh333TBwfgaaJYJ5UvUFbtoydobuH RGIkhAayNlYN3E7bu3.F0msTGMfH_T1izVicvtRoVvXrGC6y6gOa58EMZ_mPmSPw8H5ICDf1Jojp rkXgNAE3IydE_PWqEzHqpz0tcBXyDNuFTrm.O8Gt3LJQrc4Sl2XvtaZyx.aKl6RqpVJNBHbXTxSJ daw2nfcA6HlyqMcOtTMnOM0kxfqREB6XOzxNUGaYlymBxjn0aSFpT9U0_5W.3b86iXmSby6YiX93 vKwKmjNAmnEa6kdmS2Tcndd8iMITCKMbsQNNBKmhLtx7g80Wtn0Yo6YD2HgVU8.2_DS6c6_iOvTP lmz3aGbZrt_XOfK6TgP0HuOzK81v9wMFwA4gRzqyvLWREvy252fRoRxvfaWwUx7qT6TDcW4qNxfk _klWkbEykjR4pY_5rz8YCmYNfzNsLBc6OrJgk6xap5f.pMEgDs3yJBNH_75ypMBmvg1Md9K2koV_ cUMi7pnGUvcdSgn.s1oDFkZ4kEWicebJmfpQ.uVbs5i0- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Tue, 23 Mar 2021 07:56:14 +0000 Received: by kubenode506.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 2d09a7cc219872f9c5b2220c1b923b83; Tue, 23 Mar 2021 07:56:09 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: Filesystem operations slower in 13.0 than 12.2 From: Mark Millard In-Reply-To: Date: Tue, 23 Mar 2021 00:56:08 -0700 Cc: Adrian Chadd , FreeBSD-STABLE Mailing List , Konstantin Belousov Content-Transfer-Encoding: quoted-printable Message-Id: <380B1597-C4E7-4CF0-AE04-085D4745BC65@yahoo.com> References: <12705C29-53EA-4484-8291-C409AF4B3DE5.ref@yahoo.com> <12705C29-53EA-4484-8291-C409AF4B3DE5@yahoo.com> To: Kevin Oberman X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 4F4Nwd4PR1z4Ssj X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Mar 2021 07:56:17 -0000 On 2021-Mar-22, at 22:51, Kevin Oberman wrote: > On Mon, Mar 22, 2021 at 8:19 AM Adrian Chadd = wrote: >> On Mon, 15 Mar 2021 at 14:58, Kevin Oberman = wrote: >>=20 >> > > >> > > It appears that the messages are associated with reading >> > > the disk(s), not directly with writing them, where the >> > > reads take more than "hz * 20" time units to complete. >> > > (I'm looking at main (14) code.) What might contribute >> > > to the time taken for the pending read(s)? >> > > >> > The reference to hz * 20 woke up a few sleeping memory cells. I = forgot that >> > I cleaned up my loader.conf. It was largely a copy of the one on my >> > decade-old T520. I commented out "kern.hz=3D100". I don't recall = the details, >> > but I think it was actually from an even older system, my T42 from = before I >> > retired. >> > >> > In any case, restoring this setting has greatly improved the = situation. I >> > now have really bad disk I/O performance on large disk to disk = activity >> > (untarring the firefox distro) instead of terrible performance and = the >> > system freezes have vanished, though I do see pauses in response to = clicks >> > or text entry, but the display remains active and the pauses are = short... 1 >> > to 15 seconds, I'd guess. No, I have no idea what this indicates. >>=20 >> ... which drive controller is this? Is it just a laptop ATA disk? >>=20 >> > I'm still not seeing the performance I was seeing back in February = when 40 >> > MB/s for extended intervals was common and I once untarred = firefox.tar.gz2 >> > in under a minute and performance seldom dropped below 1.4 MB/s. >>=20 >> Did you find a resolution? I wonder if setting kern.hz is kicking >> some process(es) to get some time more frequently due to bugs >> elsewhere in the system (interrupts, IPI handling, wake-ups, etc) >>=20 >>=20 >>=20 >> -adrian > No resolution. This is a Lenovo L15 ThinkPad with a 2TB ATAPI drive. I've not found documentation indicating the "which drive controller" answer. That may have to be answered from boot messages or boot -v messages or other such on FreeBSD. (I've no access to such a machine.) You might want to put a copy of such a log someplace that folks could look at it. There may be commands that some folks would like to see the output of. (I'm not all that likely to be one that could put such to use but other folks might be able to.) Intel=C2=AE Celeron=C2=AE? 10th Generation Intel CoreTM i3? i5? i7? > The current drive is a Seagate. All testing has been done since I got = it back from Lenovo in late January. I can read or write the drive at = reasonable rates that exceed 50 MB/s. Extracting a tar distribution file = is painful. I have had firefox extracts take over a half hour. Worse, if = I do other operations while the extract is taking place, I often see a = 30 second (and, occasionally 60 second) display freezes I thought that you had reported that use of kern.hz=3D100 had lead to "the system freezes have vanished" and "pauses are short... 1 to 15 seconds". Did more testing show that to not be always the case? > as well as log reports that of "swap_pager: indefinite wait buffer:" Unfortunately, I do not know how to investigate what is leading to those message being generated. Figuring that out would seem to be important but I do not know what to monitor to at least potentially eliminate some possibilities. One possible thing to look at is something like "gstat -spod" output spanning the time of the untar. It would at least indicate if a large queue backlog was accumulating on the device. And the ms/r and ms/w columns would give a clue if commands are sitting in the queues for long periods. (The "d" may be a waste: no BIO_DELETEs possible? Also, the r/s vs. ms/r are not rescaled reciprocals but distinct measurements. Similarly for: the w/s vs. ms/w.) Given the "indefinite wait buffer" messages, I expect the ms/r and/or ms/w figures to be large at least some of the time. Knowing how large may be of use to someone. But I can not eliminate anything with such information. > This is a bit odd as I have 20G of RAM and am pretty close to no swap = space activity, but, of course, paging does occur.=20 With 20 GiBytes of RAM, what is going on at the time that leads to paging activity? I'm thinking of just untarring the firefox file, not building firefox or such. Can you test such an untar in a context that is not otherwise paging (nor swapping)? If yes, is the behavior different in any readily noticeable way? > This system is CometLake and graphics are not supported on 12. I am = not absolutely sure that there is not a hardware issue even though the = main board, the disk, and the keyboard/mouse pad have all been replace = since I received the system back last June. I now wonder what else could = go wrong. =20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)