From owner-freebsd-stable@freebsd.org Thu Mar 4 22:16:35 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CF8A355C583 for ; Thu, 4 Mar 2021 22:16:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-22.consmr.mail.ne1.yahoo.com (sonic303-22.consmr.mail.ne1.yahoo.com [66.163.188.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ds4w25qVKz50bR for ; Thu, 4 Mar 2021 22:16:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1614896192; bh=o2r4s+JeJgQ+n9N9CMsKEcPD10GBai8+CLTxF+7VBK6=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=m8rP/lxJJVpAVIokN6Bkd3GZ4NSC+Bo5N2u70mr1TCMzhm/Kt0PpQS7TMSjcVEwQLJMvTjirjmCDXTj3YFGItohW10h1xvzdvZwO1be5TJAZO0S8UIrIsez+1qjapck/Xh4fbC37+d75nJAhEdSs4zLyU1tCIUosIJixb2JHpSvgNwqnG1v/REukrRzriUtQeSLxT2Wona8bzvwxXpnuJ3pNKInZgZ03+XbiOBl301nsbVpw842hpveNcZ+2KBbiaTDPOmxbvsFc/FDgbBPJOU3Yo1f+DsBUo84xYtuVjDrTvJ9/UhWlcm34FckLXj7ss+2NXDdLwoPp+Ki8Y08M9A== X-YMail-OSG: OmBtWyAVM1nY4HXatd7YSXxG3mUTcIL3WCjud6iHmDC6m_hRMrKhaJSKVOP_4zg 6Op0uHZxMuVY24I.gMVvcrSltdFwx94XJD.YEH5CxIXmUPyvSRWbNZzHPg9bQbHmOGkNPgPJIxaq H0uEykirvDPnIsEGl9_P8QqeJvXJ9Bt5aEQvSqzdLwhMFzgzbyPjuoIHplLIO7QuP4Hrz98sFFIA 4JXEWE1KRntgdMTT397bDo9ZOLm8IqK2lr1nh_a4_3Ph8r5xQfPfWvKAIBo841fSMse5RZXx.dBN 1heBTrvQII4EaJ7UjmjQvP_hit.Pf1iMyGEUXRLWNUmhbDfdRAIMtsF1A7L0aO0yGgkQYzCKBLNI xXux3n2pngbEfPLzOo_S4kSy7urdDws5RAEPbBqpz2Gm7vtTLPDhnE5bGXc1RrffSnHVv6j3.kio kHjmjcvcA9h4p1wOAYlYKs.OLEBd6AFyTYM_AaO2Yonr2li29pDwT12j54suxoQq76N7yy5sJ9nT N3iRaBb9bbhxtdmmLpB_EIoBAtBKokYHUO8ANeCE.gZNcR2VLqgXR2sTMRiu_Ub4jccv5QvQxJ68 Pq52BPs748._Mgti_L7A2t5zSgd7e5mCunzgKM3iGTiI90SqXJV2rjohouP6w0.c6RWDo5gCLOvF _puFbUDL8nL6lgA_jQOvUAlCxQ9OO_bPaRh2FRisfxKsMETbH9Rb0RVUxrblehJVMs2IiATT8nHW H9B0VKYHWkR8JbWnkKUYoPen60Ppmu_.c_Rd9E8MFR.LZIt3TWzTelODRtBN.HVoRiroqNlULenm RZkIQAnjIo7gGXasNi80DGLYOYQ35Usmdg.9tweC0Fuwc37Y6bSCUDr1z90kQCVljFhz_wmZ0OdQ i1aRQwsU.TJ1DfBwYeHALIOkQekMUXM36Hgr395SgkkNgt14QRZ6JU4oQpZZQYw298KZ5ajDmv5S QqC4Qr6bLWds1DLmSH2jq6P8NkIsjlPx1k.rZgFTcq_E.MAgjzYrMQ_fqlpKbS.FwLjbWOiCvmIb iRwl4G2xz3C64l0hhxJTbxDaSQIAt5gjParxi8vQ7pPk0NACkeUFwUcigSQci.WAPdj3CZOsqbke stbI3T1U44SbfmCrch87_G1odtPlE_hT05WuxBl7_4PexXmuIZiSJjU3axQNEWG.wDW2Ukht.fZH l4KC7YCB.JrgO2wu_qoMSFteK6LOsYE5.MPMxc.8Pg434ipMn2PC7dyHctlXK60Upo18y4claIqv 38g4jyd2OoW.DzZAq8R.h9Wa8UkE.qO7NQ90WqbxVWjWlfxpJJ46A5AlM10byR9DrjvWDo.zaU3P lovp049BMy6PRh.CFlcHPBFgFgfCokKlpKZt2muAMh8mjFUYeOnl0CwJnY311_8AYETz5xBCpRDO HArb9EKJRXx1X.B.s8ONDeRB48vK67NvgqCYb73eyUqG.2h.AwdNS0jAr1gUFh9dBEcLF5YZr0aC jWtvXDK.FcvxEGNV_32I3PftMBL9QbSqFe4E9VBCIpot0u_IxzdhR1TbN0IaUgCn3uOzfamSl2WT .Rvhp_c71gzxq2n.F1nx5NVok6g.NdkLDrQ5hnD6xq0TDtOlUiCd8cc0EwvoYUfOnTVNuS.LmSup jJjL8Pum8Kv183hfyFK3d.a.JeFN88E_iwSYF1310GJFSGRFHhp2U3AXVIYdRBzIs6.EPJEEqtOn O9I9_mWZ5x7tElQjXZVxexfrxYMJORL3Dxu.LKQ2NWic0fWRVdM0yKnR5vSS8CgoPtwusCwOB_qr cO9mCDm.CIItOXgtOsSD3RTaYm6XioIj0UXIQYr3uQjqh1FQrNu7YBrOV8jQ4.LpDfsCOZJfPiB7 HnKY9YK_3_Fm_eg78_zOaC8IYwF_k_uhListaHlU0n.h7mgiXPbcL0pFkgVWMhcxM7jL50qSesag 68BXyUPm1YgZkVGA29esgn9xTBLjalmRRUvAwuXgetpOF37h1HtwFmpkgdKV8Qo.j1Mng9d8AiBR PLb4dO_cex5.i0px7aQfWCGvPV.BDMnRYyTOBGBxcmf_tIyIf_rzqC3f5a0LZurp4owvE7FxkZ0m SrXZg8kQqup4huAfYSLDmDlaXmtn0amBm0iUvZ1mAsZWOY0KkbyUl4Xutnc4TKwr_vPTiUjVs9FY 82aouU6mfuzWD2icugCQ2WkRmW149_R1L9u_kgM7Rd5rljNGsDBGY8O_dHX__MmS46W0YdRNeiWW NoL69rimpK93..yh6cbz5yS.jVB2EbeTaHzrom0gi9NvEETCyt_v6Qa3sUAbY.rZlzQY_t_gbOFM SnxJ0ZAZGNsojWmeSt1ATS3DV_NhNWqyKADS8KYdsvaGnY8pSBKwp.U1RCXctD37kNlYyVZD08Qe yxXH8m2MNU_Uc9MzxDHyy1tetsi6NOMY8.4q6D52Ae3QLJC412hy39SzJxi1C0_rLjI0k1kelMAv oBlGvghmbcM_M6uujRyZGnCYxcw-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.ne1.yahoo.com with HTTP; Thu, 4 Mar 2021 22:16:32 +0000 Received: by kubenode569.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 65138c9888337fcfdc957b109a945a9c; Thu, 04 Mar 2021 22:16:28 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: Filesystem operations slower in 13.0 than 12.2 Message-Id: Date: Thu, 4 Mar 2021 14:16:26 -0800 To: chris@cretaforce.gr, freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.3654.60.0.2.21) References: X-Rspamd-Queue-Id: 4Ds4w25qVKz50bR X-Spamd-Bar: - X-Spamd-Result: default: False [-1.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[66.163.188.148:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(1.00)[1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[66.163.188.148:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[66.163.188.148:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[66.163.188.148:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Mar 2021 22:16:35 -0000 Christos Chatzaras chris at cretaforce.gr wrote on Thu Mar 4 21:41:01 UTC 2021 : > After finding slow filesystem operations with 13.0-BETA2 I did more = tests. >=20 > All tests done with same hardware (Seagate ST4000NM0245 4TB HDD - 2 = disks with RAID-1 using gmirror). >=20 > Filesystem mounted with noatime. >=20 > Command used: >=20 > /usr/bin/time -l portsnap extract >=20 > but similar differences I see with "/usr/bin/time -l rm -fr = /usr/ports" I doubt that "rm -fr" gets large differences of the type: (from 12.2p4:) 0 messages sent 0 messages received vs. (13.0-BETA4 and 14.0-CURRENT:) 4412 messages sent 2536379 messages received In other words, large variations in Inter-Process-Communiciation counts, especially "received". It is not obvious that the "portsnap extract" issue is dominated by file system I/O vs IPC issues. portsanp is a script and does something that looks like the following, with the "while read" happening over 29000 times: . . . | while read FILE HASH; do echo ${PORTSDIR}/${FILE} if ! [ -s "${WORKDIR}/files/${HASH}.gz" ]; then echo "files/${HASH}.gz not found -- snapshot = corrupt." return 1 fi case ${FILE} in */) rm -rf ${PORTSDIR}/${FILE%/} mkdir -p ${PORTSDIR}/${FILE} tar -xz --numeric-owner -f = ${WORKDIR}/files/${HASH}.gz \ -C ${PORTSDIR}/${FILE} ;; *) rm -f ${PORTSDIR}/${FILE} tar -xz --numeric-owner -f = ${WORKDIR}/files/${HASH}.gz \ -C ${PORTSDIR} ${FILE} ;; esac done; then I expect that the "tar -xz . . . *.gz" sort of commands also involve internal IPC use. (It looked like the portsnap script has not changed noticeably since something like late 2016.) (13.0-BETA2 showed a large "voluntary context switches" difference as well, but I ignore that middle step in the version sequence here.) So I expect publishing the "rm -fr /usr/ports" figures from "time -l" would be appropriate. I do not know if the reports should be via separate topic or not but I doubt the figures with large differences will be the same for most-modern vs. older: I do not expect notable IPC from "rm -fr". > ---------- >=20 > FreeBSD 12.2p4=20 >=20 > 98.18 real 35.31 user 59.31 sys > 49064 maximum resident set size > 21 average shared memory size > 3 average unshared data size > 86 average unshared stack size > 11821398 page reclaims > 0 page faults > 0 swaps > 29 block input operations > 2783 block output operations > 0 messages sent > 0 messages received > 0 signals received > 354648 voluntary context switches > 322 involuntary context switches >=20 > ---------- >=20 > FreeBSD 13.0-BETA2 (2021-02-12) >=20 > 497.88 real 76.06 user 120.03 sys > 49032 maximum resident set size > 22 average shared memory size > 3 average unshared data size > 91 average unshared stack size > 12288156 page reclaims > 23 page faults > 0 swaps > 29890 block input operations > 621229 block output operations > 4412 messages sent > 2536379 messages received > 0 signals received > 1004790 voluntary context switches > 251 involuntary context switches >=20 > -------------- >=20 > FreeBSD 13.0-BETA4 (2021-02-26) >=20 > 163.81 real 71.93 user 107.32 sys > 49032 maximum resident set size > 21 average shared memory size > 3 average unshared data size > 89 average unshared stack size > 12288156 page reclaims > 5 page faults > 0 swaps > 716 block input operations > 868 block output operations > 4412 messages sent > 2536379 messages received > 0 signals received > 355244 voluntary context switches > 277 involuntary context switches >=20 > ---------- >=20 > FreeBSD 14-CURRENT (2021-03-04) >=20 > 255.43 real 74.94 user 148.90 sys > 49032 maximum resident set size > 23 average shared memory size > 3 average unshared data size > 96 average unshared stack size > 12288156 page reclaims > 23 page faults > 0 swaps > 31207 block input operations > 175 block output operations > 4412 messages sent > 2536379 messages received > 0 signals received > 385527 voluntary context switches > 369 involuntary context switches >=20 > ---------- >=20 > Differences between 13.0 and 14-CURRENT maybe related to debugging = features. >=20 > But 13.0-BETA4 is slower than 12.2. Does someone have more information = about this? Again, I expect that the "time -l" figures may point in different directions for "portsnap extract" vs. "rm -fr /usr/ports" in your context. The question may need to be split because the answers may be different. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)