From nobody Sat Aug 19 18:40:59 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSndB3mqcz4mKKF for ; Sat, 19 Aug 2023 18:41:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-55.consmr.mail.gq1.yahoo.com (sonic308-55.consmr.mail.gq1.yahoo.com [98.137.68.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSnd868JQz4Xp7 for ; Sat, 19 Aug 2023 18:41:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ZrlW0Egb; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692470473; bh=bUgIEGa41pnqkDwg4aEaM4HPHHGGv5WOy4pjJRDm+XQ=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=ZrlW0Egb1rp/jNevygAZT6pCU3EEi0uK1HKCEK9x7pG/vvg2kWYxcM13/6ktQwv3zXtEWuw5SJCnkeWyDNz/NKMkN3cVbsL33+2YAb4CIlbW4xKeUG9JqJfw1fnFUPCkDS4v7Asy6FvNGW1kkO43f8PqmTBKDUvnMO7Mdm+aFXxrOXc1tpRK+qRC1s2eBQBi+VAWosUPtf5uyDUeExgZptDQpnUksfNttiFTzvYHgN/GiEkjBnZ4bOlrIg8EAN5SMeEv8AA3GfPA5NSYxj0vTNsHUk94+eExpYuQzZumeY2WS3O1W8LrRNSuUME2OWUC/yx9a8qCMYtSVQXh6idP/w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692470473; bh=H0j6+dCLyhIS2fOlCaUJYdqkNOvLWZ+wYMCXtMOFkJI=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=rZVC5/SiffYIu2wR9hwTJ7DLVhsYiAFYFcI0I7PLKhP9xqtbI9X4lMGraDcxkQVOWnzX4ckJPQnU9EEXjqXqa3wD0fU+16QlTZyZiRxC6EDChQlhYsKL40H/PYvF5gHrgV7VfJ6G8WfT4aUS4g7goC/pp1gxvRxfUgyo2anUPle/rXKSAgj/3F5D3gMISKacgg+4Vs4wGk4eNaiZiWx9IfdOcdlNQR6vJ3eLVt6OtRP9vHZtWBFnLYaQ1Ov0A5pGkiNyBOeKHE9OEmTMobSgkYfQp0rwiZGL09NM6sySLUAHujANAxZAHTIhIl2oYLxVN60DLXYTCS6GVrjZma4CsA== X-YMail-OSG: 0R5ALFsVM1kU1qDx8p0SE_ml4s8X1gwlLFLmBq.wDIz.CYgsb88LbcV9XUqiYu3 wwqmU3LUGzDFbY5MoCIaRm4qrpa6uWopw83NGk6QP0FaC36f0kwEWZD9SY7p.lx5uHsbO1VCwxGj 12t21PmTPsvrfJyYMcuZ76gm3XYjA75.rQCugA7UpkRVnDwTk_yiv4GanuwxwFaiM2dZWSiT2A1v Jk8q9b3.n0DvIeW.CTE.sorD4hhobbeif7_Oluc8XfcVN5NfvlSvVt49OqdmPO2E0J0jSNNRXKts 8Kf5GcgYDi66MvyjHSFU1w21.og8sSWvCGBu1q1HWLHZgo.cNNBdf6XSwnvziXtk172b22nfJ9O0 j64Dhm4yRq6QzusVc2mMjti6_cMzfRIfU.JJ1KjOciiibzRDFzOZuYaEOFvdPkXB8w3UGikVl.2M Q2diKElgvP2VI5S4BCirp1JgHBK02fQw.zNXRqBFBAUSc8MCG3H70D3pFtyBKCHphSyodWAaQEp3 s7TX_bOfrVhX6YxjhqSgp_3tjXlyV5.3y_D_e2.fVEqsxyRsZcxZVTbieWYjTprSs3rl5YXwOscI C0K3NxlFchK_yKrE_exYXhpZqTtA5NrenTPvtQofF7cLX8cBKZlTl5ld9QXOLM7LQDMZWrTauyjp BG1_NKnJkgIq5vUUYjfnmeAtA2_7eJhUnyQAfBuqMbPZUBUd8UJL3VKoRa4FJnhf7LrcW8Tsrp0z UZxHxacngeVWaN98AizOfiP4XWR3JQlLMR_6bRqzxw9Ki1fA1g3dxZd1FPoQ15Hx0rj4ROkY5um5 jRSJiHzbBQE8zuxFBlrK2L0OZgDEXn.hYNkGO5Tks6N79B.HCfJckJs5lueTpr3RAfju7sLb_ZWR wGBDExbfmki3myC1IoLNVQPlOhCKrDhTUoGXRNMpdB5eYFsp77mWb1O52Jlw2GnFjUhTlDS3YIAB e1bUYM8neW86bN._Rb8ZXw0tFpDwLmQahEftI11W5bTJe0M9UTxuPAEF260rT1hoJiV79mf8IJQs wIG.06kREnPLBj5qaeXi68aGJLP6niyoxf7LjWVaL6W_LQonIzTAhAOOJGHg7vDQtMBTp8f2lMtO jZYNFSp9qjCy9KhiURyYMmBic5Hi1n.dQgyq6XwyX0NEeHuCcFuUefP3cV5zm7v.edBFjWgQ8O7x e9HL7TbouUF1254Dbf1.JGte8VyOltmHuoQSxLGYKNX49459ib43UUHP7O2HnwtdgO0ykbii9Gv0 csm6z_74zYJ4I.VSxgDGwI4twfaYIyBggKBwFegjGZdppgA8Z2s_YuqvJmlpic3VkCJNWQHguQkm QS8alIdNygvxwPVxta19CoKA80bT9JgRd59VwODaPwim5vOm.XgOEk3YBLLcQ2ceh6Gy77Nk.VaJ DgO4zCEeor_RTDuD1ukPckhnPUG0WQCNGMEy5xy29O4rMPvMFH.GmqaBm6cpYeLZ6fDvpiVByA2F NlvWMqXFGXncpRpi9J.qKjy6EszsVpqxlydp6ThQxBJkWdy1CQ9Yo_aytQkN5ZRDKbBErJ8jupQp rREAbnarhRgjIRLfiHmGNE71x9kJWTJhsFqlUrC4Zax.XZbx7zpt8ixUxcxzEepWcLAsUnjPpPer 1Mb_cv4z6gax94i4oLptk7O.T.iAA93Vk1EVqaoJI4CpUHUITX9xfeS4EXxwTwXz9OOR2SSGqjUy BFFanFotMkeg3VRJJPrTzPEJVBhd8Zoo4KLwLGEf5Ba.fIK7DILPbFcbDhtTDv6Nla_8nTv94kig s8f.x5EHHxqjwTiP3tO_A3Ehx4a_I75CLzYwM50kOMfS0uV4bXgh6T786zTuPRR8w.vsFAVuFFHa ZJcbLugnm00ww.QK2feDvSNSxNJ82Z2jqEd8NIem1rerxnurY6_zy8l5ihbsMdEQozVUPnNN4i_J GcShtsXu7qVxNXAXi.ElKLHkpFmtLrcAwoLyNtc4REuQFRs.50sHLqcnFhxtUvpJ2mQx4aD2yxr5 2oW7K6Yob2HEAqETh.ehiRMCHXg.UJzTpuxCrcd9SimpquO77z_rC_QF.e68WLDRohM83iZlnuiw .laK56lOW8DYi_FftowJdEpj1xcnYUzi1QBTvK8CQUqdtUxjx3e8XZMHFizBf6vUiFBC.ozlZAsV 265K0Eh4JJ9oBeDMwW9jt942TL44hszTMh1Z0Ztrg8mCxcwDZAw5ZOXg3rdS4r96mVpQwv3UvNgQ 2YoobT8spNic50yqcHNEhHZ.ZspAJSJP7ORHwd6tcibnEAsxbHH0Y0.CoBxQEBg-- X-Sonic-MF: X-Sonic-ID: ac5d9cee-ca32-4303-8a20-fe4b326e65b3 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Aug 2023 18:41:13 +0000 Received: by hermes--production-ne1-7b767b77cc-7tm2h (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 0a0aaeab01d107c73ddc8b75c91cfc1f; Sat, 19 Aug 2023 18:41:10 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 Date: Sat, 19 Aug 2023 11:40:59 -0700 References: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> To: Current FreeBSD In-Reply-To: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> Message-Id: <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.31:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.31:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RSnd868JQz4Xp7 We will see how long the following high load average bulk -a configuration survives a build attempt, using a non-debug kernel for this test. I've applied: # fetch -o- https://github.com/openzfs/zfs/pull/15107.patch | git -C = /usr/main-src/ am --dir=3Dsys/contrib/openzfs - 13 kB 900 kBps = 00s Applying: Remove fastwrite mechanism. # fetch -o- https://github.com/openzfs/zfs/pull/15122.patch | git -C = /usr/main-src/ am --dir=3Dsys/contrib/openzfs - 45 kB 1488 kBps = 00s Applying: ZIL: Second attempt to reduce scope of zl_issuer_lock. on a ThreadRipper 1950X (32 hardware threads) that is at main 6b405053c997: Thu, 10 Aug 2023 . . . =E2=80=A2 git: cd25b0f740f8 - main - zfs: cherry-pick fix from = openzfs Martin Matuska=20 =E2=80=A2 git: 28d2e3b5dedf - main - zfs: cherry-pick fix from = openzfs Martin Matuska . . . =E2=80=A2 git: 6b405053c997 - main - OpenSSL: clean up botched merges = in OpenSSL 3.0.9 import Jung-uk Kim So it is based on starting with the 2 cherry-pick's as well. The ThreadRipper 1950X boots from a bectl BE and that zfs media is all that is in use here. I've setting up to test starting a bulk -a using ALLOW_MAKE_JOBS=3Dyes along with allowing 32 builders. This so 32*32 or so potentially for load average(s) at times. There is 128 GiBytes of RAM and: # swapinfo Device 1K-blocks Used Avail Capacity /dev/gpt/OptBswp480 503316480 0 503316480 0% I'm not so sure that such a high load average bulk -a is reasonable for a debug kernel build: unsure of resource usage for such and if everything could be tracked as needed. So I'm testing a non-debug build for now. I have built the kernels (nodbg and dbg), installed the nodbg kernel, rebooted, and started: # poudriere bulk -jmain-amd64-bulk_a -a . . . [00:01:22] Building 34042 packages using up to 32 builders . . . The ports tree is from back in mid-July. I have a patched up top that records and reports various MaxObs???? figures (Maximum Observed). It was recetnly reporting: . . .; load averages: 119.56, 106.79, 71.54 MaxObs: 184.08, 112.10, = 71.54 1459 threads: . . ., 273 MaxObsRunning . . . Mem: . . ., 61066Mi MaxObsActive, 10277Mi MaxObsWired, 71371Mi = MaxObs(Act+Wir+Lndry) . . . Swap: . . ., 61094Mi MaxObs(Act+Lndry+SwapUsed), 71371Mi = MaxObs(Act+Wir+Lndry+SwapUsed) =3D=3D=3D Mark Millard marklmi at yahoo.com