From nobody Sat Aug 19 19:18:42 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSpSj3l8Mz4mMMf for ; Sat, 19 Aug 2023 19:19:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-20.consmr.mail.gq1.yahoo.com (sonic306-20.consmr.mail.gq1.yahoo.com [98.137.68.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSpSh26Vvz4dM2 for ; Sat, 19 Aug 2023 19:19:00 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=szrMRsNh; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692472737; bh=VOHbdOGh7883w2s+W48/zWYhaclTBlP4JRm4jgHhPew=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=szrMRsNhNWQssKD+GswcVrVzg2UIQA3Oegp8DVE+WSfJ+Vm+TN4Pmqi51qZpE9AsXWRYhU47+IS8Z58LqbZ7X6NzTmpn0mVHvrx+owWUj03xtnG6y1NOgaptejeP1BR/Efj0A8MGG183Jd1aapbYo6IoDM1v+SsQ4BlWgF5PSK3PS2E6i7Z8stuIFrSH4nCiQ+gFoJ2MU/x28i7RroIHl32KKWibY6O4T3toiBkg//VNt57aoFYz7SqMyL25a6y32uX3pCXoh1ws2RzhMK0OQ5YH/xqb0znoTeoMSfroYtMSJdkOMatM7h/ty45WJzpbd7uYWAYSQUgSUnormxr4eA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692472737; bh=Jio29q0KzfbO+nE6HkORXnW3a9SpzxNJY4OLTbMVNpc=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Ee9WVmSW63rAqdjMHxCCpPiRkDUlZDHkU+EnhNB3nNU1hUPlMtaNi8+lHki3kD9C1uYzr1wj2WJC6bJ6nWE6Xh8SAxLE87SB4po5ixQUZXecEIbhFhHXttLwr4tUs6nNjh7lxj5M+O+Ags9VNOdP9nO4zxaZRqWCHYrh9fegZ4VnGey2p/pia/IMpqNAazGLW8RK1RbIieF50BIhSYZ5Cs8PawPCwhywfGTxZ38exugcwmTbNZAhU068Q4Yz/5ywU7PsxQIqM7Od56XWFYcKheteNa4vQ4b1AN07MUZbH/RZH/NboX8yQDpHEZO7ENAzravPFwiaO7XkY7dH0LNOhw== X-YMail-OSG: s.8hsuUVM1ndF2lCC5rKBk8tPhbA9dsw.HKKXGa1SegC0SxnjP6ZWxZ2Lw.yiWk uYZhTfHEAhAew8WSXL8eUPya35gHMcOJ39Z0lQoISrAiZR9aXiANmTPJkO_PTTldm5QLgPYUZBry 8_g5XwXerLx5U6rvDMxImZrjQ_ayxg7omPrcMQr5PUz_C8YKMl6k_utHarONARaGrPbO5j8ryxpE ox5sEw5j7E0KxkiZ_DGnq0HNDM9czgxb5b6JAak9qZc5ryo3HaCQG1XByKXi.dBYdt5.qS3svvHy Ivd6gdV1Iemu.LcHqVS7D4LESM10dlRFWtKfysXcg.LbTIOoq4F0d6HgYgmFeapCQbrcTeHiy61G mkz6t8h_aERGgCKMnOY3gmx_R9BRtQd4Besd2IolrClDq2tmwKTMiQ0pfIXLu6i96v7gW9IT33g8 LodvUt0K4BJYer5OxmujRO3BHoF6a5C0OO4mBOYV15KTIGkqpWp7uTfdVSc58Rp592cW5DjEHhvM iO.DNPIVSss0ORZ.KX2bVBObRxMKDFLSej7nxv7Ggv4y5r5t644wkzw_wSbe3AoobGjbXemLfzjE KoxV61CtvlFzmA1dTHykwaWOijhhPB.7mhbz3oX0rR80FM4WAnHSU2bw2cMNOmUxc3vXh1uFf4Zf ovCXk_o8i9UxpeyKUHjWKx.6vcbUXQnJhMfDnaAsKfA8vSrXAHC2YbNAFtZmNoWFQ9emPpFzorjV wL4His6NsxninbMoN879PUkjkVs6iDzZVo2JRGO0PlS27C6FZNtKJrUfmMPeo057X7o_TC5_tJgV 46Av.RjZz58n4HFyyL7BV2kuWTjCCqzTDqaeUv7haoHTELq6M2jwdU62UZOCOHQjTfj5rDLMw3_z xaitF5AK6H3fPgjRXm2MDgvKj.JcyosuGvPCHtsv9vtPUjKA1L8Im6_VY7STIT4_MWr8cjc4ay95 V1DDLZ_erJJnSXIXsiAZcatFeKTxDZoJAYa8QECrvQN0ryv1i9q2ZQdO3kRDzU6CmxHQCR8rO6FC Hk5m18q85aT91zMOlsKx2DRrxoEHkr7getxWiJRSOgstaJIrEw.pGGZykYexf1aE4vByJzhh8wv1 6Ekfxe4fro702WGHILd3IRzkokt7VZGOOJEnORQAxqYQxwdmqnGdCM.o16XN3kY3D8MG82pqf8sg JT0qHRt0NhVLGa5hXPf3mAM2s97DRY5sV9rQVS2KHeKwyj0mE8y.uq4wvC0O65PoQrVh5hNIzF1h _SUyrAaycgosdhdHDfPJK9wqUE8gfOQzL_AbkSOMun0zPgHWoG7mYJCTrkUAHYfuDKg2OL4JjFEY BLVJNFFmhDtsmMSzXea1gSi3pGA5CP1Ze0GZN0gHK48TQ9wbzyplUe.e03eL4KU0Vi7uabaqDEv2 IjNa5Yjf6q3CKIHBuB5u4vr0UpyWrx2x9x9zNXTkOEzi7fknBuBiDLrb2hyOXpuUtOGms0_kBRmz JuJYeTiojydcfBou3yzNG1obkCiITy3pOgc0fSvy6uwfBT16eFfnB2WoJEHHUhUiitifbeFnk0KX PLtwKBmj_vB7CWqfPivsqV.3_L7EhNEVYr7zDHEga1b.TdK5ywSYwG2XvxEQhnIZiDVS.umyZojw SR15DCAxhs588y.9uEEtUcMkfdg2DFgynofN3CRtBAbimFV949aafGXx4S4g4tjDYITm5acnA0ff I7X2LfhOkE8E6wSoxTLlWG6KjWsSr9vMaf7TWRSDzDii5nDxi8Pp4QlrdsWCex8jGgH2x8iHATTp lf14Ke34sJxRXpzPDUO_4l9.05rxGNwU.VD3RWHGo_ojckxeSpWQqD1T.wBnSMYidAfEXWFJOmMZ V1tIk3dsKEu5.ktpQ1_ZmnYdobP04xXBANTm95mJvBZtMhilpOfyDvDa5DpShl_JwoVnDOXoE6jj V3sIEVAWujvzkHNvNy6tJ62yZT.B9eUm.5Gpi9qUVSoNA5xKqfWIR2B8qijR8.feI9Nb5I4SCrmw ezMaLbNsu1Xf2qaFhMxqu_nAEZAFHmWEBP8WTrHMVkPEt_BB8J6az3xHQmhsgMpqi78xA1eoa4kx BAZZ_s5J.nrLq6GQzVFnpCZuZRpK0ez_J5Y3o0bOZ.tQBrM8cdQuS819coaOj8.JN2mECVuc0S.r PmmrNdA2WWohqlEo_i8eZRVOphkYFdL3VvIbbjDRHHXVtLao3mdLLxbh3RhoaKqy_Pu_twoCuBLB K_cAR4LyJpG9HpdDHgxMAFlfrXdoDYdWFRZOyUBTju5zRAmMZAiv0fSBipI1x X-Sonic-MF: X-Sonic-ID: bbf5009a-e1d0-4086-8703-503483729189 Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Aug 2023 19:18:57 +0000 Received: by hermes--production-ne1-7b767b77cc-c8mvb (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 2f951008d3317f83f9ccb5871b5c6d61; Sat, 19 Aug 2023 19:18:54 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 Date: Sat, 19 Aug 2023 12:18:42 -0700 References: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> To: Current FreeBSD In-Reply-To: <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> Message-Id: <3AA253E3-C4F0-4AA3-9C37-D77E7527A458@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.83:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.83:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RSpSh26Vvz4dM2 On Aug 19, 2023, at 11:40, Mark Millard wrote: > We will see how long the following high load average bulk -a > configuration survives a build attempt, using a non-debug kernel > for this test. >=20 > I've applied: >=20 > # fetch -o- https://github.com/openzfs/zfs/pull/15107.patch | git -C = /usr/main-src/ am --dir=3Dsys/contrib/openzfs > - 13 kB 900 = kBps 00s > Applying: Remove fastwrite mechanism. >=20 > # fetch -o- https://github.com/openzfs/zfs/pull/15122.patch | git -C = /usr/main-src/ am --dir=3Dsys/contrib/openzfs > - 45 kB 1488 = kBps 00s > Applying: ZIL: Second attempt to reduce scope of zl_issuer_lock. >=20 > on a ThreadRipper 1950X (32 hardware threads) that is at > main 6b405053c997: >=20 > Thu, 10 Aug 2023 > . . . > =E2=80=A2 git: cd25b0f740f8 - main - zfs: cherry-pick fix from = openzfs Martin Matuska=20 > =E2=80=A2 git: 28d2e3b5dedf - main - zfs: cherry-pick fix from = openzfs Martin Matuska > . . . > =E2=80=A2 git: 6b405053c997 - main - OpenSSL: clean up botched = merges in OpenSSL 3.0.9 import Jung-uk Kim >=20 > So it is based on starting with the 2 cherry-pick's as > well. >=20 > The ThreadRipper 1950X boots from a bectl BE and > that zfs media is all that is in use here. >=20 > I've setting up to test starting a bulk -a using > ALLOW_MAKE_JOBS=3Dyes along with allowing 32 builders. > This so 32*32 or so potentially for load average(s) > at times. There is 128 GiBytes of RAM and: >=20 > # swapinfo > Device 1K-blocks Used Avail Capacity > /dev/gpt/OptBswp480 503316480 0 503316480 0% >=20 > I'm not so sure that such a high load average bulk -a > is reasonable for a debug kernel build: unsure of > resource usage for such and if everything could be > tracked as needed. So I'm testing a non-debug build > for now. >=20 > I have built the kernels (nodbg and dbg), installed > the nodbg kernel, rebooted, and started: >=20 > # poudriere bulk -jmain-amd64-bulk_a -a > . . . > [00:01:22] Building 34042 packages using up to 32 builders > . . . >=20 > The ports tree is from back in mid-July. >=20 > I have a patched up top that records and reports > various MaxObs???? figures (Maximum Observed). It > was recetnly reporting: >=20 > . . .; load averages: 119.56, 106.79, 71.54 MaxObs: 184.08, 112.10, = 71.54 > 1459 threads: . . ., 273 MaxObsRunning > . . . > Mem: . . ., 61066Mi MaxObsActive, 10277Mi MaxObsWired, 71371Mi = MaxObs(Act+Wir+Lndry) > . . . > Swap: . . ., 61094Mi MaxObs(Act+Lndry+SwapUsed), 71371Mi = MaxObs(Act+Wir+Lndry+SwapUsed) Status report at about 1 hr in: [main-amd64-bulk_a-default] [2023-08-19_11h04m26s] [parallel_build:] = Queued: 34435 Built: 1929 Failed: 9 Skipped: 2569 Ignored: 358 = Fetched: 0 Tobuild: 29570 Time: 00:59:59 Not hung up yet. =46rom about 10 minutes after that: . . . load averages: 205.56, 181.58, 153.68 MaxObs: 213.78, 182.26, = 153.68 1704 threads: . . ., 311 MaxObsRunning . . . Mem: . . ., 100250Mi MaxObsActive, 16857Mi MaxObsWired, 124879Mi = MaxObs(Act+Wir+Lndry) . . . Swap: . . . 5994Mi MaxObsUsed, 116589Mi MaxObs(Act+Lndry+SwapUsed), = 127354Mi MaxObs(Act+Wir+Lndry+SwapUsed) =3D=3D=3D Mark Millard marklmi at yahoo.com