From nobody Sat Aug 19 23:27:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSvzf6g8rz4qMMB for ; Sat, 19 Aug 2023 23:27:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-8.consmr.mail.gq1.yahoo.com (sonic308-8.consmr.mail.gq1.yahoo.com [98.137.68.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSvzf0QkXz3Wv8 for ; Sat, 19 Aug 2023 23:27:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=DRGarmtQ; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692487659; bh=es5JdmrdzOfdbgG9AcmbfIwAPVOkI7wOD3mKxq3mLL4=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=DRGarmtQ+LEgI4RNS8/UYxllQ22jvN/4dknnqcSDldqvAw9FlhE+3+kad6Xvq/yB2vfRfvxALDSln6Tc3ncgVO853mJVeKN8b/4jOvLoNH2veo1YidZiGlmwQUQwDlqJP6ujoumq4HxlxTf6JDDA0+uwir0Oezd0teN7XJskQVXdGJc01ut7hFe7V3KM6ySBvFRnLwcjIHqEOPzG3LflgQS1w6FOhSBlK6jncH0U/IRFLhgw+2N3ahVeRt7wphs40oDlxNjbxVZAxkKu8j5YGIFGiiXtPy1ze3Kh8elJhqk2CqB7YBqHyy6xfvmSd4BlGw2AgKM3Vz90JMijk75g2Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692487659; bh=6XoOtFcwuRh09Vw8/HYPjGPz2jY/IVIMNCDqJfGUXL/=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=TWD15C6gIyp3pj2wRaIQB6n+04yBXxjWNh7dx6tAenx1H26gGvHUpLic+6nf2dRuKeOsj2RyxIwrrEkNItHvYYyoAobvXVhVRUEhPUFmqBgonQnQyu22sd2yrLVDQ0NQH2E2z1l01N+NyOtVaA7026l5lh5lzFZDqEFm233kn/smVE92bYSvr7GIjqAH/WtVXYrJ6UFkqbwgBM+R3Ps2PPbXi4UhBCwXhKI7mpjss8MDoMSY9UOEbrbylTl6l5pSHds9YVOfxzJ+4E8kjYrtQ3aBv5LBGF+wYLpORsXcmi9E+BTWtodyqv4c5opk0V11jIk+M+chmbt3taTxr0Q+gw== X-YMail-OSG: fsd2u6cVM1katebCF8jY.vJdOJb3CPmS0zN4Ee.hgsVonEkAtZKo3qJWl0xYPld GZhBxVo4Fb.ZDT6K_wgip1x0xo6yOFEptebg3ASNpb.y8ZmRIfAt1fvwW0P2xZRi.iEZAarQkTGh rCVzqUxqhPRNQAs4k.tOh_NybrOMHWeCfCoQTsx7157R64n8M6vGPXr6cvm5OXoHgmh9wLj3EexQ Z5gQUmYwLVdHksXLazSP6rLosXYaCzRfNRFb_Va1Hub_kQzgDKOvifqD1yjlQyEM014mbp1kwVnf oTygI6ENkql0VAg.cS9_QfF6QH_Jm.6XWgfd15mgkDM9r0CqXQ0VwK4nnmzrKeQ_zR4F3wa.lH9M bou9_kUNC4tWDdo9weRj.T9qsRXAPiTqDzMC3HgSCVrOC1H3HJqkG7Yj0UnODY4dxV0VJn8gWq0z .7xWqRg5BRJenvdW7MRSvcUndDyTWxgJRflecf8ytAC5ylCsa1SF1BjUhNrT2_qO2gXY4skX3RNs IOk11MLlwkIWv5IQGmcRCJqcytLe_bg8iZbt.h6W5Puut10LnmchbdSGzA3.n4bPaUvkrmIq6vW0 YXD.mAwm700xFiVLm6.2gq2LWxNAZeiEiAzIwE.leADHrRcR3TzMSlYz67xo_z.kIHjxNdJbQtKW 5LoQqLh4DtBYXoFdaQwFanuIV0SuojYVY2Q4UpkMoGKA4rdiLgO4vn0BIuOTQ4ZlWLyy_Me0fRts ATMTlTdF_Qzb.wlec4MX4D2M2eGyW.VKIN1IXev4gwqnUAgyJ1sXuIthtpLWqLg8Pim4OJdbPzUl zGeKZeWdjil.y742bMudCs6zFAHAwuwKiYNfNmwZgIexZNg2HCRp45WkGEjDX0aAy20ggpQNvZfR d9wiLDlAaxRxTlSD4bwQ9VfBKpDacAGY_3c6cwiJhR0kHpvSlFShGBkYoIG9Fcx6WiRARFkVWftr kZDImxiy8qYqK8xcqTUDa1Tld5DHcFWggYM1Ju9RYeEpciF4iVGEQjeEGyVXKy2z_Zi7iDvGQIgK vEILns91HyPylSIhbcrEn6h41TJo8LsuO4_2NUzdjF91UyoPGtk6NLqH550sGDhaZGApSu1NZq0z 3U_4ULKQqm9Sbl9qIEDTZgTNBK7x59MoudIxkVsbe9c2O8d3Q240AJGZ5SRPGCqzkShAqFN.EwzG 89BuDqlJ9MvHneLaNTrKbZ2jLFu0vCgENE4SAx1heDhVl230nRxZAJ1O162drF.bqWzNPTXgyYQW IQFT27oYPKAN_vzAaiC3_2N.KoYqW1kH0OSpxF7DmAF0s.4DG_62JqPwEUmQGxqkirnOH5.gQ_EX v21Ip2zy__FYReKsgOjhWF9_YgZ02lWaEqfCcWplXso1Rmfl5oGYnvUYjNrV9pQwFb.kf2UcDLkK BS0sxHHoS2NruicVxyHmbbrBhHFP_xv1Ar4tt7bO7OxEBpRD4KBCVdG2sMDbmDe2twS8x8MaDQdp MPmWiwnFa2K8Wu4yC41BxbbbuBx_DcB35NqwyhfV3zCWjBEHGHN1Yf2OvvjaQp7ck8kTuygmjbxT cQ_405g96qn._zbtkOQehZ1xBCuh8mQG7yfljfDxjDpRlYakVTTX3ZKhGTFIyJPFWBUtZuEtXGwh VnXgCCo_PdG8s_uUAA0ETJyC6iXa3ElmDddoyb7vipaC1KNeVa_5mxTJOaJBHFhAB2IUrzTl8u7d gWUkWPPRibpAm3oZv3rHCIO1KqaIGpdhnP9RYfHcIaoanBaw6e2TM0K2y2eClcwm8YURI7pZAwqs xIJ0vas2AHadVjh7Cwk1rtGFbZUK3SxJ7DcovzV5gKlrSwAb4E.uOxuTlz7z1Clm6Pz_KyJi.h_R __cgDYN7TV287uO8ofEFAW_V9cno9Cs8.RSatLZIg9fRa1uyJN40jLcHCr11_LlqSmA.DdRC2a9z xRbQkwSzcjKowsJmqXVSdMmhI3LYJgwJs.GCmxAxBmdxQYrRJPDu9Ky7t5UBOaDspD.1X4RWHaF7 OJhtDbIU1osu5srwPtemRfRzUMUnpdjp0jdql_ObMvqzPg1I48FL93wwyjdovWqCmZPuULefDGPj 8ViBlVmMF0YvIyiE15J68f7xNINgTHjcw9UzGdYHHjCoTqfgfQBVU9UdD8gmdCeNX0KGhPLOb4Jb LaZBz69DsOXM43HPTR4gph4ZGLfGCqJO8srur71Txmm48krW5cXZ3CmdkUcbwhCSzKg6ddXiyhTQ 92tP3tJqHxPhoqUj2R.IrfZ7__fv3lNma4OuH3ulLD2HeS.wG5m4R56UN5uYd X-Sonic-MF: X-Sonic-ID: 73496822-1ac1-4f78-8e00-283fb58073e6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Aug 2023 23:27:39 +0000 Received: by hermes--production-ne1-7b767b77cc-9p4qj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 10bb64efe3f1897f813ca28fb48133b0; Sat, 19 Aug 2023 23:27:38 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 [USE_TMPFS=no poudriere messed up from the start, lots of "vlruwk"] Date: Sat, 19 Aug 2023 16:27:26 -0700 References: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> <3AA253E3-C4F0-4AA3-9C37-D77E7527A458@yahoo.com> To: Current FreeBSD In-Reply-To: Message-Id: <8D0C1422-CE60-4266-8051-2296C3E9B7D7@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.32:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.32:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RSvzf0QkXz3Wv8 On Aug 19, 2023, at 15:41, Mark Millard wrote: > On Aug 19, 2023, at 13:41, Mark Millard wrote: >=20 >> [I forgot to adjust USE_TMPFS for the purpose of the test. >> So I'll later be starting over.] >>=20 >> . . . >=20 > I finally got around to starting a from-scratch bulk -a > again (based on USE_TMPFS=3Dno this time). This is with > 15107.patch and 15122.patch applied. This is a non-debug > kernel experiment. >=20 > Interstingly it got: >=20 > [00:01:34] [01] [00:00:00] Builder starting > [00:01:57] [01] [00:00:23] Builder started > [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 > [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: = Success > [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 > [00:03:21] [02] [00:00:00] Builder starting > [00:03:21] [03] [00:00:00] Builder starting > [00:03:21] [04] [00:00:00] Builder starting > [00:03:21] [05] [00:00:00] Builder starting > [00:03:21] [06] [00:00:00] Builder starting > [00:03:21] [07] [00:00:00] Builder starting > [00:03:22] [08] [00:00:00] Builder starting > [00:03:22] [09] [00:00:00] Builder starting > [00:03:22] [10] [00:00:00] Builder starting > [00:03:22] [11] [00:00:00] Builder starting > [00:03:22] [12] [00:00:00] Builder starting > [00:03:22] [13] [00:00:00] Builder starting > [00:03:22] [14] [00:00:00] Builder starting > [00:03:22] [15] [00:00:00] Builder starting > [00:03:22] [16] [00:00:00] Builder starting > [00:03:22] [17] [00:00:00] Builder starting > [00:03:22] [18] [00:00:00] Builder starting > [00:03:22] [19] [00:00:00] Builder starting > [00:03:22] [20] [00:00:00] Builder starting > [00:03:22] [21] [00:00:00] Builder starting > [00:03:22] [22] [00:00:00] Builder starting > [00:03:22] [23] [00:00:00] Builder starting > [00:03:22] [24] [00:00:00] Builder starting > [00:03:22] [25] [00:00:00] Builder starting > [00:03:22] [26] [00:00:00] Builder starting > [00:03:22] [27] [00:00:00] Builder starting > [00:03:22] [28] [00:00:00] Builder starting > [00:03:22] [29] [00:00:00] Builder starting > [00:03:22] [30] [00:00:00] Builder starting > [00:03:22] [31] [00:00:00] Builder starting > [00:03:22] [32] [00:00:00] Builder starting > [00:03:30] [01] [00:00:09] Finished print/indexinfo | indexinfo-0.3.1: = Success > [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 >=20 > and is still that way minutes later. >=20 > ^T shows: >=20 > [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 > load: 13.02 cmd: sh 2187 [vlruwk] 570.19r 0.62u 38.60s 9% 3948k > #0 0xffffffff80b7701b at mi_switch+0xbb > #1 0xffffffff80bc976f at sleepq_timedwait+0x2f > #2 0xffffffff80b76770 at _sleep+0x1d0 > #3 0xffffffff80c5b435 at vn_alloc_hard+0x2a5 > #4 0xffffffff80c50b72 at getnewvnode_reserve+0x92 > #5 0xffffffff829b9b12 at zfs_zget+0x22 > #6 0xffffffff829a6a8d at zfs_dirent_lookup+0x16d > #7 0xffffffff829a6b5f at zfs_dirlook+0x7f > #8 0xffffffff829b6410 at zfs_lookup+0x350 > #9 0xffffffff829b182a at zfs_freebsd_cachedlookup+0x6a > #10 0xffffffff80c36a0d at vfs_cache_lookup+0xad > #11 0xffffffff80c45141 at vfs_lookup+0x581 > #12 0xffffffff80c44238 at namei+0x238 > #13 0xffffffff80c63b5e at kern_statat+0xee > #14 0xffffffff80c64237 at sys_fstatat+0x27 > #15 0xffffffff81049a79 at amd64_syscall+0x109 > #16 0xffffffff8101f11b at fast_syscall_common+0xf8 > [main-amd64-bulk_a-default] [2023-08-19_15h14m10s] [parallel_build:] = Queued: 34435 Built: 2 Failed: 0 Skipped: 35 Ignored: 358 = Fetched: 0 Tobuild: 34040 Time: 00:10:52 > ID TOTAL ORIGIN PKGNAME PHASE PHASE = TMPFS CPU% MEM% > [01] 00:07:29 devel/gettext-runtime | gettext-runtime-0.22 build = 00:06:32 25.4% 0% > [00:11:25] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-19_1= 5h14m10s >=20 > Note the 3:31->11:25 . >=20 > Top is showing lots of "vlruwk". For example: >=20 > 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 > 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 > 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 > 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 > 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 > 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 >=20 > In other words, it is messed up from the start, not > just later. >=20 > It does suggest that the dbg kernel should not end up with > resource problems: not that much gets very far. So I'll > probably stop it and substitute the debug kernel, reboot > and try again. Still for nodbg kernel . . . The "vlruwk" processes do occasionally instead show a CPU?? . Nothing seems stuck in only one STATE. (Live lock?) As for using the dbg kernel instead . . . Most of the time that processes are showing CPU?? more progress is made in building, but basically one builder. vlruwk dooes show up, gradually showing a larger fracion of the time. ref 02 .. ref 32 are still in cpdup -i0 -o . *vnode is showing up some as well. N process looks to be stuck in just one of those. (Live lock?) The debug kernel is not reporting anything during this so far. (some time goes by) At this point vlruwk is fairly commonly what mostlt display for the cpdup's that are not finishing --but none are stuck in vlruuwk . Looks like I should try without the 2 patches (15107 and 15122). =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Aug 20 00:33:47 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSxSF553vz4qR3x for ; Sun, 20 Aug 2023 00:34:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-20.consmr.mail.gq1.yahoo.com (sonic313-20.consmr.mail.gq1.yahoo.com [98.137.65.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSxSD3CVDz3d1g for ; Sun, 20 Aug 2023 00:34:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=DVOguTjN; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692491641; bh=Em1xBU2uVQ+k1yrMk5AG5s2djwPsxZXg05iWrtSy62k=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=DVOguTjNblJuxrzID3ejWCJdw5uAOu2TTk1Oo/C99HD0O1vnrkE+1K2BaSPM5aOHzwnOCnOYnbVYCZQGbUoSbp/FLumnE3mhT9nEWF9v4AyoBlctjaSYw3+vYCHsXCXjf3dhI7+cYuGHdFQaErGG0mVSpeeJBYyFvgccgo8l14+hMVeLC9I04uCNunEvNfWUwjosfx9bScctX4ezrbO6RCsSAOK+H9zGJEvMZC1XpUfZo2Sfg+N/4Ae8ZNDuRi2M5RuuDnipMiOn22F32lBHdjLrATkDE87Ok2HlJjKWeMyrgoNPz88+wEc1m62uOB8J+INjveYyQ297DPpKENrE5w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692491641; bh=oPYAcMfyZGP4bNBfI/d8kMgjQVfuvTrzgFXgrAMxX5v=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=qQKWuDhHTeC+0TOL8svsC5DR3LpW2m3BLKK0JYuTdVslTVooSpVe7lk9JTuNHNF10bpB8FhT1Y/S0pVKHz2jbdTH+nUxkIcEvfPJCKBcvQze0UYj0D2lLMF6/F4Teyu9hpTfiZ/9PXU16qmDDlER5NLa/ktYSspb18YzYS4UKYwBxIk0ocpKoEmA3Y1RYAylmb06DrL5QBUacy3vBxnWFCYmut0fNohLBzU34TlJIls8894xyDEGn+2/6AsrQw+DURUvLq3rByKcpGu0zjjQsTEXTgagvwkQL2P4ijpf0MkxcvkM3E3qW6v0jy13oOS33Gg9MvrfyR4JpS/z+TM+fg== X-YMail-OSG: RsL86a8VM1mtiWr8PPuo9o11H5vDmsvICA4IVCx6SR59YGNANsmO0_Ig4Qrxj1V y6KaxbXCzAe_CZbP1kZG6g2H4fsOKmQmt6VVIVhv3F1Hwh0KlEHLhlYLG0orvCpS2eLNoon.oDnY c6CeJMwAP9a1nTQe3JnUrfV83wUuP4tDIN0Ms4sVjAeKKTu609WUy3FxE3CXSTcp29RCw.fX3WEq lRq4AwCEEBDr5bBfSHsDlX_JnNu_ylNBrjpQAxxtffEeDnTGmDJwmdr5xLNyK8hHliCKbxpoN2cS 7ieon2sF9DYiPJS26OLgwYJgPNs5E1gj7C9XSnqpAUqzvjrCa1L7R_Q0ralMJCws3dTg3.7bSKJ9 IbNH7bNNYr96VcoIpMmhZrU6iZ8VbbuLpqBErCLNv.rxHdxLdfcbZIhmDAlhkSbloKwHtB5_rXMp Qp9STdl8Gldr.pLyWkTL7YUa4mfxLTOg3p4ztLoze5uUvLfqjo4ytcywp_GfwLypvAVdynaCs6Rc 21OWA5_.iYpxEaYCSa45H5aCSOkKkvw.W0piFfsrbUkVMlFALTcD_BquCJwIiL9rEAmRmwfNaAnF 5uuZY426iIK_YR9BFmcK0kvbAueMzcwwuvicc8YVgkD3W7sGWBqPjYGnJuT64sG8HNKVHD9VIVB4 Lpc13lJijlVVRMIMUubOnGU.p5PInp2w._DCMMwJF6ZTybMAOvWRaMo5_8.P.WDnqP7HJsyjAYtW rWg3KMT3nRhDtymcFqW8.8DmlGMf8VDbfNOaBMYjOIDjbuZBiZEMdMtgMjs5k1ZeK1I814HGCxf_ GjIx.oUyZrIuGDdW1vBeQnoPSheiz7V8yLMFj7SzfuCYe5ogqFrfkTPdGdChjTc.0pxyWcTbDWqb p_gtoZ2Ouhk8EKnuk3Lepu1MKthHrttKwWYTUrpWQZfF6GYheusHwqrsUjT2FYC.qaUK5zpNxX6R 5iSsabjS5ox8Tw_aK2PTOfl6IYNT3VEgNdak9M2w9ncTX2pwSm9av1g6fgUSOQB7mJVou72wO0jE Ja3uQm4Rr.pHebqfBUO3d7F.HeldQ57KIOmNeSpKwQpvd5HCDOvjcRKgWxj7OQy5udHv4RNiU1dm VB9nL..MGbA5kSVe_OwuFAQuPTKPTIq5RXCepM9Zye7WxIJF1KWtxpnhh2rGp.njdUyVB3cw8fl4 n7MvG7vpaQXjZ7h.BjzDUkXb40I58hShvQz6IW76TavsmZkcVwqfnCzwve.pLiSTAtmx3C5ZX5.1 kqkRgeinNJFY4ZaCvwInrm6.RLpWuK6MONGKxF6evQtVMvCpP.YkUr8.ldq0_N13aNJdiVIRyx_p mrhLojdMWzpCJI68LrESipUBmQ3Hd8TZB92YisTS5Gf8nU2tu3WB12x0t9UZW5PfJ2Jx9TFc_GmJ HeP4Ucd.9EcR2r.YPL1kCIsG9GJ0nXeYmZ6Ma0ZnfpLQH_EZpWf50XbMncqykeTtuBJMXDVHex6Y EiOt527OPOrWK5wsdxocDLTaXJIBW9juDOMYw.S3nqPBqt4_ssEpv6EpSAQWWZ1NHqiu59_nTJqR BM4mJgKaUIdSlTpRuOm8R60Cjq3hRmt5LcV1XCCUG1mRh.Ae.IBWCKTKfH1v7LdqFMdM4UKEmnZm rGoZthLNeNyUbUBXcJe8CdwiS104H4qMd4D5LcSc_SVvoSUIwxrKEewrirXVLeHzZxJtO4l4fUb7 Cv.SixLwuFmIr2llDUiHIx8hwyU1nsVGpElsf1Uxvws.D4.3DngtQ_ItiVewAhGT0B2.uxl1MLb2 mkRKJ9hAT6R5lRQ13LWIzyHTILi.Cf1oYrFTgr3FkSvpsKj0R.qSfXUDVDA386H0PyBk5y8xqr0o z_wwTpVl0MOrxfcStKDaWNHJlN2cyM_aMwtrZYkuPMQnbvbKSOowE77I4R1B2ymb1laIJyZtL7gn .otyhMYvKSV7_BWMuQD29wdIxMbD6ESe1HMvj0CQrkk.G4qFZNz.J_R9gNHWLwQMIQwGouTnEDuz L6JSajGaO7kKHQO4WebEDZQZMtVlPCPSqlDb0ofloA07WBD_73P6kf7mfvu2LL3iu2H5WWxP05IO 8HU_LagxVjP9HSBOFbCU0D3ax58tKSw9g.9eXyxRuUSZnGieuywWLNzN4JspAiIsuVWlm2poUxPf GjRBTa74IrbxkntwjGtfL8x3X9TWDFkHmkX2.4hrVHJrd7U2jNmWp_Og2pB2jyhzS4ZRFfTBbscl dcBgbfrGzN.CIyIsDFKJUJD2GXo3BnXNFmHcJJmsPzTYxaT590hiSwAafK2fZ6Lla14b0UNo.5TB I X-Sonic-MF: X-Sonic-ID: 195d1693-ede8-4d22-9031-2a116035a6b7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sun, 20 Aug 2023 00:34:01 +0000 Received: by hermes--production-gq1-6b7c87dcf5-wlch2 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 169ce5b3ada94fae3eca25ce000ffde9; Sun, 20 Aug 2023 00:33:58 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 [USE_TMPFS=no poudriere messed up from the start, lots of "vlruwk"] From: Mark Millard In-Reply-To: <8D0C1422-CE60-4266-8051-2296C3E9B7D7@yahoo.com> Date: Sat, 19 Aug 2023 17:33:47 -0700 Cc: Alexander Motin Content-Transfer-Encoding: quoted-printable Message-Id: References: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> <3AA253E3-C4F0-4AA3-9C37-D77E7527A458@yahoo.com> <8D0C1422-CE60-4266-8051-2296C3E9B7D7@yahoo.com> To: Current FreeBSD X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.83:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.83:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RSxSD3CVDz3d1g On Aug 19, 2023, at 16:27, Mark Millard wrote: > On Aug 19, 2023, at 15:41, Mark Millard wrote: >=20 >> On Aug 19, 2023, at 13:41, Mark Millard wrote: >>=20 >>> [I forgot to adjust USE_TMPFS for the purpose of the test. >>> So I'll later be starting over.] >>>=20 >>> . . . >>=20 >> I finally got around to starting a from-scratch bulk -a >> again (based on USE_TMPFS=3Dno this time). This is with >> 15107.patch and 15122.patch applied. This is a non-debug >> kernel experiment. >>=20 >> Interstingly it got: >>=20 >> [00:01:34] [01] [00:00:00] Builder starting >> [00:01:57] [01] [00:00:23] Builder started >> [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 >> [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: = Success >> [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 >> [00:03:21] [02] [00:00:00] Builder starting >> [00:03:21] [03] [00:00:00] Builder starting >> [00:03:21] [04] [00:00:00] Builder starting >> [00:03:21] [05] [00:00:00] Builder starting >> [00:03:21] [06] [00:00:00] Builder starting >> [00:03:21] [07] [00:00:00] Builder starting >> [00:03:22] [08] [00:00:00] Builder starting >> [00:03:22] [09] [00:00:00] Builder starting >> [00:03:22] [10] [00:00:00] Builder starting >> [00:03:22] [11] [00:00:00] Builder starting >> [00:03:22] [12] [00:00:00] Builder starting >> [00:03:22] [13] [00:00:00] Builder starting >> [00:03:22] [14] [00:00:00] Builder starting >> [00:03:22] [15] [00:00:00] Builder starting >> [00:03:22] [16] [00:00:00] Builder starting >> [00:03:22] [17] [00:00:00] Builder starting >> [00:03:22] [18] [00:00:00] Builder starting >> [00:03:22] [19] [00:00:00] Builder starting >> [00:03:22] [20] [00:00:00] Builder starting >> [00:03:22] [21] [00:00:00] Builder starting >> [00:03:22] [22] [00:00:00] Builder starting >> [00:03:22] [23] [00:00:00] Builder starting >> [00:03:22] [24] [00:00:00] Builder starting >> [00:03:22] [25] [00:00:00] Builder starting >> [00:03:22] [26] [00:00:00] Builder starting >> [00:03:22] [27] [00:00:00] Builder starting >> [00:03:22] [28] [00:00:00] Builder starting >> [00:03:22] [29] [00:00:00] Builder starting >> [00:03:22] [30] [00:00:00] Builder starting >> [00:03:22] [31] [00:00:00] Builder starting >> [00:03:22] [32] [00:00:00] Builder starting >> [00:03:30] [01] [00:00:09] Finished print/indexinfo | = indexinfo-0.3.1: Success >> [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 >>=20 >> and is still that way minutes later. >>=20 >> ^T shows: >>=20 >> [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 >> load: 13.02 cmd: sh 2187 [vlruwk] 570.19r 0.62u 38.60s 9% 3948k >> #0 0xffffffff80b7701b at mi_switch+0xbb >> #1 0xffffffff80bc976f at sleepq_timedwait+0x2f >> #2 0xffffffff80b76770 at _sleep+0x1d0 >> #3 0xffffffff80c5b435 at vn_alloc_hard+0x2a5 >> #4 0xffffffff80c50b72 at getnewvnode_reserve+0x92 >> #5 0xffffffff829b9b12 at zfs_zget+0x22 >> #6 0xffffffff829a6a8d at zfs_dirent_lookup+0x16d >> #7 0xffffffff829a6b5f at zfs_dirlook+0x7f >> #8 0xffffffff829b6410 at zfs_lookup+0x350 >> #9 0xffffffff829b182a at zfs_freebsd_cachedlookup+0x6a >> #10 0xffffffff80c36a0d at vfs_cache_lookup+0xad >> #11 0xffffffff80c45141 at vfs_lookup+0x581 >> #12 0xffffffff80c44238 at namei+0x238 >> #13 0xffffffff80c63b5e at kern_statat+0xee >> #14 0xffffffff80c64237 at sys_fstatat+0x27 >> #15 0xffffffff81049a79 at amd64_syscall+0x109 >> #16 0xffffffff8101f11b at fast_syscall_common+0xf8 >> [main-amd64-bulk_a-default] [2023-08-19_15h14m10s] [parallel_build:] = Queued: 34435 Built: 2 Failed: 0 Skipped: 35 Ignored: 358 = Fetched: 0 Tobuild: 34040 Time: 00:10:52 >> ID TOTAL ORIGIN PKGNAME PHASE PHASE = TMPFS CPU% MEM% >> [01] 00:07:29 devel/gettext-runtime | gettext-runtime-0.22 build = 00:06:32 25.4% 0% >> [00:11:25] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-19_1= 5h14m10s >>=20 >> Note the 3:31->11:25 . >>=20 >> Top is showing lots of "vlruwk". For example: >>=20 >> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 >> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 >> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 >> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 >> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 >> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 >>=20 >> In other words, it is messed up from the start, not >> just later. >>=20 >> It does suggest that the dbg kernel should not end up with >> resource problems: not that much gets very far. So I'll >> probably stop it and substitute the debug kernel, reboot >> and try again. >=20 > Still for nodbg kernel . . . >=20 > The "vlruwk" processes do occasionally instead show a > CPU?? . Nothing seems stuck in only one STATE. (Live > lock?) >=20 > As for using the dbg kernel instead . . . >=20 > Most of the time that processes are showing CPU?? more > progress is made in building, but basically one builder. > vlruwk dooes show up, gradually showing a larger fracion > of the time. ref 02 .. ref 32 are still in cpdup -i0 -o . > *vnode is showing up some as well. N process looks to > be stuck in just one of those. (Live lock?) >=20 > The debug kernel is not reporting anything during this > so far. >=20 > (some time goes by) >=20 > At this point vlruwk is fairly commonly what mostlt display > for the cpdup's that are not finishing --but none are stuck > in vlruuwk . >=20 > Looks like I should try without the 2 patches (15107 and > 15122). >=20 I restored to eliminate the 2 patches and rebuilt the kernels installed the nodbg one, rebooted, and tried again, still using USE_TMPFS=3Dno . . . The cpdup's get the same sort of vlruwk/CPU??/*vnode sort of general activity. ref 02 .. ref 32 still do not complete, leaving 1 builder active. In other words, the patches did not make a difference that I've noticed for what I'm reporting. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Aug 20 13:00:27 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTG1W2cLgz4r9t4 for ; Sun, 20 Aug 2023 13:00:31 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTG1W239cz3g78; Sun, 20 Aug 2023 13:00:31 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692536431; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=eBY4GihQL+HyA9GHUKv72/TvAa+k9uHysyjq9jGaHoc=; b=phmIJKiJ/MJewKvB464xS9W7HYggLv8cL/jo+ea3kGuvVZlVHSCIEKd864Zu9zNZrMrVXB WA9uJEg5PgFNLEcgNTDAuI5z7pQsW3Ot5IAphKiAVDKsaCLxZFjq9USeNtY3JBO7Fdnzbb G/yg+CIGXxQjxFtFK6uHQAV9LJCAeLcODazuewZ0ppcmJly7ykHpITytHcV1PxTiYxtF6d PEpESCPzhuBLPngnC3RZFE7EuqIG7dRe8P8iM1qOE2f3nauq+/oQ8Q2sPc6jG5A+BVnNik vW/7GZhPc8l7gSYrXDhHNaq8Ljas6F+JlvROAHKAtO4v4d3gk1RV/nByQOjNPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692536431; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=eBY4GihQL+HyA9GHUKv72/TvAa+k9uHysyjq9jGaHoc=; b=XX98v2Azmx+ahPqps+qP30t2au9po2BfXcxnH57LAoWjYAyfKvNbWf7OjxV6sAn2zol+Yh JGJGkkblFdvnMaqPl5PVVSYJUKb1eQ6cBxYa+YdZGyJyuQ7O4lYb5rPFXy0ndnyZbqV2iW VuxPRp/AA93jF2EPLgg6WjoqfyNEtg6RHKf4IxHAyxW/H+wC9rpP4lmK/zprRmo1ueDFYs mQSiH5PTdIeF30o52LhqGOxvIBPpDXXMwoZgPQwnny6tI00gtBkRTN+QTSmkP4I44swWsI uPqLyrfd7T5nqlbBTlBruJkp2sdQCgpWr55PPN6dv4AtNpAHSRP3ymvxXXcsJw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692536431; a=rsa-sha256; cv=none; b=sv0EAMYeC19Z4SEDXSvKxFWp4ULag1qo8GIGpgqCa3nIp1LTbIQ+IZ4sQaBm+No+ktFxTL aQiUM3nyta7U0qrPy/WRROJea1Mmi/wPOmPuoD7wn0ae0H1zIPKcknXn2MNTKICLgeMn3k Xclfa9I7YKKIYOkq2uVjKWluBJ5BGOU/qf1Ui3zfXcYAztQmHsWDuOUX45Tki4Zxp17VBv H0+8c23/uVWFe2cKG0toeD4yVKo/1Z93O5yG+YrA3DrGTPmiP4wTsl20D0IaZGARMd5rKX BMNaAWyN/5tuaIyIB3MmlHN8nUh65lB5/k8gDhWo9XXQi5TX52poIf+p70nv+Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from ltc.des.no (ti0187a400-1976.bb.online.no [85.166.95.197]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RTG1V3KBkz14b3; Sun, 20 Aug 2023 13:00:30 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.no (Postfix, from userid 1001) id 20AF22900A; Sun, 20 Aug 2023 15:00:27 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Alexander Motin Cc: current@freebsd.org, Mateusz Guzik , Martin Matuska Subject: Re: ZFS deadlock in 14 In-Reply-To: (Alexander Motin's message of "Sat, 19 Aug 2023 14:13:36 -0400") References: <86leeltqcb.fsf@ltc.des.no> <86h6p4s64h.fsf@ltc.des.no> <86a5utrafp.fsf@ltc.des.no> <86350kqokl.fsf@ltc.des.no> <86y1icp95t.fsf@ltc.des.no> <86ttt0p8wv.fsf@ltc.des.no> <197ead1e-210a-6be6-7e24-5c56b14bb777@FreeBSD.org> <86350hpiby.fsf@ltc.des.no> <0fd64567-59cd-9276-cc0c-17a09611f11a@FreeBSD.org> <86sf8hnp3k.fsf@ltc.des.no> <86fs4go2qb.fsf@ltc.des.no> <868ra8nfo3.fsf@ltc.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (berkeley-unix) Date: Sun, 20 Aug 2023 15:00:27 +0200 Message-ID: <86il99na1g.fsf@ltc.des.no> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Alexander Motin writes: > Unfortunately I think the current code in main should still suffer > from this specific deadlock. cd25b0f740 fixes some deadlocks in this > area, may be that is why you are getting issues less often, but I > don't believe it fixes this specific one, may be you was just lucky. Yep, it took a while but I got a deadlock with unmodified 4f14d4b6b7. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org --=-=-= Content-Type: application/octet-stream Content-Disposition: attachment; filename=ddb.txt.gz Content-Transfer-Encoding: base64 H4sICCYO4mQAA2RkYi50eHQA7L1dbyy5ci34rl9R8JPtc/vc5DezAZ+LgQ0MZuALzBzfwTw0DCE/ mLs1W1tSS6Xu3f3rh2RGVuUHs6oiyPJTt31ql6oUS5nkikgyYpHs2x/5j1/79q/u5eje//rWvDx1 fzsc3j9fDs+v3denl+H1oW9/FD9OP/3t8PHz62/xy4/1N+NXzfPzhW/Dz67/9eW1/3j49/j+EH5w 8ddTlxKt3rq3zwf/v6f+AP/9y0Gah/73l+bbUxe/9p9U34fwn6u4Vrw3Q1c9dJ/vx5/fXdOPNuff aKXRjWiqHw9vHrQ6HP0rqypl+aF7fzq+uI/jgR3+ofemz1/f3cc/BKi3rj39+RWU9X9seJv9uX85 vLy+uIen/tmdrmBhJWrdG+kvAP50parDP4Rf//Hgb0eaf3j4cM/D4bD+i/4/yxsxeIt4Td+at8Rv sNq3QV09HD8+3vYxhJUP7x9v1fY3/BV2jRM8/JWv3btIYkz/PXxe/Y2P9W9UD18+BH9LGcVLk5V8 eO6PhyTs+BtShtu79BtChib69cUdT7/x8PH2NJL74/Cze+5/3CNee3z4X+9N9/TyZUuRY5+g0oPH eIwY//hPh+Z4OP34l+q74P99eG++uVXzVh2vHn6Nf3G0Gd97A6ZF2qJX1cPMYPp9uffrbfXw+Pjt +P0x3PJjuPnH4bn58jGaJ78Kf543O3hD9fBb89V9vj02L7+PIOefvaVL2znmudh8fP3l03063yzj v2M0GFH2vg1Xo/odVJlCPT59c6+fx8eP9rgHPfuVcM1mB947kPd9dzy+9tCppx/DZVVyx673ZHDv L4/vrn19hYuYfeBtPT3TtoNvKE+c+HdGw+knb9Xv9MogolH3+u1b89KfzODnwL9qpwWHIVp693x6 Oz667647Wc8+Czfb2CQCq+o5gif9zAW2nweqpn2BVX0EOr43byfr8EPoIJY2YXr0urPN17MRV+km Ztz7xNkCftt2dfq3hb+szj/UzhbTT8Fq3+aHH36I2P478d8OvhGW8alq65YP3H/z8bYJu0w2lf+m TX9z8NCEYBNMccHG+x0m2HiYosGGaUMKNkzXdwg2THf3DDbMMFKwYUaRgw0zNT7YMNMSgw2zJjfY 1KxQsKkVOtjUjhJsWoEJNp3CB5tgkxVser4XbPw31GDTY0c2zKFGNswFGk7Dc+jD6cd4XTv0df6J N7y+f/VcewLSn34MjbxzP+ERG3/PN/O3t9fnpxc3Mz5/uB+VAsKsn3TNmP9CnbtL82rgUvKedbVT jTx3l5FuqAfXWuWks86eu8vfpjKD6LgwfSPqsbt2RrRvHw+HOJY9vI2vX97D+DvOqj6OzdH5H377 5j6+hH+7n5uXw/K/7lv/4B1QhumC0ez0evCD48O/wS/9MYwjcrh5WzHuKt0a43/n/eP3ly5ACJ0P IQ7+lVt4Df/xJcTRNWcIoVXXdEzJw2GYIMDYLCH+A2YUvzVPx9lVuEpJ5R2p6aowKX0YLf312/r0 Gu/iP+AS3p7e3Hs/s/fzlbbu/ITKt+T760tAqFT421W8Ff9+aoeP6zcxXkElkFcgjLdxdrqCqvZW 4VXCK74ZT8ZiCbHbjEb4ZhQVh2aMlqib0FyxirHqfBNaZNLJQ7BsCGXyIWJTKn4Y3yP5EC1RTTn0 WnNrZ3yQKvsmpMyH4Ncg9trhDMFCI/pAM75SiA3GagmxS2wpusZzU5+ILRSuN6QQwvF+1htcH4xU Ps4wZjk/aKWEv8B/+8v/efUmvn3tn94jhDpwJrk8KMMEEUIerLTCwuvUG38Z7+Mfx/SW6//psPnv y9PxQata12L73akhPqElz4yomsZVzlYAYHzISAHc0pkBoK6sCdFJxw4Nr6tWOH57i6w8AWijtK1r 4a/g28eX4duIUeVjaF0AQxXAKNAemuVjqLoAhimAUaBfYvDOxeAFMAr0iyzQL7JAv8gCXJcFuB6f JjEE1qzm0bC6MYo2X11EKBA9/N+v/fhJLq/iPwBjO2AdOtnU55GWFQV6RESW+0eS0X5ATHme2DDu zr0MXuBWeAFn4wW6lRVoD1Yg+FQF2rQq4LBVvsOaOv9eTJ0fSI3N71tj8/vW2HyeGlPgXgoMxIzO fziZAgMgU2DQYAo8aI0scR0FOCYL+Et8TBofiaRR9WnOdMvj5bnvXl+Gpy8BRRRo1figM6xOP2y3 szfhlJ9stdPD1gh2sIxrcxjfIx6U3V/+EgB4gZso8Jg03N+IqHyTGlUpi7mRX2DwYwo8Jc89cdvg R0pjDVMhWzddBfM3omp/C7JipsbcyNijrMBdxGes/3zJ7/270Nw/h6pTTsFU/h7s+RXjH0/fno4f EaM6WB8A42DSv0czU9f5zNR1GEtr65uDC4nqjKeXj2Pz/BxBIr3r2C96A3LpQjogha5j4POPWFVZ qzEN+u3r8Ppy/Oia5xEojiZNfNUbN7l4LdCq020sbmaixqU0y3gzUtaa8e23FzJOPmY1bsz/RQht fNSpL0BsczVS8KZpqmoGIaoKBcFdo6xbQTD0VZjWLiEutcUaglvLtBbrq8Dkrbg0vJG6W12FRLYF t4NaXYVCQfhnkBHrHtG4q5BmaMTqRgwKoqr50Larq7AoCN+ptdErCBQ7RW39rGZ1IwzFzn7wd1Kv OpXh2NnzvmOrtrjoqRuIqmLWY6wgUOzsrXCcrSFQ7IxX0btVW+DY6TvVtN3qKlDsbJXpNF85O8Ox U4uGNeu2QLGzb0Vrm2oFgWJnL3lX16vm5LjYWdWiMSt2chw7q0aKni9vhKPYOYYctroKXM6fSy6H egWBi51VbVndr27kIju/Hb+/zCC4dszxakUtjmEnl70Y2m7dIzh2hgdis25OFDujjzCzgsCxs9N9 r1fNKVDstG3Fq27l7ALHzkitlZsJHDuruhu0XLaFwLFT+8mDXbFToNjpvVI6veoRgYudVV1xP1JZ QuCe7JU1bd+sIDDsZIJ3cujUCgL3ZOfODt26R9DjTtuuhmsSFzu1b38nVhD4cadeEVzi2KkFa9t+ 2RYSyU5ZW7W+CuS4M4Tf1aNIYp/sVevWV4GKneMYXJ4hxpmZn075ALqYn+1PmMOUpo01j7evX34I sqanLuKwg2HcxjmrJaRj/HztNGsO7wkA09RwMUG8fX6nuZX2Uq9+/WVcZzL3dttVyy7hCjk6CDFn +SziCsVw7XpVrbjFkwX9S2GLtb1a3QiO4cFVrV5BYONvNyi5uhHszIqrmHWZQyBnVv4uDFtBoGdW tl43Jy7+Dpz55/CyLTRy7No0HmR5FRrDTu9dvkv6FcE1dnTgR9D96kaw7Bz4airANYadnDee3c6u rgLLTj/jrldXgYq/hg+qdeu2QM6s/LNoTS2NY2eYLvM1L3DsjAqi5RiFGww7uXSNadkyjcINhp1x KtB1wwoClZUaJySrHjGoeb9zFeer6TI3yJmV9ZewYqfBsTN26rpHUOzs60b1rFlBYNjpr1kwJuyq R5Bj19p0fOXsBsdOP/CUcjlFjJEUAeFk3ddyeRUW92TnjvtosYJAxk4f+Nr1VaBiZ5wirsbx/OIo J51S6tZXgc5K+StZQeCe7HE2sfJUi3+yr8daFsfO6Kmr8Gsx7PT951pTr56pNTKjX8tOrkJOjR93 ylVz1jh2xhHfytmRSlSn+qZZXwWGnWxotVR69TSrMexk0lV+xLd6FNXYeb+n1voqcOyMg8bVuLPG sVOb3s2LRad6F9fpqldq9YCQtZoWYUzzKbDfzKo29kw0qlYxFw/2o6U8GM7gqXhj/W85v9RBlad5 vcS5fB8mMU/V5+JseI+fZsrZRFduJrrrOiRvdFM53bXnOuRY1+Wr6u50Jx/u2XXHxfiikb1uu3mn Rs28jNqW8Hpq1JEXx+8//O34/XEZL5xWIX3jb/Bh1BfWfvy2UhleLPbrWZl8vPpQYBcx/CFa8eXp 5f9rHkbjUKnXS4j9S/BBdxjOxFI+0kjl53uH8T26I5UPmLaulTqM7/EANjaiubURtZ/EOO3HzSfF hPIsOl2C3RT6b7iE0x8fX09ecdhpRGt7Y6tzI0bvVqK+VS4hfYB3c/vIA++bN3ai8OOqhWjkbCtO D9KLdyDZYIVqzldQnZuQ4s9KzfpAUfpAiWjK4RUB8Obenx9OttGdZwgXRStsiOsExzbwltYI7wTj 65WIUvVmMKYe5KwXPANUzdmtshkjlZT9WfmsfECyVll2GN9fCYqs1a0PYs6cgqLi/NwLuJU4AMDs GYChVFQTgJoBEPKPsq5PAOE9AUCeHk3h/bUni394GNaJ4dSI0oozgCWEROkfrHF1P7yunitXFVzh WTRaLp5Lu86slW2GGZElqw5hMVKgUXh/TeNjGPPj4HkLVPIkCgzvr/GQM/9MUTMeitCJ0IRi24nX +0D4Gd50BeE9/goEOz3XxFbXuGmDtpaDEs35CsJkObyKg6lrKMNRRlwBwdbWoy1wLkvp+hjblzia nwZM4T16wMR8M4ZMW2gRdr1Jq7ppOt7MWoT57pxoFd6jaVWFYSfcQniPvoWKn3kZ3iM904cVOwGM 7/EARsMtjO8JAAoacXyPbMRgVEE3ju+R3TiGVv/qhznSP3muDpz9zRqptQmEHAedwTjcf9A9mFrO l1rets7ot589pz8exovwrxVPX0pi3KVtPUzPy9EyvLIb7Q2zduib07hpbju+Xgu1fgTvbDeN3Ea/ Dq/iVu9uWuvk+Q7OlrfZK97Wfmpoz3cQ1llaH2s9rXy4M+S+sPrsGuH9NV51Vgy6Ue2JV1ZyGLyM 75GDF1sbaSBkj+/RvhVU+eNTZ3yPB2Bn7w7vkQFqFArbWrF6KRe+MBEQQvLpyT1OzP2rH3jdOD2X SnX9NKGcW95mrxz3s+K5Q4Rr93QKQ+mb7iAu3rMR4QtAiDAfhggV3qMjVBBCTkwK79FMEsEboB/D e3Q/CuX7UXpvO4zv0UyKAX66BXV1+JG4BanOANsFjDcARD8K6uJFjN4N823VVk3P5HLcEXDE+UL4 ZjR8w4WEIG3t3oWkUk6elDFDsbiQkLGfLoTjh+XeyM+yTwD6qndv7ySs5xjH5au1HXsPCz+U8959 8s7IJ4hPFW6KO15AxfmJ19V2eneV19Y/PyfXjO+xrhm70YcHc2tnSqm4608BbmZ5m30MUO6c7Bmf dOGVL593u7xmmnVDU6/G0yOCtcLzIImzvRLVCd3HbTOWOCq2iPeMG0N+2H5DTZnh0dJaWeub7Z1q mnnAjStsgJfz1Ta7vByfGudLiC5hQwLGR1slr40IPXVrxW0/GxF640Apj3vQUtSocfF4CSo6N+Pp S0gECT8obU7jMBtyFTCstom8xdVhdfzT4ZXdeAFScunGPCr0wtl2fL0aHVSo/8ehXLyFQAH/am4l gmoqHhrxfAVn2/H16hWIvu/r5sSD0H3hVSw78QoVq1kbzGzj61UmMi2G0b3HK/Cjj6kbw3t0N8ow gjsB4OdnMSD4V6VuDAshxovh3IZnS0xYmbdhEGZOtyCuj6O3bRCWRsFIPr5Ht0GQacHoJ76/9pjp O9E2/ewxw0QIRZ7eB+njEe4KoBWqWp1uIrxH30T463525km4uIYLHRGG77OZ1cx2fL1K5hCT1Olx H7TCB2VsuJH4/lpH9tb0qj8/bMeRgglFIlEbdXWmHorRSrv4sIW4HI39q2+LJETiHkJEONHZ26vT PYT32G6If9q/VvWNF6ArNqixRAbdcLYdX691Q9iyXHanp5uRYV7qp4P+ucjMtsi26QbXiUH148S2 Y2/Pnx8BhIWksgiavQyQ8Jz2AaJeguzHlo7XrIl38v1LbM2YVx0909ySV117pjdSM4DrM6MEQCyP +MiWLJKkq3VmKjR5m5BAG0eg8T2aUMKwEyPDe7RXifC3pya4nuhINIFnAaYbjfDzY93Mu3FsvhhZ 6u2DNuHY1rY+yp+dQsz8Umz98oZWCPP8KFVJXkJ6vONOo5VoaaNy6DZ7P3RmbV3J8y2cbcfXa34t vQO59jRW0DWTE5Pie+xcRltbnwDsDT69BVBm6oT4HtsJOmztcwK4Pp/bAoR1+sDl+B7L5bjNgB8q hmLJfLOBSwkrPYhJPuAfl/63IWcX31+bU8uqFY05z6mVMKeHZHyPbQIltJiaIL7HBpS4VyDMnub7 Bu7yULmuMqf9E72Fv3Zogvge2wQiOAI0QXyPbYK46YUf6YXCwHzri4tZw8bMZg7R1r8KfSOC9C5b jVF9QvC2lktd34wgVF/Phls81KuhH+J76Id51XSG4LnMWDPYuDNB6If4t/2rHzffeAVh9tPOZnAe 4ZS4jO+xAWH2x8fXa1SK9f/uNNrinj8Tl+N7rDvzIBL2r2FdFnmYEs0xzzdVW92oGNrH5xuvQljy rzJ9GRd98nQZwRx1GUMzKBlnUuNlsDoIMUbPjO+xnsnqWAe3QT1Ibs5ojhr1eV66PvoW3IcNvcps WKSWcRneHHcZHWvG+RBchh+Jn5rT4DVC3kjwM8B2k9fr/RGng97eE0OHoHdN81f39SA0m02IorF/ 5VUaIjl4a/jJRaNlkBvUN9qH+b2pZvOZme34ejVICMXlMI28xjLhGCdXJcMd+yAxak5Z6DHtOtnL G553ngj1wKeS5ZiZGDeGEFXN2LVOYLIXoh4rRGMnjMajvDgJkY4weprSjZajwvk2+7EJ4y3ETpjb jq9XGzGUuZpp3DOqBKERU4rB1NTc8s5AHnxMDYH9Mk202wkh1TYNOsbNlv2rp9Jiy+WLyUItz4/8 ue34egMNWGOmjOtYXoQ7WJYa92gcBh186sNxrax/9XjJFbPbFlRtLXqzyuOPCLoKjLhx5a2fyPRu rWgeEfyruX0Fb9ewbpqOzC1vsx8z2OoUGEbdoPLPKZlWD25aRGjTtR1btciIoKxit6tRWW9UDDBz nIig/JiY36pmjFrxqUXmlrfZK1kbORsSju04cmzVpnteGlheTRwdp5WjfXKKuXle8Nq1vWjlZB+V 2WC/uIM9jseZ/uSlY9Zwsk9kELccH7ipXTf9/ahIAfuUOmVz/a2PJX5UDPaQNBuFiKnU2SZUG936 /3HwUShmjPapksbGvhPc+UnY9PdHPfJon1Ilb66/MrVlapjsx4fkaJ96VO5EWT1d/xjfR/tUlN/Y G2nbcKOj/bgffXgVy13p9xnsm0m25yTH3HZ8vcaguO7j9KQ7iLiUU8T1sdSh32iOSlj5aUkzREeO Q7+xmgSK1FRlaVt0ZYOf39nT4y5MhqbHXWJilOrIajBnXUzYNzjsa7IzobhpBBzNcROKqnLqPAIe bX0DaJFG2PC5t23bS3uu1Y5VsbElkhWybUt4bzKWTdre+Mdrzbm8dRgvvXU/KxdLoePSx7jgDF4P 4R6n5VCXdnj4+Xh86x80q+3FxXrhkLJh1h/+2ciapon7CQBEiNOXVtptGsIPv4waYmyYQVxaaZdo SzXUTdWdICTn7NwKpLaQKhZu969is9uFUN2gx81YphtR9uJ6v50nv5RLCImAYA3XLqxHPbeF9zmW 1xaeF+rifnFbXjjP5SaulT7fiLy03m+nU0W3hLi03m+nbqbPEGNGlBnvpzfnRY0VbgpYoyXTYXvv W+2lathUJhktD6yubrcXemDTyGd81sDIKfXc2RH5TgF7/Mu+ASp149+PkpIuzo/GMDMaH8JZizff gh+Y99PgabT0r1Gae9MlhAG1jU04XUIwPjDlPePmXnR2qE6XECzDq7j5EqyfpJ3XZc1tx9dVPyTn ea2ZrmC0mUaACfsNlVvph2CmgvEDVypuypkT3/wI5mKUXfu0fyoxK8Z9dEYIPyMRF7cZ2Zstd0uI S4uRNxDWiF4wew6RBz8Ct5ltoSS7uC3cui0q1gTxWVXNbqRSF3co3dwI90P/VrNuCYGLbzycoWVm bSHH7TAz2oKFVdKXemQT63vd91KqRVuIi5sXbOc3rJZV28klBO5x4d2j7dpu0RYyty2svLin26Yt tFCt0L1d3gjBR2S3hMD4CAub2zVSLNsic2zIwlkSlzp1Ey+cHwYMQ7NqC+TALgySq1VbYIZUYdeW WnG2bAue2xaer5fGyemRyHyQG2/kErW2EH4kIJxcQWCG2tz3hdZVs2yL5Dj576i2uLi1xiZ2Gv8c Nb1Z8uLiJjg7baGWbZHcBOfvny//d+Lj1I3EbOn4TE5mThNjqz7szjxNZsf6nn+c84N3lvrqLDKG rFo1sypGNLZC1TYNsZPDP2knomV41bfbM+8f53rvzHZ8XTVCWtV/qqOMVdbwyg51rczVxdbcGxvj Zo0wGhsemJmESK+ar045/GjpX/2z9Fb7rtHinJ+c246v1xohls311A2jkt+/xo31LLuhEUTtH6Ns VkqJxiaefZiESA8w5amQEC3Dq7jRPqbxq7Mgf247vV5lgpBmqmeNwgf/GiK+8dOYq/Wkoa6F6+oz E0ZjZVTYdD0FsVNibU+zpWAZXu3N9k7aWT1pbguvV5kgwsKtKdMbi6L+NeYrZ7rlS5VNP5xVZyaM xrIO2wbcBBEGlkHCMdsQJaaVahX3D76xRuxUMy4/G6dco3EdahC3l5kb5pYQ4aAxFdYg3pinC1l/ dtpFIVqG150sXXrFtT0Ht7ktvF7pzSha1VNwGwMiPCFSwXH7hGDOO0U/zb4jg2DrgBSbEi5hOavN lDWO1wz2qevfsMm2LWd8qhuMbjxtXZBw6Z06+/T3x1gI6eJUXEyM34R1dQv2I4kPeryKBJWTQake 4jSrfX09foSjyx9GgIOWit8IE7fnGbdlefv9+PPri/hr/TAC+Fe545up6NK0+lxrnduOrzdFh1Nt cbIcN5RI2G/6g8mWCTPVUayNhxIZG69Cz9Lvh4v9MdahotrzbHojgAxPqlhIO7ekFbw61XPmG2NM 95E4+KVrlI0HgYd24JXmKfsbJWIny7N3XO8HIavpYXmytBZQvHuHJPZ/fPwl7ZfaiZ43Yz3xs399 GG0PTI11qfh+RNj1DOnnMaKaI0AnyFVXfOwgqFoI18bk5rkrRgDMqc+qGmTXn8/gtkGTEcJDHU+r Cu+Xt/LWfH64eWOOMjGtwq107869/CD/Wv+1ejiM2hTfKP4V3i+a9Xj8/Wl+/EE8CsJH7FDd+WPM 04Xa3ggA7xcAm0sZiyJhacIJYCTDGQYA9rslrAztNvcS2yO8VtA2Vy8lBAsbW+WPMfaOpnE/RTtj yAVf5VoM4Kv9w2Rr4paj1l5nSOSY5HMEOEsi3go/x5v9DqmdbmXokC/Ofxsh5AxC3gLR+wH5EkLM IMSNEHYJwWcQ/DaIanUVbAbBboRYXUU1g6huguDD8irg1JcIweyNEMurgLNnRghzG4RbXcWMF+wm XniI5VVUs7aoZm2xR86Q5urdLHd5ClcBYh66AOKlefl4fjtBxEBcN1bJDkLX5BuAYMz1i9B+vgKj 9dFDjD6zYnp/EcE3g9Mx1fb+8ftLFyDEDCK+Z4HkE8SllMoLZFQkKq8DFxFThi9jU2pIqgeY6f2V UFH14yqOj98/nl+/BBA1c3V4H2rsl7rUB+CoMnl+7ZrnHz5f2tfPlxGKzaDO5wYC1CWJafvcdF+f nz6OAUdGsUscuMHrIUwnppZp34YRelZEbF0ft/d9G+C2wC5ey/R+3jbtOFyYJyFd5VjXzTHOopX5 67/9+4jxNhy/za4jjllcU9kw8PvJX+Pb5/sX958R5xwK5UzzuU9Y4Qf0XWji3v0aruSg1KUraZ5e 3/vf18+lcR3eT/7LXv5nBJEEkDhLjSBiBBFYkL4baygRhI8gnACiTrfDIsg5hKVAfn1+//ztOANh wg31uALpp19f/LcjyMWGDc7u3pfRqLGuC2Pan8YvI8rhIlEuBYOf2s+hb9y315f/9DEh7sG5/e/f 4N+3j2fn3taX08fjkzZAqRz8BPQD/DsH0pVgE9DHW9O5Ee2HCvBS9Y0reDH5vcZjgJcqM1zB06nr 44CXyi1fwbOp6xMRTyePl7yC16auTwLepY7dwetT16cAD9+/4/KKNZ4eGXwalRwSDE4Sz/FBRWf6 NvEu/Kq5hHPRE96aL27O4KRI6aIrdOG4PI/Uv36bWJsqI13BqNoOMA4/u2c/k5qwUuU5LNbE/lTZ c8J6bvzz9GN7b513wZ/il++//3iY3eSlhvr81rx3zwuwyjezCtPqn/yXY69dDKUpNpm2q8OJGz+9 Ny/948/N+6/u4/jLCHYxpF6kgB8ifHXv0P/J8+ym2zr+8Lc/PNJ8XNKxTqqgHfrJ3/Gj+/WpOwLQ pdCwAyRPQP5q3wDnUjuvGmnEGXhUiv3Ut5/zC0oecDcB+d8d/C8vhhXh9K74zIEvPdLj2I4AiAgu 05WNweqPx1/9GONxeHp2j0Cn5MF3aDAgevIIPDQYxHiBeAbtgwkAu+TON4NJAEM8zfbBFIBdIv7N YBrAEM/FfTADYAQP2ILZESwpukCD1QBWxAMYuEDydD08GvhA8qA9PBo4QfLMPTwaeEHy+D08GrhB 8iQ+PBr4QVJKikcDR0iqSvFo4AlJZQgeDVxBFXEFBr6QPHwPH3PBF5Ln8OHRwBeSR/Lh0cAXkqfz 4dHAF5IH9eHRwBeSZ/bh0cAXksf34dHAF5In+eHRwBeSh/rh0cAXkuf74dHAF5JH/eGHDOALyVP/ 8GjgC8kDAPFo4AtJkS4eDXwhqdfFo4EvJE8IxKOBLyQPC8SjgS8kzw3Eo4EvJI8QxKOBLyRPE8Sj gS8kDxbEj3jBF5JnDOLRwBeSxw3i0cAXkicP4tHAF5JKaTwa+ELyPEI8GvhC8mhCPBr4QlKgiUcD X0hqNU9JDh4m18N6mlvFpAJ86Vy/mOYmtbRXLq8ZL2/4+HX4AJRLZZc/3K/u5fi4uqxOyDHj+xG/ jjgmiXPhauq2bwU01h+/vj4DCtKNAgoHlLd3N7hj9zPM3E1SNI3FYoCFdKEkFgcspAMlsQRgId0n iSUBC+k8SSwFWEjXSWJpwEI+RJJYoxOa5KGKWCwLWATGb7BqwCrBe5ivm4tp3pvBgPmYSsc+GFD/ YtL4ZjDgPqZmsg8G5L+Ygr4ZDNiPqb7sgwH9L+azbwYb+T+pFDLBLICVcACYpltMBWc/wlYAVsID YJIeV+rng3EAK+EBMEW3VQkPgBm6TS7lQ4MpACvhATA/t1UJD4DpuWUlPABm55aV8ACYnNvkcnP0 uAA8gJXwAJiaW1bCA2BmblmR4Q94ACvhATAvn4RkmWDgAayEB8Cs3LISHgCT8km1lwkGHsBLeABM ySdVYx4YzMgnlWUmGHgAL+EBMB+3vIQHwHR8krRmgoEHXCwy3wwGHnCx0HwzGHgAttqcBgMPuFhx 3gFjAPb59uW96afysMXWmlNQwDFspTkFBQzD1plTUMAvbJU5BQXswtaYU1DALWyFOQUFzMLWl1NQ E68IjF9DQVzF1pZTUBBVsZXlJEWB7tjCchIL+I4tKyexgPDYonISCxiPLSknsYDy2IJyEgs4jy0n J7GA9NhichILWI8tJSexgPbYQnISC3iPLSMn4ynwHltETmIB77El5CQW8B5bQE5iAe+x5eMkFvAe WzxOYgHvsaXjJBbwHls4TmIB77Fl4yQW8B5bNE5iAe+xJePkwx94jy0YJ7GA99hycRILeI8tFiex gPfYUnESC3iPLRQnsYD32DJxEgt4jy0SJ7GA99gScRILeI8tECexgPfY8nBypAq8xxaHk1jAe2xp OIkFvMcWhpNYwHtsWTiJBbzHFoWTWMB7bEk4iQW8xxaEk1jAe2y9NWCNawz6t8ewPObx2Hx8/WWa OVIKpik4YBmlcpqCA6JRiqcpOOAapX6aggO6UUqoKThgHKWKmoID0lEKqSm4iXeEeJuCg5BLKaem 4CDqUiqqSRqDW1Cqqkk88AtKZTWJB45BKa4m8cAzKPXVJB64BqXEmsQD36BUWZN44ByUQmsSD7yD UmtN4oF7UMqtSbzRP2pKxTUZlyvAK+QfMButKXXXJB4HvEL+AbPSmlJ9TeJJwCvkHzA7rSk12CSe BrxC/gGz1JpSiU3iWcAr5B8wW60p9djkQAP8g1KSTeKBf1Cqskk88A9KYTaJB/5Bqc0m8cA/KOXZ JB74B6FC27kJ74+n58fu2TUvAAqQeBe5Con3kquQeEe5Bkmo3F6FxLvLVUi8x1yFxDvNVUi831yF xLvOVcjy3kOo8V6FLO89hHrvNUhC1fcqZHnvIVSBr0KW9x5CVfgqZHnvIVSJr0KW9x5C1fgqZHnv IVSRr0KW9x5CTfkqZHnvIVSYr0KW9x5CvfkqZHnvIVSfr0KW9x5CLfoqZHnvIVSmr0KW9x5Cnfoq ZHnvIVStr0KW9x5CDfsqZHnvIVS0r0KW9x5CffsqZHnvIVS7ux4qYn+8u9PGODWh1L0CggwLoc69 AoLUCqHIvQKCnAqhwr0CgmQKoby9AoIsCqG2vQKC9CKhsL0Cgrwioaq9AoKEIqGkvQKCTCKhnr0m JFCbUM1eIwG3CbXsNRKQm1DJXiMBu7HLmhNIQO+LS5pvQwJ+E2rrayQgOKGsvkYChhMq6mskoDih mL5GAo4T6ujrWAkcJ5TQ10jAcUL1fI0EHCcUztdIwHFCzXyNBBwnlMvXSMBxQqV8jQQcJ9TI10jA cUJ1fI0EHCfUxddIwHFCRXz9GAeOE2rhayTgOKEKvkYCjhPq32sk4Dih8r1GAo4Tat5rpMhxVhGq 3WskDUjZHB/1iix9sDIOyQJSNsdHraJHyub4qFT0SNkcH3WKHimb46NK0SNlc3zUKAY9RjaSBKRs jkvgOKFivUYCjhNq1Wsk4DihSt11gPT58vz08tX1j/178/TyCLwi1Kn3EIFfhEr1HiLwjFCr3kME vlGq1TuIwDtKsXoHceIfwSd2ECceEnxjBxH4SKlU7yBC7KUUqncQIQZT6tR7DAenodSp9yDBayh1 6j1IcBtKnXoPEvyGUqfegwTHodSp9yDBcyh16j1IcB1KnXoPEnyHUqfegwTnodSp9yDBeyh16r1o Dt5DqVPvQYL3UOrUe5DgPZQ69R4keA+lTr0HCd5DqVPvQYL3UOrUe5DgPZQ69R4keA+lTr0HCd5D qVPvQYL3UOrUeyMX8B5KnXoPEryHUqfegwTvodSp9yDBeyh16j1I8B5KnXoPEryHUqfegwTvodSp 9yDBeyh16j1I8B5KnXoPEryHUqfeG6WD91Dq1HuQ4D2UOvUeJHgPpU69BwneQ6lT70GC91Dq1HuQ 4D2UOvUeJHgPpU69Bwnegy3XNobVMkJ+c8fm47lpH7+8v36+xYItQCJJdAskkkS3QCJJdAskkkS3 QCJJdAskkkQ3QGJLvbdAIkPwLZDIEHwLZHnvwZaEb4Es7z3YIvEtkOW9B1s2vgWyvPdgC8m3QJb3 Hmxp+RbI8t6DLTbfAlnee7Dl51sgCd7Dr0ASvOcaJMF7rkBiS9S3QBK85xokwXuuQRK85xokwXuu QRK85xokwXuuQZb3Hmyp+xbI4t7DsMXvWyCLew/DlsNvgSzuPQxbIL8Fsrj3MGzJ/BbI4t7DsEX0 WyDLew+2rH4D5MVy3PH7D387fn+cQfJaOh1Pn/vp+P0LnLXy6F6O4VzriHeJlH88Pb9++eFvi0t0 vBPtBbxLjPz27e2Hv337NuG5SmijGybDOPwn/2UK7xIdt0e7hvt1PJ7l8sfj00v/9O6642P3+tK7 lw8HkLhjZ3lt+roGyOenX104xf6xdx/H99ffARF5/mxjtJJrxNVFXmJjAtLywTQA2f3suq9vr08v /jKfProGjjC/eDj8xVN7u+ZbVFVJjNuFC2PG+t4IbOlfX9wvFaAgPO2E0p1QRv2axDjXhCLO18IB BXtqd0A5X4sAFOxZ3R5Fnq9FAgrimXNCOV+LAhTsOe4eRZ2vRQMK4slyQjlfiwEU7BnwHkWfr8UC CuL5MaGweC1N2L0hgijMbGUCiZOHn7wHvby40YnYRSf6cL98HJvjIsVaD7az8Wwp98une+mcd9tq dEhxCeuyQ77//nZ8jTcmLuaRx19cnHbFGn89ISO3RLnkkyuUeF+NqGJZH1AO7+74+f7ycRh9XFxc 37SHZxN4DPAu+ekeXp/A44B3yWN38BhP4AnAu+S7e3gmgScB75IX7+F1CTwFeJf8eQePswSeBrxL nr2HpxN4BvAu+fgeXpvAs4CH8IoJbzy7bYVXAx7BP4RK8Rkc5GJOfA+wSQGCh1zMiO8BDilAcJGL +fAdwPGhtgYEH7mYDd8DrFOA4CQXc+F7gC4FCF5yMRO+A6hSURCW7oiLefA9wGQYBD+5mAXfA0zF QVjIIy7mwHcAdSoQwnoecTEDvgeYioSwrEdczH/vAaZCIazuERez3zuAJhULYZGPuJj73gNMBUNY 6yMuZr73AFPREJb8iIt57x1AmwqHsPJHXMx67wGm4iEsABIXc957gKl4COuAxMWM9x5gKh7CciBx cQS5A1in4iGsChIXs917gKl4CIuDxMVc9x5gKh7CGiFxMdO9A9ik4iEsFRIX89x7gKl4CCuGxMUs 9x5gKh7CwiFxMce9A9gmB4bgKRcz3HuAqXgIG5+Li/ntPcBUPITdz8XF7PYOYJeKh7AFuriY294D TMVD2AddXMxs7wGm4qGcUhAET+lT8RB2RJcXs9p7gKl4CNuiy4s57T3AVDyEvdHlxYz2HmAqHsIG 6fJiPnsH0KXiIeySLi9ms/cAU/EQtkqXF3PZe4CpeBjUUN6E0WfsX9zrmEMTmGx4uC5WazHEmPrl 8Xw4tUCn4iKMjTCfb4CBTcQFDDleSv/628uYxuCpRvl/b0pjPL92Ma9d8WQW4/9IfJYGOfxj9U8A lLonDBADoGQGAwPEJ6BUaMAAiQkoFRIwQHICSoUCDJCagFIhAAOkJ6CU62OAzASUcnkMkJ2AUm6K AaonoFxms4nayRwECmnidjL5gEKayJ3MOqCQJnYn0w0opIneyTwDCmnidzLBgEKaCJ7MLKCQJoYn UwoopIniyVwCCmnieDKJgIqVE8eT2QMU0sTxZNoAhTRxPJkvQCFNHE8mClBIE8eTGQIU0sTxZGoA hTRxPJkTQCFNHE8mA1BIE8eTWQAU0sTx5PQf9RifOJ6c96OQJo4nJ/wopInjyZk+CmnieHKKj0Ka OJ6c26OQJo4nJ/UopInjydk8CmnieHIaj0KaOJ6cv6OQJo4nJ+6oEebE8eSMHYUEHBfJqToKiU9I uRyXYkLKHonLCSmX41JNSLkcl3pCyuW4DBz3PzP61PDp5RgVQBVLLmO76Wo+fntSPx6G5uN4iNtF Ah7Z7zye/jFCHX75dJ8O8Mg+M+L970/Ny/wCeXKt2a2A7MfDizs+fbxDIV4mt9a4Ce3p/RdZ/Xj4 9en9+PT6+NY9TYhk1w6ILIGY3FrjZkQ+R4R0XHJrjZsRRQqR7PIBUc4RIcOXlPLdjKhSiOQQEBB1 CpHMRY/ou7o5fm37qZdzmMh8L799fJugyBT0LuI5/dm8H+NLFNsp+pWNHvc2HD7cSz/GvJF5q5j3 95tiXv88xpT0rlx//3yJ//7r//X/HFIOc8L48dC9fVaAlKLEHCnlKEskBkipp8IcKeUgSyQOSKkI PEdK3f0SSQBSip5zpNTdL5EkIKUoMEdK3f0SSQFSiptzpNTdL5HG9HWVVDjOkVJ3v0QygJTq5TlS 6u6XSBaQUr08R0rd/RKpBqRrHE/e/gIJtDTpbbgWUDssn0EBy5O6xwXUDs1nUEDzpPhxAbXD8xkU 8DypgFxA7RB9BgVET44HFlA7TJ9BAdOTA4EF1A7VZ1BA9eQIYAG1w/UZFHA9+ehfQO2QfQYFZE8+ 8xdQO2yfQQHbkw/7RfS8ynbQw6S3z1pAXWU7KGHS22YtoK6yHTQw6e2yFlBX2Q7ql/Q2WQuoq2wH 3Ut6e6wF1FW2g+IlvS3WAuoq20Hrkt4OawF1le2gcklvg7WAusp20Lekt79aQF1lOyhb0tteLZ7w V9kOmpb0dlcLqKtsBzVLepurBdRVtoOOJb291QLqKttBwZLe1moBdZXtoF1Jb2e1gLrKdlCtpLex WkBdZTvoVdLbVy2grrIdlCrpbasWUFfZDhqV9HZVC6irbAd1SnqbqsUo9CrbQZeS3p5qAXWV7aBI SW9LtYC6ynbQoqS3o1pAXWU7qFDS21AtoK6yHfQn6e2nFlBX2Q7Kk/S2Uwuoq2wHzUl6u6l/bV4m tN3/ZlCj2uQwn0me3v/Hv3/Et781T8f40bSWjreilm30tp+eXp6O42x0NQ89LBQrzWf/dHz8bb06 pA8prZ/il+OFJEH+DhdycUr71b2/uGeYGF3SXH381rw9vazVJlJGEVf48m1cEVihN99SfdezKN16 Gh6fXh+Pv3yB817Qm26puFfxEgq8DbvZlodybgUF3obdZCtA9Sso8Dbs5loBqltBgbdhN9UKUO0K CrwNu5lWgGpWUJO3IdcBB6h6BQXPFuzmWQHKrqDg2YI93ClAmRUUPFuwhzoFKL2mKNAde5pTwFJr LOA7dl+wgCXXWEB47IZgAUussYDx2J3AAhZfYwHlsVuABax1pJnmyti9v1Q89WqFBaTHbvrlsYZ1 rJlmy9jdvgLWOthM02XsNl8Bax1tpvkydn+vgLUON9OEGbuxV8Bax5tpxozd0StgrQPONGXGbuUV sNYRZ5ozY/fwCljrkDNNmrGbdwWsdcyZZs3YXbsC1jrmTNNm7HZdAWsdc6Z5M3afroC1jjnTxBm7 QVfAWsecaeaM3ZkrYK1jzjR1xm7JFbDWMWeaO2P34lJ9X61jzjR5xm7CFbA2AxzgPXb3rYC1jjnT 9Bm77VbAWsecaf6M3W8rYK1jzjSBxm60FbDWMWeaQWN32ApY65gzTaGxW2sFrHXMmebQ2D21AtY6 5kyTaOxmWgFrHXOmWTR2F62AtY450zQau31WwFrHnGkejd03K2CtY840kcZumBWw1jFnmkljd8oK WOuYM02lsVtkeSy2jjkSeI/dGytgnWJO9/oyPH2ZzRwZdl+sADeGnY/X4fj0/ssJCM971i6AGADh Sc+aBRAHIDzjWb0AEgCEpzuzCyAJQHiuM7MAUgCEJzrTCyANQHiWM7UAMgBEoLhcAFkAIvBbLIDG QIzesyoA8SUhgdrYo6ACElsiAbexR0EFpGqJBOTGHgHlh0Ri2Uww0WTYo58C0qqdgN7YI58C0qqd gN/Yo54C0qqdgODYI548knRLJGA49mingNQvkYDi2COdAtIy5sLkkmGPcgpIy6ALU0uGPcIpIK2i LnAce3RTQFqGXZhWXt4ibgdpGXdhUnl5c7gdpGXghSnl5W3hdpCWkRcmlJc3hNtBWoZemE5e3gdu B2kZe2EyeXn/tx2kZVSBqeTlbd92kJZRBSaSDHvkUkBaRhWYRjLsSUsBaRlVYBLJsAcseSS1jCow hWTYc5UC0jKqwASSYY9TCkjLqALTR4Y9RSkgLaMKTB4Z9vCkgLSMKjB1ZNgzkwLSajAHHMcelRSQ llEFpo0Me0JSQFpGFZg0MuzBSAFpGVVgysiw5yEFpGVUgQkjwx6DFJCWUQWmiwx7+lFAWkYVmCwy 7KFHAWkZVWCqyLBnHQWkZVSBiSLDHnEUkJZRBaaJDHuykUfSy6gCk0SGPdDIu28DSZuXb4/Du3Pz hQbYs4wC2uh7Y0V1sWgBy/Q2bJsSsJqn1/7x61P3dQGHpbuHG12wd4N7f3f94+fLt9fPuHABILG8 95CjLz4/vXx+//r29Bi64rfpCtGUbdrxht+6p8ef3xZ3iyZt0453+zWu83jsjt8XeGjCNS3kl170 lif4wmPTimXLffz8+n70bTdlFfAFyH1IiGv4QuQ+JAQ4fD1yHxIiHb4suQ8JIQ9fndyHhNiHL1Lu Q0IQxNcq9yEhGuJLlvuQ8OjHVy73IWEMgC9gXqU6vo55HbKg90yQBb1ngizoPRNkQe+ZIAt6zwRZ 0HsmyILeM0EW9J4JsqD3QFTHV0OvQ5Z/9uBro9chC3rPBFnQeybIgt4zQRb0ngmyoPdMkAW9Z4Is 6D0wgMHXVK9DFvSeCbL8yA1faL0OWdB7JsiC3jNBFvSeCbKg90yQBb1ngizoPeNYnePLstchC3rP BFnQeybI4vMeji/fXocs6D0TZEHvmSALes8ESfAevoR8fn35cp7hc3y9dxdxHApyfOF3F5EDIsF1 dhAFIBI8ZwcRegZfEN5FVIBI8JsdRA2IBLfZQTSASPCaHUQLiASn2UGsAbGcz4AigeMryPuQ4DX4 UvI+JLgNvqa8Dwl+gy8u70OC4+CrzPuQ4Dn4cvM+JLgOvu68Dwm+gy9A70OC8+Ar0fuQ4D34kvR+ NAfvwdem9yHBe/BF6n1I8B58tXofErwHX7behwTvwdev9yHBe/CF7H1I8B58RXsfErwHX9rehwTv wde49yHBe/DF7v2RC3gPvuq9Dwnegy9/70OC9+Dr4PuQ4D34gvg+JHgPvjK+Dwnegy+R70OC9+Br 5fuQ4D34ovk+JHgPvnq+Dwnegy6jXxilg/ega+kXIMF70CX1C5DgPfilyfuQ4D34Jcr7kOA9+KXK +5DgPZRq/R4keA+laL8HCd6DX8K8DwknnVDSD6MeZnh6//Zb8z5XAgjKFHzUxMBhFnAOEGXivcCB 438o0+0FDpz6Q5lkL3DgsB/K1HqBA2f8UCbUCxw42ocyjV7gwIk+lMnzAgcO8qFMmRc4wGrKPHmB A8f2UCbHSx4CoSlT4iUQMJoyEV4CAaUp098lEHCaMuldAgGpKVPdJRCwmjLBXQIBrSnT2iUQ8Joy mV0CAbEpU9glEDCbMnFdRkZgNmW6ugQCZlMmqUsgYDZlaroEAmZTJqRLIGA2ZRq6BAJmUyafSyBg NmXKuQQCZlMmmksgYDZlerkEAmZTJpXLZzUwmzKVXAIBsykTyCUQMJsybVwCAbMpk8UlEDCbMkVc AgGzKRPDJRAwmzIdXAIBsymTwCUQMJsy9VsCAbMpE77l6BGYTZnmLYGA2ZTJ3RIImE2Z0i2BgNmU idwSCJhNmb4tgYDZlEnbEgiYTVFtL4FGZkvsQKt2DNav/Hp8ccfq8P79l2l3fuxYK2DpOdbxjIUd kwQsWMPy+8fRfXuMM9HHCQ1J8h002D0fOzrZQYNd6bFDlB00OBgTO07ZQYNTMbGDlR00OBITO2LZ QYPzMLHDlh008APs2GUHzQIaMszvoI2xXmJHMXvsBWfAjmX24MAbsCOaPThwB+y4Zg8O/AE7utmD A4fAjnH24MAjsCOdPThwCex4Zw8OfAI76tmDA6fAjn324MArsCOgvSgMXoEdB+3BgVdgR0N7cOAV 2DHRHhx4BXZktAcHXoEdH+3BgVdgR0l7cOAV2LHSHhx4BXaXzj048Arsgrk9OPAK7GK5vdEEeAV2 odweHHgFdpHcHhx4BXaB3B4ceAV2cdweHHgFdmHcHhx4BXZR3B4ceAV2QdweHHgFdjHcHhx4BXYh 3B4ceAV2EdzeqBi8ArsAbg8OvAK7+G0PDrwCu/BtDw68ArvobQ8OvAK74G0PDrwCu9htDw68ArvQ bQ8OvAK7yC3AyTlc756b3wEUIAmecQUSu8jtFkiCh1yDJHjJNUiCp1yDJHjLNUiCx1yDJHjNNUiC 51yDLO892EVuN0BiF7ndAlnee7CL3G6BLO892EVut0CW9x7sIrdbIMt7D3aR2y2Qxb0nPjtLQxb3 HoVd5HYLZHHvUdhFbrdAFvcehVWZ3QJZ3HsUdpHbLZDlvQersbsFsrz3YJV3t0CW9x6sHu8WyPLe g1Xp3QJZ3nuw2r1bIMt7D1bRdwtkee/BqvxugSzvPVjl3y2Q5b0HW6Hu2MDHnTab9+7x7f3zZTzT WiUno4sz1vaOAexd0z9/fXcfAIRst3BF4756Tff2FG8OitwKO09MQTGAIrTTGooDFF6p0I0bGv7r //Y/Z3pyjR4wtQMcAPLH48vn8/Pj08cHIBHa3C2QXo6AhKVo29cckN77cEXQeRo9KtoiMUDChvMt EgckbBTfIglAwgbvLdKYPtToEc8WSQESNlRvkTQgYSP0FskAEprjfc1mSC/Hx2piFHo8k8ICTqEH MiksYBV6BJPCAl6hhy4prIlZBLZvsCZuEfheLbHY1I/oUUoKC/oRPTxJYUE/osclKSzoR/SAJIUF /YgeiaSwoB/RQ5C2b9wSi5/6Ec/7LdbUj3jeb7GmfsTzfos19SOe91ss6Ee0AjCFBf2I1v95rH6J JaZ+RKv/UljQj2jtXwoL+hGt/EthQT+idX8prKkfCbzfYE39SOB9t8SSp34k8H6DNfUjgfcbLOhH tNovhQX9iJb6pbCgH9E6vxQW9CNa5NdG/fIcS039iFb4pbCgH9HyvhTW1I8E3m+wpn4k8H6DNfUj gfcbrKkfCbxvAOu39/n8B63m2yJBH6KFfFsk6EG0hm+LBP2Hlu9tkaD30FK7LRL0HVplt0WC+Q9a EbdFgvkPWgy3RRoFRBqtg9si1YCUzydQXmu0/C0BBSxHS98SUEBztOwtAQU8R0veElBAdLTcLQEF TEdL3RJQQHW0zC0BBVxHS9wSUEB2tLwtAQVsR0vbEtET2I6WtSWggO1oSVsCCtiOlrMloIDtaClb AgrYjpaxJaCA7WgJWwIK2I6WryWggO1o2VoCCtiOlqsloIDtaJla4gkPbEfL0xJQwHa0LC0BBWxH y9ESUMB2tAwtAQVsR8vPElDAdrTsLAEFbEfLzRJQwHa0zMxD1Uuon6eRMVpflsICZqGFZSksoBZa UZbCAm6hpWQpLCAXWkPmsewMa55DR4vHUlhT2xNIv8Ga2p7A+g3W1PYE2m+wprYn8H6DNXq2QSvD PJZZYk05dIOWhKWwGGAReL/B4oBF4P0GSwAWgfcbLAlYBN5vsKZ+JPBeL7H4qR8JvN9gTf1I4P0G a+pHAu83WNCPhArnFgv6kVDj3GJBPxKqnI1aYk05dEOoc26xoB8Jdc4tFvQjoc65xZr6kcD7DdbU jwTeb7CmfiTwXi6x5KkfCbzfYEE/EuqcWyzoR0Kdc4sF/Uioc26xoB8Jdc4tFvQjoc7ZiCXWlEM3 lDrnBmvqRwLvN1hTPxJ4v8Ga+pHA+w3W1I8E3m+woB8pdU6+xJrmCoZS59xgQT9S6pwbLOhHSp1z gwX9SKlzbrCgHyl1zknzMUD94tT2BN5vsKa2J/B+gzW1PYH3G6yp7Qm832BB21PqnBss8CFKnbNa Yp3mCpQ65wYL+pFS59xgQT9S6pwbLOhHSp1zgzX1I4H3G6ypH/G8b90S6zRXINQ5t1hTP+J5v8WC fiRUOrdY0I+EWucWC/qRUO3cYkE/Euqdbb/EOs0V0HuVpLCgHwnV0y3W1I943m+xpn4k8H6DNfUj gfcbrKkfCbzvllinuQKhIrvFgn4k1GS3WNCPhKLsFgv6kVCV3WJBPxLKslss6EdCXbZtl1inuQKh MLvFmvqRwPsN1tSPBN5vsKZ+JPB+gwX9SCjObrGgHwnV2baZYcG6BUMozbZTvr07r6QwhLpsa2c4 p+shcH3KNz69dsf5JRGorpdQp6siMH3Kuhzfn77N9E2GUJXdYoHXEMqyWyzwGkJddosFXkMozLZy gRVavm9/5D9+7du/upeje//rW/Py1P3tcGien4/vTeceHv6X/+fp5cuhe/32rXnpD1/d+4t7Prw9 9YfqcPSv/lrCAPPYT38s/D1Wa+nngQ8f3c+uf/z47enY/fyP/3Rojof5J3+pvqu++e/De/PNza15 XTs/2Hn49rQwPf3o7ZgRO3bhrz479/bL8s/OPwr2ldyx1yf749M31//WPB0XEKdPPYpsd0Ca6uEx /v5oOr73Brzt0xaD9H/2t+btzb3DXxt/8DZG7Zg0Ywsdm/fj59u5icafvaHgO/c4DNVDe3Tf4cbi 23Bx4sbu9gPhc3e7irei1n1F6m5XMaetNr3DdfdkF2hC6e6TvW/3b/G3Hz/enl4eP1polNWHAUP2 Oxhd9fAlrp37dJ/u8fjzu2v6x+fXV+iU9HcXr8p30PD6/vXRfZ/Yd/rR21meNhsEmHnf/fb2+vz0 4mbG5w89hNtH+OGHHw7+d98O1X87vD+9Hf4lvvmY3rTxzcH/1q1s4Vu2dHWVwZaOyJYuky1dAbZ0 xdnS0djSZbOlu8aW7zY8Gk6c+c65GR8WCPKIBHmYo5PHMBp5DMsjj7fPJo/HKEwej0ghj2G55BkR iocauWVL23QZbNFEtuhMtugCbNHF2aJpbNHZbNF3YYtKsEU0GWxpiWxpM9nSFmBLW5wtLY0tbTZb 2ruwRW/Z0nSWzhZb0dhiqzy2ePtstniMwmzxiBS22CqXLSPCmS3f9VBVvBvsiTbfe9U7adpGV41t hGnOA5ymYc1Qsa4aZGeH+VjHNoOpjdXM9MZZLTFMMxumGZUzYLaKyDSVyTRVgGmqONMUjWkqm2nq LnHJJtiSM0K2DZEtTSZbmgJsaYqzpaGxpclmS3MXttRbtsicEbIdiGwZMtkyFGDLUJwtA40tQzZb hnuwRVcJtuSMkGtJY0st89ji7bPZ4jEKs8UjUthSy1y2jAizMQ9zXEhj2HnMU3d9U6mm6XtpmoF3 5zGPEVyyTqhadX4IrNh5zCMdZ4ZXvbBtXRvjMEzbppSNyBld1zWRaXUm0+oCTKuLM62mMa3OZlp9 l7i0TSkboXQGWxyRLS6TLa4AW1xxtjgaW1w2W9x/QUpZb1PKhoduoJKnETTyNCKPPN4+mzweozB5 PCKFPI3IJc+IUDzUbFPKhpuMWmdjiWyxmWyxBdhii7PF0this9li78KWTUq5qUTOgNmPv2hs6TPZ 0hdgS1+cLT2NLX02W/q7sGWTUm4qnjPobTmNLS3PY0vL89niMQqzxSNS2NLyXLaMCMXZskkLe7bk DHpbouqmzVTdtAVUN21x1U1LU9202aqb9qrqZjXoxXBmkxxuKpYz1m2J2ps2U3vTFtDetMW1Ny1N e9Nma2/alfbmO78+VWLoidImV+zJkzP07Yjamy5Te9MV0N50xbU3HU1702Vrb7qr2ht6wDGbjLHn TJVRu+yICpwuU4HTFVDgdMUVOB1NgdNlK3C6lQLnu6haJ0aSXIk7gKJ5Hfbhhwj0P/7H+Oer9R87 /dpD3z5GDfrjh3seHn97n4mjk1/9Jcir13C8a/xlCt8NX73Ru4sy9xFk9kEw7XZM++qhG+Xw0GfT T6G1921mTdWds+oD/GdZxWplZo22AJDNtv3Gb5QY2y9eu6dL+/vRfZwa5fyRv7hm07YniGD+xR0f f22eP93J+vSJNxZsx9hE47bpvsYeOBmfPgnk7esd62HWrSPhVx06fhguYBMNRgjNqodH/+sfR/8H wch3yAiT+MJDDRvnBKg2Xg3Es9OFwM/hGqodSzO2wkfn+/XoXc91J+vZZ6Elmh2KeALMEDwVZ8Tc fu6R5MaxAUhNTfo2b8rATrfTh9aN3nC2+Xo24moTTkerxtPmbDE5QLfT063Cu02wmUeYhNtUbe0n V3zHbTq+5zb+m+g24UZXIWBqYLHTwN704de4Lma0Gd+HvtU7LdX7+5gZTL8v937d0/Dx8dvxu4/9 nrnxsTI8N1/ArZNfhT/Pdzyk9072W/PVfb49Ni+/jyDnn1MBfrRznpLnx5F7Gf8Nf9iBc+x9G65G 7bhKeIpu7cIimtfP4/mZevFXwjWbHfi6enh3H+54fO2hU08/ph+jYOdDdBjA+JDZvr7CRcw+CJ7Q 7HTYMPpu/Duz0Ot/8lbbUQwYCWqwGYbMYMOqukywYVWPDTaMaUKwYbxFBBtGeEaz1TMaHWzY7jM6 fEMMNsEUF2y832GCjYcpGmyYNqRgw3R9h2DDdHfPYMMMIwUbZhQ52DBT44MNM9SRDbO5IxtWFxrZ +PExOtjUlJENazEjG9bhRzbRJivY9Hsjm/ANNdj02JENc6iRDXOBhtMJjNCH04/xunboi57jjmY5 c9wzwlwJXjPmv1Dn7tK8GriUvGdd7VQjZ6oo6YZ6cG3QPDvr7Lm7/G0qM4iOC9M3oh6765//+Z8P 7v399f0Q5jshrTK8v347NH3v2+bjAFc1TYbtwf9+yQmx4uQJ8U3Tx/XC6D+nj3PLP6ePf04f/5w+ /jl9/HP6+Of08c/p45/Txz+nj39OH/+cPv45fbx9+nhjMT6xqEabDPVPR9zgoMvc4KArsMFBV3yD g462wUGXvcFBt9rg4HunrFwU479DVGCV7pgaertVHp5/paldPBdmUZffICwvZGZ3KxkTa3Z0zqZh PXH/hD5z/4S+wP4JffH9E3ra/gl99v4JfXVHNdFmqU5T6Rz5Yk/cCaHP3AmhL7ATQl98J4SethNC n70TQq/W8kU/Zev9w20TwARnVWdlt1bKhG+qrg3Nuolo/puhCQvr0hHtBLmOaCe7W9m5WRrUhHM4 M9hJ3Hmhz9x5oS+w80JffOeFnrbzQp+980J/deeFjIiWWCCkc/SRPXH/hX7I5EyB/Rf64vsv9LT9 F/rs/Rf61f4L31lR1iQWCqmcgbwj7sPgMvdhcAX2YXDF92FwtH0YXPY+DE7uRJrzINsIJtPjdz3w cP6T0lz3Rkm3N34HhNPfX9vdysHE8iOVM353xB0aXOYODa7ADg2u+A4NjrZDg8veocHdZYcGk1h4 JHP2MHPEHRpc5g4NrsAODa74Dg2OtkODy96hwV3doYHElsRKI5mzh9lA3JJhyNySYSiwJcNQfEuG gbYlw5C9JcNwly0ZbGKNkcjZw2wgbskwZG7JMBTYkmEoviXDQNuSYcjekmG4y5YMdpMEb6omZ+w8 ELdkGDK3ZBgKbMkwFN+SYaBtyTBkb8kw3GVLBrvJUnu2ZIxy64q2JYO3y2JLsM9lS8Aoy5aASGCL N8tkCyAUZ0siP11njHLrirYlg7fLZEv+lgwBozRbSFsyeLNsttzlIBSbyBfXGaPcuqJtxuDtMtmS vxlDwCjNFtJmDN4smy1XD0IhsSWRKbYZo9ya0XZf8HZ5bGH5uy8EjMJsYaTdF7xZLlvYXU4+sYkM sc3YeKxmtH0XvF0mW/L3XQgYpdlC2nfBm2Wz5S4nn9hELtdkbDxWM5owyNtlsiVfGBQwSrOFJAzy ZtlsWQuDZEoRNLAuXVGQuxKg0eT0J+WtPEtkgU3GlmU1p2l+vF0ez3i+5idgFOYZJ2l+vFkuz/gd NT92mwtmPGeUzGmaH2+XyZl8zU/AKM0ZkubHm2VzZq35qbWQtX/obEJUU8uO267aan5U7Rqp2s3u QsHGSDO0dkfzc4Jca35Odjeys97mnhnLGZVzmubH22WyM1/zEzBKs5Ok+fFm2ey8y2kr9Tb3zFjO qJzT1D7eLpMt+WqfgFGaLSS1jzfLZsta7VPXUglxXfMzgXhqeItwGs/FzdBOv3Yr37bZ66B1pPNN 0HRC3i6PbyJfJxQwCvNNkHRC3iyXb2JPJ5QXnbbZa1bljM4FTdHj7TLZkq/oCRil2UJS9HizbLbc RdFTJ7LXQ4YWvxY0RY+3y2RLvqInYJRmC0nR482y2XIXRU+dyF4PGdr4WtIUPd4ujy0yX9ETMAqz RZIUPd4sly3yLoqeOpG9HjJU8bWkKXq8XSZb8hU9AaM0W0iKHm+WzZa7KHrqbfaa1TmjXElT9MTD DLPYkq/oCRil2UJS9HizbLbcRdFTb3PQrM4Z5SqiokdlKnpUAUWPKq7oUTRFj8pW9Ki7KHrqRPbZ 5oxyFVHRozIVPaqAokcVV/QomqJHZSt61D0UPf7/E2zJGeUqoqJHZSp6VAFFjyqu6FE0RY/KVvQo /PEqGNYkssI2Z7Sricoenans0QWUPbq4skfTlD06W9mj76HsYVUipysy9Ou1Jip7dKayRxdQ9uji yh5NU/bobGWPvoeyh1WJnK7I0a9rorJHZyp7dAFljy6u7NE0ZY/OVvbo9i5s2eZ0Gc/RrxuiPsdk 6nNMAX2OKa7PMTR9jsnW55ir+hwSW7Y5Xd7njHYNUZljMpU5poAyxxRX5hiaMsdkK3OMugtbtjld 3ueMcg1RKWMylTKmgFLGFFfKGJpSxmQrZcyeUmb8JVsx4bjTSYXpjr50MriVWYn8b5czIjZEVY0Z MplVQFVjiqtqDE1VY7JVNWa4Focy5tyJLHCXMy62RGWMzVTG2ALKGFtcGWNpyhibrYyx91DGsCqR BW5zxsWWqIyxmcoYW0AZY4srYyxNGWOzlTH2qjKGHmFYIhfc5ujWLVEfYzP1MbaAPsYW18dYmj7G Zutj7EofExhSCzbfbJdX/jMuxr1JOsZmbOp7P74Zxq9mezqLuNBhmMwwNEskj5scAXpNFNbUmcKa uoCwpi4urKlpwpo6W1hT30NYw1giedzkCNBrorCmzhTW1AWENXVxYU1NE9bU2cKaeiWs+c5iRJGz 9X7WGh9e5Bh6rJjFpDbEnWqMQW1zjko9G1G6Zvwew7Nt2tn/4YzhdU2U5NSZkpy6gCSnLi7JqWmS nDpbklP364dfz+YrtkR4gskwZLLDuJQLSDYyiXfjv6w+s4yNjGx1JBnDkGybrRZVzqi8ISp5mkwl T1NAydMUV/I0NCVPk63kafiaZNzOSWZVfw5YZ4qNPxtlw1dsNrgaV5HWbPxXYxi2zXCLKmcM3xDV P02m+qcpoP5piqt/Gpr6p8lW/zRmzbBWhcg0C2SRTIMZyTRfesrcSCQ/mRPLUfz4ueomcmKIlkiO Dzmj+IYoHGoyhUNNAeFQU1w41NCEQ022cKi5uhVQckHg+T+HIdE2D+4jacbgviXqiNpMHVFbQEfU FtcRtTQdUZutI2rvoiNi2wy4H0JkqOZboo6ozdQRtQV0RG1xHVFL0xG12TqidqUj+t6rEGEknw2g ZOM/YeMzTqjZs23cOiostYlPsPkoPY7fFYzfLYZo2+Q5dzmC+5YoQWozJUhtAQlSW1yC1NIkSG22 BKldby5khzDz03Oe1WeezeeCnhbS+u/UnGLWNdVpWI/gF98m2nmfI9HviKKlLlO01BUQLXXFRUsd TbTUZYuWupVo6buoTnlzIFjIpk8prW5Y5BvqcwqejSkKoBlX52+4sJhHJt/m2kXWFowdUfDUZQqe ugKCp6644KmjCZ66bMFTt96KqKus05zP5oMsloGdU45z3yB2nj7lkFmQc5o5NeazwjeuwtFsm6QX WXs3dkSlVJeplOoKKKW64kqpjqaU6rKVUl2zyTqE/AKbPTB5SB7E5yO32qwHZi0Mv1o+my+KycJ/ 7hgmf8oTSfqsXR87om6qGzJ5VkA31RXXTXU03VSXrZvq1rsRtVUzSLVIocoqZrZEINo8hTo+SgXb FKgheYpO0vNEkj5ry8eeKLTqM4VWfQGhVV9caNXThFZ9ttCql8mKY93NSNb1U/rUc2p2uLUdmvO4 bVDrWaY2hAwqT6Tqc46GrXuiRKvPlGj1BSRafXGJVk+TaPXZEq2+Xs8Cxix7v87VN2Z6EJ6JpsKP baVlHPTPakKW8/Njs+owPNtm6kXOIa+eZUSeZcq6+gKyrr64rKunybr6bFlXf3XbI7oUkG8T8yLn kNfaETVaLlOj5QpotFxxjZajabRctkbLiXWqFWp/s3lj5biIn0o+j0xx5FU5UYXAJOeHVvOqb5om 6ig47gm4TegLlrMMwhG1XS5T2+UKaLtccW2Xo2m7XLa2y91j0yTGt1l5kZWVd0SFlstUaLkCCi1X XKHlaAotl63Qclc3TaI/x8Q20y6yMu0DUXA1ZAquhgKCq6G44GqgCa6GbMHVcHXrpIIqBZFIomet Gh6ImqohU1M1FNBUDcU1VQNNUzVka6qGjaaqk1rLuagq/nY8gtwMdfrIEQBUddfWXVXtrRCdENYX crK7lYyJVHvWouSBqLsaMnVXQwHd1VBcdzXQdFdDtu5qWG/Y1NleikZvydiGg2y4SRwu0bLeW2i+ PVyirfuqDSqJNDtPkMsrm9ndys5Egp58VK237ipC7nS0o7MT7LPYCRgF2QmIWHaOZjnsPCPMj2US xijVIs4LiBa8v3ZewPRra5KOVyLloLcnoGjmJ6b8Voom0vvk83Fj2xDSrqNdJkUz066AUZqi+LTr aJZN0b2VsedHsLZswOzqMBncyqxEQp98lm68JUKidbTLZFZmohUwSjMLn2gdzbKZhU20TsZS9eGs nPVTd/03Tr92K88SCX3yKbzhBhkhOTva5fGMZSZnAaMwzxg+OTua5fKMXV1Am5EI2Sb0Q7I2gzOE VOtol8mZzFQrYJTmDD7VOpplc+ZqqjWDM9v0vCSn5+PFEhKuo10mZzITroBRmjP4hOtols2Z9ZLY vrd6GBJTTd9htq+rbjvVHL9ZyFSnb4RTjfTBZWdQNUGuR/Enu1vZuS0HSPKC7dAsnJDaHe3y2Mkz U7uAUZidHJ/aHc1y2cnvsSs+k9tCgCSvvI6XScjhjnaZbMnM4QJGabbgc7ijWTZb7rIrvtxm/CV5 FXW8TEKSdbTLZEtmkhUwSrMFn2QdzbLZcnVxK4ktiZQ8eSl0uExBWMU62uWxRWSuYgWMwmwR+FWs o1kuW8RdVrHKRIqcvOY5XiZhFetol8mWzFWsgFGaLfhVrKNZNlvushu+TGSryWue42USlqKOdpls yVyKChil2YJfijqaZbPlLrvhy20GWqqcUa4kLCwd7fLYIjMXlgJGYbZI/MLS0SyXLfIuu+HLbR5Z qpxRriQsDh3tMtmSuTgUMEqzBb84dDTLZstddsOXiQyyzBnlSsIaz9Euky2ZazwBozRb8Gs8R7Ns tuzthp/HlkTuWOaMcuXQdyS2xBWeGWwZHK8eHuNvj4bje2/Be71j4a/0AjX2mNGxHTgSMbIWZZ4R ihMjkbYVOQNaJWnEUDKPGEpiiaFkUWIomhhIZYuB1F12rFeJDK2gyvvjZdZEYtSZxKjRxKjLEoMm wVHZEhx1dXN6EjESyVhO1fDHy3REYrhMYjg0MVxZYtAUNCpbQaOuKmhIxNjmXSWn6vLDZWqiDkZn 6mB0AR2MFkSq7F8ThSs6WwWj77KNvNrm0WSXM+zQPS2I6D4viOgeG0R0XzSIaJpsQWfLFvQd14mp ROKsyxl8GE6jh+F59DAcSw/Di9LD0HQDJls3YNZ7cPesb2Ybr/E2LJOp/L/tmTC9kCZ+KuRsU48+ KAPCp5XFrBNTiQxJm/M8Mh2RRF0miTo0ibqyJKIViE12gdjcpUCsEnNenSOks5pGDKvziGE1lhhW FyWGpVX3bHZ1z272qLVstg2aE33cgtb/q8/RxQ1djDn+X3mOLkYMLMSX8C9mXyqdmCDrHMGbbYks ajNZ1KJZ1JZlEa3qZ7Orfna9AW3PjJw/o0JqOzyjmu76M6qS4zOq4hgSbSfTSuZMpuuKRqK6yiNR XWFJVFdFSVTTioF1djGwXu8yOwy9m+/f0piwVU/4d3ZmEqtco+LHrplvxtjUYUepamhQQx29nXor mTPUqRWRRiqTRgpNI1WWRrQqYZ1dJazXW8j+18eirRJKyZw1KXVDJFGTSaIGTaKmLIloxcM6u3hY rzeIFdKJYbbRnXQ2ECb8q64GIyE7yeJvd2LAECmR3Blyxtc1sdhYZxYba3SxsS5bbKxpxcY6u9hY r3eA5YPVs435/Y91iA3h39nppOuPpx3JhlCcCx/LHkWkRBpoyBliN8TiZJNZnGzQxcmmbHGyoRUn m+ziZLPeqeC//rGWkFW5HBFeQyxkNpmFzAZdyGzKFjIbWiGzyS5kNncpZOpEftDl6O0aYiGzySxk NuhCZlO2kNnQCplNdiGzuUshUyfEUn2OtK4VNGK0Io8YrcASo6WWLdPEaGlVyza7atmuN1adPXPi r8UtwHgr05uIyd1dw0aT05+Ut1Jqm3JWNmdI3FoipWwmpSyaUrYspWhL+9vspf3t3tL+GTuGGreh DRjcSCKzzTgrmzMcbolF8zazaN6ii+Zt2aJ5Syuat9lF83a91t+0fTubV5k2Docr//Ewq4uadoi7 CoZ/Z4dqmLZrxo+7pkU84Ewi62xyhsQdsbzeZZbXO3R5vStbXu9o5fUuu7zercvr/snk1IxIPBSr PTX8v82MSL5Zhvhxr/szkfTQizYcvef/VRgiJfLOJmcI3RkikUwmkQyaSKYskWgr9rvsFfvdetdV zft6RiTtXSwwxv9b9Wci+R87NX7czubo3ljp+HGvJIZIidyzzhlyd0StRpep1ejQWo2urFajo2k1 umytRrfeMVUZLmeFMP+jDlmd8O8s9+x/jDIx/68ZzkTyYWQY4sdswJyuYra5Z6VzFrr0jEaknuUR qWdYIvWsKJF62jr/Pnudf8/WczffxLPTeiSzMhzYE/7lZyL5H7UaP9YzeYc35jp+bDjq0bbNPSuV o1DtiSKhPlMk1KNFQn1ZkVBPEwn12SKhfi0S8mOcena0iv8xPsPCv7PDxfyPAx8/HmZFDNH4yVr4 2P+LGmxv889K5WhZe6JOqM/UCfVonVBfVifU03RCfbZOqF/rhHhnzLwa1pko2wj/zuTP/scWPm7n 5/N0uu7jxxpHpG2+WmftguSIWiGXqRVyaK2QK6sVcjStkMvWCrm1VojbYX7Yk/8xBKTK/yuHGZHs oNn4sZ4Tybp2iB+7FqMWMtv8tsraIMkR1UIuUy3k0GohV1Yt5GhqIZetFnJrtRBrjatnqrN2HCOF f2djJP9j044fN82ZSN5Y1PFjIyyGSImsdtbeSY6oGHKZiiGHVgy5soohR1MMuWzFkLvLdgM2kanO 2ibJERVALlMB5NAKIFdWAeRoCiCXrQByV7cboC/vson8M/nouHCxA1HXM2Tqega0rmcoq+sZaLqe IVvXM1zddCCDHomsMvmUuHixRMXOkKnYGdCKnaGsYmegKXaGbMXOsD502Vb1fIGOH36MJ8TXgs1U 8daHhngWM1Ph3/MIZTw1PnwjwynNmESOTaSWycfFxXsjynyGTJnPgJb5DGVlPgNN5jNky3yGPZnP XJgxIEvvo8GtJEqklcnHvHnrvqJJgoJdDom8PZJEwaIgiQIcgUTeLJNEgHCnJ9Y2WaxzdlHqK5q8 J9jl0QMr7wkWRelBkvd4s2x62HUd1AjVNtxugs0ghDCqGbanMHTS6CocWTlb0AU22jT1UOudcHSC XF7ZzO5WIm6TzTpn16a+okmEgl0eEbESoWBRlIgkiZA3yybiWiIkikaqRBI5Z/emntGkP8EuiyAM K/0JFiUJwkjSH2+WSxCGPWwZQ49taljn7OHUM5qgJ9jl0QMr6AkWRelBEvR4s2x6XD2CI4Me24Sv zjm2rGc0mU6wy6MHVqYTLIrSgyTT8WbZ9LjLlir1NuGrc84m6zlNdhPssojBsbKbYFGSGJwku/Fm ucTgdzleo96menXOsWA9p8logl0eMbAymmBRlBgkGY03yybGXU7SqLdJXp1zAljYUIxGjDxZjLdH E6OoLCbAkYiRK4sBhOLE2OZrdc62bb2gyVyCXRYxBFbmEixKEkOQZC7eLJcY4i7nY9TbHKxucwaf giZbCXZ5xMDKVoJFUWKQZCveLJsYV4/CoM9N6kQONmdXv17QxCjBLo8eWDFKsChKD5IYxZtl02Oz fY3RQisxEzb5TzRTQ1U1DRdWz7RNmmsdf1uIWtioqpuUu/5jwYXx/1eHdxhSJfKpOZv89YImZAl2 eaTCClmCRVFSkYQs3iybVOutbIrmU+tEPjXniK9e0qQswS6LIBIrZQkWJQkiSVIWb5ZLEHmX8zPq RCY15zSvXtJELMEujxhYEUuwKEoMkojFm2UT4y7bztSJHGrOwV29pOlRgl0eMbB6lGBRlBgkPYo3 yybGPbad4VUih5pzRleviBoTlakxUWiNiSqrMVE0jYnK1pio+2lMeJXIpOZshtcrosZEZWpMFFpj ospqTBRNY6KyNSZqbwuZEvRI5FNztrnrFVH5oTKVHwqt/FBllR+KpvxQ2coPdb8TVXiVyKrmbHbX a6LuQ2fqPjRa96HL6j40Tfehs3UferPli+9WI81s/aBWWuraxwJR1VUz32bB+E+1jt91sAHDtGVH +LT3BLGCC8ZRpEpkZHM2yus1US2iM9UiGq0W0WXVIpqmFtHZahG93v7F00RpO9+jVTP/SeepJowI 72akErrWMnwTiKXNjFS9/79BBxoGIiJWp/Iqkcftc4bBmqgx0ZkaE43WmOiyGhNN05jobI2JvofG xBNsS4ycM+N6Q9SYmEyNiUFrTExZjYmhaUxMtsbE3ENj4uNPghg5omdD1JiYTI2JQWtMTFmNiaFp TEy2xsRc1ZhkDH0Tmdg2R/JsiEoTk6k0MWiliSmrNDE0pYnJVpqYeyhNeLXNxJqsTKwlKk1sptLE opUmtqzSxNKUJjZbaWLXG6r4+U+vhZqdeRJ+0q0Og1etBjPbVVV3RvmIOxgd5k18tvvc+fc9njOI 5aacbbO4JiuLa4kqFZupUrFolYotq1KxNJWKzVap2KsqFUq0Ydv8rclaI2iJ+hSbqU+xaH2KLatP sTR9is3Wp9i1PqXh3PSGXyAI/IqfJvvpMNeI028422Z0TdZaPkvUnthM7YlFa09sWe2JpWlPbLb2 xF7dRIUUSba5XJO1hq8mak7qTM1Jjdac1GU1JzVNc1Jna07q9bFIDa8qxtlc6db4WaxmnFcibEc5 2w1Ocf+fDUSouaxmB7YZ6QcyfiplZBXmU6hhyzaVa7IW/tVEuUqdKVep0XKVuqxcpabJVepsuUp9 Va5Cn0OzbVLWZC38q4milTpTtFKjRSt1WdFKTROt1NmilfouohWWSMrmnPXYN0TRSpMpWmnQopWm rGiloYlWmmzRSnNH0QrbpmaNzRnDNkTRSpMpWmnQopWmrGiloYlWmmzRSnNH0QrbpmaNyRnJNkTR SpMpWmnQopWmrGiloYlWmmzRSnNVtEIiRiI1a3KGoy1RrtJmylVatFylLStXaWlylTZbrtLubVMy /lLcg61SVfoAPlbBf7vbt42mpz98MriRYDyRpjU5A9qWKF1pM6UrLVq60paVrrQ06UqbLV1pr250 Qok8PJGmzVpG2BLlJ22m/KRFy0/asvKTliY/abPlJ+1d5Cc8kY7NWgrYEeUnXab8pEPLT7qy8pOO Jj/psuUn3fpkIV53rGNynnWruA1n2AvuW0+fH03fWTiBgVVMNRVjirnzs6pjNWOmciGZyadH1o2c SmRys1YPdkTlSpepXOnQypWurHKloylXumzlSndH5QpPJGWz1hB2ROVKl6lc6dDKla6scqWjKVe6 bOVKtz46KChoNdezvbB5pWsttDX9KEQ4EyeKZ1XHKyGCLHemT4gQQcftTQYtMJzaZnIjP8mc6omi lz5T9NKjRS99WdFLTxO99Nmil35zipCfDvX6XIb+zqStZJwO8Wqo2jmjwr7qrQnfin5+ljCDX+4l wz3Ctglg0+aMl3ui3qXP1Lv0aL1LX1bv0tP0Ln223qVfHyYk4mrUqjkTyipeV0NXKVMN8DSbtj6u Kxm/tPHfM6GkUh6m5bpqnbMCxalE1rjNGWr3RKlMnymV6dFSmb6sVKanSWX6bKlMv5bKsCA6l9Vs O21mhfWftsJ/GvYWnY214477/ssYk+ZpIcZdCFTMuvgthlOJVHOTM9TuiWqaPlNN06PVNH1ZNU1P U9P02WqaHnsk0SldKPykTfLeLrOJy78x+7VbKZVIUjc5w3NH1OG4TB2OQ+twXFkdjqPpcFy2Dsfd 7xgjLhIp5jpHXu6IkhqXKalxaEmNKyupcTRJjcuW1DispOa7H7uc6xY38iSRca5zFOOOqK1xmdoa h9bWuLLaGkfT1rhsbY27qq3JCCOJvLPN0Y0PRIXNkKmwGdAKm6GswmagKWyGbIXNcFVhQyLGNnls Rc6IdiBqa4ZMbc2A1tYMZbU1A01bM2Rra4Y7amvENnlsRc7odCBqa4ZMbc2A1tYMZbU1A01bM2Rr a4Y7bggjEnlgnjE69S8kegS7HHr4FyQ9gkVBegQ4Aj38SyY9AGFRW3BOj3y4ZWLsKqmVZM1GZrP+ Y6dfu5Vb25yw5RkjWlfRxDXBLo9bWHFNsCjKLZK4xptlc+t+pwhxsU3vWpYxonUVTWIT7PLogZXY BIui9CBJbLxZNj3uIrER2xytZRlqccdoEptgl0UMhpXYBIuSxGAkiY03yyUGu7rDS0bc2OZbbZWh FneMppYJdnn0wKplgkVRepDUMt4smx53VMvIbb7VVhmaccdoaplgl0cPrFomWBSlB0kt482y6bFW yyjBrO/FjXZ8aExt+3ZRiB6/aaq6Nf7LbRVosC0bBv/Nzhm8E+S6LHSyu5WH23yubXNGv5ymsAl2 WTzkWIVNsCjJQ05S2HizXB7ytcLGcancIN2ZiEa5umlM6+KGpHMi+m+0rUVtek+boe/PRLSGd00n HVPcsxQl25LbLLBtcsbMnCazCXZ5pMLKbIJFUVKRZDbeLJtUd9lWRiaywE3OmJnTtDLBLo8YWK1M sChKDJJWxptlE2OtlWmbqmsFuzmNU1XBwqr6chrn9Gu3UiuRQa5zxtucJpkJdnnUwkpmgkVRapEk M94sm1pYyQwm8iQyyHXOeFvQ5C/BLoseAit/CRYl6SFI8hdvlksPcUf5i0wkgesMobkTNPlLsMuj B1b+EiyK0oMkf/Fm2fS4444yMpEElhm6cSdoqpdgl0cPrOolWBSlB0n14s2y6XGXHWVkIgksM8Tf TtL0LsEuixgSq3cJFiWJIUl6F2+WSwx5F72LTKR/Xc5wVNL0LsEujxhYvUuwKEoMkt7Fm2UT46re hUIMlUj8upyBqKQpXYJdHjGwSpdgUZQYJKWLN8smxl12kVGJTKzLGYIqosZFZWpcFFrjospqXBRN 46KyNS5qbxeZPGIksql9zuBTEQUqKlOgotACFVVWoKJoAhWVLVBRd9n9RSWyqX3O4FMRpSkqU5qi 0NIUVVaaomjSFJUtTVF3kaaoRC60yxBbO02UpuhMaYpGS1N0WWmKpklTdLY0Rd/l8CGVyIJ2GTJr p4miFJ0pStFoUYouK0rRNFGKzhal6KuiFBIxEvnPNkdgrYlyFJ0pR9FoOYouK0fRNDmKzpaj6Hsc OyQS59i1kp7HkJUjlfKjXQYxgj2OGNGiHDEiHJ4YwSyPGBNCaWIkjg5qBX3wKauBVDGJdlnEGJAV k2hRkhgDpWISzHKJMVytmJCIsc1jtJw++PSXSaqVRLs8YiBrJdGiKDEotZJglk2Me9RKROKIoJbT B5+SVaRaSbTLIYa3RxIjWBQkRoAjEMObZRIDEIoTY5PHaKuWnvn0l0mqlUS7PGIgayXRoigxKLWS YJZNjHvUSkTinJ/W0jOf/jJJtZJol0cMZK0kWhQlBqVWEsyyiXGPWolInPDT2ozBJ2OkWkm0yyIG Q9ZKokVJYjBKrSSY5RKD3aNWIhIn/LQmY/DJGKlWEu3yiIGslUSLosSg1EqCWTYxVrWS73bgmqME xt5CDdcFxvBrt1JrqwFrTc7wlZGqLdEuj1rIaku0KEotSrUlmGVT62q1hSwRFInTgVpNz6BKxkk1 l2iXRQ+OrLlEi5L04JSaSzDLpQe/33JgkTgjqNX0dXb+YkmVl2iXRw9k5SVaFKUHpfISzLLpcb/l wCJxwk+r6Cvm/MWS6i/RLo8eyPpLtChKD0r9JZhl0+Nq/SWDHpucalu5nGGtIK3SjXZZ9BDIVbrR oiQ9BGWVbjDLpYeo7kiPTWbV0yNnaCpoRbpgl0cPbJEuWBSlB6lI582y6bFXpBt/KZ4iZ4ROHkC3 e+zcaHAriRJZ2D5nACtIa3OjXR6JkGtzo0VRElHW5gazbBKt1+byolFmk431BMkZwgrSCttol0cQ 5ArbaFGUIJQVtsEsmyD3W2Ertmf1tFWXM4SVpBW20S6LHhK5wjZalKSHpKywDWa59JD3W2Ertkfv eHrQt37wF0vTiwS7PHpg9SLBoig9SHoRb5ZNj/utsBXbU3Taqs3QmTFJU40Euzx6YFUjwaIoPUiq EW+WTY/77SsvtgfieHrQV81JpojaEZWpHVFo7Ygqqx1RNO2IytaOqKvakQx6bLKrLcvYw9dfLFFB ojIVJAqtIFFlFSSKpiBR2QoSdb/d5cX2cJuWZezh6y+WqCNRmToShdaRqLI6EkXTkahsHYm63+7y YnumjadHztBUE9UkOlNNotFqEl1WTaJpahKdrSbR693lmWyZYBjVQLDg7qpqYPq1W7m1Tc2yjG2A /Z0SBSk6U5Ci0YIUXVaQommCFJ0tSNFrQYrcJme5bFQ6Oftd7qRnJ5PTn5S3UmqbqGUZWwf7GyQK UXSmEEWjhSi6rBBF04QoOluIou8oRNkepdOyjDOzJTNEIYrJFKIYtBDFlBWiGJoQxWQLUcwdhSjb o3TaasjRVBuiEMVkClEMWohiygpRDE2IYrKFKOaOQpTtaTieHjnKakMUophMIYpBC1FMWSGKoQlR TLYQxdxlIfD2HJz/n7136XXkSKI09/oVWnZXzyJe/lo1MI9ezWKAwawT/iwISmUKmVdVqv714x4R fFy6R5JmxylxBleFKtUleZwM8oSHhfln5m4USCCrmQiKBhEUTUZQdF8ERfMQFA0jKPq2Ubyyxg+O QlZnhQvhLlm9v+xRa9XJ3VEgAa1m4isaxFc0GV/RffEVzcNXNIyv6Ls15sAlqZHcFUhAq5lgigbB FE0GU3RfMEXzwBQNgyn6FkxRwShJqukoivl+Tcf+sgetVe/C40agr28+UCbSokGkRZORFt0XadE8 pEXDSIt+ItJSb46T7YEEw4aJtBgQaTFkpMX0RVoMD2kxMNJilvvJu0EpI2jJu5OEnLyrt8ZxI7AP dj5AJgZjQAzGkDEY0xeDMTwMxsAYjHkiBlNvkFOWMQCS2zAxGANiMIaMwZi+GIzhYTAGxmDMEzGY epMbNwL7YC+jZWIwFsRgLBmDsX0xGMvDYCyMwdgnYjD1JjduNEg4a5kYjAUxGEvGYGxfDMbyMBgL YzD2KY1U6u1tsjGQQNYyARgLAjCWDMDYvgCM5QEwFgZg7FMaqdQb27hRI+GoY6IvDkRfHBl9cX3R F8dDXxyMvri7jVSAC0ojM6uRcNQx6RUH0iuOTK+4vvSK49ErDqZX3N3W84A9GplZhYSjjkmiOJBE cWQSxfUlURyPRHEwieL8bWZWOW1pa0JZoR5YE9pe9qC16m1ysrUQotszKRYPUiyeTLH4vhSL51Es HqZY/DNa2M/1NjlulAjL7Zn8igf5FU/mV3xffsXz+BUP8yv+ifxKvVlOtgfCcnsmv+JBfsWT+RXf l1/xPH7Fw/yKfwq/Um+W48aAhLKBya8EkF8JZH4l9OVXAo9fCTC/Eu62UGEZo5FT9UgQG5j0SQDp k0CmT0Jf+iTw6JMA0yfhKTsc1JvlZGMgIWhgcicB5E4CmTsJfbmTwONOAsydBPsUYzSyqQ4JQQOT GgkgNRLI1EjoS40EHjUSYGok3KVGWMZoZFMdEnxGJi8SQV4kknmR2JcXiTxeJMK8SLzbAoVhjKXR G2dOQIwxz7yFuaJDjJH1RGMURUdjlOEYxsgy0Bj7CFeJshDL5TzEC090euTqme+Hz7j2MxRT1Zch PQCrvfPMW9QrOsxU1EW9ouhqKtaiXpbBpgqvZqpGqwQVkZlK8PKuRQeZSlDzrkXR01SClXfNMtRU Ynw5U9U5WxWBO6kcgTNNheVss55sqq452zIcy1RoznYfoXdc1OiOoAJwJzULXra26DBjULO1RdHV GKxsbZbBxnhGtnZp9DhQAbiTmiUvW1t0kDEkNVtbFD2NIVnZ2ixDjSGfka1dGt0NPLCDdf6YvGxt 0WHGoGZri6KrMVjZ2iyDjfGMbO3S6GvggR2s88fkZWuLDjMGNVtbFF2NwcrWZhlsjGdka5dGRwPv kOBT8rK1RYcZg5qtLYquxmBla7MMNsYzsrVLo6OBt0jwqXjZ2qKDjKGo2dqi6GkMxcrWZhlqDPWU bG2jH4G3SPCpeDV6RYcZg1qjVxRdjcGq0csy2Bg3NXp/ajtG8UDBOcUmNf3qDdARZVa8Wr2iw2xC rdUriq42YdXqZRlsk2dsdL40ugp4A/QzmTWvSq/oIGNoapVeUfQ0hmZV6WUZagz9jI3Ol0Y/AW+A TiazZi4DanAZUJOXAXXfZUDNWwbU8DKgPqrP215UWgNYZy1pn6Zd8KiJ6pypT0jYqpnLfhpc9tPk ZT/dd9lP85b9NLzsp49q+S6eUCY7nLTZ1yZ41ER1ftUnJMQ1vLq/ooNMZKh1f0XR00SGVfeXZaiJ zG3La2Gjslsse+OmfBW1S7vByT7gsuR/FfGBvU4j3H6Qs+6n4Mpn9vHT9/g5ffp3/ki/x2/bMM2n yg/kquPyNs7DnH+fX7PoW4xf3k6DXD1QpP5AGoafvP38uXwlm+70V/kljjVXX6O/+f7KFzAOoxHq 6qbh3QCLrb/Q7Rkxb1/o+tmzk9x/3uL385dyeSh/OFuVTJ2HKPJ/xrdP/7Kf/4hn9fmRLJ5ra25i tYqd9b+uv8BZfH6knFnBHKjT1c+6nQs3P+j2YPkA9aSwDiHHfHLml39/y2+4i/IPsp+p9RN5qBQO hnLrp9mntfMH2f8un2E4UKrtW/ju8+/6lk/L6M/qq8fKN2EPLJINcDVCtuKVMevH80hLfdJvA4nT V/r79VdZ3BkPfkMdt7Phovn1IppEPauuKpttc1GcTgB/8Es7QT9tiubqtJkbp83gjJvSdHDa+Ono tMnPrKdNOdCbKeD0Bc8HX3CW/vSv3+2XX/bfePv/5beVB99UyMdxJTi9fjl6ebbhp0+/vf2ZLwvZ ud9//+XLp/TZ/nM/rZtPlbefDs6QkE+yf9tf4x+/f7Jf/rMNcvm7OfmvujheX8Til+3f5Y1juL2O vX+2fBpxcKqU9EWte/vlt/j1j7dP393b0dBXLymfWR0Mb4afvsXv8e3ta9h/1POfB9f4TZen6BLH 5CnTff26f4irB8qZYA9+sLSdu+v7XE29+a+sagQzm2jmTjYpgZPNOJg+k804BOpkM46SMdmMkyNM NiPjGj3eXKPJk814eI0uzzAnmyKlTTb5vKNMNnmYrpPNKBVrshmlecJkM0r/zMlmVCNrshmVYE82 ozL0yWZU3Mhm1GhkM5pOkU2Oj8mTjeFENqOjRDajp0c2qwaabMJRZFOe4U42gRrZjJEU2Yyx2DCH 859//Xa+STn9uX6uA/vS739XGXT/ex7h6neSZhzzE+Lyc8lpSNOyTGH0Jgq7XH6usmJpUnRaxCXq qC8/Vz5ModLsp1kFO5vt5/rHP/7xc/z27eu3n8v9TsmupG9ff/vZhpC/m+8/75/qdDOsf86v73lD LCb2DfFDt4/L0ft+3D5+3D5+3D5+3D5+3D5+3D5+3D5+3D5+3D5+3D6+G+Hj9vHj9vHj9vHe7eOD a/KN0paI8ISG12y16LA1eWqz1aLouibParaaZfCaPLXZ6nlK2P+hVNU2ur37iGCGhtd9tegwv1C7 rxZFV7+wuq9mGeyXu/sAszDDRsVLRDBDy6zht2ANvyXX8Nu+NfyWV8Nv4Rp+e3t1mpYkovG6hYrN bmzDPWMcRpfW3Z+P0Z6T/vwx3qse9VyjmCYgzUgss8TfgiX+llzib/uW+Fteib+FS/ztTYn/VeTa oVqi0UreB2ALitkyS/0tWOpvyaX+tm+pv+WV+lu41N8+pdS/0UTee2ALitkxS/0dWOrvyKX+rm+p v+OV+ju41N/dLfXnzxuNRvBhRqJcxyz4d2DBvyMX/Lu+Bf+OV/Dv4IJ/97zNgZdGO/gwI7GuY5b9 O7Ds35HL/l3fsn/HK/t3cNm/u1v2D9ijLpYJExKWOmbxvwOL/x25+N/1Lf53vOJ/Bxf/u6cU/zfa wYcJCUc9s/jfg8X/nlz87/sW/3te8b+Hi//93eJ/YN6oc7FhRIJSz2wB4MEWAJ7cAsD3bQHgeS0A PNwCwD9vm96l0Ro+jEgXKs8s/fdg6b8nl/77vqX/nlf67+HSf/+8bXqXRoP4MCC9qAKzAUAAGwAE cgOA0LcBQOA1AAhwA4DwvG16l0ab+DAgHakCsw1AANsABHIbgNC3DUDgtQEIcBuAcHeb3uZCnx7m YRR2rLLx79/j6mWPWqrOrgaFhLOB2RQggE0BArkpQOjbFCDwmgIEuClAuLvBLzDj1DnWIJFwNjLL /SNY7h/J5f6xb7l/5JX7R7jcPz5vm99FNnKsEglnI5M8iSB5EsnkSexLnkQeeRJh8iTeJU9Yxmhk VwUSyEYmYhJBxCSSEZPYFzGJPMQkwohJvIuYAPNGI7sqkEA2MUGTBIImiQyapL6gSeKBJgkGTdIt aCImn2Nb3+oiY2O+bR5uu6Do0hslyTzz1dHuKHyY1SAP28rsQ96Gv2fdo0ZsZHMXpCdWYtInCaRP Epk+SX3pk8SjTxJMn6TnbQq8yEY2d0G6XSUme5JA9iSR2ZPUlz1JPPYkwexJumFP/oxKj+m629W9 O+8UpVjS3Tvv88se9VYjFTwD1LYdeAtJRYd4K+uJ3iqKjt4qwzG8lWWgt/YRusfOjSSwBWJnO/CW kIoOMwZ1CakouhqDtYSUZbAx7i4hsYzRSP9aIGq2A2/xqOgwY1AXj4qiqzFYi0dZBhvjKX2jZSOJ a4Ao1o68ZaOig4wxUpeNiqKnMUbWslGWocYYn9I3WjbStwaIX+3IWzAqOswY1AWjouhqDNaCUZbB xri7YMQxhmokbjUSfI68ZZ+iw4xBXfYpiq7GYC37ZBlsjLvLPixjNBK3GqCm7cRb8Ck6yBgTdcGn KHoaY2It+GQZaozp7oIPyxiNlK0GeGk78ZZ6ig4zBnWppyi6GoO11JNlsDGestSjGilUBZDSduIt 9RQdZgzqUk9RdDUGa6kny2BjPKWaWDWSpxEJPmfeIk/RQcaYqYs8RdHTGJxeF6sMNcZtr4tOxmhk PgMSfM68RZeiw4xBXXQpiq7GYC26ZBlsjKfs6q0amc+ABJ8zb7ml6DBjUJdbiqKrMVjLLVkGG+Mp pb6qkfkMSPC58Ep9iw4yxkIt9S2KnsZYWKW+WYYaY3nKrt6qkfn0SPC58Ip8iw4zBrXItyi6GoNV 5JtlsDGesqu3amQ+PcA124VX3lt0mDGo5b1F0dUYrPLeLION8ZRdvXUj8+kAornsbsUzBlbYm/Vk Y3Qt7C3DsYyBFvbuI3Q3RiPz6QCW2QomjyFAHkOQeQzRl8cQPB5DwDyGeAqPoevMZ/6CgRlDMHkM AfIYgsxjiL48huDxGALmMcRTeAxdZz7jiASfgsljCJDHEGQeQ/TlMQSPxxAwjyGewmPoOvMZRyT4 lEweQ4I8hiTzGLIvjyF5PIaEeQz5FB5D15nPOCDBp2TyGBLkMSSZx5B9eQzJ4zEkzGPIp/AYus58 xgEJPiWTx5AgjyHJPIbsy2NIHo8hYR5DPoXH0I3MZ0JgYMXkMRTIYygyj6H68hiKx2MomMdQT+Ex dCPzmRAYWDF5DAXyGIrMY6i+PIbi8RgK5jHUU3gM3ch8RgQGVkweQ4E8hiLzGKovj6F4PIaCeQz1 FB7D1JnPKJDgUzN5DA3yGJrMY+i+PIbm8Rga5jH0U3gMU2c+o0CCT83kMTTIY2gyj6H78hiax2No mMfQT+ExTCPzuSDBp2byGBrkMTSZx9B9eQzN4zE0zGPop/AYppH5XJDg0zB5DAPyGIbMY5i+PIbh 8RgG5jHMU3gM08h8zkjwaZg8hgF5DEPmMUxfHsPweAwD8xjmKTyGaWQ+ZwQGNkwew4A8hiHzGKYv j2F4PIaBeQzzFB7DNDKfEwIDGyaPYUAew5B5DNOXxzA8HsPAPIZ5Co9h6sxnnBAY2DJ5DAvyGJbM Y9i+PIbl8RgW5jHsU3gMU2c+o0GCT8vkMSzIY1gyj2H78hiWx2NYmMewT+ExTJ35jBoJPi2Tx7Ag j2HJPIbty2NYHo9hYR7DPoPHEEMj86mR4NMxeQwH8hiOzGO4vjyG4/EYDuYx3DN4DDE0Mp8KCT4d k8dwII/hyDyG68tjOB6P4WAewz2DxxBDI/OpkODTMXkMB/IYjsxjuL48huPxGA7mMdwzeAwxNDKf CoGBPZPH8CCP4ck8hu/LY3gej+FhHsM/g8cQQyPzKREY2DN5DA/yGJ7MY/i+PIbn8Rge5jH8M3gM MTQynxKBgT2Tx/Agj+HJPIbvy2N4Ho/hYR7DP4PHEEMj8+mR4DMweYwA8hiBzGOEvjxG4PEYAeYx wjN4DDE0Mp8eCT4Dk8cIII8RyDxG6MtjBB6PEWAeIzyDxxBDI/PpkeAzMHmMAPIYgcxjhL48RuDx GAHmMcIzeAwxNDKfDgk+I5PHiCCPEck8RuzLY0QejxFhHiM+g8cQYyPz6ZDgMzJ5jAjyGJHMY8S+ PEbk8RgR5jHiM3gMMTYynxaBgSOTx4ggjxHJPEbsy2NEHo8RYR4jPoPHEGMj82kRGDgyeYwI8hiR zGPEvjxG5PEYEeYx4jN4DDE2Mp8GgYETk8dIII+RyDxG6stjJB6PkWAeIz2DxxBjnflMAxJ8JiaP kUAeI5F5jNSXx0g8HiPBPEZ6Bo8hxjrzmQYk+ExMHiOBPEYi8xipL4+ReDxGgnmM9BQeY2xkPhMQ fLqBx2MUHWKMrCcaoyg6GqMMxzBGloHG2EfoboxG5jMBwacbeDxG0WHGoPIYRdHVGCweI8tgYzyF xxgbmc8IBJ9u4PEYRYcZg8pjFEVXY7B4jCyDjfEUHmNsZD4jAAO7kcdjFB1kjJHKYxRFT2OMLB4j y1BjjE/hMaZG5jMAMLAbeTxG0WHGoPIYRdHVGCweI8tgYzyFx5gamc8AwMBu5PEYRYcZg8pjFEVX Y7B4jCyDjfEUHmOqM59pQYLPicdjFB1kjInKYxRFT2NMLB4jy1BjTE/hMaY685lmJPiceDxG0WHG oPIYRdHVGCweI8tgYzyFx5gamc8ZCT4nHo9RdJgxqDxGUXQ1BovHyDLYGE/hMaZG5nNCgs+Zx2MU HWSMmcpjFEVPY8wsHiPLUGPMT+ExptvM5yJGCRjDM3kMD/IYnsxj+L48hufxGB7mMfxzeIyqyfyi ZNnb/ef/8r/9X//Pz4v6rz/9Gtyn+OUtfts+7fnP/Dnn+lC9jfPg80/0r9/tl1/8ptn+f/lRZW2K VRHE8NOV4PT65ejlLnvg029vf+bfyv/66fvvv3z5lD7bf37fLdF6qrz9VHt5Gy//pP+2v8Y/fv9k v/xnG+Tyd/MXWXVxvHZW/LL9u7xxDLfmev9s+TQiHIy6tEZ9++W3+PWPt0/f3dvR0FcvKZ9ZHQxv hp++xe/x7e1r2H/U858HJ96mC8NPxUOfvkX39ev+Ia4eKKegPfjBUv6isnHW99mEp7+yqjHDbKJ5 Fe3+Pcv2v4v/hoNvMKVV+d3nU+Qtn5jRn9VXj5WDtbo5wjiY6xGy6a9OgfrxYtX2uTAOYR2onLNn dfmj/ED1BLNKRjmsZ91F8+tFNIn2VzxO+Zy4KPZXa2/ar57zx/L28+eL4vRXUR1rLvPPn/N5Btpe lv/RgzNuStN5Sno/wGIvc1T1zDpp0SebIqVNNvm8o0w2eZiuk80oFWuyGaV5wmSTo4BnTjajGlmT zagEe7IZlaFPNqNyzMlm1AqdbMzYabIxgjzZmMiZbHKYSphsvKBPNkUDTTZhOpps8jPcySZQI5sx kiKbMRYb5iD086/ZsPtvePpz/VwH9qUHpasMCkrPI1z9TtKMY35CXH4uOQ1pWpYpjN5EYZfLz6WW mEyKTou4RB315efKhylUmv00q2BnQwpob3MjOaA17DS7t8mtOQ7inc6qiwP7TmfT5zjwt/XV25Xl PBnfPFjGWFqTUxmDd+9z/JkYPssy0Gf7CNc+y1+uldPlLujPaUx+mIdhHI3LRvAXn41qWKQf0zgP TqnhalrQLt8aa+Xz67NymCg+u021ZJ9pdhVlPkQ/8HzmB8xnfsB95rnJl+PPxPGZx7Iv5xGufBbW +oIQby4/OsfocTRDCvrm8lOuS/s/F5+dRjn9e/XZf//v20erhzx/slvdo96ssj3Zm+xCzvK1CKY3 BehN0cGb3PzP8WdieRNLAJ1HeOdNNdpJXl1rvYpqWUyYRjHna665eFNk+RjntASX580oL9700ucf ydsg8wzpxkSZA2+xueIzdl1oOUROMc+m42cVVz0pq7gpumUVt+FYpsKKec4jXJlqGqSKQaWLqYIe 53nJP6udbMhT1MVUWmev5a80DCKOg7sK4PRs3KBnK5bRqkQxlWx0upnZ+4jMs8qepptq17FNtekp ptoVvUy1D0c11SZDTHUZoZGqvnrRbFPr4nn+530y++ZaeTXCo66ql0znyL0tWI+RsZa+6zBXkdbS d0VXV9HX0jcZ7KpnrKXLRpucOXDj+NkOxjBAzl3HN8aqJxljU3QzxjYc2RirDDLGeYTuxqiLRebA DaLzx3SGUV626xBjZD3RGEXR0RhlOIYxsgw0xj7Cj65DKrtHQdeh0wiPuqquNJkDN2Rej5FRm7br MFeRatN2RVdX0WvTNhnsqmfUpslGj51ZcatZZ2sEZ8+jXQcYo+hpxlgV/YyxDkc3RpFhxjiN0N0Y daVJnnTY1yEvPIPp2nWAMYqeZoxV0c8Y63B0YxQZZozTCIfXIT0U3MAugztyym2q8CJ40EQ3/Xjy NOaNYN9U2zQnRuXrrgNMVPQ0E62KfiZah6ObqMgwE51G6D273PTj2YzBvi/OV0fLMsaqg+IRSzTG qugZj1iOMYoMjUfskTGuY1Q7olHuOsKjrppqVy3ATbWzjHrqXYe5ilRPvSu6uopeT73JYFc9o55a 3jTz2YzBvql2znD62+86vjFWPckYm6KbMbbhyMZYZZAxziP8aLoxyTkHTTenER511dJwFfum2kWT ODm8TQe4quhprloV/Vy1Dkd3VZFhrjqN8GNXmdRcbye4ahvhUVfdZIZ9GJeFf0eep1TOXLXpsIsY ba7aFF0vYoy5apXBF7G7c1UJbLC56jTCo66SDVfxb+cH53jrDUUHucpR1xuKoqerHGu9IctQV7mn rDfc9iAqxpi5zanWj8loHLHrMGOQGkfsiq7GoDeO2GSwMY4aR1xPFk6g0806wqOu0rehUXAGWcVy jK4Tuw5zFanrxK7o6ip614lNBrvqGV0n5E0Do80YyEKU52A2mw4yhqdhNpuipzE8A7NZZagx/F3M pkwWGGZzGuFBV02NPLNGYmbPwWw2HeYqGmazKbq6ioHZrDLYVU/BbKZGnlkjYa9nLW+uOswYxOXN VdHVGJzlzSKDjXG0vHk9WXiDTjfrCI+6qpFnVkjMHFhro6sOclUgro2uip6uCpy10SJDXRWe0e9C To08s+L21Fo/JqPfxa7DjEHqd7EruhqDXu6wyWBjHPW7uJ4swoxON+sIj7qqkWeW3IZc6zEy6h12 HeYqUr3DrujqKnq9wyaDXfWMzUvkTd+lzRjchlzrx2RsXrLrMGOQNi/ZFV2NQd+8ZJPBxjjavOTd ZBHg6SYQphtZ36IHJGaOjJ1Pdh3kqkja+WRX9HRVpO98sslQV8Vn7HwiJ1UbwyNhb2Si6RFE0yMZ TY990fTIQ9MjjKbHR9D0CKPpkYCmT41UsUdi5shE0yOIpkcymh77oumRh6ZHGE2PT0HTp0aq2CFh Lw8eXXWQMajw6KroaQwWPFpkqDEO4dHrySLBjGAiMIJzlSrOrkJi5sRkBBPICCYyI5j6MoKJxwgm mBFMT2EE5ypVHPJsw2bV88dkcjcJ5G4SmbtJfbmbxONuEszdpEe4mwRzN4nA3cxVqji7CioA5yxA bDqsAJy2ALEpuhaAMxYgVhlcAH53AWJWs8MWIE4jPOqqKs8cnGGD7skqy7kT23R8V616kqs2RTdX bcORXbXKIFedR/hhcZaWk02KUJy1Cx410U1a2ef/1ez4Op8orAq/TQdNTcQKv03Rc2riVPitMnRq OqzwwyIh0TAGO0RO+SaAs96w6YDZpehps8uq6De7rMPRZ5ciw2aX0wg/bCh32+TtcvFqPPOjhnIP mkrWplLs8LocImetYtNhpqKtVWyKrqZirFWsMthU6eVMpRqmQqJrz1lv33TYJYy23r4pul7CGOvt qwy+hN1db8+xscfW208jPOoqXbtKsqPrcoyc69+mw1xFu/5tiq6uYlz/Vhnsqqest8+mYQwu1b4M QURGRnrXsY2x6SnG2BW9jLEPRzXGJkOMcRnhla5hy9AwFZuIL97nBEabDpttaIHRpug62zACo1UG zzZ3IY5yBcIgjtMIj7qqkc0ObCI+H2PgQBybDnJVoEEcm6KnqwID4lhlqKvCUyCO5SYhndIS2RsV rB+TkzrcdJgxaKnDTdHVGIzU4SqDjXEX4siTRcAgjtMIj7pqrl3F3pZgPUYOxLHpMFfRII5N0dVV DIhjlcGuegrEsSwNY7BD5jHMnB0gdx0/ZF71pJB5U3QLmbfhyCHzKoNC5vMILxUyi4apkJCZcx+2 66DZhnYftit6zjaM+7BNhs42t/dhzWtYxMig0wiPuqqRobZIyBw5ZNCmw1xFI4M2RVdXMcigVQa7 6ilk0NLIMls2EF8+JocM2nSYMWhk0KboagwGGbTKYGPcJYPKZIGRQacRHnVVI8ts2DR9PkZW97BN B7mK2D1sU/R0Fad72CpDXXXYPQybbhpZZsMG4svH5HTk2XSYMWgdeTZFV2MwOvKsMtgYdzvy5Mki YR15TiM86CrxPs28TNGOlhszZ7XhrL/vOv6N2Kon3Yhtim43Yttw5BuxVQbdiJ1H+BEyNlk/HDcr 1NebAx4wZKcRHnXVWN+JeSRmTpw+T5sOm6tofZ42Rde5itHnaZXBc9VT+jyJRprZs2PmwSXLMcam 4xtj1ZOMsSm6GWMbjmyMVQYZ4zzCK+V9RCPL7Lgh8zLrYDnJxE3Hv4atetI1bFN0u4Ztw5GvYasM uoadR3gpUzWSiY4bbq+HyAmMNh1mKlpgtCm6mooRGK0y2FQvByYKWZvKctn6coiOsfy+6yBTOdLy +67oaSpHX37fZKip3HLXVHvQvIyzjroVdg/1ntzp/T/tPbkvQ77/ZFe6R42oG0bksvzr18JYmN11 mBFJC7O7oqsR6Quzmww2Yny52c00GBIuy18O0XM4yE0HmcrTOMhN0dNUnsFBrjLUVP7lOEjZSFAF JLj3jOW3XYeZirT8tiu6moq+/LbJYFPplzPV2DAVEtx7xtLdrsNMRVq62xVdTUVfuttksKnCy5lq qk3lkeA+MFbudh1kqkBaudsVPU0V6Ct3mww1VZhezlRzw1RIoB4Yq367DjMVadVvV3Q1FX3Vb5PB pvrxqp8eRjGb+YeQwd31mdMIj7pqqV3lkEg9MNLwuw5zFSkNvyu6uoqeht9ksKuO1meu1uxSWCBX nUZ41FWi4SpuDUE5xsjYxGPXQa6KpE08dkVPV0X6Jh6bDHVV/PEmHvtMY+G5yhJcJRuEAreGYD1G xiYeuw5zFWkTj13R1VX0TTw2Geyqo008LjNNsgq7Ap5GeNRVquEqdgFCOUZGA65dh7mK1IBrV3R1 Fb0B1yaDXfXjBlzb9Sv+oAGXf4R72UZ41FW6cpUVSLSeGL2Tdh3kqkTqnbQreroq0XsnbTLUVWl4 uVtAU5tqQYL1xMQbEog3JDLekPriDYmHNyQYb0gvhzeoOq1uFyRW5zTz2nWYqUjNTHZFV1PRm5ls MthU95t5XV3GWv1N/dBYiW6aqnllXIc8f7Jb3aNGrFPxdkbCexaAvOkwI5I5m9SXs0k8zibBnE26 DyAX5AEDkPcRHnVVnYu3M3uj2cElx9lPdNNBnKmj7Se6KXpypo6xn+gqQzlTN77cNbPOxduZXV9c DpGzneimw0xF2050U3Q1FWM70VUGm0q+nKkaqfjILpUoh8jpBL7pMFPROoFviq6mYnQCX2WwqdzL maqRiY9ImQWrh/Omg0xF7OG8KXqaitPDeZWhprrt4fwCpqoT8ZbfzKccIqer6qbDTEXrqropupqK 0VV1lcGmer08RJ2Ht+xGQOshcpqqbjrMVLSmqpuiq6kYTVVXGWyql2sqrhppeH4joHKInN6Zmw4z Fa135qboaipG78xVBpvq5Wp3VCMNr5BAndU6c9NBpiK2ztwUPU3FaZ25ylBT3bbO/PtNpRtpeIUE 6qy2m5sOMxWt7eam6GoqRtvNVQabyrycqRopdcluIbRu3MY0FdJ1c9WTTdWx6+Y2HMtUWNfN8wi9 uy/oRlZcslsI5Y/JapC46SBjEBskboqexuA0SFxlqDFer1G9bmTFBXu/sHKInP6Imw4zFa0/4qbo aipGf8RVBpvq5Qq0dJ0Vt/wGQuUQOb0VNx1mKlpvxU3R1VSM3oqrDDbVyxVo6TorbqHmQ6zWipsO MhWxteKm6GkqTmvFVYaa6ra14guYqpEVZzcfWg+R05Zx02GmorVl3BRdTcVoy7jKYFOplzNVIyvO bj60HiKzTRrUP2/Vk03Vt00ap3/eKoNN9XJt0nQjK85uPjTFUTlGrmnXsU216Smm2hW9TLUPRzXV JkNMdRnhhw1krXLgFi/7CI+6qpEWZ3cSWo+RkWzadZirSMmmXdHVVfRk0yaDXfWUZJNppLbZ3YDK x+R0A9p1kDFo3YB2RU9jMLoBbTLUGLfdgJrTjQd349hHeNRVjdw2e5+y9RgZ2aZdh7mKlG3aFV1d Rc82bTLYVU/ZjcPUuW23cEPm9WMyMka7DjMGKWO0K7oag54x2mSwMe7vxpEnC3A3jn2ER11VJ7fd jMTMnJ4+uw5yFa2nz67o6SpGT59NhrrqtqdPp+mmTlC7GQl7OX15dh1mDFLaZ1d0NQY97bPJYGPc 343DqgDuxrGP8Kir6gy1m5CYmdOXZ9dhriLlfXZFV1fR8z6bDHbVU/ZNMHWW2U1I2MtprbPrIGPQ Wuvsip7GYLTW2WSoMe601tkniwRPN4kw3dRpZjdyif71GBkFbbsOcxWpoG1XdHUVvaBtk8GuOmyt M2ITTp0sdiOXy18/KKMsbddh1iCVpe2Krtagl6VtMtgaP+6Ps00XP+qP89iEQ+mPY+pksRu5YH45 Rk5/nF0HuYrWH2dX9HQVoz/OJkNdddsfp0t8o4Y6WezYDSnXj8moLdt1mDFItWW7oqsx6LVlmww2 hnhgukkzOt2sIzzqqjpZ7NgNKddjZBSX7TrMVaTisl3R1VX04rJNBrvKPmW6aSSL2T0l14/JKBDb dZgxSAViu6KrMegFYpsMNsaPm87sk0WAp5tAmG4ayWJ2T8nZpzBwink2HZ/aWfUkamdTdKN2tuHI 1M4qg6id8wjdp5tGspi9ye76MTkFOZsOMwatIGdTdDUGoyBnlcHGeLXdltTQyBULNiKfD3HkFPNs OshUI62YZ1P0NNXIKOZZZaipxlcr5lFDI88s2Ih8OUROMc+mw0xFK+bZFF1NxSjmWWWwqW6LecZW TnD9v7tfRopBGinjhY27l4/LKczZdJhBaIU5m6KrQRiFOasMNshtYU4KPwhz/pyUGa7+oXilkUNO SDw8ceptNh3klYlWb7MpenplYtTbrDLUK9MRPPGnU/PgL9PJNE5h9qzguJETTkhwPHEKaDYd5hJa Ac2m6OoSRgHNKoNd8moFNGps5JP5XRXLIXIKaDYdZipaAc2m6GoqRgHNKoNNdVtAsyxLXMblcrH6 U0YpZj8Np/9cTCXkNM1BChnkWNaGL6Yqj+YoxqglKybpKaZqpJP5XRXzIc6cpsKbDjLVTGsqvCl6 mmpmNBVeZaip5tumwlOapsUt88VU743zzlRpmRYlJjGLNC9KX8VIgwxKXOxIMVUjFc3e33Y9RE5T 4U2HmYrWVHhTdDUVo6nwKoNNddtUeDHLlM0iLqZa5mIrtRSLZGuFi6nyXybbR0glRzXJdG2q6/8M lmKqRiaavb/teoicpsKbDjMVranwpuhqKkZT4VUGm+q2qfDitovalal0fiwJky+KXgxivphKyfcz 0sVUeVab8wVwUsVyC22mamSx2fvblkNcOE2FNx1kqoXWVHhT9DTVwmgqvMpQUy23TYXlfI6cfrmK j4T0eTZaTXQx1SKyY4yYliXPbdmMzZlqmueZkk8aG1ls9v626yFymgpvOsxUtKbCm6KrqRhNhVcZ bKrbpsJX4fgvp8gpx0zzMor8AwtxPVNlK2U/lagq/28QV5e/69cXw1FMVWex/YwE6gunqfCmw0xF ayq8KbqaitFUeJXBpnq1psJqrDPffkIC9YXTVHjTYaaiNRXeFF1NxWgqvMpgU71aU2E11ilyPyGB uuA0Fd50kKkEranwpuhpKsFoKrzKUFOJV2sqrMY6o+5HJFAXTA5JgBySIHNIoi+HJHgckoA5JHHb VDiH4uPsB3sxlVrWO7lZTir/7zRd3f0tas7/1fnZRU7SXCU/8wjlP3PMAf4yJYKppjqj7kckUBdM hkmADJMgM0yiL8MkeAyTgBkmccswFaPItAbdJ1NNMmXzZGNlc83yiheYYzbTUgy0mecqUE+LF8sy CyNyDD9TZqqpzqj7gYv9l0OUTIZJggyTJDNMsi/DJHkMk4QZJnnLME1p9mpSy9UyzSiFmLYZSSzy atW45KCun72aqaZsspAfy/OX9NNEMVWdUfcDl/pfD5HJMEmQYZJkhkn2ZZgkj2GSMMMkbxkmaYqt 5quYSqY8P0UppmHLqr+bqbKdyuvXuepqprp9hmKqRkY9sTe/KofI5J4kyD1JMvck+3JPksc9SZh7 krfckxxvTbUZY56lLs9M5mqmitKsl0tT5iV5Rc/lZ0bp19TnVGIyiqnqjLpnd2Qsh6iYgJQCASlF BqRUX0BK8QApBQNS6rYhsVy2GeZdTLVf5tR6UbueqaapzEhi2F5zHaiX+Unt8ZYiXf7qjLpnN2Rc D5HJUymQp1Jknkr15akUj6dSME+lbnmqSQ8mG+hq7a/EUSU+mub138vVTFVHW2f0ZQ3t82xVln2k oJiqkVFnN3NcD5HJUymQp1Jknkr15akUj6dSME+lXq0hcZ75GqZCAnXN5Kk0yFNpMk+l+/JUmsdT aZin0rc8lVryBcutkdNpppoXs8T8X7n4fEW7MtW6wCzXi+OK5F2lFAZhxCQK8CcWIyiUwtTIqPN3 qS2HyOSpNMhTaTJPpfvyVJrHU2mYp9IVT+WXcck2ekd+mmKnSU+lSePVgvL66DD5eSj2UlcxlVpW +CWVYL3cM1JM1cio83epLYfI5Kk0yFNpMk+l+/JUmsdTaZin0rc81Qq42GtKIc82shS7nWafK1PN S8qPDGWqmidxRX5uZst2TOX+UFLQl7mRUefvUpsP0TB5KgPyVIbMU5m+PJXh8VQG5qnMLU+1oipC XCU/S3a8mEQWGM9IeYW+DEtY0lIUQUzLNfqSh7g8KzTFVI2M+owUPhgmT2VAnsqQeSrTl6cyPJ7K wDyVueWp1oBKiuuEer4hUzLle/0h38ypq5u/VNDOPSdq1FVCfZ2mwkVJ8VSdUA/s5pPrETJxKgPi VIaMU5m+OJXh4VQGxqnMy+FUc51QD+zek+shMnEqA+JUhoxTmb44leHhVAbGqcy9FjzLoIP5UQue eNNOpWrBcx7hUVfVGfXAbltZjtEyeSoL8lSWzFPZvjyV5fFUFuap7HLgKqgFz1xnxQO78+T6MZlM lAWZKEtmomxfJsrymCgLM1H2Kb2Z5kZmm7/TbPmYTK7JglyTJXNNti/XZHlck4W5JvuM/avU3MhO 83eLzR/TMdkkB7JJjswmub5skuOxSQ5mk9zR/lWYMRoZZv6Or+VjMvkiB/JFjswXub58kePxRQ7m i9wztqBScyNLzN+1tXxMJiPkQEbIkRkh15cRcjxGyMGMkDvaggoyxlJneoNEgk/P5Hw8yPl4Mufj +3I+nsf5eJjz8c/YRUotdbY2QD0cPZPV8SCr48msju/L6ngeq+NhVscf7SKFGaORcoX6MHomb+NB 3saTeRvfl7fxPN7Gw7yNf8ZGUGpppE2h/ouBycwEkJkJZGYm9GVmAo+ZCTAzE26Zmb89F780sqbs vVPXQ2QyMwFkZgKZmQl9mZnAY2YCzMyEW2bmWU09l0Yilb0j6vrJmShMAFGYQEZhQl8UJvBQmACj MOFoYyjsylQnUgN7R9TyMSMTZ4kgzhLJOEvsi7NEHs4SYZwl3uIsf/+VqU7CBvaGqOshMnGWCOIs kYyzxL44S+ThLBHGWeLRvlA3i8JxGGcrrF0cb/qps7LBIIFxZEIqEYRUIhlSiX0hlciDVCIMqcSj vZ6uaYCoj3mCODzCE6wjPOqqOqUbDBIZRyalEkFKJZIpldiXUok8SiXClEp8hFKJ8thV+jFXycdd JRr5YI3E0IlJqSSQUklkSiX1pVQSj1JJMKWSbrv+CJuWYbhyV9mhbJSzgIJq0UgQaySoTkxsJYHY SiJjK6kvtpJ42EqCsZV028qncooeRhGl15hTGhljhUTKicmxJJBjSWSOJfXlWBKPY0kwx5Jebo8x 0cg2Ky75vcw6/w/dVLuObapNTzHVruhlqn04qqk2GWKqywgvZapGtllyye/1EBnXtF2HmYp0TdsV XU1Fv6ZtMthUt9e0v99UjbS05ILf6yEyLn+7DjMV6fK3K7qain7522SwqV7v8tdIaSdu9F0OkbPF 5q6DTEXbYnNX9DQVY4vNTYaa6vW22BSNlHbkBurrITLw0V2HmYqEj+6Krqai46ObDDbVbXu6v99U jex3RAJ1zracuw4zFQk93RVdTUVHTzcZbKrb9nQEU/XxUCPXHZC4nLNd566DPETbrnNX9PQQY7vO TYZ66Ha7zr9/YpKNVHdA4nLO7p67DjMViXDdFV1NRSdcNxlsqpfb3VM2suKBy9Ovh8igY3cdZioS HbsrupqKTsduMthUL9eNTjYS6JbL4pdD5OzuuesgU9F299wVPU3F2N1zk6Gmqnb3/PtN1UigWy7H vx4ig6zddZipSGTtruhqKjpZu8lgU92StX+/qeoEeoQS6JzdPXcdZioSgrsrupqKjuBuMthUt93o /n5T1Qn0CCXQObt77jrIVLTdPXdFT1MxdvfcZKipqt09/35T1Qn0yC58XQ+Rge/uOsxUJHx3V3Q1 FR3f3WSwqW670f39pqoT6HFEAnXO7p67DjMVifTdFV1NRSd9NxlsKqAdXR8P1fnyOCJxOWczz12H eYjE9e6Krh6ic72bDPbQy23mKesEehy4JQjlEDmbee46yFS0zTx3RU9TMTbz3GSoqV5vM09VJ9Dj wK1AWA+RSUshm3luerKp+tJSjM08NxlsqqPGdduL9DBObhL2uALB36lAuBrhUVc1MuiJW4GwHiMT l0J289z0ZFf1xaUYu3luMthVR13v3nkiHbpKD4+5KhFcVafQI5RC52znuesgV9G289wVPV3F2M5z k6Guqrbz/PsvgHUKPUIpdM52nrsOMxWZl+q4nec+HMtUMC9Vbef595uqkUJnVwuvh8jkpZDtPDc9 2VR9eSnGdp6bDDbV381LqUbGnF0bXI6Is3vnroM8RNu9c1f09BBj985Nhnqo2r3z75+YGhlzdmnw eohMXgrZvXPTk03Vl5di7N65yWBT3eelruJr41sR+rkz08VU6f0//jhkX4d8/8mudI8asZFlZ1ce r18Lk7FCdvzc9GQj9mWsGDt+bjLYiC/HWKlG2p1dpFwOkbPj566DTEXb8XNX9DQVY8fPTYaaqtrx 8+83VSPtDhUpc3b83HWYqciMVccdP/fhWKaCGatqx8+/3VS6TrunCQnuOTt+7jrMVGTGquOOn/tw LFPBjFW14+dfaapfvvzytlpqPFlqGN5nRyc3m4Wdcx9nO6fJjlRL7bqZa6ldr856b/NrP33/5Z9f 7Ofv74Z590yxm6rsto/mz6P92/7yViTvBjo9mMcwByOEI4t731a4fATlxF8Hl5vq8ndWLrpy86b0 JUv0n+/rC5f9c57+zDoV2rKYZfa3IJdP+dXefv68Sd89VL7yWbf169lgv7+dXvupOO3rl/2MqJ/I Y6UfDFVOi13w839R/8vP/+NbjP/r//2///x//J//Qy7/y8/r0fzXy4ky6aC8vZweehq8T2q+2gl3 f8geNK3xn7/6X9czYjqdEdNN1WI+I6yZuKndqez2IqKlnhG7rnjwa3pbP+Y+r+3vfPPofyu7Tx8M Qp3oNhky0V1GAPq5Nn8bU/82Y8mQPvJJH3uLm9qw7S3KNbbjW4z1Wwxlvun4FtW0nt+i3Ch3fIu5 egudym1Tx7dYGm9RguiObyGqt7Bz559b1m8xdf65VeMtOv/c9bRox84/d31227Hvz63qs9sOpUqh 41vUZ7cdCrPe8S3qs9uWWK7nW9Rnt5V9f25Vn91Wdv65G2e36PxzN85u0fnnbpzdovPP3Ti7F9X3 i2qc3cvQ9YvSjbN77hse6MbZbfv+3Lpxdtu+P7dunN2m78+tG2e36fxzN85u3fnnbpzdum94oBtn t+obHujG2a36hge6cXbHvj+3aZzdoe/PbRpnd+j7c5vG2e37/tymcXb7vj+3aZzdrm94YBpnt+sb HpjG2W37hgemPrvd2Pnnrs9uN3IXRcute2l6z0lYpNAhYZHo6M4mQxMWCd1ls/nb1NOiG7qeJ/NQ T4tu6HqezENjWkxdz5O5zkPnt+gaV81DY1pMXeOqeWhMi7FrXDUP9bToBPBzf/vP729f1/eYz+9R p0DznMWt5Z/cMkYTHH1GWXWBuyiw6eP4+DrTrsifdPtWPoVfvv++rhdcz2Tt58pnMAfHwJjQigyb 0E4jIBNa0xvVjJa9wS3JXz9mZHojgt6IZG/EK298i28tW1weLu+8HL0zyxFQdcZlhN6OqDPm2RHc evryMdPMc0TWQY5IM9URWdHFEYleWbHJUEek+SmOqOIFNw1comr9mJrpCA06QpMdoTs5gl4Wsclg R+h7jvhT/5ABPWAP2kapor5sFC4ltX76wDRKAI0SyEYJnYzCuV8qMtgod++XboxC8UUVqrsxAQHo NEwsXxQd4ousJ/qiKHr4oozD8EWWgb7YR+h+SanurLIjgLBzGhTTEQp0hCI7QnVyBL32YJPBjlBP cUR1I5wdAYSdOUJhOsKDjvBkR/hOjqAXAWwy2BH+KY6o0rlujNxK7/Ixx5HniKyDHDGOVEdkRRdH jHSCf5OhjhjHJ0YTVQ4++4JP9OUPK5m+kKAvJNkXspMv6BD+JoN9IQ988eeCzRWNFGfg1lqvH9Qx PeFATziyJ1wnT9AZ+k0Ge8IdeWLEPFGnNqcFiTGngeeJrIM8MQ1UT2RFF09M9Dajmwz1xDQ8xxM1 MJg9gUSZk2B6QoCeEGRPiE6e4CDoRQZ7AkbQm45oJDdnJMqcLNMRFnSEJTvCdnIEvcXnJoMdYZ80 SzTymDMSYU6J6YkEeiKRPZE6eYLesnOTwZ5IT5kl6gzmNCHx5bzwHJF1kCPmheqIrOjiiJneb3OT oY6Yl6NZYn2RHsZB6Vlct8i4rIbINA1qkkJOsuxKHQ86YpxGOL//re5Rn9V50YldErwePA/VKDrM Z1RUoyj6+IwFaGQZ7LO7gMbtUpscVb6qSJ//vcgkFsqEVKdLp5Fb6Lt+eh63UXSYUajcRlH0MQqL 28gy2CiHXTVVHK7+wa5YdfZ0AjDSZVp4GEfRQQZZqBhHUXQxyMLCOLIMNchyhHF0NUidRp00EuQu PKqj6DCDUKmOouhjEBbVkWWwQY6ojq4GqXOqk0Ji3oVHcxQdZhAqzVEUfQzCojmyDDbIEc3R1SCN BKtCglXBxDoEiHUIMtYhOmEdgod1CBjrEEdYR0+D1AWcbpJIkCqYlIcAKQ9BpjxEJ8pD8CgPAVMe 4ojy6GqQRvJVIkGqYEIfAoQ+BBn6EJ2gD8GDPgQMfYgj6KOrQRqZWAGgx5NkMiASZEAkmQGRnRgQ yWNAJMyAyCMGpKtBGolZASDHk2TCIBKEQSQZBpGdYBDJg0EkDIPIQxikp0EaGdUFYY8lkwyRIBki yWSI7ESGSB4ZImEyRB6SIT0N0sikOiRIVUxMRIGYiCJjIqoTJqJ4mIiCMRF1iIn0NEgjk+qQIFUx mREFMiOKzIyoTsyI4jEjCmZG1F1m5E75k6cYpZFRtUiwqpgoiQJREkVGSVQnlETxUBIFoyTqECXp OZM0MqoWCVYVkytRIFeiyFyJ6sSVKB5XomCuRN3lSvhEe91tyU0GiVE1ky7RIF2iyXSJ7kSXaB5d omG6RN/u5qq0zqf0o1eaMsi0LKWu5T1aUr/X/rKfgisfy8dP3+Pn9Onf+V1/j982VfOp8uW7ajhv 4zzMOcz9NYu+xfjl7TTI1QNFWjW33qXZKKWzcznq/dfa/ypf9rHm6pvyNyRO/kePw2iEuvrS3g2w 2Mb3tz4j5u37Wz+7DZ/cf97i9/OXcnkofzhbf7enIYr8n9lt/7Kf/4hn9fmRLJ6rjt27WK3ifOPw 6/oLnMXnR4p3Q9VbfFenq5/1+gy4ebB8gPqEX4eQ5cTLL//+Zv2vu+jcAbzxRB4qVW3E96Hc+mn2 Kev8Qfa/y2cYDpRq+xa++/y7vuUzL/qz+uqx8k3YA4tkA1yNkK14Zcz68TzSUp/X20Di9JX+fv1V FnfGg99Qx+1suGh+vYgmUc+Yq6r0ub8oTieAP/ilnaCfNkVzddrMjdNmcMZNaTo4bfx0dNrkZ9bT phzozRRw+oLngy84S3/61+/2yy/7b7z9//LbyoNvKuTjuBKcXr8cvTzb8NOn397+/LQ2Gvv++y9f PqXP9p/7ad18qrz9dHCGhHyS/dv+Gv/4/ZP98p9tkMvfzfl91ZVL2mU7ivhl+3d547ifHEfPlk8j Dk6VcnmvdW+//Ba//vH26bt7Oxr66iXlM6uD4c3w07f4Pb69fQ37j3r+8+D6venCvmfBt+i+ft0/ xNUD5UywBz9Y2s7d9X2upt78V1Y1ApVNNHMnm5TAyWYcTJ/JZhwCdbIZR8mYbMbJESabkXGNHm+u 0eTJZjy8RpdnmJNNkdImm3zeUSabPEzXyWaUijXZjNI8YbIZpX/mZDOqkTXZjEqwJ5tRGfpkMypu ZDNqNLIZTafIJsfH5MnGcCKb0VEim9HTI5tVA0024SiyKc9wJ5tAjWzGSIps1oXdsrXT51+/nW9S Tn+un+vAvvRb3FUG3eKeR7j6naQZx/yEuPxcchpSvj+dwuhNFPZSzP2nyjfmJkWnRVyijleb+eTD LJCJn2YVSl/39ef6xz/+8XP89u3rt5/L/U7JnKRvX3/72YaQv5vvP58+1X4zrH/Or+95Qywm9g3x Q7ePy9H7ftw+ftw+ftw+ftw+ftw+ftw+ftw+ftw+ftw+ftw+vhvh4/bx4/bx4/bx3u3jY+vt9dZD bjJIZyjNrLLXYJW9JlfZ605V9ppXZa/hKnt9VGXfE9SpN45ypdsHYBBmdb0Gq+s1ubped6qu17zq eg1X1+u/orq+3vbLTRrpGmWY1fUGrK435Op606m63vCq6w1cXW/+iur6etO2HIYgqJ9hVtcbsLre kKvrTafqesOrrjdwdb35K6rr6y333BQR5s8wq+sNWF1vyNX1plN1veFV1xu4ut78FdX19YaJ2SBI kGqZ1fUWrK635Op626m63vKq6y1cXW//iur6ervLbBAkSLXM6noLVtdbcnW97VRdb3nV9Raurrd/ RXV9vVmpmwISpFpmdb0Fq+stubredqqut7zqegtX19u/orq+3mo2GwRpAeWY1fUOrK535Op616m6 3vGq6x1cXe/+iur6er9bN3mkBZRjVtc7sLrekavrXafqeserrndwdb37C6rrl3q34mwQpAWUY1bX O7C63pGr612n6nrHq653cHW9+wuq65d6r2k3z0iQ6pnV9R6srvfk6nrfqbre86rrPVxd7/+C6vql 3inczRMSpHpmdb0Hq+s9ubred6qu97zqeg9X1/uj6vquBqkzqfOEBKmeWVXvwap6T66q952q6j2v qt7DVfX+qKoeL5pehjqBOo9IbOqZxfQeLKb35GJ636mY3vOK6T1cTO+fV0y/DHXedB6RkDQwi+kD WEwfyMX0oVMxfeAV0we4mD4cbdXQwxd1unQekD5PgQl9BBD6CGToI3SCPgIP+ggw9BHubq1wzxH2 t9UOy8UOjdxXQpr1xDlYjh2yziN2iHOOtUh2yIr8Sf/8/e1T+PolfnrbjXD1QNaqulJlk3IcEOFF +3izaP+n9HGS8/hwm41hKIpZmB+32Ti/7CEP1emxWSIhSNRMD2nQQ5rsIc33EG9dP8Lr+vFmXX/d c7TaCkiYub0V0HK4988mOb/l8oh3xjpzNkskTImB6Z0AeieQvRP43uEt+Ud4yT/eLPn/mWddF8bQ 2E1qfabuYXN6Rk/6ZnLKzwifZx4bhwOLnYZ8/8mudA85rpGKE0gAlCae47IOclyaqI7LCq7jEo8h SDBDkI4YAmIsXPugkXETSD/LpJg+UKAPFNkHiu8DHiqQYFQg3aACf9rZmv2Hv714STPa9sXrcrEK yY36aKI5jVBNNCfdQwZrZOwWJLROnmkwDxrMkw3m+QbjoQYJRg3SEWoATzSNDN0CMLDzMLJ8UHSI D7Ke6IOiYPqgSBk+yDLQB/sI150MyzQzP9rJcNsFM41y+GEnw6uXtffR9H4wTU000/DY/NPIAM4A WjsPkuk7CfpOkn0n+b5jgQpZBvvuEFS4/PhmUPeua1YpcWu9aoSGrzbdQ75qZBBngMidc1DP85UD feXIvnJ8X7H4hiyDfXXEN8DXtZqrnBMQQJcGACwfZB3kg3Gg+iArdh/Y7//54t8Z4fRIccLR23Gc MKIgwz7CdQid0rBMtjHVhCBV0rZx874+Y3xj7pF5mh8HfzT3nIas5p6T7gHPiap1+CIFH5PwNgd9 xTtEz226yPbcqo8TwXObwm+e++7tly/x27vli/rx8t6iVTGeOOudqwwsgU731zvvTEL/M30vjQ1W U4izKabbiWgRowQC7HzzEgxjIio6B0xERZ/O+tI0Ivzbnn6f20f/W7OL1jpILF04/L8ur730pqge Lh9G1y7Zxslz3P/MZvr0+7evPoY/vu2/9vvHygjm4HA4E12WgRPdPkJ/n1WTjxgFEFDnO12mzxLo s9TDZ6mTzxLsMxa/kWWwz540n92upxWfAQG2TwvjIrfp+Be5VU+6yG2Kd012tkvZp89fv576p7Se yuP5umfLNhzHGQklOPYRujtjvl0tK84AMGKfTJAsZ5igIWeYq5CdOwMls8bt8AyUx1n7Ev2RPsV/ /eLfcyHVw3mcWdfde7ZxWG5DuZB9hOPUQWlTMvgxh98H9ruN1i+Ch51Zr6olAfDLYYqsOSusC7F8 Z2Y9cc4qio5zVhmO4aIsA120j9B/zqrX2ZIAwOUwzzxnZB3kjHmmOiMrejpjZlFHWYY6Yz5qFQI6 o14gSwvADYVZM52hQWdosjN0X2ewWKIsg51x1CPkMEc9LsJHL/Kl5YfLI5eXPeym22W2Rc4JiJrD HJhuCqCbAtlNoa+bWHRRlsFuOmooAs4ztwthxRlA1ByWieeMrIOcsUxUZ2RFT2csLAooy1BnLHcp IJ4zbpey8v2UQaLWRTGdoUBnKLIzVF9nsLigLIOdcdRCBHRGI6dskKh18UxneNAZnuwM39cZLKAn y2Bn3AV6eM5oZIE1ErWKkeeMrIOcIUaqM7KipzMEC/HJMtQZ4hbxmczkSkF+L4s0ErgagNqDkEyL SNAikmwR2dciLBony2CLHNE4mDOWRgJXAfB5EI7pDAc6w5Gd4fo6g8XTZBnsjLs8Dc8ZtwnU4gyA pgly4Dkj6yBnyIHqjKzo6QzJ4muyDHWGPGoUcpwMGcKstTPmx6zo5WUPu+k26brIZUbCVymYbhKg mwTZTaKvm1hdRbIMdtNRVxFwnrlNumZnTEj4Ki3TGRZ0hiU7w/Z1BgtvyTLYGc/BW5Y6gbpMSNQq OWzdpsOcQWPrNkVXZ7CAlCyDnfEcIGWpE6jLiEStigekFB3kDEUFUoqipzMUC0jJMtQZ6jlAylIl ULMzkKhVGaYzDOgMQ3aG6esMFjySZbAz0KYiB86oEqhyGYDqx6CYQIgCgRBFBkJUXyBE8YAQBQMh 6jlAyFIz/MsA4NpBM4EQDQIhmgyE6L5AiOYBIRoGQvRzgJClypvKOQGAddBMIESDQIgmAyG6LxCi eUCIhoEQ/TgQQnGGqPKmcpFIBKqZcIcG4Q5Nhjt0X7hD8+AODcMd+jlwh6jyptkZSARqmHCHAeEO Q4Y7TF+4w/DgDgPDHeY5cIdo5EAFEoEaJtxhQLjDkOEO0xfuMDy4w8BwhzncH2Z9kZ6C3jb7ObJK BbafBQ+7qJEvFUi0apggiAFBEEMGQUxfEMTwQBADgyDmOSCIaORLFyRatUwQxIIgiCWDILYvCGJ5 IIiFQRB7tHtMVfc+SZHPv62ChmGVRgJ1QdhkywRCLAiEWDIQYvsCIZYHhFgYCLHPAUJEI4G6IGyy ZQIhFgRCLBkIsX2BEMsDQiwMhNjnACGikUCdETbZMYEQBwIhjgyEuL5AiOMBIQ4GQtzjQAjJGY0E qkVCUseEOxwIdzgy3OH6wh2OB3c4GO5wz4E7RJ1AXQwSkjom3OFAuMOR4Q7XF+5wPLjDwXCHew7c IRsJVINEoI4JdzgQ7nBkuMP1hTscD+5wMNzhngN3yEYC1SARqGfCHR6EOzwZ7vB94Q7Pgzs8DHf4 58AdspFA1UgE6plwhwfhDk+GO3xfuMPz4A4Pwx3+Bu74cxrcFEaje1mkkR3VCGfsmZSHBykPT6Y8 fF/Kw/MoDw9THv6I8jilzJcwCW2MOcbYhzgGtwg36XbS/TzC+/e/0j3st0bOVSH0cmCyIwFkRwKZ HQl92ZHAY0cCzI6E57AjspFiVQi9HJjsSADZkUBmR0JfdiTw2JEAsyPhOeyIbKRYIxLgBiY7EkB2 JJDZkdCXHQk8diTA7Eh4DjsiGynWgAS4kcmORJAdiWR2JPZlRyKPHYkwOxKfw47IRoo1IHFtZLIj EWRHIpkdiX3ZkchjRyLMjsTnNAaRjRSrRyLQyORBIsiDRDIPEvvyIJHHg0SYB4nP4UFUI8XqkQg0 MXmQBPIgicyDpL48SOLxIAnmQdJdHqRuZ+cGb30U99rZnV72sJsaaVmHEM+JiYwkEBlJZGQk9UVG Eg8ZSTAykp6DjKhGWtYhxHNiIiMJREYSGRlJfZGRxENGEoyMpOcgI6qRjbVs4tnbOHCWcjYdsjtK 1hN3RymKbs7YhmNskZJl4BYp+whXCfvSjnDU8yUruwyDXNZwYxjGC6coyprdePrv5TKlsl7m/wqb H1eeZKc62Vqa4AF24qz/bDrMTrT1n03R1U6M9Z9VBtvpOcW9qk62ipEd6pbPyVn22XSYM2jLPpui qzMYyz6rDHbGzbLPn9vcEexlptke8dP27+udvMfB7Y+K64jYyvWxOIxj+TfJUXWSVgzscDcf38hZ 2Nl0kKNG2sLOpujpqJGxsLPKUEeN862jrqx05Z2LYUaSQ+pkrRjYYW/5vJwFnk2HOYS2wLMpujqE scCzymCH3Czw/Bli4QtDvF1zHpzV4ySu9zrdnklpGNbdKqt9dy//xPYGhOch33+yK93DTmwkhxMS Zo+cBaVNhzmRtqC0Kbo6kbGgtMpgJ94sKP3ph8GLfF92caLXZpqEX2aj4jCYixHlEsSwyGhksGLb GHo3oh5EjhtF0i4u0kdHmt4aeeXEhrrzIU6TnzmmyjqBmCrrDba10zZIgLd2WscJy/DT58l+859S zN/a9dZO1cPF90q0x2EZdcLWt84jXF9U5RyHK0hHTMMye5dvHHPIJS4uVXG94ytPLoO+uuaWvbHl OBe/L25UFI/qRoY7svHycnScVbFNB018E21VbFP0nPgmxqrYKoP9pO5egk+PXD3z/fAZ136G5KpG pjuy0fRyjL5MHgxXeZ8gV/kwn/Xe5td++v7LP7/Yz9/fDfPumXXKaTvOX22xt810v/zz3UCnB/MY 9X502whmm0Tfy68fKUcj0oE6fxv5J/v0P+O/4pe3T5dp/ObBMov79hBxzIfwn+/7a7NF47f9EN4/ mIeYD46Bd6Jgi4TnEZ51N6NuTL9M0SZ2S8Vl1tEwqvt2HXsq3fSUqXRX9JpK9+GoDtlkiEMuI7zS VKr01HAV9x55PUZGZeCuw1xFqgzcFV1dRa8M3GSwq8TruWquXcVu5bgeI6OqcNdhriJVFe6Krq6i VxVuMthV9vVctTRcxb3hXY+RUZG46zBXkSoSd0VXV9ErEjcZ7Kr0eq4Slasiu4VkOUbLWALfdZCr LGkJfFf0dJWlL4FvMtRV9nYJ/AVcJRuu4t6irsfIWAnfdZirSCvhu6Krq+gr4ZsMdtVtJeQLuEo1 XMUtTFmPkbGKvuswV5FW0XdFV1fRV9E3Geyq21X0F3CVruMqjUTrjrGSvusgVznSSvqu6OkqR19J 32Soq9ztSjrBVZ1MZBomQoJzx1hs33WYiUiL7buiq4noi+2bDDbR/cX2v3pqMkPtKoUE546xcL7r MFeRFs53RVdX0RfONxnsqtuF8798ajKNzLlCYnHPKNrcdZCJPKloc1f0NJGnL2pvMtRE/nZR+wWm pkbmXCGxuGcsbe86zFWkpe1d0dVV9KXtTQa76uWWtpVpZM7ZPTzXY2QUi+46zFWkYtFd0dVV9HXg TQa76nYd+AVc1cicszd3LccYGIWmuw5yVSAVmu6Knq4K9ELTTYa6KtzuQP8Crqoz54m9Mex6jIyC 012HuYpUcLorurqKXnC6yWBXyddzVZU5d/lRZK5iFKvuOsxVpGLVXdHVVfRi1U0Gu8q9nquqzLlb awLYropMeiqC9FQk01OxLz0VefRUhOmp+Hr0lGlkzh0SrUcmPRVBeiqS6anYl56KPHoqwvRUfD16 yjRS6Q6J1iOTnoogPRXJ9FTsS09FHj0VYXoqvhw9pYdGKt0i0Xpk0lMRpKcimZ6KfempyKOnIkxP xZejp/TQyK1bbl1/OcbEpKcSSE8lMj2V+tJTiUdPJZieSke94LcX5Z94clOqSmv9cPXP+54QVSXt eYSHbdVIrhuk2CEx8akE4lOJjE+lvvhU4uFTCcan0lEjkfN8M+yV2X2X+/TQSKEbpKQhMSGpBEJS iQxJpb6QVOJBUgmGpNJ9SOp6Zkmtsv+kb+emtqvak1W6Lvu/1T3sxCrt7oYFCOTTwAOrig5xYtYT nVgUHZ1YhmM4MctAJ+4j/OjiONtFz00Dnkx47+J4GuFhW1V5dzfMQCSfBh5qVXSYraioVVF0tRUL tcoy2FZPaVyvh0bufAai8TTwcKmiw5xBxaWKoqszWLhUlsHOuI9LXa5T8xDrmcc3wvLHL33bkPil r5Fvn4AAPo085qroICeOVOaqKHo6cWQxV1mGOnE8apR/5ZbBtGOvtDvx/n3hNsLDtqoS7tlWQGyf Rh50VXSYrajQVVF0tRULusoy2Fb3oauTOwatrG639KomuPfOO2rpdR7y/Se70j3sxCpJ74YRKJpI Iw/UKjrMiVRQqyi6OpEFamUZ7MSjrv7Xl8MxNAyojwy4a9w4tzc2uwz5/gNd6R414Fjl87MBgYKL NPGYrqKDDDhRma6i6GnAicV0ZRlqwOlo84BrU0zTD66wj2RetxEetlWV0HfDAFRcpIkHdRUdZisq 1FUUXW3FgrqyDLbV0S4C70yhjm0VH7OVotiqSujnyA+5H5h4VFfRYbaiUl1F0dVWLKory2BbvRzV pccq1Z9dhdwOzDyqq+ggV81UqqsoerpqZlFdWYa6an45qkuPjbQ9Ug+dZh7VVXSYq6hUV1F0dRWL 6soy2FX3qa73S49/LlNzPRszUiNRj9REp5kHchUdZiQqyFUUXY3EArmyDDbS64FcYyPJj9REp5kH chUd5ioqyFUUXV3FArmyDHYVAHJ1MlEjP4/URKeFx20VHWSihcptFUVPEy0sbivLUBMtL9f1So+N 9DxSE50WHrZVdJirqNhWUXR1FQvbyjLYVS/X9UqPjVT7AlRZpIUHdBUd5ioq0FUUXV3FAroKmYq6 6uW6XumpkT8PSHAumHCWAOEsQYazRF84S/DgLAHDWQLoevUsVzXS5x4JzgWTzRIgmyXIbJboy2YJ HpslYDZLvFwbLD01suceidYFk+sSINclyFyX6Mt1CR7XJWCuS/zdbbD01EiWeyQ4l0wkS4JIliQj WbIvkiV5SJaEkSz5cm2w9NRIliMl0EkyiSwJElmSTGTJvkSW5BFZEiay5Mu1wdJTI3OOlEAnyaSr JEhXSTJdJfvSVZJHV0mYrpIv1wZLT43MOVICnRQTmVIgMqXIyJTqi0wpHjKlYGRKvVwbLD01UulI CXRSTGJKgcSUIhNTqi8xpXjElIKJKfVybbD0VKfSR4FE64oJTCkQmFJkYEr1BaYUD5hSMDClXg+Y mupU+jgg0bpmAlMaBKY0GZjSfYEpzQOmNAxM6dcDpuY6lZ4fBqJ1zQSmNAhMaTIwpfsCU5oHTGkY mNIv1wZLz41UekKidc2kpzRIT2kyPaX70lOaR09pmJ7SAD3VyUSNzHlCgnPNhKU0CEtpMiyl+8JS mgdLaRiW0q/X9WpupNIjUs1gmPSUAekpQ6anTF96yvDoKQPTU+b16Km5kUqPSDWDYdJTBqSnDJme Mn3pKcOjpwxMT5nDplfri7bCKdcqKX2gpvkHtViuLim90j1swEbWPSCFD4YJWhkQtDJk0Mr0Ba0M D7QyMGhlXg+0muus+6iRON4yQSsLglaWDFrZvqCV5YFWFgat7OuBVnOddR81EthbJmhlQdDKkkEr 2xe0sjzQysKglX090GpuZN0VEthbJmhlQdDKkkEr2xe0sjzQysKglQVAq2e5qpF1V0hg75jklQPJ K0cmr1xf8srxyCsHk1fu9cirpZF1l0i07pjklQPJK0cmr1xf8srxyCsHk1fu9cirpc66jxIpi3BM 8sqB5JUjk1euL3nleOSVg8kr93rk1VKn4UeBlEV4JnnlQfLKk8kr35e88jzyysPklX898mqp0/Cj QMoiPJO88iB55cnkle9LXnkeeeVh8sq/Hnm11Gn4aUCidc8krzxIXnkyeeX7kleeR155mLzyr0de LXVufUxItB6Y5FUAyatAJq9CX/Iq8MirAJNXASCvOpmokUpPSHAemKBVAEGrQAatQl/QKvBAqwCD VuH1QKulkUqPSHAemKBVAEGrQAatQl/QKvBAqwCDVuFvB62WRuY8IrF4YIJWAQStAhm0Cn1Bq8AD rQIMWoXXA62WRuY8IlUQkQlaRRC0imTQKvYFrSIPtIowaBUf2V5w8L2RmG1IGIkRjSS7QQomIpPJ iiCTFclMVuzLZEUekxVhJiu+Xkcr0UiyG6RgIjJBqwiCVpEMWsW+oFXkgVYRBq3iLWi1PjcNup7f BhO0e3CLQm/0soxKbKONR7tInIY8f7Jb3cNOrBPz04zE/okJZyUQzkpkOCv1hbMSD85KMJyV7m5R OLk5LU0DPr5/7zbCw7aqM/PTjNwNJCadlUA6K5HprNSXzko8OivBdFa6pbP0KL2ax4a/8mxk5Dt/ XTbIYW5Edxry/MludQ87sZHNn5E7iMQkuhJIdCUy0ZX6El2JR3QlmOhKt0TXPMZT9/bGTNfYEvH0 j792onZLWB91KttEmx9Mfe+2RLzVPezEegVgmvi3EmYYWBTYqgOcWPQ0J66Kfk5ch6M7scgwJ55G eK1biXpJYJr4txL5GFkU2KrDXEWkwFZFV1dxKLAig131ehSYqJcEppFfs5GVLAps1WGuIlJgq6Kr qzgUWJHBrno9CkzUawTTyK/ZMMPIosBWHeSqkUiBrYqerho5FFiRoa4aX48CE/UawTTwazbyMbIo sFWHuYpIga2Krq7iUGBFBrvq9SgwWSf+J4VE6yOLAlt1mKuIFNiq6OoqDgVWZLCrXo8Ck3Xif1JI tD6xKLBVB7lqIlJgq6KnqyYOBVZkqKum1+u/JRtJfIlE6xMLC1t1mKuIWNiq6OoqDhZWZLCrXg8L k40cvkSi9YmFha06zFVELGxVdHUVBwsrMthVr7d7oWzk4wUSrU8sTmzVYa4icmKroqurOJxYkcGu ej1OTDZy64Jfs2GGmcWJrTrIVTORE1sVPV01czixIkNdNb9eQy7ZyK0v/JqNfIws+GvVYa4iwl+r oqurOPBXkcGuej34SzZy6wu/iCMfIwv+WnWYq4jw16ro6ioO/FVksKter8uWbOTWgUanJv+X56oF ArmKnuqqpSfItQ7HcdUCglynEV7LVY3cOtDoNB8ji+NadZiriBzXqujqKg7HVWSwq16vy5Zq5NaB Rqf5GFlM1qrDXEVkslZFV1dxmKwig131el22VCO3DnQvNYNg8lUC5KsEma8SffkqweOrBMxXidfj q1Qjtw5sE52PkclXCZCvEmS+SvTlqwSPrxIwXyVej69Sjdw6sE10PkYmXyVAvkqQ+SrRl68SPL5K wHyVeD2+SjVy68A20WaQTL5KgnyVJPNVsi9fJXl8lYT5Kvl6fJVq5NaBfaPzMTL5KgnyVZLMV8m+ fJXk8VUS5qvk6/FVqs6tz0D30nyMTL5KgnyVJPNVsi9fJXl8lYT5Kvl6fJWqc+sz0L3UDIrJVymQ r1Jkvkr15asUj69SMF+lXo+v0nW+aga6l+ZjZPJVCuSrFJmvUn35KsXjqxTMV6nX46t0na+age6l +RiZfJUC+SpF5qtUX75K8fgqBfNV6vX4Kl3nq2Zg3+h8jEy+SoF8lSLzVaovX6V4fJWC+Sr1enyV rvNVM7BvtBk0k6/SIF+lyXyV7stXaR5fpWG+Sr8eX6XrfNUM7Budj5HJV2mQr9Jkvkr35as0j6/S MF+l7/NV66vXlh5xmnht3o67hGxDvv9kV7qHndjIcQF7TefvhclkaZDJ0mQmS/dlsjSPydIwk6Vf j8nSdY4rSSTCN0wmy4BMliEzWaYvk2V4TJaBmSzzekyWrnNcSSIRvmEyWQZksgyZyTJ9mSzDY7IM zGSZ12OydM2PzkCj03yMTCbLgEyWITNZpi+TZXhMloGZLPN6TJau+dEZaHRqBstksizIZFkyk2X7 MlmWx2RZmMmyr8dkmUY+HthROh8jk8myIJNlyUyW7ctkWR6TZWEmy74ek2Ua+XhgR+l8jEwmy4JM liUzWbYvk2V5TJaFmSz7ekyWaeTjgR2lzeCYTJYDmSxHZrJcXybL8ZgsBzNZ7vWYLNPIxwM7Sudj ZDJZDmSyHJnJcn2ZLMdjshzMZLn7TNb66rUFbRK6b+b0NCScOTV1Dj9ZJMJ3TI7LgRyXI3Ncri/H 5Xgcl4M5LnfDcbVS92ZJx22V9f0G8qcRHrZVnZBPBgnxPRPk8iDI5ckgl+8LcnkeyOVhkMvfB7lO Xbyvu8bLu93iH7VQI/tukHjeM6ktD1Jbnkxt+b7UludRWx6mtry4NzNNbvHycGbSD21tsY3wsK0a 6XeNBPSeiW15ENvyZGzL98W2PA/b8jC25V8P2zJ1+j1pJKD3TGzLg9iWJ2Nbvi+25XnYloexLX+L bSm7jPnmsRFO2UW09+NJ7Xi+bHmYRxtCONql4jTk7Qc762415ck89S1e3NhHT8ENYXLqYevWOf6k kEqOwGTDAsiGBTIbFvqyYYHHhgWYDQu3bJie8q8or3dYeR+qdZn0cjjfcA5SrRGY/FcA+a9A5r9C X/4r8PivAPNf4eX6a5mhzuMniVRrBCbLFUCWK5BZrtCX5Qo8livALFe4ZbmsDVo2NnyapFnGSV/f RO7PzMrHUejqUpqfCYORS57N2pfS85A3l8WL7mEj3qT+ZzMJidwpRCb+FUH8K5Lxr9gX/4o8/CvC +Fd8YG/FJR7EcunhG9BI2FvRDHPDVsitQmTyXxHkvyKZ/4p9+a/I478izH9FfWCrRoB1mX2CmLy/ cVFjktpe9rCbltpNAoneI5P7iiD3FcncV+zLfUUe9xVh7iuGRyap9raI1Q7EP5ikImWSEg1bIaF9 YoJfCQS/Ehn8Sn3Br8QDvxIMfqWXA7/MIGtXLUhon5jgVwLBr0QGv1Jf8CvxwK8Eg1/pPvj1lFSD ajgHKcVITLgrgXBXIsNdqS/clXhwV4LhrvRycJcZdMNVQCnGOPDgrqJDXJX1RFcVRUdXleEYrsoy 0FX7CNepBim1bmXtJyeGZv3ij7L2ZTRxmLU/DVln7XfdUdY+ydvIfxLzsEz60ay9GRpZewvUe4wD jyArOsy6VIKsKLpal0WQZRls3Zfr6pU/Uz0heiDsHwceDVZ0mKuoNFhRdHUViwbLMthV97t6XU9f x1jYu9xX21XtGTFdz4i3uoedONZOdMCtwjjyALKig5w4UgGyoujpxJEFkGUZ6sTxPkB2ZZtx7O7E dUjciY1lAAfceowjj0MrOsyJVA6tKLo6kcWhZRnsxPvdw65t0wbSqlQbxYmyixMbKwcOuV0Zeeha 0WFOpKJrRdHViSx0LctgJ74cumbGxgqCRu4kRh66VnSYq6joWlF0dRULXcsy2FX3O45dTUbtLj7Q lfZ9Fx/2/NZYdNBALco48Ui0ooOcOFFJtKLo6cSJRaJlGerE6X6XsmvbiP5OFF2c2FioUED5yjjx yLaiw5xIJduKoqsTWWRblsFOfLyzWbbN0t+JSx8nNhY+FFAFM048Gq7oMCdSabii6OpEFg2XZbAT 73c2e8aS2VgvbpQ2U3znzDx8regg58xUfK0oejpnZuFrWYY6Z3657mVmNPV8lJC7hZlHrxUd5ioq vVYUXV3FoteyDHYV0L2sj4mmxjJDQgL9mQetFR1mIiq0VhRdTcSC1rIMNtH9ZmVXsdDcptfWfzwv vFqHhMOrqbHMEJFAf+FxbkUHOXGhcm5F0dOJC4tzyzLUict9zu0Z4dXUWBaISGC+8Fi2osOcQ2XZ iqKrc1gsW5bBzrnPsl1NOEuzhcv+AuYtou4yhzWWBQJQpzIuPDau6DAnUtm4oujqRBYbl2WwE1+P jZsaywIBKFMZBZONEyAbJ8hsnOjLxgkeGydgNk4Ajc86maiR0fdAdcoomJSaACk1QabURF9KTfAo NQFTauKWUjPLMAm7NIo55+BcnJe6mDNf6wYRB12XRY2FrygtDg6KOU9D3tCSF93DTqwz+nJGAn3B JNsESLYJMtkm+pJtgke2CZhsEySyTfjuFMc6JB6u1Rl9OSM3DpJJtkmQbJNksk32Jdskj2yTMNkm b8i2pgHVMdD2SMe90wgP26qR7p+QuwDJxNQkiKlJMqYm+2JqkoepSRhTky+3yaWZ6nS/nJC7AMlE ziSInEkycib7ImeSh5xJGDmTt8jZNA92zreczVlLugezHGqWZhqklEIuMipxPI2tQ54/2a3uUSfO 9ZqBHJFbCcnE1CSIqUkypib7YmqSh6lJGFOTt5jakl24TGpuXz+HB504DZNa8t3A6ioj/Q8uqMO1 E291DzuxXjOQI1Iko5iYmgIxNUXG1FRfTE3xMDUFY2rq5TbTNHO9npDjW2BhXTGRMwUiZ4qMnKm+ yJniIWcKRs7ULXImBqGnYXaNFpI6Dg929Cjz1Lv/HLWQ3Ie8nt/e/edhJ9brCXJACl4UEzlTIHKm yMiZ6oucKR5ypmDkTN0iZ0qqWaSpfaeqzeNX2vN/lMt3nIdX2nXIphOL7mEn1msQ0iF3H5qJsGkQ YdNkhE33Rdg0D2HTMMKmXw9hm+tFCWmROwnNRNg0iLBpMsKm+yJsmoewaRhh0y+3AaeZGwsMFrkr 0EymTYNMmyYzbbov06Z5TJuGmTb9chtwmrmxWGCQuwLD5NMMyKcZMp9m+vJphsenGZhPM9MDiwVm RhcL1hEetlVjscAgIb5hwmsGhNcMGV4zfeE1w4PXDAyvGXXPVoNO4cGmCoe3lNsID9uqsVpgkJIT wyTRDEiiGTKJZvqSaIZHohmYRDMAidbnkrc0Ev0LUmFimeCZBcEzSwbPbF/wzPLAMwuDZ/ZvB8+W Ro5+QSpMLBM8syB4Zsngme0LnlkeeGZh8MwC7dE6mahOyasJCb4tkxmzIDNmycyY7cuMWR4zZmFm zN5nxp5tojqbriYk1HZM3MuBuJcj416uL+7leLiXg3Ev9wju5bBQ+zTCw7aqU+PZVECo7Zi4lwNx L0fGvVxf3MvxcC8H417uFveafErKyCscYl8MHMszbnvme/1M3J5x9TNDfoY0WTVS4wGJvR0T93Ig 7uXIuJfri3s5Hu7lYNzLvV6HsaWRGg9IMO6Y6JYD0S1HRrdcX3TL8dAtB6NbLj1wCfTHxNaDScx1 hIdt1ciNe6SkwzM5LA9yWJ7MYfm+HJbncVge5rD8LYflx2WekhVtf9kHOQXtFznPyc7KLT7b4geG s9ecwq3uYSc20ukeKenwTHbLg+yWJ7Nbvi+75XnslofZLX/Lbi1H7QPez2v5n/oiuSNcC+ki2cid O6R+wzOpKw9SV55MXfm+1JXnUVcepq78/UZf56lnmX3zhnEYKv7vxm6HBWr7kO8/2ZXuUSeKOgGv JHITEJjUVQCpq0CmrkJf6irwqKsAU1fhgX0vRTje9/JdU57jq2eg7Hsp6pS8kshdQGBiVwHErgIZ uwp9savAw64CjF2FW+wqGyGfpGlp++tRrDRaFYMex3HJRzZ79QPDvcNKb3UPO7GR1xfIjUNgoloB RLUCGdUKfVGtwEO1AoxqhVtUy2g3JeNT24mPovYmDbMutRhJmaT8D0qJwjvU/lb3sBMbiwMCuXGI TLwrgnhXJONdsS/eFXl4V4TxrvgI3hXbeFcV6x37LVLwLtFYHFiQe4nIxLsiiHdFMt4V++JdkYd3 RRjvinfxrmKK45ZkxVyP2EpTbFWvDqgFKceITLwrgnhXJONdsS/eFXl4V4Txrvh6jcZEvTqgZqQc IzF5rwTyXonMe6W+vFfi8V4J5r3S+MBklZrbjkTCNXAd4WFb1asDakbqMRKTAEsgAZbIBFjqS4Al HgGWYAIsyUdspVDuYh3hYVvVqX4FpfoTkwlLIBOWyExY6suEJR4TlmAmLLlHbNVuH0axlafYqs7+ KyT7vwy8tcyiQ2yV9URbFUVHW5XhGLbKMtBW+wgvFVrJRiYfqZ9eBt66ZNFhrqKuSxZFV1ex1iWz DHbV/W2M/nJXNRL5SP30MvBWKosOcxV1pbIourqKtVKZZbCr7q9U/uWuaiTlkfrpZeStOhYd5KqR uupYFD1dNbJWHbMMddX4er0eZCPBjtRPLyNv0bHoMFdRFx2LoqurWIuOWQa76vV6PchGfh0pn15G 3gJi0WGuoi4gFkVXV7EWELMMdtXr9XqQjfQ6Uj29TLzFwKKDXDVRFwOLoqerJtZiYJahrpru70X0 l7uqkV5PSLQ+8dYCiw5zFXUtsCi6uoq1FphlsKvu71P0l7uqkV1PSLQ+8ZYCiw5zFXUpsCi6uoq1 FJhlsKtebylQNpLrEYnWZ95SYNFBrpqpS4FF0dNVM2spMMtQV81A64dnuaqRW49ItD7zVgKLDnMV dSWwKLq6irUSmGWwq4BeEE9ylWrk1iMSrc+8hcCiw1xFXQgsiq6uYi0EZhnsKqA5xLNc1citIwXY y8LrFlF0kKsWareIoujpqoXVLSLLUFctw+u5qpFbRwqwl4XXLKLoMFdRm0UURVdXsZpFZBnsqtfb G0g1cutI/fWy8JpFFB3mKmqziKLo6ipWs4gsg131es0iVJ1b1wqJ1hdes4iiw1xFbRZRFF1dxWoW kWWwq273+XkBV9W5da2QaF0w+SoB8lWCzFeJvnyV4PFVAuarxC1fJWwSG4p3rtOfZjmO676xSsyy 7uRGskydONcTEooLJjwlQHhKkOEp0ReeEjx4SsDwlHg9eErViXM9IaG4YMJTAoSnBBmeEn3hKcGD pwQMT4kbeKrRedtqOairjbDvNtreBQ+7qE6U6xEJvSUTlpIgLCXJsJTsC0tJHiwlYVhK3m3RMGi3 rl887qJd8LCL6sS4HoFahkUy4SgJwlGSDEfJvnCU5MFREoajpP6hi/KLBu0PtkLfX/FjW12N8Kit dJ0Z1wNSyyCZdJQE6ShJpqNkXzpK8ugoCdNR8paOGs/GuvFRHApuagqpe9OxaHVO9GVN5L2xqjc8 veyn4Mpn8/HT9/g5ffp3fuvf47dN1Xyq/BDudrjJ2zgPc/4dfs2ibzF+eTsNcvVAkfoDaRh+8vbz 53Lom+70V/nGjzXXbexuzsMyP4/DaISqv79tgMXW39/2jJi372/97Nkx7j9v8fv5S7k8lD+crb7b 8xBF/s/49ulf9vMf8aw+P5LFc2XBXaxWsbP+1/UXOIvPj5QzKJgDdbr6WTfP3/yg24PlA1Qn/zaE HPNJmF/+/S2/4S7KP8h+RtZP5KFSOBjKrZ9mn7/OH2T/u3yG4UCptm/hu8+/61s+/aI/q68eK9+E PbBINsDVCNmKV8asH88jLdXJvQ8kTl/p79dfZXFnPPgNddzOhovm14toEtXsualsts1FcToB/MEv 7QT9tCmaq9PmdiPoctoMzrgpTQenjZ+OThu/b5RaDvRmCjh9wfPBF5ylP/3rd/vll/033v5/+W3l wTcV8nFcCU6vX45enm346dNvb3/m6T879/vvv3z5lD7bf+6ndfOp8vbTwRkS8kn2b/tr/OP3T/bL f7ZBLn+3JvlNF8fri1X8sv27vHEMt9er98+WTyMOTpUC7NS6t19+i1//ePv03b0dDX31kvKZ1cHw ZvjpW/we396+hv1HPf/ZvpbvujxFl2glT5nu69f9Q1w9UM4Ee/CDpe3cXd/naurNf2VVHbTsopk7 2aQETjbjYPpMNuMQqJPNOErGZDNOjjDZjIxr9HhzjSZPNuPhNbo8w5xsipQ22eTzjjLZ5GG6Tjaj VKzJZpTmCZPNKP0zJ5tRjazJZlSCPdmMytAnm1FxI5tRo5HNaDpFNjk+Jk82hhPZjI4S2YyeHtms GmiyCUeRTXmGO9kEamQzRlJkM8ZiwxzOf/712/km5fTn+rkO7Eu+z91kyH3uZYSr30maccxPXPUf l9OQpmWZQulNLexy+bnUkm98U3RaxCXqqC8/Vz5ModLsp1kFO5vt5/rHP/7xc/z27eu3n8v9Tkmj pG9ff/vZhpC/m+8/75/qdDOsf86v73lDLCb2DfFDt4/L0ft+3D5+3D5+3D5+3D5+3D5+3D5+3D5+ 3D5+3D5+3D6+G+Hj9vHj9vHj9vHe7eOji+91AZEekJZvitlEQoFNJBS5iYTq20RC8ZpIKLiJhHq9 JhK6LiDSDqEWFbOJhAKbSChyEwnVt4mE4jWRUHATCXXUUL4JdOhlcTxsWtdVQdoiaKJidoZQYGcI Re4Mofp2hlC8zhAK7gyh/H3AdZT5EkgBXDfBwy5qVAFZBE3UzE4QGuwEocmdIHTfThCa1wlCw50g 9N2m8JspRqqLxsdd1Kj6MQiJqJmdHzTY+UGTOz/ovp0fNK/zg4Y7P+gf94Df8FR1vDFsegxwXUd4 2FaNyiCDxNia2fpBg60fNLn1g+7b+kHzWj9ouPWDvm39wAVck54eAVy3l30Arh+A68cK5ccK5ccK 5ccK5ccK5ccK5ccK5ccK5ccK5ccK5ccK5e0zHyuU/99boXwYcF1vhj8A14/bx3cjfNw+ftw+ftw+ ftw+ftw+ftw+vpd+3D5+3D5eaz9uHz9uHz9uH///dvv46OJ7o4GiRvazMswO+QbskG/IHfJN3w75 htch38Ad8s3rdcjXjYaKGtnPyjA75BuwQ74hd8g3fTvkG16HfAN3yDeCArhOlg24NromKmSTKsNs e2/AtveG3Pbe9G17b3ht7w3c9t5YilXcyLWKqTshmgnhDw2zl70Be9kbci9707eXveH1sjdwL3uT HuAPTUD5w3WEh21V1/iYCeEPLbOZvQWb2VtyM3vbt5m95TWzt3Aze7s8Yit3bCv9mK0cxVZ1kY8Z kcjaMhveW7DhvSU3vLd9G95bXsN7Cze8t7cN76s9ElR+bb6ZFdjeCKYu8jEjEi5bZhd7C3axt+Qu 9rZvF3vL62Jv4S72Nv5FVqkrecyAhMuO2arega3qHblVvevbqt7xWtU7uFW9m1/uft3UlT1mQHaJ cszW9Q5sXe/Iretd39b1jte63sGt691R6/rWTVj0/JuwulrHDMjWT47Zjt6B7egduR29+3+Ze7Nm zXHkSvC9fkU+SlZSGwlifRhr02jUZm2mtpZpumXzdg1rZnTFpliyUvXrByABfgsOM4JwhuqWlqjL e8/h95HuDofjwHFtO3o31o7ekdvRu/Ad2bL7nWz5+yZh7lS2DOrQjnI8lB9stOCJjRb86UYL/tpG C36s0YInN1rw7HvMylAnYSvDd5tVX4g2kpJZ+8FOC57YacGf7rTgr+204Mc6LXhypwWvXl+61Nes jaAk4X6wKYMnNmXwp5sy+GubMvixpgye3JTB+1dmVXPZ99pbFSUJD4NNGgKxSUM43aQhXNukIYw1 aQjkJg3heYH/FVgVqG4LSr4eBps2BGLThnC6aUO4tmlDGGvaEMhNG4J8fVYFituckq6HwZ4Ngdiz IZzu2RCu7dkQxno2BHLPhvDcs+EVWBWog3NKX7Q4KEaKRDFSPC1GiteKkeKYGCmSxUjxtYmRslWB kvlCaaEWB8VIkShGiqfFSPFaMVIcEyNFshgpitdnVaBkvlBaqsVB3VIk6pbiad1SvFa3FMd0S5Gs W4r29VkVqK5HSrYeByVOkShxiqclTvFaiVMckzhFssQpptdnVX1x3RhKtp4GFU6JqHBKpxVO6VqF UxpTOCWywinx12dVoLZuKNl6GhQ4JaLAKZ0WOKVrBU5pTOCUyAKn9CxwegVWBWrrmpKtp0EtVCJq odJpLVS6VguVxrRQiayFSs9aqL++Vc2gtq4J2bqYxmRTBUexqow/aVUFcaFVFboBq8owolVVhtdl VaC2rggbEsQ0JpsqOJpVnZVNFcSlVjUkm8owslXp12dVoLauCPsRxDSmsCo4mlWdVVgVxKVWNaSw yjCyVYXXZ1Wgti4J2xHEPCawKjiSVc1nBVYFcaVVzUMCqwyjWtX82k6yyf/T19btTMjWxTymryo4 mlWd1VcVxKVWNaSvyjCyVb02fVX+n762bmdKtj6P6asKjmZVZ/VVBXGpVQ3pqzKMbFWvT18197V1 O1GydTamryo4klWxs/qqgrjSqtiQvirDqFbFXp++au5r65ZyQqBgY/qqgqNZ1Vl9VUFcalVD+qoM I1vV69NXzaC2nijZOhvTVxUczarO6qsK4lKrGtJXZRjZql6fvmoGtfVE2A0hljF9VcGRrGo5q68q iCutahnSV4mFqq+qDK/KqhiorUfCbgixjOmrCo5mVWf1VQVxqVUN6asyjGxVr09fxUBtPRJ2Q4hl TF9VcDSrOquvKohLrWpIX5VhZKt6ffoq1tfWLeWsQbGM6asKjmZVZ/VVBXGpVQ3pqzKMbFWvT1/F +tq6pTTRFHxMX1VwJKviZ/VVBXGlVfEhfVWGUa2Kvz59FQO1dUoTTcHH9FUFR7Oqs/qqgrjUqob0 VRlGtqrXp69ioLZO6bcp+Ji+quBoVnVWX1UQl1rVkL4qw8hW9fr0VQzU1hUlWxeD+ipB1FeJ0/oq ca2+SozpqwRZXyVen76Kgdq6IuyGEGJQXyWI+ipxWl8lrtVXiTF9lSDrq8Tr01exvrZuGWE3hBCD +ipB1FeJ0/oqca2+SozpqwRZXyVen76K9bV1ywi7IYQc1FdJor5KntZXyWv1VXJMXyXJ+ir5+vRV S19bt56SrctBfZUk6qvkaX2VvFZfJcf0VZKsr5KvT1+19LV16ynZuhzUV0mivkqe1lfJa/VVckxf Jcn6Kvn69FULqK17SrauBvVViqivUqf1VepafZUa01cpsr5KvT591QJq65TGoEIN6qsUUV+lTuur 1LX6KjWmr1JkfZV6ffqqBdTWHSVbV4P6KkXUV6nT+ip1rb5KjemrFFlfpV6fvmoBtXVL2Q2hB/VV mqiv0qf1VfpafZUe01dpsr5Kvz591QJq65ayG0IP6qs0UV+lT+ur9LX6Kj2mr9JkfZV+ffqqBdTW DWU3hB7UV2mivkqf1lfpa/VVekxfpcn6Kv369FVLX1t3EyVb14P6Kk3UV+nT+ip9rb5Kj+mrNFlf pV+fvmrpa+tuomTrZlBfZYj6KnNaX2Wu1VeZMX2VIeurzOvTV3FQW0+UbN0M6qsMUV9lTuurzLX6 KjOmrzJkfZV5ffoqDmrriZKtm0F9lSHqq8xpfZW5Vl9lxvRVhqyvMq9PX8VBbT1SsnU7qK+yRH2V Pa2vstfqq+yYvsqS9VX29emrOKitR8puCDuor7JEfZU9ra+y1+qr7Ji+ypL1Vfb16as4qK0Hym4I O6ivskR9lT2tr7LX6qvsmL7KkvVV9vXpqziorQfKbgg3qK9yRH2VO62vctfqq9yYvsqR9VXu9emr eF9bd5KSrbtBfZUj6qvcaX2Vu1Zf5cb0VY6sr3KvT1/F+9q6E5Rs3Q3qqxxRX+VO66vctfoqN6av cmR9lXt9+ioOauuCkq37QX2VJ+qr/Gl9lb9WX+XH9FWerK/yr09fxUFtnVOydT+or/JEfZU/ra/y 1+qr/Ji+ypP1Vf716atEX1t3nJKt+0F9lSfqq/xpfZW/Vl/lx/RVnqyv8q9PXyX62rpbKLshwqC+ KhD1VeG0vipcq68KY/qqQNZXhdenrxJ9bd0tlN0QYVBfFYj6qnBaXxWu1VeFMX1VIOurwuvTV4m+ tu4Wym6IMKivCkR9VTitrwrX6qvCmL4qkPVV4fXpq0RfW3ek2noY1FcFor4qnNZXhWv1VWFMXxXI +qrw+vRVoq+tO1JtPQ7qqyJRXxVP66vitfqqOKavimR9VXzSV92Mqf3RpL1nKeibMaX7/+hp2o1p tZ3/+l+3D9AzfLdZgeI6afNyHBRYRaLAKp4WWMVrBVZxTGAVyQKr+CywEjaJabozLz2xRZg4lX+V E/NuXo/W9L0mAyrnpJ3JcVA9FYnqqXhaPRWvVU/FMfVUJKun4utTTwlQOSftTE6D6qlEVE+l0+qp dK16Ko2ppxJZPZWW3x3fdH7L2i45Ek0HAehpOLsDfLcVgUq5ouxtSINqqURUS6XTaql0rVoqjaml ElktlfQ3rcg4VaZ9329FFfC9ViS7usASAmGEk9OYjqXgKFaU8SetqCAutKJCN2BFGUa0osrwqkY4 2dUFslURRjg5jelYCo5mVWd1LAVxqVUN6VgyjGxVr0/HIvu6gKfs5pPTmI6l4GhWdVbHUhCXWtWQ jiXDyFb1+nQssi8LeMpuPjmP6VgKjmRV81kdS0FcaVXzkI4lw6hWNb8+HYsElQPKbj45j+lYCo5m VWd1LAVxqVUN6VgyjGxVr0/HIkHlgLKbT85jOpaCo1nVWR1LQVxqVUM6lgwjW9Xr07FIUEmg7OaT bEzHUnAkq2JndSwFcaVVsSEdS4ZRrYq9Ph2LApo7ym4+ycZ0LAVHs6qzOpaCuNSqhnQsGUa2qten Y1Gd5m6JMyVbZ2M6loKjWdVZHUtBXGpVQzqWDCNb1evTsahOc7fEiZKtszEdS8HRrOqsjqUgLrWq IR1LhpGt6vXpWFRfW48TJVtfxnQsBUeyquWsjqUgrrSqZUjHkmFUq1peX58gBWrriZKtL2MyloKj WdVZGUtBXGpVQzKWDCNb1evrE6S62nq2Kkq2vowpXQqOZlVnlS4FcalVDSldMoxsVa9P6aK62voS IkEfLPmY0qXgSFbFzypdCuJKq+JDSpcMo1oVf319glRXW89vl6APlnxM+VJwNKs6q3wpiEutakj5 kmFkq3p9fYJUV1tfQiDIgyUf6xNUcDSrOtsnqCAutaqhPkEZRraq19cnSHW19SVqSrYuBvVVgqiv Eqf1VeJafZUY01cJsr5KvD59le5q69mqKNm6GNRXCaK+SpzWV4lr9VViTF8lyPoq8fr0VRrU1hUl WxeD+ipB1FeJ0/oqca2+SozpqwRZXyVen75Kg9q6omTrclBfJYn6KnlaXyWv1VfJMX2VJOur5OvT V2lQW5eUbF0O6qskUV8lT+ur5LX6Kjmmr5JkfZV80lftxkTbs6f7+niUlB0NclAjJYkaKXlaIyWv 1UjJMY2UJGukpPsxltHXuKOk7EpQgzonRdQ5qdM6J3WtzkmN6ZwUWeekph9jGX2dOs6UnQVqUKuk iFoldVqrpK7VKqkxrZIia5WU+DGW0dea00TJXtWg3kgR9UbqtN5IXas3UmN6I0XWGyn7YyyjrxfH RMlA1aBmSBE1Q+q0ZkhdqxlSY5ohRdYMqfRjLAPUfBMlA9WDuh9N1P3o07offa3uR4/pfjRZ96OP +tfQLMOAum2iZKB6ULujidodfVq7o6/V7ugx7Y4ma3e0ObCM/Y+mHFj8cWejlPw3OxtVhu82K1C4 tZT0VQ+KdzRRvKNPi3f0teIdPSbe0WTxjo4/JuCA4qulyOXNoADHEAU45rQAx1wrwDFjAhxDFuCY o1YzRMsABVRDkbybQRGNIYpozGkRjblWRGPGRDSGLKIxR+1jiJYBCqiGIls3g0IYQxTCmNNCGHOt EMaMCWEMWQhjwo+xjL6AmjglfbWDYhZLFLPY02IWe62YxY6JWSxZzGLZj7GMvoCaFkoGagcFKZYo SLGnBSn2WkGKHROkWLIgxaofYxmggLpQMlA7KCqxRFGJPS0qsdeKSuyYqMSSRSXW/xjL6AuoiVEy UDcoDHFEYYg7LQxx1wpD3JgwxJGFIW7+MZbRF1ATo2SgblDc4YjiDnda3OGuFXe4MXGHI4s73A8R d8xTX0BNM0VO7QbFHY4o7nCnxR3uWnGHGxN3OLK4w/0QcUf+X2AZFEm0HxR3eKK4w58Wd/hrxR1+ TNzhyeIO/0PEHfPU10DTRJE1+0FxhyeKO/xpcYe/Vtzhx8Qdnizu8K+uEc089fXTpCjZqx8Uhnii MMSfFob4a4UhfkwY4snCEP9NYcjzyl0cC0B9QTUpSjrrB5UinqgU8aeVIv5apYgfU4p4slLEHylF bmu2OqRAPM9mY/huswLVWEnJhcOgzCQQZSbhtMwkXCszCWMyk0CWmYQfIjOZJ1CNlZRcOAzKTAJR ZhJOy0zCtTKTMCYzCWSZSTiUmcyBTXf/ucWbOSxPv6hpTgqnjAcUbAUlXQ6DYpJAFJOE02KScK2Y JIyJSQJZTBKeO8Gk8DuR5TdW8tbbf04ZC6jhCoo8Og7qSyJRXxJP60vitfqSOKYviWR9SXx1DV7m CdR/OUVaHQe1KZGoTYmntSnxWm1KHNOmRLI2JT43ePn9EDQegWZQEeYUsXUcVKtEololnlarxGvV KnFMrRLJapV4pFbZjObz3X8/nwuzrk7jrKakM8nnBGzAPjKOU+wj4/OnfvG/vvzZtrfTfiooKQ5Q 2Sq//PZzM4Eb+OlieTsOU8Rlo/j3r2/iZ9+s6cbyeL18GH7wFYYsLJFXsNOra4sws64o5KyhzNG2 6uOAXSrDSHapjBywS2Vy7PrLm7cvn//jvd9Q7aeMMtgSlc0PKLz7+vLB/Z/P8csd9ulieUzzAYdT mePz25ePHz7c3/3hUvngczjAZxv+/NHeQdtPJbRKjVEluBdfKX/XOdDdxRLuD/xwTBmUyMqg9Nyq 5odNJxg4+VtSiuqJeTfkFMwHklOwMO/4L2/exXBzjuermYXjF86C2Dxr/9uXz+7OxR4ulw+jsfFt Kst3Hx/s7vZzee/24BOMmRxZppiOZIo/vPzBwCnhiqBXE2nNCweOVc0w0rGqa/PCk3G5oHKI/Us2 jJePnz74GL5+qm/u8VphEMjaCsPQaarUNLIyfKtGPx/X6OP31ejnEzV61hXTcvZJqI+oaR6yJlX6 zhOsqeDPW1NB0aypMAxYU4YRrakyXF6aZ13BzFlFKG2oSQ4ahCQahBwyCEk2iCFFW4aRDeLHKNpY V+vKBkGoX6jJDRqEIxqEGzIIRzaIISFbhpENgipk+5T/38sv9tOv8fOXf1/NQlazmMRTfzPmuJJm tGzBvE1Oni9bVNxw2aLiNSkTriT+qDbmumnahoh5lC1P+MO7lz/d572P1zKDUwcEZ81qg1HM6sZw X6vQjmt/l87wkLw2PMz5lQW1mFs6o9I0zU4Epzm3StzNyVjOmlX+ZczzmkyIDfKj/TkGG9992IKU ulnjY5BiK8noqMVKAY2b0/Oyihuel214S5uXVRJxxhpXRDamX9+9lCf84euXlz9nW4ifNnB3ufBo j3miu+d5JihfPABzXpHnzbnAaObcGAhR8tAon3Yw55vNfBmVVeX5+qJNsKeNcsP5YaNc8WXk/d7K f0XYB3v6Jb79COxpu5x5bFfxrzSnjWKFkYxiZ/idqdoyGaXV7zXRSN+Yqu0MZyxK9BbFRhVVLM7K yAGL2nDjFrXiT1nUhiBb1EZz2qJWGMmidoYfEmYUMIrRQn35qHayasQoMs5QjCLjwzmjsFOpld+9 /bf26/vw6T+Ohq/HX5fUiiXMO2QllrTD4MbwQ6yky5BmPg/n6+WjipI3D1iJKCGHYCX3mxSGMqSV 5HgtG2RIGyJHvK/v7Mun6N/aN+8ebKy/XpZauulgJRqyLdIehRvD741pi1V2IY1pjaE3zF/f3Zml vpnlBBL30epCSecWM2CWG27cLDd8PBG8VkTUa/DaHswerLYfDzPegUMBNxg1UX4+FLCzn2xkhmY/ O0NvP+5rujMgczOgGRjQaMuP9WvGIIcMKN7VEUYMKN6lVKMzvyUe51V45pcR2e7z030ww9vP5QOz PpXakEOGSFKP3hgIg+ShLbHelvxokxDm0yJ5qU2dtaUVJ8dtacVboi2tJIcRDdlSQZSIlp/u54/W x2eDur+YORYGzGPlOG9VBUazqsbwI6yKA6saXbBdpxLzgFVtuHGr2vBEq9pIzljViiBa1cYxNO0j rdLdGL6RdOlETbpWhhMmKXuTdKMylvVrukGTdESTdFeYpDttku4Ckzy/LLTByCZ5tCx0b1DGUE1y ZThhkhqY5HCxNHrlhqLkiiOY5IqnmuRKcsokC4JqkivHeZMsMJpJNobfM0nJnaVFycbw3SYpJzA3 tZRqq7VjUTLjSFEy4+lR0tqzUdJaepS05/febzBytYR6KMOhVYEJqyWVa9OgVSWiVaUrrCqdtqp0 gVWd36a/wchW9a1t+msF7Xe26X/f2GvxNv1DkwTz3nEtR/6abmTeu+FIJunI896N5KRJOuq8d+MY MUlHm/fuDIRAZ7+GN19Wiypb3TaTmp6jnFuMmEbTuXmxSwwhnDWpikujJrXhy+aiM6qxijLTH9Yn 87BOcH+lvNNu9amCz5rCBqOYwo3hMDrpaZ6ki/E+Ov1uMLoD9GbzJryNm9XMN6t5FpUtJlunnX76 m3/8l//90/S3f/Afv37+8uHjyy+Z4W17qE8Xi58+SUzKZ5nzgC6ym775+OblfTacB4qni8VN7QFF fq/l6Wy48t+KzUr8x9JNfyi03r59ewPdXylv8xh7J7Kal6e3sf7dNClm+dNYsb3MVBoU320yePhN 2N5XfnIv5T28WP/xzf4wb5fKV4NesVLs8EdkWXvvFjwqKLUAUP7wS7gPANuV9Y4HtxzyCtruohvD 73Zh0g8bOnCYhPbOe3uPeeDd7H0etvfywWM0rDzvQXtvFGU94Zv23v44sfP2fo8l2bs/tHdPt3c/ Yu+eZO/n96JuMLK9H3VTvg/qj62kbo9dJKUVl0Yk6aVR9nhYWBn2+z/jvtN/BPAfIav/MKL/2Jns P3Y+4T92HvefDUvwHyX4gf+U3xD9Z6M46T8FNO4/GT3iPxlG9J/KQEiooaXL3tLL/rnN0heipbuJ bOluOmHpbhq39A1LsnRzaOmGbulmxNINydLPK0I2GNnSjxQhJEtXwNIVq5bOqZae6Jaezlh6Ilh6 Ilt6PLT0SLf0OGLpkWTp5yUnG4xs6VTJCbR0DSy9NCRYLV0QLd1HsqX7eMLSc7o6bOkblmLpcjmy 9PwbqqWvFGctPYMIli7Pd1vbYFRLl9TT/KClm74auOzzXEm09BDIlh7CCUvPtxu29A1LsnR9aOma bul6xNI1ydLPd4DbYGRLp55OiCz9qdvbZun7jFQRLT16sqXnWPv9lh79uKVvWJKlh0NLD3RLDyOW HkiWPlTXzzCypVNPW4SWDlZ42D4j1URLT45s6cmdsPTkxi19w1IsXbEjS8+/oVr6SnHW0hWjWLo6 35Zpg1EtXVFPj4SWzoCl7zNSQ7P0NFmqpWeK77f0/MfDll6xJEs/WpUqvyFb+sCqVAFRLH1oVSrD yJZOPQ0TWnq//irYPiOdxxdgN/OZDdnUZ3PC1GczbuoblmTqRwtS5TdkUx9YkCogiqkPLUhlGNnU qcd7QlPvl17FrFr6MhPXXhPTZFNn+oSpMz1u6huWYup6PjL1/Buqqa8UZ01dzxRT1+eF0BuMauqa 2tQNmnq/SirmqeUvM3GZNC2KbOqLOmHqixo39Q1LMnV5aOqSbupyxNQlydTPN8TYYGRT/2ZDjGFZ zdwvlorJ7GkMcbU0cUk2eC5PGDyX4wa/YUkG7w4N3tEN3o0YvCMZ/Pl9VxuMbPDfbMf3ZPC/MaZ+ p0M1tPx+8VS4fao6E1dPkxBkyxfihOULMW75G5Zi+WY6svz8G6rlrxRnLd9MFMs357vabDCq5Rtq Vxto6v3qqXC3uSpx+TRJTjZ1yU+YuuTjpr5hSaYuDk1d0E1djJi6IJn6+SY7G4xs6kdNdkimDpZP 5W2uSlw/TWohm7paTpi6WsZNfcOSTN0emrqlm7odMXVLMvXzOyQ3GNnUqTskkakzsH4qb3NV4gJq 0mRJfKY4Yep6XBJfsSRTT4emnuimnkZMPZFM/fy2zQ1GNvWjbZskUwcLqOI2SyWuoCZDVq9nihOm bsbV6xVLMXV7qF63dPW6HVGvW5J63Y6p1y1ZvW5/hHqdgRVUkY2rmjp1CdWS5euZ4oSp23H5esWS TP1Qvm7p8nU7Il+3JPm6HZOvW7J83T7J13/Ty8L4Ml9YiWRgQZVb3zYoURdULVnNninOGP64mr1i SYZ/qGa3dDW7HVGzW5Ka3Y6p2S1ZzW5/hJqdgQVVvu/dZtQFVUeWs2eKE6buxuXsFUsxdXcoZ3d0 ObsbkbM7kpzdjcnZHVnO7r4pZx+P7WBZNe1TVUZdVvVkVft6tO53G7wfV7VXLMngD1Xtjq5qdyOq dkdStbsxVbsjq9rdN1Xt4wYPllXjPmFl1GXVQBa3Z4oTBh/Gxe0VSzL4Q3G7o4vb3Yi43ZHE7W5M 3O7I4nb3TXH7uMGD1dS4T1sZdTU1kjXumeKEwcdxjXvFUgzeH2rcPV3j7kc07p6kcfdjGndP1rj7 b2rcxw0erKmG23SVuqaa6FL3dEbqnghS9/R9Uvd55inM2OCV5wcGX36zvpW39uMb/1LOhP9iP1Ub erxWngTv+u/vLH/YHlj8Nb7/8nnD318pr2LG5qtKg7/SDe+Td/VZbj/8sZzTjS1PlUPHt8+33WVt p/fp/nPfX/9jOWDngCgbwv+3/tGb9/V77z8W2NE3Tts7yX8VP336+vHL2SB0uP3A07cf+JHtB560 /cCPbT/w5O0H/kdsP2BgnTvcSgi0dW4rJuo6d6H47vBT/ng0/DQsabw93H7g6dsP/Mj2A0/afuDH th948vYD/yO2HyxgndvvO+IZbZ3bipm6zl0oTpj6PLzO3bAUUw+H2w8CfftBGNl+EEjbD8LY9oNA 3n4QfsT2gwWsc/tbkzbaOrcVjLrOXShOmDobXuduWJKpH24/CPTtB2Fk+0EgbT8IY9sPAnn7QaCe xwlNHaxzu31PPKOtc1uxUNe5C8UJU1+G17kblmTqhxsPAn3jQRjZeBBIGw/C2MaDQN54EL658WDE 1PuVbSn3UsFCW9nO5kNd2S4UZ0x9eGW7YSmmHg93GkT6ToM4stMgknYaxLGdBpG80yBOz5IOzvQ0 3SQdvy3FePWyzPlffVcyk3pefyNi+ftlub0NPentUgEIfcJF+hVxKffp7EJbEbeCU1fEC8UJF+HD K+INS3KRwx0Kkb5DIY7sUIikHQpxbIdCJO9QiOLZRdZoPt31eW5XJh1M9Z7P996T3OoQ8/1eNR3t Clv/4ISP9IvoUuzz4IW2iG6FoC6iF4oTPiKGF9EbluQjh1sbIn1rQxzZ2hBJWxvi2NaGSN7aEO2z jxitnx2knFqnZ1f2LDysvLDNB7zQD5s5tZ2b18x6NidcpF92l2KfPy+0ZXcrJHXZvVCccBE5vOze sCQXOdwSEelbIuLIlohI2hIRx7ZERPKWiPgjtkQs/YK75Lcu57QFdysUdcG9UJwwdTW84N6wFFNP h1siEn1LRBrZEpFIWyLS2JaIRN4SkZ62RPwmk1rc7W3kH1XZqF/+lXcziqRKel0ux3B7DzLJKNfL MvITrtEvzUu+7+xfaEvzVmjq0nyhOOEaenhpvmFJrnG4hSLRt1CkkS0UibSFIo1toUjkLRTpeQuF D7M2YXp6H8UNAp/DwqfufUxTxpiF6e59rL/RMml80OFO+fzBdhzASCttdEuPmbmYlMgvDtyn0nUP gIly7opE92l0HWae4uSnBO6z03WfjRsWhTcI0+g6jGWpLOChz9boOoyLOqoFPoNG12GYYdOUGLpP o+ufAZOa48/W6Lr76EkFLhx8BpWufz9+UcrBz9boOsxkZeAa3qfRdZ/N+yRTmhCm0fX3EXLmDj6D RtfdJ+Zv6qwDmJ2ufz8pOek8uk+j6/0nezXLgxR6P5Wuw6gclyOb0X0aXf/cwuS5ndF9Gl2HCSl6 biJ8bpWu9+0YlVEQ0+h6n5sYnzV61jtdhzF6SZkMfZ9G13+fzOY0tING12FEYvPM4XNrdP37KWVq 7RGm0fXvJye7Thr02Rpdfx9nk1/gc2t0/TNYRA5K8LM1uv5Zc868gLG30fV2EOK0MIbu0+h6nxNW Twk+60bXP7c8pzbCIkyj6+8Tk/P4WTe6PsZPs5MTfAaNro8HwQWh4H0aXR+rnJiVQfFtp+tjb/Z6 E9E73em6z7ZI401E72en6zFGc4PH7UYHxoXZOQW/T6PrY6LLObdDdrDT9c96cSFGC2NipQO2wydd jpwFz7rSgfimvFMwXje6DiMX65jXOL5tdCD2+lkIlLvsdP198ljGHYexqtIBuy4xDI4lja73n6T0 7NG4vdP1z00zJyPMDxpdn7sYK1wU8LlVugN7Q3660wHbiTYJZAc7HciRNPczzHsbHXgGS7YPlFvu dCBeqzkP6TAmVjqQuwhtFjjONbp+DE5Tdnr8fipd76cpsKIZRWNwpevfaVBpZgv000qHP1tCPrfT dZjEhHMW5+SVro8HS84aHIzXjQ7fR8P41uh6P41cCa0O76NBfMujs5hDgH5a6UDuojMGx8RKB+Kb F9k+EKbR9fbmeCi702B82+j6Z6240I5De6t0ve1kEGMaPutKB3IXE7SHc8BGh+Y/bhYwt2x0IHfx UkeY+ze6/p3alBYN32mj6/OD5LmTMMY3un6uabUPAX22na6PO5NdhIR5fKPrbSfnj1LD99Po+pxC zfMsYDxodH08sJNjCuawja7/PtJLZaCNNjowP13iwqEdNDow346OTXBO2+iAjaqYzQrPtze6ftz2 M/cc5geNDs/RLXxuja5/P4lxA210p+ufgUyOMQHfT6XrfU4kGz2KvTtdb285VC4zzEMaHbZrPP9p dP19/JwNC46Nja4fF5LUasH3qXQgjgY+YZ9rdDgnN3DO1OjwPAvnyo2uv48TxnmYizU6kIeoZZb4 PpWuHxvzd7UW2nWjw/kBjPE7Xe8/oczBYO7f6Ho/XVgQBmIaXT9HD/M8BZj7N7o+7hjtfYBzjEZ3 kPMpGHcqHfCfnAt6mIc0uv4+Uvgl4lpNpettJ6g8n4KYRtfn5JzFRUK7bnR9DGF8Srim2uj6d1pq BAnn5JUO5ORa5w8AY2+l6+0gMsdmmO80ut4OtAsK5/6NDtQGjSvyIGQHlQ7Yjo7CQT9tdGAuMwVj Dmxno0NrBSpnqng+t9GBnGLhfIE5UqPr7U3k0D/DeN3o+rHRcjdHWN9pdCjn82vPEzA2VroeY6Rd 4PrCTtfbtTTOORwTKx1Y+1Ay4PGn0fWLTNlCBJ4HN7r+PknkuS6y652uf9Y8269FOexOB+7DmZ5h Dtvo+jFYxbVwCO+z0fXP2moZJMxHGx2o2VlpHX7Wla7/bHPp6QrznUbX5/E5tAQG89FGB3KKWRgO /afR9fdZ+GLgs97pekxckoBrbTtd79tmtg7P5xpdbwcuf08Oc6RGh2tCGtUpdjpQww9OxeMapAZ1 CrVIxaC97XT991FW+gDtrdGBtZwc/jWe01a6fszSVhm89tHo+neaFutmmIc0un7c9jm4CDjONbre RllSM85DGl3/fYSTNsGxpNH1vj2XtRS8jlHp+vswzZYFzzEqXf9+8hw0WjxvrHQ9hi+lqAtrG5Wu H+ci10rDZ93o+ljlvSvSVTTOVbrerrXWHOZVO13/2aaoZgnnC42ur1fl/2EGry9Uut63ReASrtPu dP2z9k45DWNvowOxSkW3wJyi0aGaXcrpLZwDVrp+DJ5VkgnG3kbXz39Y0F7D3L/RgRgyhTzMwPlP pet92wgX8FpBowN5IjcWr0k0ut5P1ZSDJY7xla6PIdFzy+D3aXR97OVBLB7W8Btdb9dLsHPCY0ml w/PTAH2h0fXPIPqceMP5T6M7qJPjGmSlA75gp5hg7t/ocA2FIXvb6XBtw8Bn0OjA/EcwnvD7qXT9 fG4O3sC5806H8tGFGVxLq3S9L8xBpAnqQxpdf58ck9ejLoEvVLr+PiIIDtdLdjqMwRqMRgcwQuYU /xCDNBiTMbNS8Lk1OhBHneYB+mmj62s15XtinUOjA3VLZxdso40OzE9j9kf4fhodsoOF4/pBo+v9 NA8WjB/ZwUrXj8H5e4qA61WVDjwDP084V250YC4TVJ6/42ew0fVx1OS0TsF5VqPr406Sni14Dljp QK2TKwt1DjtdH0dzNswDriNVuv6zLVxKhXO+StfHxMhyDo3zg0rXx0QfxYzrIY2utzft4yTh+2l0 /TMIyxwsjPGNDtQcSloDMY2ufz85Ukx43bnR9bYTk5EBxvhG1z9ra+XsYc2h0fXv1GkdoL3tdGAd Q0+LgONpowPv1IRlgfG60fW+LU3OhfDae6UDcWdmWsFcrNGBnE8oFbAWtNKhei+bIxwXGh3Kr/Nn huvojQ74nIgcrzs3un5ckFHPCuYUja6P8SGHCQ7nWY0O1NJKewQYDxod8NOceMM54E7XfzaXcpKG 58GVrl8v0XzWEmIaXT8GB86kg+NPo+t9W0s+MXifRtfP59KiJJ7XN7re3haVZqwLaHT9c0uGJYtr qpWutx0V1eKwBr3S9ZglzBbXuBpdj9FLMBbGkEYH4puc1QRzpEbXP2vGvY3Q5xodWpPgaob+0+j6 /MBq62Dtdqfr896ceEeJ4ttOB8YFFxl8pztdbzsq8AjzxJ0OvB83+wnW3xodmjvnuR5eB6x0oHab 5+JwfXunA5/N2AXriBtdH3dK4z2of9vpejuYuFN4vb7RAV8QXOB1mUYH5uiBeY8/W6UDOYXNySr2 7UqHMR7WLRvdAQb6aaMDvqCmEOA7bXQYE3EtrdLhz2ZQnrjTHTy3g7xqo8MYrA9pdBiDfaHRYQye nza6AwwctxsdfNZxhrGq0R1gUN6702EMg2sFje4Ag8bGne4Ag+dZlQ5j8PpcozvAHGiYNrq+7sJN mbcd3gfVvDcMfKeN7gADbafRQYydYDxodD1G+FlgjUyj6zGOW4bnC42uHxfMItkC/bTRgRq+ygkX zF0aHcZEON9udL3P2fxNYa680/W5pU+u9F1APlfp+vzN5pCM9RSNrr9PdDl7wnWXStdhknQ5FcS6 gEoH6vGzsxbWkRpdP54GK9KExtOdrs93HFcOrxU0uj4P0TwtUE++0/XzhcX7gDXOjQ7M54IrTWrQ fKHS9XlIzh4XXItudKD2ZMMkIabR9d/HLFEr+KwbXW8HmuU3jvc8VDqQw2a+CWvDK91BDIE+1+jA nDZnnAH6T6Pr64l5XjTB8Wen6+8TlxInYN2/0uHvg+NOozvId46/D4g7ZVEx5qh8mCNF8H1WsoTH 00p3gIHPrdFhjIV20OhQTJQa75tqdBiDdSiNDn82f4xBOpT1Fwrm8Y0OYzQcGxsdxmCdaqM7wOCa Q6U7wMD6aKPD+c4En0GjwzkF1I/udPizcWw7le7AdvC+90qHcyQP53ON7gAD11gaHcbg9ZJGd4DB ethKd4A50LpvdBiD9441ugMMnP80OozBdf9Gd4CBuUujO/BTvHe50h1gYO7S6PrcxbsYD9beKx3E JI5zl0p3MC5Au250B3k83sdS6fB9EoyJje4AA9f4G91BDEHvZ6fDzzrhHgOVDj/rCWuCK90BBvpc ozvAQJ9rdBiD64mN7gCD96NXOoxh0Oca3QEG+lyjw5gF5qON7qBOgddlKh3G4P4Hje6gJoTXqivd wX3wvpxKd3AfaAeN7qCOhPUule6gXgXfaaPr45sVUfnjZ4D6hzhj5zwPhTGx0vXzn4Vzj3VCjQ7M zZxnWDvZ6DBmgvdpdP28JBitGO7RUen6WKWjFFgX3ej6NQmXHLdwX3Wj6+eAkxZ8wnseKl3/DCY7 Sw/XPhpdPy74xITFz6DS9TUHZYQ+GIMrHagF5BeO/afR9XbNbZwjeqc7XT83K1LZg70vla6fBzPP I4f21uhA3YWp2cPn1ujwZxMH+083uv795LThYK260fW+rWfrFrwHt9IB2ynmAXOkRtfHA5GnBA7m B40OrMuEYHAvqkbXP2tX9sPDuUyj6/2HSc4YHBsbXf/cAvdTwtqvStdjuFQS90dqdP07XSJjeF9B o+vXf6RjxuP5dqXrP5uLXuOaaqPrMDrlQK7xXLPS9d/H6jz+QbtudP1nM86s3azB96l0IB4kFSyu QVa63t6MlE7DWNXowDvlYsZzwEaHMbi23uh6u9aOO437U1Q6jFEw32l0BxgceysdxuD9tI3uAAPX JBodsjcTcR+ZRgdq0SHNEcbERtffJwoWAnwGjQ5/Nrj3f6c7wMCxpNFhDM6RGt0BBu+9rHQHGNxn rtJhzMH+rEp3gIE5bKPDGLx3udEdYODcrNEdYKDPNTqMMdDnGt0BBtpbo8MYDX2u0R1goM81ugPM sf+gvhGloCmxjqvRHWBgDtvoDurKOHepdP3YyMpu8OM6bEIa2skof7AHt9L1z42FJSwQ0+hATJx1 gpqFna4f56wTgsPaU6Pr5yX5E3ucizW6DjM7c9BLdKfr7xNVVFAXsNN130eUU4xgXrXT9fmbNMvB vKTR9fdh1h1oQRtd/05tjsh4z12j6+cL3k2Lhs+60fX3CSzP2vA6YKXr8+vo8jQPx6pK179TY43B 9tbo+vmpnhPnWPtV6XobXfL/4rX3Rte/05xxJdxTqdH1vq0Sm/C40Oj6nC+U1jdYE1zp+mddhGQe 5vGNDuSwXi9YX9XoOgz3ngnYH3an62NImpXF68GNDsyZlqKHwhrNja63AyGilXC9sdH1z01Fw7CN Nrq+7mIcx+vOO12PSWrSWAfZ6PpYlfh6FBLCVDoURycFY/xO1/tcadwXMKbSgbkM93kOCn2u0vXv x3DNObTRRofmweagv1ij6z9bfp5+OZoHG9RfLAewTIb7RjQ6UNsImsNnsNP1sTeU5BvWuBodeD8i 5AklrNVUuj7uTExIuO9wp+vfD1v0pPCeu0rXj1nOW4v3SDe6/p2KIk6HOVKj62OVWJY5wnjQ6Hqf YzkeQQ3GTtffh4WE+z3tdOD9lIMk8Dy40oExyxplYG+tRncwP4V5b6Pr7UCkubRTOJyfgryXuajM BMf6Rtc/A8VFXKAvNLp+jWWZ88iEc7FKB2re1suDtcNK138fa5WBmp+drr/PrLnD66eNrredOHEJ 9zfudH1epSRPeP7T6ECMn/3BXuxGB9+pxn00G12fh0iX0wa8f7vS9fEtvzmO9S6Nrv9sWtgyoKH4 Vul62wk+KTxnanR9jE9eHewvaXR9jJ9zEj3j3kCVrp+bJRvlgWah0gE7SJHhvRWNrsdoFaOC36fR gTrsxBNeM2p0fazii40wP9jpgJ9yuWBNSaPrY0gOfYzhfKfS9fmbsTzCXkc7Xf9+8gcWBubxjQ58 NuYO+tU0uv65qewm0A52ut5/UmlhiWN8peu/j7eT1DC/bnS9b6c5Ro73z1W63n+81MnhfdWVDtRq FhUWvJ5V6UAMcU7MeH5a6frvo7JR4XX0Rte/UyaDmvC6TKXr4wGbYpgP+nVudCA/kIlZrEGvdB1m KR0OBJ7TVro+P/CBMXzmSaPr8zee53q4ftDo+u8TFD9YY2l0/Ts1eeoKdYM7HfAF46KBz63RgZrD 4hLWwzY6UNuY+Yz1VY2uf26y9DzE+50r3UHshTGk0fVjI1eJOzinbXTADriS+D6Nrs8ptAszXs9q dP33MYuKsAfETgfijrUW5/GNrsf4wMWC17crHciRvDd4/2mjA/kOD3hNYqfr48HicqSAY0mjA/UD 5SSeLzQ6UA+JicN9bTtdnyPF2XiscW50wHZml7DWvdH130fIqBaY7zQ6UAsIs8T7jBpdn1/nTCNP wuDYWOn6MYupMAmcX1e63q61U1binK/SgXpv/gh4btboekzkLr89XO/d6EDeu3g+wblzowPxWoSI fa7RgTp59DPcy7PT9ZjMxuFZFzsd8B+fpx9YW1TpwNxM8GTg+2l0/WfzPA/n2H8qXf+sy/Zkhs+b qnT9Z7MyzyXw/LTS9c9A5TcnYQ7b6HqMiElyOK9vdL1vq/w08dy50fUxcVE24j7bjQ7MS7QVAmt+ Kl3/3PySLR6vUVa6Ph7oYIODdctG1/v2HM3MsQaj0vWfLVo/4T4lja7P32YhdcB72CsdqIv5xcL5 3E7X24Gflce9Whod8IXFHJzH0uj6vIpNeZyBvU0aXZ8jTUu2XVhHanT9fewSJV4HbHS9vUkeLa73 Nroe4w3Pv4T2Vun65yZZzp/wWnWlA7m/lRb2otrpwFxGCbzPdacDaznMM47rLpWuH4OlMAbvB2x0 /Xi6zI5BffxO1z83ZkOwENPo+tyfqzwPxz1yKx2Yz+VZgcd1pEqH5zK4Z3aj6z+b8xrXOnc6MEdn 3uP+sI0O5714z1Cj69/plBifsfar0nUYvjBtEtbqVrreT7mQIsFxodH1sTfMJk647lLpwJyJqxyW 4Xyh0vXPzfmcR0M7aHRg/pPyyITnC5Wuu08ySTl8Vmaj65+bKtIZqEdqdCA/UPPscB/nSgdsdFks HrcbHajVCB4EtJ1G1/uPFVziPVCNDtxnikrBvTyNrh+D8+2VOtCLbXTY5zzeg1vpsM9h7WSj623H 5Uko3hPZ6PB94F7FnQ7NAZnG9d5GB8aSyCx8Bjsd/mxYH9/o+rE+SrXgM9EaHcjfUhI47jQ6jIl4 j2el6zHSWj8fnKO20YG8V1iD62KNrvdTHh23OPZWuv6zhdKWB76fRgfi6JQzYjxuV7reT11ZI8Rr epWux0jFJ6gT2ul6jJjDBM/B3en652acS3jNtdGB+YJRDPdPbHT9Z1PRSXzWUqPrx9OU8tQdf7ZK 1/tCWEQ8Ootxo+tjomVa4n0Sja7PxeQkDrT7ja63HWFdxJr6RgfiTp6BwnnWTte/H6fcwfpCowOY FJeE58GVDowlKky4116j699pOeAGn+XT6PpnoOO84LM/G11/n1LIwvOFRgfWPqI+0Kk2OrAmkSYp cE+/Sod94eCM3krXj3Nm4g6e/7PT9e/HuXlJuDdQpQPzEu+kx/1HKx3Ie9Uk8JpEowO1gDQL3POq 0YGaah4AYU1op+vtukQJ3Nuk0YEYEnLkw2evVTpg12U6BeuwjQ7MG2cXYb+Nna6PO54tDJ9l2uj6 2gaLZfESxp1K1+dvPKcAOBdrdGA+l1813p/V6MD6j5i8g31kGh2IB3lqCHPYna5/1i7MKeC1tkrX +7b0Pud18FlXuv65peiZxbr1Stf7djmu00I7aHT9fUwyXOG+UpUOrMtEdqD1aHS97WjhAxwbdzqQ V+VpOO4B3uhwjMfnKja6vs43KblEmIc0ut7ecnZrHdQJNbo+H83TIg3PZ9rpwFqB0TzBeWOj62sB lhmFzwRodGCeZbTH9d5G179Tz/mCe0g2uv79mMQC1qA3OnwfievXla6fo0sbNO4f0ujAeJpkwDXV Rte/HxaUOch7K13/DLyfnYC5WKMD44LNgww+V6TS9WOWyANgPBgXNrr+s9mwTHCdaacD9VE2MY73 uVY6MC8pgyDufVbpwJrE7BaoJ9/pel9QdsJ75Xe6/rOZJZTjkZAvVDrgp34JuLdJo+s/m5Slvwxe 09vo+phoeWACn4td6cCYJUs3ATjXrHT99zGz0geYSgd8u7RgwOdnVboek3JKjGv4jQ7MZYw3uEdU o+vzxHICK4PxrdH1/hN08HjfYaPrcz6Rxxi4L3SnA3WxkGMY1lJXut4XZmcV3FO80/V2kCae8Jnq ja7/bJYLK3CvsEoHxtOYBDzjeqfr50zGMw/XF3Y6NDfzC+5x0+hQHI0c92VrdGBu5vJrw725K11v b35yB9rjRtffRy/c432hjQ7YtVcOr881uh6zyIOzmne6fgwWzOZ/IabSgZyP+bJVHY3Bla6PO5Px E+6D3ujwc8N7ihvdAQb3D6l0Bxjc67XSYQzei93oDjC4D22lwxi8h73RHWBwP7tKhzEJ94OsdBAT cG+GRocxM95PW+lA7m/dgnXeja6fy0QZDBy3d7o+JjIlmMZ7VitdP9ZrnyTU/Ox0KKdQB3vLGx1a Y1FqxmcGVTrgc84oCb9Po+vnCyzPPuC8cafr36mP1uPaRqPr6xScOzPhd1rp+mfAtbAB7wOrdHgO qHGtptL1zy2bu4b7NXe6fpzzxi94nanRofoBd3jNqNH1eUjZb8AP+u9sdLgO63H9utL1OYUSwuNe /Y0OfLaixMT9LStdP26bmeOefjtd/9m05gd7FRtdP2blAT0d9SDa6MC8hGuH97k2ut52Fi4V7tXS 6Hrb0XnyDnvb73So7u8E1pg1OpD3BrscnPFW6YAv6IP+ITsdmAMKwXE/rkaH5j/GJbzfrNIdYHAO W+l6O5iUjvic1UYH1n9Kczjc/7rSgTxRyzyg4/WfjQ7EKiMCPme10fX34a4Ui3Cs2ugwRuKcr9Lh Z43Pm2p0fV0spCAFjCGNro8HRWm/wPpOowP5QUjCYN1TpcMYPGdqdP3YmPPhg5pDowN2MIsUcA+8 StePC06UTg/YDjY6UHPIs2Pc96vR4fkC3lfQ6ECt0y8Bn8HX6HrbkcZ53N+/0YHaU8qJHV7HqHS9 7XCvJmjXOx3I+ThzCq8zVTqQW04ySagFbXTgWSthpoM+JRsdGLfz5B1r9xsdqq0bBs8l3en6XEwL yQTWbVS63kajEgrPARtdP87lL8lxfGt0oB4idTTw/TQ6VFcWCu5V3On6+0ySR7z20eh6jMxk+Byb Rtc/N5PnyAe9qCpd/36WJBLDz63S9eN2nlFPeL2k0fX3yXZl8dys0YE4qpXG+8AaXe8/y+zyrAXH 0Y0O1ISMLb9H/lPpQD1+thaf8dbo+vv4KMSEdQGVrr9PVNoKOHdudCgnlxKfJdfogC+UXSQHvRA3 uj7uuJTwvo+dDtTJpRcKa0oqHX7WcC/PTtf76SJ9OqgRV7p+DPazM3iO0eg6jMsGZQ/G4ErXjwvJ zQfnvjQ6YKPWJ9xTttEhe9MLPqu50YG65bJIWKfY6XrbUdnt8Xlgja7HmJzW4fPnGh2+D+5b1Ogw Bms9Gh3wH2YMPGt2pwNzpri4I23RRtf7guUzi3j/XKXr58FJLVHjc3ArXe8LhscFnpOy04F1mcVF fM5qozvA4DW9Snfw2XDeW+nAuOCMt7Dm0OjAfEE6hdeQGx3G4LlZozvA4D2rlQ5jGNbQVrredpw3 acb7PirdAQbuuWt0+FkfaGQqXR/f5ikEfBZwozt4pzCHbXRg7hy8Edh/Kh2Yo7vZWNyztNLheODw 3stKB2pPXKt40K9zozu4D/TTRgfidRDiQI9U6Q5iCN4jUOnAWJKswD1yGx2Oibg/RaMDuWU5xwCf 8VbpDu4Dfa7RAduJfMK6jUZ34D8wF2t0+D4H/bgqHb4PXjtsdP0cfebKw/MXdjrspxx+n0bX50iJ aQXz3p3uwBdwDb/SHWAO9nxvdCCGTDLOWAta6TAG6/AbXf9OvVgmfJZ2o8PPzeD9MpUOPwO8N7bR Yd/G6yWNDtWrvMM98BpdP99W3C+4p2yj6+3Ns/wbWLdsdL1d58wuzHA/YKPr7xN9mvG40Oj6eeMs fKnUo/tUuv4+WhgHz9Pb6WC+g8/H2OkOMDBeN7o+jiouJrxPotH19Z2ojYI9bna63t5CDE7idZlK B2pCWicDfa7R9TlSYkwu+BybStc/tzyX8njcbnT9fE7zJHF/2EbX24HQIkhY32l0vV1zZfC8ZKfr n4HPn1jAz9boQM1Oe2/gPLjRgXlwBuF9LI0O2U5MeO2j0fU2ahVz+HymRgfWf9x0cH5jozt4P7h2 W+n6eGByqg5j707Xf7agF5nwGn+lO8DgnLzS9XUXm1827hvR6A4wcD7X6Ho7CFPyGtaRGt0BBp+t UulAvhPyfB/OzRodxkBt607XPwNhJ6/g+Y2NDmPwHo5G19uO93P5gyMM2MOxYeC58jtd/6xLJgjn tDsdxuA5baPrfdtyr2HP0p2uj71CBYH3bzc64HNzONjz3eh637ZFH4j7W1a6Pq9aF+4O+q5sdBiD +4c0uoPvA/OdRtePC1EYAedmO10/B5QqBrw3qdEBe7Mh4TXXRofvg3OxRgfsraQnWNNY6XB9FJ+X 0+gwxhz0zN7oUM0hMtz3q9Ed5P4H+z42OowxWLdR6focSZTeIngvXKXr32mQCz/oX1XpQJ3czwru b9zpMEbiHKnS9f7D5mjwmlGj6+cYxusAa087Xf/cbJ7mTLgfZKXD8Q33YG10OL7hfjWNDtS8S76B n0Glw98H72FvdP2YZfKbw2d/Nrr+PjLK7ItwnKt0/ffhwTqs42p0YDzlymGtVKPrc/LZCSGwpr7S 9fbGY9mbCnPyStfnFEbZBesGG13/DLSyAvZ13+kO5uh43azSgZxcM4tz8kYH6mJOH/RHanR9Tm7k kjNIXH/b6EBOLo72fDe6g88G9zw0OhATrTs4V6TRoVqA4Vhn1+h6345TmOAZOztd/04Fnw725De6 /vukoOeDvqCVrn8/efCzCZ+zWul6n3MqLViD3uh6DFsMx7rORofmZtmHob01OjBHV0Z7qHFudH28 ThkA18B2ut5PJ6v0gue0lQ7kbylnt3D+0+j6WLUkdXD+QqPrn3WYFw3Xdne6Ph5EFgw+N7bR9ZiZ TxHvGWp0/TvVbnEGn09b6frYO6XJwVrATtfXNsoBhXi9sdH17ycqZnF9tNEBnzNqnmH9rdGB2kbI 83CsR6p0+Lnh3s+NDuRvmgkD8+tG1z/rkNgC+3HtdH0Mccwd9f2qdGCtQDl/cAZ5peswSw4tbobz 00bXPzcvcqIGtTiNro+9Njp5sI+y0oG5TE4dJtxvsNIBO0ilaobnTBtdP86F6JnE+rdK12GSL0o/ vHe50vUxxJdwCT9bowP+Y9TBO210/XNj+engvnmNro+JduL84NzlSgfyqpxFwzi60wF7kzlvwPq3 Std/HzfLBDXBO10fR12e60bcg7XS9XEn5umHhPO5RtePjUrHBffDb3T9nCnnQRGuSex0vc+JcjoS 1kFWugMMXm+sdL1d57CnoF3vdAcYGKsaHcoTg8HjXKM7wMD1rEbX24HVZYvlQW650h1g4PdpdL2f Ord4GBN3OozRuFZT6fpxIWR7h3uXd7r+++hSlcL7MSodxuAzyBtdH3uVnR0+i6TRYQzun9joUE6h GRyzdjp8H9xvsNGBuqWOkePedJWuj4nWe3+wFlrpQA4bosXzuUbX5xRzmvOnwDXIja5/BnJKAteV Gx1+1ngfZaPr47WSXMG1gp0O1CmS1ArXOitdP2aZcpoD1gBWugM7ODhfZqM7sGt8bnml6zEuzwmw RqbR9fZWTmjEfVsbXT+WcJlDMtacVro+hsSgFnymRqPrPxtPgjm8plfpetuxYp7h2ZI7XV9bF3ku gdf4Gx2Ya+YwEaCfNjpgO4brGc4bG11/n5zSLBjT6A5qKPiMg0oHxgUlGK77N7p+DNYsqQU9g50O zDVnfVATanSgxjU7g3sZNLo+HthoLY4hja6/T+Lc4j1Dja73BTUbjvtKNboeM8U8COJzbCpd/06t thLOAXc6UE/Mj4dhbWul6/Md5heHezc1uj4f5TLP82DcaXS9XXulA9wjvdOBGGJmUXqS/eF/fbL+ zfuff/If3r2z78NPb8Lb+NPHN+Gnef7pS/knf8PF/PQl3NDMLUbyfP2nv/nHf/nfPy3yb//gP379 /OXDx5dfMsXb+Gn7HE8X/zj9xlQ/o7B5Al1O8nrz8c3L+3dvHimeLmaKxWKK/KX+8OWT/bjhyn/L fywk/uOUPaDQevv27Q10fyWDwWy7Yf/+7/9+vUX+3bz83U+f3nz86f9qf5f/o+fSMsry/JvPd79Z GVQqG07zbxz6TZh+ytzlcb6UF/Fi/cc3+8O8XSpfjYOPt1Hs8EdkBln0PAooP//P/pcY1j/8Ejbg /ZX1jge3LLrD9OHTn17ib2++bND9x/IcGYalpcLyk3z38cPbN+/jHfh2MVPEY4bbq9jfw+2x62yk z8/69p+4Puvv8gA+dR4gSmOE6gGK6AFWkz0gZ6ff7wFWj3vAhh32gCj1MkEP2H5D8oBGccoDNtCo B6zo8x6wwkgesDPcvQolPJtvbyP/KMsIlf8Vy+1V5B912i7reHsPSjjH18vO+BOuMfeDgxWyuYYm uoZTZNdw6oRrODXuGhuW5Bri0DUE3TXEiGsIkmuIMdcQZNcQT64hRZruXCP/WA7Nmsq/dwNG+THU y+72HqSIOSssl2PgJ1yD9a5h4j5qGKJreEl2jTU+fK9reDnuGhuW5Br20DUs3TXsiGtYkmvYMdew ZNewz64xR53uXGMu7SrXBucu3LnGHPMrXC/HdOcamVJtl6U44RoLcA3FqmvwiegaQZBdI4gTrhHE uGtsWJJrpEPXSHTXSCOukUiukcZcI5FdIz25htAshNvbEHpZbLbe8q+5vYr8o1PbZSdu7yGDmVwv M3Zm1ODANfbZNp+JrhE52TUiP+EakY+7xoaluAbnR66Rf0N1jZXirGtwTnENzodco9Qqaa6xMdy7 xhyVu3ONPGqU6Xb5V925Rh41xHY5LXeuMa82VC5zdcI1RO8aep+Gc0Z0jbSQXSMtJ1wjLeOusWFJ rmEOXcPQXcOMuIYhuYYZcw1Ddg3zn1iIksADpj1vWigeEKWdGM0DVorv9YD1jwc9YMeSPACXYrff kD3gdCl2A1E8YKAUu8LIHvBUiv0tP3Flbm+jnHFSUpjyr769ivyjnbfL9u49rN3It8vCnHAN1buG MnvexImuMc9k15jnE64xz+OusWEpriGWI9fIv6G6xkpx1jXEQnENsQy5hliorrEx3LvGwt3dbJst grti7ItY7l5F/nEtxpZ/711j4Uasl7k5M6XQwDWyMVbXEETXYBPZNdh0wjXYNO4aG5bkGvrQNTTd NfSIa2iSa+gx19Bk19BPrjEvS5C3tzEvvNh6Tnfz5OP2KvKPZtkuK3Z7Dxms5vXyoqYTrtGvbSt2 m22T1raLuSW6a6QzrpEIrpHIrhEOXSPQXSOMuEYguUYYc41Ado3wrSnF/pS/39RFv4gtQzksYHsY n//85ov/5f7xbFfK4wnIMrVk5Y1ky7qH7j9m3KyWAxzhpUg29FIko76UjYHyUt5/+bS9FNZeyiy6 QrjlYbHf90m/8xbdnNFyFwffe0Z7pQr63HuvOJmD4ZdfPkUbXt5++FCD0/2VjDbmAHz2tW8wymu/ MdyFxZT4/ShVDiQp407+925uM09uXS7J/94vlyQtyx9P+V99wki6/M2uSeN1RsJEN3uyho0Gh4zO z2zISApu2EjSgBRjgxGNJD1LMa6IDXzuXrtTvjzVy147n7v0J98iJ0CDr50v5eSt8699xY2+9hV8 /rUXGO21N4aLXzvrxunSk/5Kb+esE+04Po97O18YH3vtGTf+2tn5xZANRn3t7HkxhPlp0ZO9vf/f uFhYksk5zYpE+K4WzEtP0yIpnPXkxXy3KuLL4X1KT1rZidnpjMl0SpP8Psu+wwtNpktUHJ/Klqlh k4mDJhMpJnO+RLrByCbzXCINsQhaQryZTLty95vPh79x+DcnTKZbZs7vc47j73NZxt5nxo2/z+V8 XW+DUd/n8lzX+6u/zy6Bc7pEmuH3ydnY+8y48ffJz0/uNhj1ffKnyd1vrgiq2XS3iiHlFCYh9Cwn we0k71a5TVQulbM55knMd+m+FdMySVYWus3EzZmIDnK/ZK5NAkDulygRgPtBi/EUi/FjFuPJFuNf VQSYxZPJ8CX4MI/O88tRxWwgl6+4sfdZwWff5wajvM8bw1/pff4cP7zb3ufS/HOZ+im544S52bR+ gvP+WXAlKryN8eO/P972/lLBT13FbsOXNv0v619vwO2/l6p56Mq2FZE/6c8vn76+f4m/xvdfPm+4 +yulYB6P7jcQEjKMGBIqA2E6CM2gL/6sS4PjZpCCGzKDVQlKMIMU55NmkEp/uZ9f3nx4Wb/a17fx 5evHZgqPV4s5+AOSWEi+fnz5+OmDdx/CfzSGu0vlcR3Bh4yJJEa9MVxtTN2Yb91CmPiXg/mGjClx mjElftaYMqIzpvDhz++ROZXrmYlDi8hEq0GVP+pM6uHioVGlocpFhlGNKnGiUX2O//41vvfx00/T tBkX3xPKfsByy1TWMgaNi5VW9QPGVXAU48r41bj8ry9/tu3ttJ8KSooDlMp3jf++DVBbslPv+3ix vGBkF4ViwC4yjGgXleEu+SlnUvhpwD7+YzWO1TJEswwpnosNXEk/Gnb4ooO1pSPO2dR0xalRy6h4 t+O/vHkXw81Cnq+WAOIOSNJmXvvfvnx2d3b2cLl8mL491MZT8uz6wPJzf3iA+eeMlArkuivyfJJd YLQkuzH8lZLsX9+//fR1M015M83noFVMc7RuUr5iaUQzYpoZt1BMM+MV2TQziTsaVl3AiNKnbX20 67i3oW4/l4le3ymmIkeM0JEE2zeGv5IR2jcfwrwaodin7pzJ56l7mOJocT2jY06v5HkjXHF63AhX vCcaYSEpiqfvN8IVkdOM/Ghfgo3vPtSc7vZzmRzMBw9qwAgLjGaEjeHOCDXL+YO3rh9yN5he1v7D Qk3fGoOLibHNxJabianexOZyPM6giaVZDZlYwVFMrODJJpZJTppYQYyZWEEOmFiGEU2sMvwwE1s2 E+M3E+sKkGFSowXl8gWYHDOxjCOZWMbTTYzJsyaWEYMmlpEjJsYk1cQ2hh9mYnwzMXEzMdObmPCa YGLzoInNRBObrzCx+bSJzcMmNo+Z2Ew2sfmHmdhvP//xj6uFFZHGFsfkrPnTbFUtgqnhKYHibNHW np4SVNzwlKDib1MCb/Pfvnx+8/N7+/bzA83Db4rtqK6AVtluJrvNTt/8/EDULpac/4AhHNmr7woi G8Llb/Cn+On9Si431O3n4hp6xkhf8uf/+Lz+Ia+fs/2YcapzkA1WXMq+C5LnOfTnopmvPnJ/qTzy vs3zhl/t3X7+0v72pVhb8zTwi8zVd+y8URXDr4Cf/kb93U//7VOM//f/+//89E///N8k/7uf1m/z t7eJi2ZLmYBYe7f5lE3KSTXr21SlXooHfuH9/PHt18+7a6jNNXj2hEfhmJjmEObh+Ku5c7q0bj8Z f1ecmobj74bn318/roj8ST+9+/zy6e0H/6eXlN+Is/5PG7q/npn6k9E2onLCyl/S55eU36X7HF4+ Rf/Wvnm3MYFfrPH8gCpPvv/tf/7Ly7/+0z/+8z/89//x8g//8m8bzdPFEti7qmWlyB72688f3sdq 6dt/LwDbr9BuiNhm/OWDvrx597FBHy8WDtV558ZhZOF4yQ/rg3/5xX4KjeHuUnmEB2+/HDjzc/zy Pv751/cfQh1R7i6sj6wfkTZsDmNf3n3Mz3m71a9tZ83DtcLQt3fdGMpRVvd/nd60HSnPVwsLTwcs rrH4T9F+eWDYrpQHePAOcpxaX/w//us//cP/+qfH93679ke0q2gjcNP6Aj58jO/L3W7Pf79SguvU heUNXXx+DcPlr+2Xu7i8XSjfW3dbojZsKc6XZ5OD38uf85j+MX5aUTVEgt8UPz6w3vFwveEvCdc3 qodwLbpwXb7MQ7Q2UoY8s7yP1nz20UwP0Xq9xL+dxTBeJ/ymNH99zGK4EGkZDdWKq9lOI1nMiiNk MSv+sixmZSNlMSvDqSymIMaymIIcyGIKjJLFFPxFWUyjOp3F5LwiPGcx0nq5PGYx+ZI6Eko/ZjFM 6Zbgm6eaf3YNLoe1r9n37byo87PIDTc+i9zwzZjf/ukO+fZPKwaNXhlT0o+Sqrz7+VO23JKz3Mbf /noZQhIawQpTcYQ8ZJQ/rq5QfyrP6uArm6XkG/FLHh/fv/n8S0s69gvFG/rJ6wbNT9nb/EqKhuxP TQpyf6V82gmNOwWdn9WvZWztGJ6vFsdCA18mKTGw/PkzfAdycXB7m2fe7/P1uut1/a8lZkSUIeW/ /64Bel4O0MMDdMZeN0BnMtIAnfFXDdCV6uwA7RxPzwO0N0I/BqL10tG+c/fhw5fPa/GihCJuxLIV tDhncnmcUHHJpjpCjYQirnye/J1fftxw48uPG/6iUbqyEUbpynBilN4QI6P0hjw9Sm8wgm+s+Gt8 Y6c6OUrnyJX0wlh6GKjXq5NceBB3LnK7eqSC//xLXZpXZqs3cFGauz2P1Ivjo5IiVQ5LUiNJ7Ioj JLEr/rIkdmUjJbErw6kktiDGktiCHEhiC4ySxBb8RUlsozrpHvMkp2BkfHSP9aqMOfCnO/fYrzJ7 4B4h/hq28WO5LeqauZP8l5MwxpdDnC9men45JOMMZTkk48NFo8fK5li/uNL5yMNvymeDKxiZTHZq rTc/Hyq2tl8dqbZWvlD96HN8G/19arZdyFjr+2xqxa6rRtl17qG3n8uL7XO6FUgYalb8NUPNTvXg S2bpnGn7Pg+DjVW8bJk1/GGwsWoJkwiTfRhstquJHdVMPqafPn799HOsHmVum2g6mYQPSshRj4oT E2WhcGATTeYb96gVH3sfOLXAWEiOd26gBcYVkS30Y3pZH+/LvQj26WKxcryV5/yhARuMvAPo+dCA /0xd2Mf09sPPNcLflGHrVOFJU51Im7pmaU9vAqq44U1AG/4uJhMj/MqmCPODjUGfmB+sCF++gbvz I1e9R0PvyQCRAR8+RR/f/Bpffo7v46c3Tab7fHktBRzwlOPxdsATfi05HMCynYQPZb5+88Lbz+Vz HzzcYLZR5k7BXn/KIA/db5aUEabgLxphGtXDCNMPMOXL3A0vi+EqT/WF1f5ueFmvmmkuB6/dHPp2 9Xh4uXNmuadrC+/VK4wi85TaphGZp9TlMMhxmWfGi8vStcJmr0rXClk88msLlZhyXcxyH9PN2usP 5SZgCXDD5CQu5+Rpc4uXVL3r7koZZg5uOOSYK/CsYxYQKfXL+KtSv0p1qWPqiBzTzEdrAvtS2aL3 pM+opSvCzXy4CCcZU2xgPWDDja8HbPhz6wEr5pL1gI3p5HrAChpbD9igo+sBG5q4HrCSDK0HbMjv Xw9Y/354PWBFD64HrNij9YBGg3+H1wRWQkJEWvHXRKSd6iEicWPgqoD98rAuEI0SenpYF1A2RSUf 1gXWS0cyK/fW+j+9ffP5S6j7QeY97Z/7aagx4xvmQhJBDexKWnGEXUkrXp3QWa0IPdN1VhuRuEJn tVFpis5qo7AndFYrwswkndXGYUZ1VhvcPeis8pP6HD/92uut2i/KrP7gbZiwvY2/ZNjtFZSfivn0 I8oKsnoDleEg2C/2BmxXMljCwp0IZcb5lzdvX97+2a2BoM3bHq+Vp8cObl9mme2v//zpzZf8vD9/ /hofae5+UeI9VDFnKrdRbTe9e5lPF8uEFU3oCkfJG1e7vW333H8sWePBtwgt9t/hbj+vpcqDoEGr GtZGyFdUDW8dkW9Vw371dv0+d2HaMBmcsUHd1wzLxdKzXtxrrPaL7CBYZ1+xb//+63v34ev7Fq/3 rS+C9xv4Zk4p0/BlpBC/4giF+BV/WSG+/N9lhfiV7MJC/Mo3WIjfBuSBQvz2f5QyCb9ID7FTnS/E q8klG314LMSXq9LmCMXuC/H7Vfn7q76acTFXlbmc5rnTZ4k4Ll2U05amn131XXGEVd8Vf9mq78pG WvVdGU6t+hbE2KpvQQ6s+hYYZdW34C9a9W1U56WLVphn6aKeNI+P0sV8SRxpIW5lCi5VG2TU86SA Ky34cPmwlFVGyhQbbrxMseHPlSlWzCVlio3pZJliBY2VKTboaJliQxPLFCvJUJliQ35/mWL9++Ey xYoeLFOs2CvLFCshSZ6V8VfJsyrVUJlC6uW5TLEYxtJjmSJfOjw25539U9yj0V401Vp0JYpkhxun yMn7EeXihhsvUWz4q5SLGxtFubgxnFEurogh5eKKPK9cXGGUCl7BX1TBa1QnB+nFZ6hWD6N0vsaT ceFeSFKvxaN1vj17FYzt24dRG4ThuWDJQ3QmHcleM05SsteMt2eG6Q1TEhLqMF2Z3KlhegOtA93Z YbpCeRum08dtXCwYm03wlzdvw/2gjX5fiGHSnYnVMzEiKwWc6eCjlZkvGIRnd/ASnG8z2y/2y8Mw ul1YXfMAG2pESPfYuwt/RIedbVBa6q7ny1J3Pfdaf8FAvWj9Rl36Lvr03YYufXdHhf18z339X8q9 e4V4OuypDJlWjPfgiZPw9nSDxIobbpC44e/0A+QqkVgVlhQxT2ZQt4aNN/j9lTXPAwXSFf0dhSCs zRFjiswCpBWCxFWKzEZ1uhC0mJRzRjOJh0JQuSpTTJr7h5X5etWKA595/+Xj5jBq2ZfBctLy5DDe JDFeVg1JqhGHWXEEhyn46xxmZSM5zMow7DAretBhCnZQwiwJ2wE2/GWLEbLfDvA9DlMkcvnrm/nB YdarS5x91PcbAvar/sBhPpW1js1j5M1jwEIEY5QdAWIZ8xix0DxGLFd6jFioHpMZCB4jvmutAXuM GFtrKEDi8t1Vaw2N6rTHaCaNe3SXnHo5ZpR6zMbKJXeUjX3+pY4sSt10/p0U03tPScUmNpaKTYyW ik3sylQssxFTsYlRUrGMbn7y8UM2qz+l8PnOVfZrZbKFVdoT8/KO4Qn8x6NjRya2NnbOTvXxBtt/ LIUQoGooMFoWN11UI9ypHqc8vF8iX7/PQynEMbkkM2n5UAxxTMQgnBAP5ZB29VubONW8VwrZxJ4q hdwIk8YrhWxRcqRSuOIIlcIVf1mlcGUjVQpXhlOVwoIYqxQW5EClsMBIWr+Mv0rrV6nOVgqX0lbb GvZYK8xXZeQ+6fucbb9qlqN1vU8f3m9D0U19zM2z1s9Pixo956cMtt65oZQt40gpm3f+wpQts92G IuomzkxmT+0KyIgitlo9w6/Vyvf2/Yc7PPpNub2H+V9mqzncE83DpVIMhSmgd7RMzrvLMrmN6sGL GJ86P9q/071yhOfZt1wsV/dS/vXqohZm7ltq7Fft0fzn531dalmE2N1JTl3NwLLxRtecLZEPSLE2 3LgUa8NfJcVa2e6kWAMdNVaGZTCz29BmaAa0YtdmuyUbe54CfdyxaOW0QCmLuAV/0SJuo3pM0Fhf k/4IZkHMMKP14yyISSEfV3HLpUOx+S9fWoFNT9M+DZL9Iq4R4+WCOGkzplvMOJJuMeMv1C1qc6Fu MZNdqlvMfMO6RW0Gy9Xa0CY62lw20dFmuJagp+daApdmeRJm5Uv8SAXcFnz3I9XYJKZu85iZh1VZ OU9NI9LfDTfuQhv+Khda2a5yoY3sOhfa+MZcaMWOuNAKpE2F0kXluJ1qwIXK4uizC+lJL7xbHNVH it+f45cv/1G9aN8aPS9PXrQYl71geCAyap7s+cbSK86NN5be8PyqgWhlM5R63MpgR+txBV1m/fmF 3Z2E134qtQIw4K2gZQWFN5/93Vbl+yvllgmMNCvabuj46xO4XvgjbleyQkd3ZRfw0K7sFXh2V3YB UcbTgr9oPG1U53ZlM6Z5THPOSh+mcuVqZCJPFO6ncvtV/R0hQd0Wf59DQln8pSxlWT0SEgqOEhJW /GUhYWUjhYSVYTgkFPTpkLCChkPCih4LCSt0uFFDBo81aijA040aMohW5LGXNWqoVOdCwhyUntws 5UNIWK8aIfnDQsLt6lG7u/uQoG+lUiCgkpQTpngcCwnlLGFKSMj4C0NCZiOGhMxACAk8DoSEDCKE hIweDQkZOh4SeBwMCRl4PiRwis56xV8WEnivs/52liCFZio85gilRuWsfWgOW699RzBg0y0Y9Ev4 lo+36gvJy7Fg4CUtGGT8hcEgsxGDQWYgBAMvB4JBBhGCQUaPBoMMHQ8GfqzD2go8Hww8qcNawV+3 CDTQYW2eZUzMqqf8IF+NydkU7uWi+9V41MjpPiTc6UXZY7dEH2Y+UfSiho+FBMNpISHjLwwJhlND QmYghATDB0JCBhFCQkaPhoQMHQ8Jhg+GBMMHQoKhrG6t+MtCgulXt76dHyTmpPfPVYRy1bLgH6YM t6vfkyXcGqiKvpOSk4Sei0mnsZCgEy0kZPyFISGzEUNCZiCEBJ0GQkIGEUJCRo+GhAwdDwk6DYaE DDwfEnSihQSdLgsJG9XJkBBSUrOW+jEk5Ksyv4OQ7tfq2tU4Hemt7kPCcgsJz4ve3s6Jsug9zyMh oeAoIWHFXxYSVjZSSFgZhkNCQZ8OCStoOCSs6LGQsEJHQ0IBj7ZmnkdaMxP2ZW7461oz9/syv90B 1kqZiqrysQNsuTorztNDa+b96ncsP9717xL9thmKBjNOQo2FBKFoISHjLwwJQlFDQmYghAShBkJC BhFCQkaPhoQMHQ8JQg2GhAw8HxIEaTNdwV+3+7TfTPfNkFD8M7H82e4jQrkYo4+zuQsI+8UjXdx9 PLh1ai/y7KdZg46j506VXMixsVnD1k9nfNaQ8RfOGjIbcdaQGQizBscGZg0ZRJg1ZPTorCFDx2cN jg3OGjLw/KyB1OtoxV93PlC/j+nbtUW9WBOiEI+1xXLVe++m+6bMt6vf3DCo9b70KAzrDwaShNJi FGYoImQcKSIU/HURobDRIkJhGI8IGV0iwviGwZXBnN4wWGDr+UpFgv64X3ADCXhshCCpaAv+srMY RK+iZVPfUuxpt+BsbWlcr4R+2MJerubhWAh372j71cM+KbujmZskvVvj91YYwnaoKOLQDvaCo2yH yvgLd7AXNtoO9sIwvoO9oEk7cwvDwM7cAhvYmVtgRFe7bOm8Up3emXvzNYl8bZHI15aj6vhfmnTd 3E4ZeN6YW1r4qdE+u2XsTquy7HyWm3GekuWmVZN2VZab2QQty80M8szhYgUR2jiWeT9//fwxvg93 LnK7uKaqMInLHKEq058oHq9lBtB+cyWgZYHpOgVa6hVovN+sWxrv308LS9+zR3F68nN4PKp7vXRU HbrrdWkWWU/pNtMin/d4zJxR2rsvI70uNxzhOI4Vf9UO9o2NsoN9Yzizg31FDO1gX5Hnd7CvMFqp ZLlqIGlUZ0/pFsmpKPzTKd3lqkvRyvtTEG5Xj4aRewdRe9JmZN+aWodhLRZXIU/SRo6xLzjKMfYF f90x9oWNdox9YTh3jH1GDB5jn5Ejx9hnGKlPcsZf1Se5Up09p1szzgOXT+d0l6veJmHnh+JBu3pU PLg5yDLVc4WlYmp6LCjKWToZRpXLiouQho40WHGUg+wL/rqD7Asb7SD7wnDuIPuMGDzIPiNHDrLP MNJB9hl/1UH2lersCGIMC9lDHgeQfNErpcK9e+wXj7zj459+/vvSc/WNr81R22Z0zhf+tAewHPth l/HGycKkgT5BFUfxkYK/zkcKG81HCsM5H8mIQR/JyBEfyTCSj2T8VT5SqU76iBVLzrDupx7zPIvZ PUw91kvp97eWGyPUbXP50klVppwYjW54kWyxemxzecaRNpdn/IWby62+cHN5Jrt0c3nmG95cbvXg 5nLS5rEVf9nmcrB57D9tc3krci1sqh2B5iIm68aVmEY7ApUIEdfuuefHlXgvpxgYV6JyFznRxuYJ HYEqw3JuXImrROLjm4/xbnVy/7GMKv3xiRtuYDW0As+thm4g2mAUr1FH3KjOrYZunvSwFLq6jZHd EVRl9ex7MzWm6gFtRuinepjmKsrR6X7+mlEHe7pwXHHDheMN7y4qHFc2ccmwVMnMVcPSxvf9K6UG nO+wMpxdKd1gJ1dKNxDJBTP+KhesVGdXSm1pnP2UEE5h8U8JYb50lBDuh8AtTFfXm42Ry/NgtuhE mSRpff4QuIobPgSu4k8dArdhrjgErjKdOwRuAw0dAlehg4fAVTTtELiNZOQQuIr87kPgtr8fPQRu Q48dArdhLzwEbiOkpQL6mrz6RjV0CJxWpjurfn48qrVeOqr7f87vLL7/e/5fzH+ZthbPYt7L/1I+ t2oJ3tlhuSRX1qQBzcaGG9dsrPjLNBsbG0WzsTGMajY29NipAyt2ZCa6AknLARl/1XJApRqYiXJr xfNMVC1hMo/5c750qCtueovsJ/OtN223zcB7O7yOzNmSwoDgYsONCy5W/GWCi42NIrjYGM4ILlYE UXCxcRAEFysBrTttuibLvVHRBRfCWGEf3aRcCt84MGCV/VUveVZbFAWgo4wmZWgbGU0yjrKYXPDX LSYXNtpicmE4t5icEYOLyZapkcXkDKONHuyijSk71V/j+O99DTm7Ba/LY0KoRwW6MD55O7qTPSeU fpmGlscKjrI8VvDXLY8VNtryWGE4tzyWEYPLYxk5sjyWYaRpSMZfNQ2pVGeP0WA2LDFzPB6jUa7m gdun++Oc2tUwHTV/aYPGmk9tEr3nkqRMUntHUCCVJkgjg0bBUQaNgr9u0ChstEGjMJwbNDJicNDI yJFBI8NIg0bGXzVoVKqz3qFd9DpN86N3lKsylK7l996xXz3SWNwOxsipVJujKzP1c/RghpeSNXcu DDjIhht3kA1/lYNsbBQH2RjOOMiKGHKQFXneQVYYwUFW/DUOslOdzqoW6fhzVhW4d0/nAOZL/htT Db0I0zb6Kg229UVG6N6vgh06m6zgKGeTFfx1Z5MVNtrZZIXh3NlkGTF4NpkqLnzWK1YYSTOR8Vdp JirVX3WuUfxi3+4qux45ITBPSKesHjmzb8OR5uD6ujP7NjbiHFyfO7NvRYzOwfXAmX0rjDYH1xc1 mtypzqZTMczOWiMf06lytezsCvdV3P1qPOohtR9pOYlbher5jLEcu6IaruMqO43sB9pwJO+YrtsP tLERvWM6tx9oRYx6xzSwH2iF0bxjumg/0E51ervDtAirnuTc61XtJzM99GlvV+cjmVD1DlE0eVUe pNWzalXmIYiPuofiwvqBqUbFUSpVBX9dpaqw0SpVheFcpSojBitVGTlSqbKeJtzJ+KsqVZXqqqTK pS6p8oer5W3MaMe8amO4eirfGpYmimZucuel3BVHUaEW/HUq1Mx2kZS7kl0m5a58Q1LuDTuwgL4B adK36Zpq1o3qr3RO2K2MZaZN/8aFkk/6N2HycCyGj3ddQgwDG+kqjrAVe8VfthV7ZSNtxV4ZTm3F LoixrdgFObAVu8BoPT3CZV2qKtX5MhYYW5ScFHsSlsjsGN+UhWqzz0i6OpacoxxOubicfPTnZaEV NywLrfhTstANc4UstDKdk4VuoCFZaIUOykIrmiYL3UhGZKEV+d2y0O3vR2WhG3pMFrphL5SFboSU 0mHBX1Q6bFRDslBWmv08ykKXOQ+mj7LQfMnxb6jc2CTv1KB9y/2wjJ/ArqwfaZ654git8jb8Va3y NjZKq7yNYbRV3oo+2zxzA402z9zQQ80zN+hg88wVPNI8cwOebJ65gmgVIX9RWrJTnWymOzv7tFcs sMk8r6yxotr5/ZU1Ps2sJupzjkHPG1VE1KNd9lVOaORQx4sVRygCrfjLikArG6kItDKcKgIVxFgR qCAHikAFRpnCFvxFU9hGdXYFIeRxLBNMjysI5WqUc+APLef3q/7APd6/ef9/bPMQsR9NI5nu1eBi OGdfjEojuyY2HKHTZcFf1+lyZSN1ulwZhjtdrmhSp8vCMNDpssAGOl0WGO0o6XTViS+N6nSnS834 FOaHNpd5xOFsea655kte/P4gtHCzzHUQyjPgbiWCT2p4GVtzN/Mx0VPGkURPGX+h6CmzEUVPmeGk 6Gnmo6KnmQ+JnmbK4Wgr/jLR09wfjvYdC3VCyhhVEo8LdeWqXxaz3C9j71f50Uxtz9H0outCnVxP U30opqZp5tNgr+U82gqdhnK0gqPkaAV/XY5W2Gg5WmE4l6NlxGCOJraj1M7maIJyUNiGvypHq1RX LdQF3S0yHIo7fvu5VlMXJpuWfJJTN3XhYhnti1ySUDWPTV0yjjR1yfgLpy6ZjTh1yQwnpy5qHp26 qHlo6qIIh2Vt+MumLqo/LOt73CI7lHh2i8TD/LQrL19i35jQL3JvYZmzs6kv69nhFpZcToEPLGBX HKEX2Yq/rBdZYbusF9lKdmEvspVvsBdZwQ71IitAUnE8468qjleqv84Ctvfzx7dfP9fhRdVd4FpJ 9ji8iCmVPk/jWRePdqSFy4ojtHBZ8SdbuBTMNS1cVqazLVwKaLCFywodbuGyoqktXArJWAuXFXmi hUv5+/EWLgU92sKlYI/W6o5X6uA63UZGGs0z/qrRvFI91kbgKt1DHDLJShUf4tCSlGKPa3Tl0nIk 0dwFA2KRS+sjJWb+1MJt8XZ440tJ4jmXpxfpNpwaXqSreH6UUwY4vcuI/Ek/vfv88mkNMym/DGdb COmvZ6aAYlkmKrHsL9n9Un6N7nN4+RT9W/vm3cYEflE+VLfcVanE9Id/+5//8vKv//SP//wP//1/ vPzDv/zbRvN0sYQYZOuFQpXQ9uF9fNuiWvnvBWDhLDUjYnHyt5++rh/05c27jw36eLFwKDj75LxE xRwW8sP64O9C+cOl8gj/f+bObdeO5DjT9/0qhgdVledLQdCFARsWBEO3G3Vs0U1uckg2pdbTT0Rk rkNlrfLeGRFr2mNgIGav7yf3IfIY8cfJTz/BTAGz7+v69x+vn5c1w3cD9C17kO1HLEyp3z99ge9z /qt+lN+A/RgqjCc/wNHuFXB2OWrgKKrYRwsRqkwXFZgXx+87hTyC38CTnwG+bePP+I9/+dMf/utP +5/7bYx+io8F3jVT2+7htYO17JkaWFWzLRSUXUlYnRu7mxQrq6Lrj1kVXXKV2RYOnaUP383YxQsa b7JD1ZPUdZ1l9ySFr3F2G2vbiJxk24h867YRGKVtIyo1bxsB4m4bEeVvG5EWbxtBhLltRLJl2wif F2wbgWZPRsDqTkYgKJqMgNeajIoUazJa/WEyGpd5s5XzHwydvV7fJiM4/ZaHtcEcPbXxtY3bpdGG JXHyTTPHzzfNfFu+KTEq+aZZqTHflCBevmlGufmmmRbmm5IIK980k+/PN6XPs/NNiWbmmxKrmW9K grIeS0kp3/QqxdsZwbxTT0adG1w1GcHQWx4OLrl4tf5xqX7kd0PPfcUE2i+M55rCSTwckNfzcEA1 mYcDKrR5OADB9HAAkuPhQD20BB4OwGt5OBSp1t5K8zzgTWbVXAlHp6Wb1uW+u9JldDurDTm49EKg lLrcPiZYpGvb/jh4/p1P6Oa13X60cALbfuS17EeLmsB+tCg02I9mQmY/WjT49qNZQLLFRV5pi3uR arQfjTDRj/NYOZAO41IdtmnoLG/sb9+/f8ldyK25WVqnVFk82BDdzC+oGuD3YBwYiwpyVrCoIB/V FhVUW0SLCiqsTYsKjE7wFXzcXsblB26Mx2+/vc5lr70fxMfM8VGTJ9DAZje3j9c47TpPQLxV/fzl 5dv3ZQfvx/AH4E++4CXmMN1u8PWPFF0n/2TR0ga81tJWpHbRCbvmQ3zSF7SLT58Cdmjc53X6ZKYq rxOGzrZ8D+LTpc6E4wG0t/zyArd1HaucHjlJOT3yeuX0oKZXTo9imuX0qMctpweWV04PoKicHnit cvoixchGQA/IfTaCNT6M687ni4a2oTGM0vEZMAieAZfJcMKIOEEYEa8WRqimFkYkphhGpMcMI2RZ YYSg6DoUeK3r0CKlFEbJ9nUYwdD7wuhaypNsfQUBk7rnZoziitvZ9oZoheM3diBeqyFaUdNpiFbE 1BqiZT1JQ7Si0NoQLWONDdEyJNsNdkrVDFep1oZoOfyirTeDcYr1ZjBOZ97GD8IPV7HYHQ5rxrDL fHy3OMZmsHCi1FSnthnMaoqpqU5xM1j02KmpjrMZzKAsNdXpbAZvUkqrmO2GehWDoZZVjMJof+cR DPr98+seXBp5FmXIiRrljpoWZaCmd6ZCMc0zFepxz1TA8s5UAMq64o1qFmVFSimMZnfYDMLQ+8LI XD3F6+64fkiLZfcv8mbzC+fqkDjB1SHxaleHpCa6OiSFpqtDJKRXh6jBujokUHR1iAqcq0PkJMsc 8krL3EXqf8HVobktc1Ujpa5fll7gxJn8xlvmgBMtc37TXOb8prjM+U13mfPvamX5eJnznFaWGZQt c14rBewipbTMme2wW4ShlmWOwujYidpGgcvztI6sMEJOEkbI64URqOmFEYpphhHqccMIWF4YTbIi HOS1wmh6UITDDiPb+Xq3iEPvCyNbdovIHDvQJP6haxgmxm6xcOzdYuGVdotFTbBbLAoNu8VMyHaL WYOxWyygYLeYFdp3i5mTxCfySvF5kfpfsFu0t2XOHpY5F7ll7wH+AR3vhQw5yQsZ8novZKCm90KG YpovZKjHfSEDlvdCBqDohQx4rReyIqW0zPX9YZmDoZZljsLIHfM1HPuFDL7VcWVc0WeOf0Wfea0r elLTuqLPYnpX9FmPd0VPLOeKnkCJnR3ySnZ2FymlMFo3X4cRDL0vjG6dp2pDOxviwG4CjY6xq2Pc LWaOf7eYea27xawmuVvMCi13i0QI7xZJg3O3mEHJ3SIpMO4WiZO5Ha9Oze04S/0v2C262zLnj3Vx ncQ6bGFdihAn2S32i+KlCKrp7RZRTHO3iHrc3SKwvN1iL2k+mnk1n7EHzUf5d4tbWOq7xQ2DrTGM wuHQ1Rl+WqJLW8+7ogdOdEW/9ZpX9FuveEUPYqpX9KDHvqLfeuYV/Say60Ne7Yp+O9Z/scPIdUt9 6MKh0zC6GPaFoXScszYNlY0l7hXjzK7R9oMZGTcXhRNkRRGvlhWFampZUSSmmBVFesysKGRZWVEI inILgdfKLSxSv5Nh38XpwIfScA5Te509RJHvuYUmmD7ZL5zOWsQJOmsR39hZCxmdzlqk1NpZCyFm Zy1C2Z21iJZ21kIRXmctIhs6a+Hn+Z21kOZ21kJWtbMWCsrynHut1JWLFMfpYEnDwXZlMsu4z3TG oelN1z4fh+HiAeX7w2MEbAu4rbWAXlLkeEARJ/CAIr7RAwoZHQ8oUmr1gEKI6QFFKNsDimipBxSK 8DygiGzwgMLP8z2gkOZ6QCGr6gGFgqLTBfBap4sixZmMtm4z9WQEU8mc9pMRDC3925ORK614TUqV Ibjf5jT0kswFy/IxJk4wGRHfOBkhozMZkVLrZIQQczIilD0ZES2djFCENxkR2TAZ4ef5kxHS3MkI WdXJCAVlaRpW68bwIsXygLLDwQMq4R3qfjLCoTcbIPsYr90JwlAXMgebJkm7wSmw3DGRk7hjIt/q jgmMkjsmKjW7YwLEdcdElO+OibTYHRNEmO6YSLa4Y8LnBe6YQLPdMYHVdccEQdnzxRTUni+yFGsy mvxwbIA82codE4fedMf0qSS7UKuUuhwcLzAFr4Bd6liTEXKSyQj51skIGKXJCJWaJyOAuJMRovzJ CGnxZAQizMkIyZbJCD4vmIyAZk9GwOpORiAomoyA15qMihSvG3tcDjsjO3VjtTOyU//GK9DQ2+FS jdhbs89IGCbr3cB2xzTLuq0Md8zM8d0xM6/ljpnVJO6YWaHFHZMIljsmke3umIRJcumQV8qlu0hp 9fiz6+FZx6WTqPg0/rJe46Ks0aF39th9edsEB4Yl8PqTIydp8oe8XpM/VJM1+UOFtiZ/QDCb/AHJ afIHmMzMKKj1Jy9Sra1hU9rgpBamfWtYHPXJ7+3db6PrG8kDsXPXLazp0yE8ZidoDRs9r+wBOUki W/SaZQ+gppfIhmKaiWyox01kA5aXyAagKJaA14qlIvX7JA9cnMnDEEvuQByMPZbIYrI236QyLJw1 hjjBGkO82hpDaqI1hhSa1hgkeGsMkow1BjGZ72TQqqq7SP2uey+8oB2u9yO+ejnyaPnFd0jplq3j OPYTJ3DsJ17NsZ/URI79pNDk2I8Ez7EfSYZjP2IyC65Nz4JrOx7W3+HYP8zDBsf1be/Yj6PeRjjm 3Dv2X0a3swApy0bo0vXIHtKjvdcgeVjtI2fZIE6wbBCvtmyQmmjZIIWmZQMJ3rKBJGPZQEz2ytdr pRxcpH7XZQPjolgl9MbVxW/UgZxd/OaHEFnLBnGSRi/I6zV6QTVZoxdUaGv0AgSz0QuQnEYvgIny AoHXygssUq1H9hAmOJfP+2WDRgc7OHffBOk2ur112qBzeqkNtfYQHontVID1dXZjhEfm+OGRea3w yGqS8MgKLeFBBCs8iGwPD8JkVZlWx6rqJtUYHpPpQrdMbhcdMOiXxTl7nzd7GXTmrSfZYIZL4fTQ dQdTxt6x+0VgpULPyA8pnCSNH/nWNH5glNL4Uak5jR8gbho/ovw0fqTFafwgwkzjR7IljR8+L0jj B5qdxg+sbho/CMpKinqd/JCbFOdJdh5DnTnrt8WnsNvJ0tDZQn13AWKGkjpL3syHoqJ14+9kIVgT p18AcYJ+Acjr9QsgNa1+ASSm2C8A9WT9AkihvV8AYs39AhCSWcBuSc0CNku19gvo+7mfXQfHxvuL eRqFcOmTud8wX0fPkkb/8fNtV+CH0sY42nC4oLfrxu+p7sZ1Yl3QIye5oEde74Ie1WQX9KjQdkEP BPOCfsRmku0X9ICJLuiB17qgL1LNNy1mm2J90zLN01J1P8Shs/XpbrPsr4+/tiozIwdz9rsV9kYd eJtl5CSbZeRbN8t4wNfZLKNS82YZIO5mGVH+Zhlp8WYZRJibZSRbNsvwecFmGWj2ZhlY3c0yCMqa GA9qm+UixcpfnJ0/1Lz2zi1VzWvv/Fkr1rvJKPTXx8JQGeRGa9wo6EriLSuZmjhJzSvyrTWvwCjV vKJSc80rQNyaV0T5Na9Ii2teQYRZ84pkS80rfF5Q8wo0u+YVWN2aVxAUbYuA19oWFSleMrU5VHak ZeqnKpl6mYZ3XCMGf615NUc34LQIDgzTFP3KMb2fYugEkxHy9myDvjw0tgcC/qVfP317+UrTzQY/ EDgnlKnkOA5Ky6M5DYRwTvsnhOEGP8jp2/LydZ0/jh8+ZaUH/wH/UdOJlOt++ut//vnlL3/647// 4d/+4+UPf/5rlqkGcap5aGsPEuik+PPn1/XjZXbD/43A+Ohsg8SKwf7x66/0D3358OnLBd0PokZ4 FPKggbMjTA/wzfo8303puyH8Fp789OGk+xPMwq/r33+8fl7WDN8N0LfsYU+BiewEvn/6At/n/Ff9 KL8B+zFUGE9+gOj9ef9pnGOOGjiKKvbRgoQq00UF5sfx+04hj+A38ORngHad+DP+41/+9If/+tP+ 534bo5/iY4F3zdgwNT+m2TM2sLozNgjK+itMai4FRYo1Y49TqGfsddkWv5+xceiNPGaYsFPOpfHe DH13cEwxy8rvUjL7mXXDg5zkhgd5vRseVJPd8KBC2w0PEMwbHiA5NzyAiQIDeK3AKFKNNzzGuXno x3nZXfLgaL+tcFK7T0++jb7jqie54ijkh2GfiulSgDMbt4sPZkfAD5lz1YOc5KoH+darHmCUrnpQ qfmqByDuVQ+i/KsepMVXPSDCvOpBsuWqBz4vuOoBmn3VA6zuVQ8IytKYZp1L55sUz8RjPDgKdSkk t1+rcejMPvt+MkqX55h+6I81eV5QL5EcYzIqnOSqB/nWqx5glK56UKn5qgcg7lUPovyrHqTFVz0g wrzqQbLlqgc+L7jqAZp9cABW9+AAgjLzZKfXxtfxJ6PBHrwWXZ/mysQDh85MPO4eh21XLp6tHVyV Mua6zm78zk9ucawKYeIkRwfk9Y4OqCY7OqBC29EBCObRAUjO0QEwmfOf06oQvkg1Pw67eV2qx+Ee ZuCwd9qiodNFeu6/fPz12yU0rk0Lh86Z+oEY4kVgLe47y3sgBk70QAx88wNxZ7UeiEGp/YG4s+wH YkAFD8RAyx+IO8t9IAay6YG4s5IH4s7yH4g7e7ZQny/Tp4/DnZU9DsOapfU4nKV2M5F7uETvJqLR BBvW3USEv8FpPxHRkH3fRGSHi4tH39UuHvPYBUF3xmViuLMXTuK0hXyr0xYwSk5bqNTstAUQ12kL Ub7TFtJipy0QYTptIdnitAWfFzhtAc122gJWaSLKYjLTFLU2sRep5onImeNEtPghVhMRDJ2dFK5P DGTxR7cWzjzos2I822HLhsk4Vt0VcpK6K+T16q5QTVZ3hQptdVdAMOuugOTUXQEmqrsCXqvuqkj9 zi4PHt/bKC4gRPbnZ7/GuES+y8PgN8sq10VOUq6LvF65LqrJynVRoa1cFwhmuS6QnHJdwGS71k1v 17odd63vKNf1afJ22sy+XBdHhx7NZO6rDy6j69uJjb6UJGJiYwrHvSu7JBGdcFfLySVCTpJLRHxT LhERGrlEKKSUS0RSslwikmjKJSJCmEuEGoJcIsKZuUTEinKJUEGeS0Qq3FwigiW5RCjAzyVCmu/r vZ7eNDB9vVfBvJ15NV/v9Thvv7fjiTdb/SQA30D4Pu+eBGDInd043Db6Q07+9EPXuf3zpINtekz8 rqR+sYyNfuFElqFWbaNf1ISWobZpo58JrmWobd/oZ0x2+rU6G/2bVKttVb/2o/Xz3jKURrsFNvf3 ZRrX0dNyytcPr/89XiLExBwhLnlj6lqN2FvBk9mWepzY2w0P033fXYbhYeqnXs/wMFGrXcGTGSqE XMy8x+9HaFJ/lBCA9PLe0uXH9ompn31j6XLGlqmULle1ywULj1IhAJN5Jya1rr5Fan/VZI+XTV+q 6uU4wGo/RL8/V8/rFPb1UDgUz0oQLuZw/eCu/UhDXZtplqWPktum0fBum4AT3TblbsJat02gJrxt yl2GW26bRsO9bRoN67ZJ1KiXeLXbpgeNet9ehNZkBzfZ/X0TDpqAJTt3kXEdPEv2vq1AEB7RXy5j 03C0dZ8Nv0DHO+M4KxBxghUIeb0ViNREKxApsFcgokUrECowViDEGCsQYqLtnlO6171JcVYg2x1W oGkxS+VqjUPjmxdXwZSaigCLUeVeE12/TvxdXuj8xnl0JU7w6Ep846MrMjqPrqTU+uiKEPPRlVD2 oyvR0kdXFOE9uhLZ8OiKn+c/uiLNvZNBVvVOBgUlcxHySnPRRYp5J3Oo7wpzD3vX3Z0MDp09M335 7fvfPr+a/5PKlBRtf/W+7OPhDbYPbEMts6xwAm5uV1E4druKwiu1q8hqSu0qiphau4qix2pXkVlG u4oMyloirTpHy5sUo11FGJatblexmC5VR0sYGs86Ol/vN1O67J17mw61khCz3FpJLDHZGO0qCify V97U2lUUNaG/8tbUriITXH/lrb1dRcZkhUmbTnbPTer3TWSIXUlkgP1uZw4ONLGX9ELq18S690dO cu+PvN69P6rJ7v1Roe3eHwjmvT+QnHt/wIQPYjo+ijep1hriKRmTJuP3NcQ4GrvU+ftLl9vom2V7 8Cl3ScLt0lDVEMe4OP6dyzAsnpPIgJwkkYH4pkQGIjQSGVBIKZGBpGSJDCTRlMhAhDCRATUEiQyE MxMZiBUlMqCCPJGBVLiJDARLEhlQgJ/IgDT30Iys6qEZBWV9VBav1kclS/EOzduhtjH287g3+KSh 0xu8uz1Nb7rrgbl+qTXLHPN6zmwW0TOMrDLHn7ML3zBnF0I+Z2chlTm7SEnm7CLRMGcXQjRnZw32 nF1w1pxdWMGcnRWkc3ZR4c3ZBebP2VmAO2dnmlnmRqxWmRuJyRqYKPW9ukm1VpdMZhuq6pLe2W3b X8r0znXd2aXMbXc9uHjpBhdDdbUZzZxGthlztOPmGbW2mePX2ma+rdaWGJVa26zUWGtLEK/WNqPc WttMC2ttSYRVa5vJ99fa0ufZtbZEMychYjUdekhQMBERrzMRXaVYphjB2cPGESajUG0cYehNu7A4 pFuPMX/cNk6GXfgPk+248raN4yrbNgLfuG0EQmfbOK5q20aQkm4bQaJx2wiEeNs4rqJtI+DsbeO4 SreN46qxbQQV/rYRYNm2cVwl28Zx5W8bx1VzxiZB2dZxXNW2jlmKM2Mv63CYsftxcNt+xoYhNMR/ Y8Y29tpr25r+aJ/P7rUdrI9Mh4QodEiIDIeEqOaQEDkOCZHvkBBFDglRwyEhsh0SYqNDQhQ5JESB Q0I8d0jgFVBFoUtC1HNJiA9cEt7b+M6lQwHVNA9r3ctjHrazkte7a0fj3K2zUFc9FaWhj85z83Tg C+W0cM4cP8Ug81opBllNkmKQFVpSDIhgpRgQ2Z5iQJgk9QZ5pdSbi1RrTfhoRjePfl9CRaPTMqzb vY/IZXTrhv85A8fbWxZbGsyxLeTMt1Xrlhg4CTjECRJwiFdLwCE1UQIOKTQl4CDBS8BBkpGAg5gk AQd5pQSci9TvkYBzSUuLxue0tN4M1OK8svAfLPepCuiwsMpuiZP4cCKv58OJajIfTlRo8+EEgunD CSTHhxMwUSUg8FqVgEWqtezWh+itqXxEaHQYw+LvL+Suo+Fszbhuqrx1lyamnixF6hYXq6BBWlg2 ZnxswvjYVONjE8fH1hwfGzs+Nl58bML40KoNuEi1xgf8rg9bP+wXDhqdwzTu0tNuo2+np9m+2Db4 ruv72qy2t07gETlaVs8u5EQ+O8i3+ewgoeKzA0JaPjsoJfTZQYk2nx0kpD47oCHx2UGc67ODrMxn BxQUfHZQhe2zg7DIZwcEBD47QLOviYDVvSYarahnF/Ja10RFinVnbdZDl0W3DWtXWe/D0JmRyP2M nS63RGbYexx4u5ptFhjtjDOrwJQ4yZ018q131sAo3VmjUvOdNUDcO2tE+XfWSIvvrEGEeWeNZMud NXxecGcNNH8ymnULTFFQNhnNWpvIixQrV7ZbD/2nZ9P7cT8ZwdDp8ep6+2BdKXbvU0y+3j26PnFv rOFLTMvIKv5BTlL8g7xe8Q+qyYp/UKGt+AcIZvFPwgq+9uIfwESBAbxWYBSp37kozl5ecoCozPCo E3IvuJWLG687DnKSWwfk9W4dUE1264AKbbcOQDBvHYDk3DrETdYdB3itW4ci1RgX2LzUbWn/kIOD 1vaTu285exn03ZtrxqVLuPPBP7hx8OymmgZ2wBPvlRM40Ssn8IqvnKAmfOUEhcZXTj9xXzn9xHrl 9JIOj8SrvXL6Y1O132HNuBgM2J4SSaubai/p1mJnxvtm4SR7KeT19lKoJttLoULbXgoI5l4KSM5e CjDRXgp4rb1UkWq9qXbdtqzbtr+oxkEYme4r8q6Db3URgT1UfzUXrhoxB4O93/h5em7N2f/N+ynk JPsp5PX2U6gm20+hQtt+CgjmfgpIzn5qldQcZF5rP7U+qDn4/79muP76umnqnBizzIIuIjbAVMny O0VO4neKvJ7fKarJ/E5Roc3vFAim3yktcO1+p4CJMruB18rsLlKt5hubT12yMezNN3A0wMlivjc/ u41Ob96Vuz6WPRWsNYdzuE2b4H4qToaZ322E+d2Gkd9t1PK7DSe/2/Dzu40ov9to5Hcbdn63aczv NqL8biPI7zba+d0CC+bM6+V3Hy2Y35vf7delvisf/bLsV2scWv2be1h3NYY11b2HS3ENC98oC7PD WXtY5CR7WOT19rCoJtvDokLbHhYI5h4WSM4eFjBZM/mkdidYpFrXagioyfq038bS6NCndXL3a/Vl dH7zWtA5ey1+SNWztndk68U2Kd36gVE8S5ygeDbzLcWzmVAoniUhneLZLCUqns0SLcWzmZAVz5IG v3g247zi2cxKimdJQVw8m1WYxbMZFhTPkgC7eJZoZvEssZrFsyQo8sMFXssPt0i172eMCQF7he1z kWh0DsH09+8519HhrHDt0nRl6IMthyxv3f5yDo5Yfb/xH3SWLTEcQAsneNAhXu1Bh9REDzqk0PSg gwTvQQdJxoMOYrIISToOoDcprcu5aTtczs1nNnJ3Db+ii/3F+jP1/pAFYAU9TF10M6vhF3KShl/A Kzb8QjVZwy9U4Df8QlrW8AsUOA2/AOM0/AJMllDgZrWEgizFabcSw6HdSrRhrtqt4NBZlF3XHtNd /L+GzvhDUWhcBW20zchzZUdO4sqOvJ4rO6rJXNlRoc2VHQimKzuQHFd2wESu7MBrubIXqd/1YQji or86mIbkDu2zsdqH/TC0mMh6GEJO8jCEvN7DEKrJHoZQoe1hCAjmwxCQnIchwEQPQ8BrPQwVqVYr ATv6sM1u2FsJ4Oga+jneH1muo+nNy6bQ9eXdNHprq3wCF7sgyVvuGOFROFHecqcWHkVNmLfcNYVH Jrh5y117eGRMlrfcqVUXFSmtZWM1h2VjPesQ+e0WFv3VwJDcNqr2qaZjW2zAmcxS9/j2k7ylpvH8 kzzwVu8kD2pJdpIHhZF3YMk0/qJzDyxFITUeWDK2rOXAsj+vZMg9vMO0VnYlYK3alUCW2sXX0B0v zarTSr8OaXXY3um+jRSNDluYhnujp+uoOVt/fr5L3QlDeQ70qbazccnPKQn65kw9y5iAOMkShLze EoRqsiUIFdqWICCYSxCQnCUIMNESBLzWElSkmpegwQ6xXoK2zfq4X4JgKJwtQfuOhT725np6GY6n emfZb4E2TBzTjsKJ2njrmXYUNWEb7zbTjkxw23gzTDsyJmvjrWTacZNqjg07jkMVG30X+3nam93j 0Jm/9G13drOyCbE/vpCz7aWDHUxkWT0RJ1gxiFdbMUhNtGKQQtOKgQRvxUCSsWIgJurYA7xWx54i 1WxlY+bOrG6trGxgdPDd1E87K5syOpzlV3355ed//fZ9/P5hvoSJv3n6dv4YJpbr+ISXGCsjTAon uvpa1cKkqAmvvtamMMkE9+prbQ+TjMmuvladMLlJNYbJ6My87XdVczdO1SsJDp15hjwIjVKWbF2I qS697NaBHRpoFeow15VhMHv/y8govQzu7n5AbjDrJndVkzWCJrGk1wga9d5/+E8P0mJIof3wj1jz 4R8hoY2tUgBepVoP/xCB41RHILpTVRFo41kB2/15P1yXJdfF6rw/BxP5tj2wJllW+i9ykvRf5PXS f1FNlv6LCm3pv0Aw03+B5KT/AiZ6wQde6wW/SDUuS0vcMG+4223ecDBsY9ruk3+vg2+ebGJ3Odn0 th8OZToju7c67VEZxueFk51s1IzPi5r0ZNNkfJ4J9smm3fg8Y8KTjY7x+U2q9bUyWR/XIe5NbGl0 6cO43pvYXke3M5elWxlb9PFis9T74bBv69j7tmD92s+sMjbkJGVsyLeWsQGjVMaGSs1lbABxy9gQ 5ZexIS0uYwMRZhnbSm1z31/GBp8XlLEBzS5jA1a3jA0ERdfzwGtdzxcpVpc72091Gdtqt6myfIOh +Q3LN5OCK4bBQ7SxO1gxJIEVw2A6g2tm+1oNXJKs1cAvemt1Z6ZB5QxZxLzWGbLoXTJev60f1/k+ uvIAsOP8qFc4sMucl/V79PZn/ME+6loOoGxh77SKQS9SuyhK5hBE+es5ZFoke8i0MHXaagfL9UkQ /e379y9LztDrLpXpzvZV5zG/+ThP7KsYP5h1HQdO4upK/fr4iavAR6WrmKy2CG4ps8Lalri6rjBR /vRxexmXH7gVGL/99jqX3cV+EKNkPN76kMY83GvUOK2zJyAWyX3+8vLt+7KD92P4A/AnX/ASc4Bu N/j6R4qvk3+yLH12VeoMeJXaxScEyCFA6QvaxadPYdmGfXxan8z+qY2GzmzxH8SnS53rwuEJ2rOf oNE7Zk6cZY44wTJHvNoyh2pqyxyJKS5zpMdc5pBlLXMIyuyJZp0KqJsUY5nzKXb7Zc6aydl1l4dO Q6cH17Mw2t96epicYQfJz3Lq554TRsQJwoh4tTBCNbUwIjHFMCI9ZhghywojBEVnLuC1zlxFSieM 7DL0VRjh0Nv3P6kbhms+VJWujjZfY2BX2Noww5Gu+f6ncOz7n8I33f9kRuP+pyi13f9kiHX/U1Dm /U+hZfc/WYRz/1PId9//5M9z738yze6ZDaxuz2wQlDmrGaWi5qsUq/8I/PwOPbPhd8bv739w6LRI 4FJx6f2tDWeXDhtjw7dv9t1ie14bTuBEbThtr9mG0/bSNpy2b23DaXtuG07bs9pwWsEinXm1Npz2 uEi//VYzjdNsB7t3HMTBYdzSruryMnhad3mr+cfwsNfwcOb4VNPzu9QOQx/ba/4Lx675z7xWzX9R E9T8FwVmzX+hBTX/WaG55j9jzTX/GRNdzgCvdTlTpDg1/90yVTX/ECGjtVUeNAyd1fzfbYdLIrQf UuerQ6XDK0iBG/TmOk7PQuQkPQuJb+pZSIRGz0IUUupZSFKynoUk0dSzkAhhz0LUEPQsJJzZs5BY Uc9CVJD3LCQVbs9CgiU9C1GA37MQae6bMbKqb8YoKLoGBF7rGrBIsc4Mm9vqM4PfBle9GcOQf6tN WOrTcHVpCfG4KXLcW0CgfZhZLi3ISVxakNdzaUE1mUsLKrS5tADBdGkBkuPSAphoJwO81k6mSP0e DmG3fQwcFsob8OC7qgTYdFPcuF3CrElh9Yxrvczxr/Uy33atR4zKtV5WarzWI4h3rZdR7rVepoXX eiTCutbL5Puv9ejz7Gs9orlunsiqunmioKQ2AXml2oSLFKuTp+v6aon2GyzR+8JrHDrtynY3GcVS c933wdV5XQ5WLUF9QjKsRvDIiQ5VyLcdqpBQOVSBkNahCqWEhyqUaDtUISE9VIGG5FCFOPdQhazs UAUKCocqVGEfqhAWHapAQHCoApp9qAJW91CVJE5mmVezzX/gZPbefhLzdOgnkSZjqkMVDNmzQ1Vd 1Albl8vhKkC8HE2bBrYFZrRTbn3SuIvMHP9wlXmtw1VWkxyuskLL4YoI1uGKyPbDFWGCLQ3xOlua q5S83nldtlSdrFayYXlrM2OG22bGmGN2rRHcEMOSximYIU5QMEN8Y8EMMjoFM6TUWjCDELNghlB2 wQzR0oIZFOEVzBDZUDCDn+cXzCDNX6e98uUnCMrWaa92+VmkWJefcQv1Oj0Ps3H7dRqHznKJj+v0 YC+3PcZVRXwRlgJ+r0Zv4P9nmC9kjm++QLya+UJWUzJfyGJ65gukJzJfyArN5guEtZovECS5akVe 6ar1IqVgvrAuYxfq7cDYvWESb9ECu0SeSdHVSci+8+x7Vj+E7Lrb/PyAHDsJufBKSchZTSkJuYip JSEXPVYScmYZScgZFL1VBCVr4JvU71Oydrentq60+YFzZdXmJ4XeDVzPbHyOSYw9deH4rxWZb3ut IEbltSIrNb5WEMR7rcgo97Ui08LXChJhvVZk8v2vFfR59msF0czXCmI1XytIUPZumrQW84sUqwg9 9f6QULCuNlUJBevq+rcnI3fNQvbdg0uvkftaAbNbWFkVEcQJKiKIb6yIQEanIoKUWisiEGJWRBDK roggWloRgSK8iggiGyoi8PP8igikuRURyKpWRKCgpCICeaWKiIsU6yJ+xTYf+8loWc2wP17Q0Jl7 1f1kFMr5wpn+WJ6Fb5h8u+ptcYZjVw2c409GxKeG1rVEeCNvXZuFgkbr2iw1SlrXZom5oXUtEcGI WtdmjYnbujbjC6t1LbGxF7SuzQrpopBn0v5eoAzhF//AxoB4XL7fPbmPJ7+OyV3Wl+1L/vDL6+f1 9fvX3+5Xmvq/4ZdlTr6pMItXio+UMJgfnEZJ4WTZ6KeTb+S7lg3Xnfxt7EI6YHUL6UBQ5uW+KXVo u0qxlo20HIyU/DZ2a50UO/ZnB+rrrdTQFSMl532wh/vgwfJ7UA2D3VhGSshJjJSQ1zNSAjU9IyUU 0zRSQj2ukRKwPCMlAEVGSsBrGSkVqd/nVupSjWqtKXdSbqjMyOhNxXGLUYMNfec4MUScIIaIV4sh VFOLIRJTjCHSY8YQsqwYQlASQ8grxdBF6ne/2b3ZwkcbDva7aeXa76L3fTex7CWQk9hLIN9qLwGM kr0EKjXbSwDEtZdAlG8vgbTYXgJEmPYSSLbYS8DnBfYSQLN3xcDq7opBUNakopvUmlRkKVYeul27 elcMv49+2e+KYQgzy9+ajGJ39TqO8TAZmZXvdexSNLzULeBEqVvAN6duRaOVugVK7alb0bBTtwAV pG4BLU/dioabuoXm3S2pW9FIUrei4aduRaOcuhVFLq3Iq6VuxWMG6Xsno7CkejJyqV+qFGsYWt82 3jIxXI23TDrc7MK6xc0csWEeOG/emRMZbw2tb97EaBlvDc1v3gSxjbcG/pt3puXGWwPvzTuTTcZb A//Nm2i+8dag+uZNgjLjrUHpzfsqxZqMxukwGdllWKqdEQytZzuju57ByZX+3NTarj/275r4KTjd Mo487y3gRN5bwCt6b42j1HsLFBq9t8aR672F/1iG99Y4yry3xlHNeytLtfcMTsbUdfTeLnXbehja zly37uLCF5dZb601to6LHgbZ5R7ejYwSqMJJemkjr9dLG9VkvbRRoa2XNhDMXtpAcnppAyYyjgVe yzi2SLX2D4IvYJ6c2/b9g3A0mcX7+/KD2+hb7bWSH0r+s4Udbe1JF2AzKKmGSoHVeg45Ses55PVa z6GarPUcKrS1ngOC2XouYZ5Ye+s5wGTnuxTUzndZqrVCEA1RUtg31IbBaV3jau4LBS+Dp2Zyt9i4 nfBcNIdEwm5gt5o3y9Zxqmczx99SZV5rS5XVJFuqrNCypSKCtaUisn1LRZjEEAR5JUOQi1Tr0hE7 18Nv/bJfOnB0me3c9/dLx3X0rG/p7QYk9P5yHTtUawfEMRxIBLltc897G0JOdAPSt78NAaN1A9Iz 3oYAYt+A9JK3IaTlNyA9920IyaYbkF7yNgQ0/wakV34bAkHZDUiv9jZUpBg3IKajbn6V9fjY+6qS FofOKmnvJiN7eRuKvu+rEqQYF3YfBNixbyv8YxnnPOQGwTkPed/iUYSE7RQ8ikjIqngUkVQQeRSR RGrxKELCdTKPItKIbI8iwsddoi18p76tX38cE24v/wEzWk5+Gm7OP41/Anb7EeCf8NfnUR9TgC65 0suHr+vr990EexhGGf/o6Iw681UHP7xTwAGc3M/+CVtG67/7trbMJ19xmPe/f7SaLLXOo/8Kqv5R 9xgQFSYPFxHF5OGiKEgezgrvTx7On+cmD2ea3YUVWN0urCAoutgBXutip0jx0iTMwfzJBVgVqpdJ GHrHvtzeXibrltBYczKz+wzAcp9Mu11f5vh2fYVvsOsrhNyuLwup2PUVKYldX5FosOsrhMiuL2uw 7foKzrLrK6zAri8rSO36igrPrq/AfLu+LMC168s0//CSNHNJsqDs8JKU7lOuUqxyD388vIyLG/ct RXFoOvMk+8fP1yk7dvZ6egl+b0PibYpO0jk7eM41PHGCa3ji1a7hSU10DU8KTdfwSPCu4ZFkXMMj JmuGHXS8h29Sza+3ZrNj/Xq7DHPVzQWHzkv55/7Lx1+/XUKjv3ZNCnW3XZiZes/2xrJhio51y4ic 5JYR+dZbRmCUbhlRqfmWESDuLSOi/FtGpMW3jCDCvGVEsuWWET4vuGUEml+XGd3ZQn2+TJ8t0iAm q8mMOpUwN6l9Z6mHS/RuIkp+HsK6m4hsnId+v0DT0Ntl/NGWtlJhGKrsKr/NaTDsp8BoxyEwJqHM 8SehzLdNQsSoTEJZqXESIog3CWWUOwllWjgJkQhrEsrk+ych+jx7EiKaOQkRq3laIEGRdzHwWt7F RYplcOS7tT4thG3qqmRPGOrfLoOJpY+kt3YYKs/CbR3cyH539fBrvrLc1pCTuK0h3+q2BoyS2xoq NbutAcR1W0OU77aGtNhtDUSYbmtItritwecFbmtAs93WgNV1WwNBmfNjWtWcH7MU6911MLGejAzW Ku0nIxyK/0OC7d//Nn7/cD2lXeqEQx+rNNtgnXGBWw2Dneq6LXDyCJGT5BEiH9r6bnYb+uMo9N0E IafUdxOkorDvJkiMbX03uy310r6boJEEfTcBn1SeX0lqyT+N+4vff95f+w7Lwwae3TbG6hmzUtiP Y7g/7EQJQrImmt0maaLZbfwmmt2mdC7NYrIGmp2Wy8lFqu1cis3k++Bnszua0qgb7NKnXRr3ZfQs /eWuziEGc71BNofaaJN6/jUZbHsXRrJq5vjJqpnXSlbNapJk1azQkqxKBCtZlcj2ZFXCJPc2yCvd 21ykWpNVu6Ez3YS5ffcBQqPd3G/31zeX0aE7OzRdc7ljuJbHeXu0DhgFdQ7erZZXBgScqAwIeMUy IFATlgGBQmMZ0Gq5ZUCrZZUBrVZWBrRatTKgLNUYHsO0GW9G1+/Cg0Z7O5vd8+N1dHrDK8uF3gzF pjTYvq4e7dc5cG1+ILjGteeYZREnMMsiXs0sC9XUzLJITNEsi/SYZlnIssyyEBRtxYDX2ooVqd/V cC7CYaW/1NLRcbj2ylr4rczNljZOCTZxghJs4tVKsElNVIJNCk0l2EjwSrCRZJRgIybrr5N0jig3 Ka1W5nY9RIV7Rwl2Kl0oYe8V+uPRZNq4cRFwbVpZey/kJHsv5PX2Xqgm23uhQtveCwjm3gtIzt4L MNHeC3itvVeRat17wQK6Dn7duxPQKJxEptXf770uo9tZe+3bsnFtnpOc3+d+WRviMAnCI06RFR7I ScIDeb3wQDVZeKBCW3gAwQwPIDnhAZgoPIDXCo8i1Rgexrt5XsK8T/+i0XEONt37d9xG1/e+MCTr S5gYW3liw96q94Gb1Q5f8LKtnBcG4iRhgnzTCwMSKi8MJKTzwkBSshcGkmh6YUBC+sJAGvwXBsJ1 XhhIivXCgKTKCwMJSV4YUID/woA0u6gIWLUXBhQTzcPAa83DRarthWFY+uS7zo+7FwYcdfNk+mG+ 36VcRs3ZLuX1w+t/j5cZ2LmSiTt0gz9ajPWCvBMTJqyuaM47QW7hn2+Rn3q1Xpmo5iXnW1II+WZo j9+P0K/uozwWpC/3QG82on10WkaF2Tc3okVsmUoj2qoTbcHCowwewET5FMBr5VMUqX2s2WO0fal6 0cbBLNFEvz8sm3nw8/6wDEPhbVfRFLqyywkhVbuctG5uEFzExsDIdi+cxOIY+VaLY2CULI5Rqdni GCCuxTGifItjpMUWxyDCtDhGssXiGD4vsDgGmp1VAKyuxTEIiq6zgde6zi5SrO7047IdPDWGzVXN 63Do7az3FNzlVaiPx6Qu+NkJjlyOkWhaOMFkRHzjZISMzmRESq2TEULMyYhQ9mREtHQyQhHeZERk w2SEn+dPRkjzDyBOM9E0C8oOIU7vEOK4iabDMq/DwdXADlNl8INDbxv8pFDcxnyKpiqRDWnoRr7B j/OjbXc1yBzf1aDwDa4GhZC7GmQhFVeDIiVxNSgSDa4GhRC5GmQNtqtBwVmuBoUVuBpkBamrQVHh uRoUWJSUCgKCpFSg2dtHYHW3jyAo2j4Cr7V9LFKsGdv4Q+9j5+axqlOCoentOqUU7WXGTkN/8Id0 id+ux8M3mzNjIyeZsYlvmrGJ0JixUUhpxiYp2YxNEk0zNhHCGRs1BDM24cwZm1jRjI0K8hmbVLgz NsGiS34QEFzyA83eYwOru8cGQdEeG3itPXaRYhVzraM5+NCYYIbKhwaGzgyvLzN26rrBlgN/9F3t eD2nLUocr/uZ4RxGnMA5LPMtzmGZUHAOIyEd57AsJXIOyxItzmGZkDmHkQbfOSzjPOewzEqcw0hB 7ByWVZjOYRkWOIeRANs5jGhm+S2xmuW3JChzYu9nNSf2LMXyAgjeHrwA4tRvlRdAnIbwjhk7XYxJ Qm8eZMVI3os4XegKJ3ovau1Clxmt96LmLnQZYr8X8bvQFVr+XsTrQlfIpvcifhe6TPPfi1S70GVB 2XuRUhe6mxRrMhrj4cBvt5AqLwAcOvMCuE1GvYkXq7aYukMVkR/YNai+m9eBY0xCnMCYhPhGYxJk dIxJSKnVmAQhpjEJoWxjEqKlxiQowjMmIbLBmAQ/zzcmQZprTIKsqjEJCora/gGv1favSLEmo9TX zcL9FpPbF2PR0Fsu2DAZxWvVyfEsu8wxcm8frR+GxPGNzJzEJQn5VpckYJRcklCp2SUJIK5LEqJ8 lySkxS5JIMJ0SUKyxSUJPi9wSQKa7ZIErK5LEgiKsvqA18rqK1IsS34X6v68fgtxricjGDqbjP72 /fuXhaYj23epvzTLcv1QJfbN2zyyr9a82cZ1HDjFocBZSXHoSLnnWsWh43qXyssqDh0phb2lOHRc J/gKPm4v4/IDp7Hx22+vc5kZ94OgMY7HaybSmId7jRqnOeIExAu5z19evn1fdvB+DH8A/uQLXmJO u91u8PWPFGEn/2RZieqo5WN2kdpFKMxxhwilL2gXoD6FZRv2ebfWJ7M3eKWhs+yShwE6uGj3PtPB BJuixIJ9jBwLBOIEFgjEq1kgoJqaBQKJKVogkB7TAgFZlgUCgjK/9lGrZu8ixbBA8LDM7S0QrAmd X3fF3jS0neWvn8bRvtrbrbBaLolZJRKsm6xjWYkgJ7ESQV7PSgTU9KxEUEzTSgT1uFYiwPKsRAAU 3aUBr3WXVqSU4mixSx1HMHQaR7fT6+AvhjzR1bVWZh594m8XB5Ms6/SKnOT0inzr6RUYpdMrKjWf XgHinl4R5Z9ekRafXkGEeXpFsuX0Cp8XnF6BZp9egdU9vYKgrCYt6STy3aRYiXxpPtzr+20aqkdG GDJnj4xffvn5X799H79/mC9zUrGYxHKQvvI3MjBLrfy3xtCPDH+jwgkq8IlXM6ogNZFRBSk0GVUg wTOqQJJhVIGYZLFGXmmxvkg1GlWMzszb/ty4LlMw+3MjDp2FxtU3z3XXF680dAfvlnXmhoT13RIY 3i2FE1h+Ea9m+UVqIssvUmiy/EKCZ/mFJMPyCzHR8wvwWs8vRep3tfzCuEiXuPDm0IJ0GQz78cWG sTccN2LiBG7ExKu5EZOayI2YFJrciJHguREjyXAjRkzURQp4rS5SRaoxLoL3dvBzt4sMHOyD27r7 Orbr4FlZ7YPt1MWSOCVbe64ObhqcZ1+UzNGxWiYgJ2mZgHxbywQgdFomoJBSywSUErZMQIm2lglA iFsmoIagZQLiSi0TUIrXMgFInZYJKCSqTpvf2crvcXXazG3ll1ndZLVZ0s4v81oXbPODdn7vOtTC tn0M9ba9j6HetvfxLWeD1BnTX5uKuu6QHLJG9r7dhtlx+vllTtJUFPnWpqLAKDUVRaXmpqIAcZuK IspvKoq0uKkoiDCbiiLZ0lQUPi9oKgo0v/u3U+3nR4Ky7t9OqZ/fVYqZw3+oukrj3Mf99RoMDWfO aneT0SVTzfo0VDn8Ka5hFGwEF85kVDhJDv/SOhllRimHf2mejDLEzeFf+JNRocU5/AtvMipkSw7/ wp+MMs3eFi2qk1EWFG2LFqXJ6CbFuusfxkM/P9gErXPl+RS3zb49GaV0veSvivZttGHauM/32LNw M7zmosCJmosC39xcdDNazUVBqb256GbYzUUBFTQXBVreXHQz3OaiQDY1F92MpLnoZvjNRTej3Fx0 k1yWEa/WXHQ7Xpa9dzJax+VQjz5v0VT16DA0vTkZ2a44iLgYbJ006+fkuTsjbCvfLaxjGnKSYxry rcc0bBStc0xDpeZjGkDcYxqi/GMa0uJjGogwj2lIthzT4POCYxrQ7GMasLrHNBAUHdOA1zqmFSnO ZDRv2H51Pxmtth/H/WQEQ9PbKVm2765vWn06vGl1HftNyyzr6hmTUeb4k1Hm2yYjYlQmo6zUOBkR xJuMMsqdjDItnIxIhDUZZfL9kxF9nj0ZEc32fQBW1/cBBEW+D8Br+T4UKVZB0bAej2nLtFU7Ixw6 2xmVvBOYi+y1nWnownEusoJmKF0aWT2DkJM0Q0Fer2cQqsl6BqFCW88gIJg9g4Dk9AwCTGRhBbyW hVWR0so72cY67yS91eQXoiIU+yq00K5edabe2YVrEYt5Nd1omksPCscuPSj86W/hA/sqImyQ21dl oUnDvipLrRL7KpJwXYN9VSaCyL4qa2xc+yrCfa/wol6kTP5p/BOw248A/4S/Po/8IxBaM7R8+Lq+ ft/tOg7DKOMfJrh1YzBXHfzwTgEHcI46+Sfg1vaf1Y7nn/c7HjOffMXRVJkAuMVaap1H/xVU/SPr BRBNoWEXN56EGG4F80e3L/nDuPkcP758+rx82H6D+el+a3n2GYyXx2l83YiluXv6kSJOWMdinKww mYf7xX46+W5PqSxe8CsB/3m357sM4fx/8i3B2p96Y5exx5u+/N/od+6xoCynstNZ225S+xpVNFis Vrf8BdXLm9+6enmD1f6wvI1nnfDuTqDRlBNoiCnUGWMJfq7cu/lop3E1XfP6lrmBvb5l3jesb0TY Tr6+ZSGrsb5lqSBZ37JEaljfiKAVkb++ZY3IXd8yPmqsb1lqblvfCPJBvL5lnZmzvmV0Y6xvRIZZ d30jUXyhk6xvWcQd1rfXz/Cd/Prbw3Wt/DeMD3Pyi4Kphez1jBRO7j8erWf0+Xfdf7ju5G9j3n8Q q3n/QYKCpZB4naXwKsW6/7BmOnjLL6ar7j9w6B0vQ+niVAzrXqqfqY1L/F6E3dIzWhMVTmI1h3yr 1RwwSlZzqNRsNQcQ12oOUb7VHNJiqzkQYVrNIdliNQefF1jNAc22mgNW12oOBGX78l7LO+YixUvg C7XVXBg7iIDqmbrDhfDtyShdO7iHutppHr0XmPBua9++Ly8ce19e+IZ9eSYU9uVFSGNfXqQk+/Ii 0bAvz4RsX140uPvygmvsy4tU2748Q/J9edHh7MsLytiXZ1J5X55FhfvyIqK4Ly+Kgn15Vnj/vjx/ nrsvzzQ7fRRY3fRREBSljwKvlT5apHjvkvGwFIZlCbbyo18wb+mtpdD1rrg/RRtDvS+Pcea7qHm7 Os5SSJxgKSS+aSlEQmUpJCGdpZCkZEshSTQthUhIl0LS4C+FhOsshSTVuhQipLEUkg5vKSSUtRQi qb4Uoqh4KSQR1aWQFEVLISq0LIX4ef5SiLSgmZZugSkKCptpaRWYXqR43RAenArd6Hx1KoSht19r XB9La5be9XU3hBCmlZ+jE/ph4SyFxAmWQuKblkIkVJZCEtJZCklKthSSRNNSiIR0KSQN/lJIuM5S SFKtSyFCGksh6fCWQkJZSyGS6kshioqXQhJRXQpJUbQUokLLUoif5y+FSHOXQmRVl0IUlPmjDTqp 8zcp1qkwxMNrzeIml/ZLIQz5/u2lcBguSyGV9FQXpIPpmEshGog7y0hcyBw/cSHzLYkLRGgkLmQh lcSFLCVKXMgSLYkLRAgTF7IGO3Eh4yqJC1mqMXGBIIXEhazDSlzIKCdxgUjtxAUSlSYuZBHNxIWs KElcIIWGxAX6PDtxgWjuWyGyqm+FKCjrM+G0vHQvUqxTobcHL92586mqIoOh8e0qMmfyBekwhCH1 BxPdYLiJC3DynQ3L7IM4gdkH8Y1mH8jomH2QUqvZB0JMsw9C2WYfREvNPlCEZ/ZBZIPZB36eb/aB NPuKCljdKyoQFF1RAa91RVWkWCWt07rWk9E2uKUy9oah9Q33YpyLShWZSybFuicVTEf8GyoXpoFT RUacxIESebUqMlITVZGRQlMVGRK8KjIkGVVkiImeMYHXesYsUo1VZP0Y0roMcdgVktHoNPV2uk/q uY7OZ/1srzbGCVtoF2euYFJthrNuxkpsjIcZZ+J2G2PgZva5lXic0bVsjEHNqfSyKWJJq5dN1sN4 oMj58hl++X7Zlm930XQdw4hKj3wVUCHdKVQwxvxDa+ZhXtYciDfq8ifauj+GZGbJg1J3+6vULgyH 7rg+4Zezi0M39N4tfbS7OIRR19kQ7S4OL6NnduK3ZcrdeuF0w6Gt9JwoHljGCyn0jL4TheObiWde y0w8q0nMxLNCi5k4ESwzcSLbzcQJk7gAIK/kAnCRalymsOvwuJk475YpGl2iD7vGhddR68/8xNev H68BYi7Xq13aJ8OHzvsxcJcp+NunwWIFafOZErjQCc6UyNumpBskVo2kGxDCo6lG0g1KOVnSDUqE pqQbJFZh0g1o4CGXm3SDeNxdrx6uVelb9sAAnFiYzL5/+gLf5/xX/Si/AfsxVHiYpwMKWDx7/2k8 Kh41cBRV7KN7BVSZLir3FuT3I/gNPPkZiMzHUYBvPo40++ANrO7BGwRFB2/gtQ7eRYpz8PbzVt8C dpNz0/5BDIfmdzyIOX/1H/f94UHMjYKDxRw5lbyZ4z+IZb7lQYwIjQexLKTyIJalRA9iWaLlQYwI 4YNY1mA/iGVc5UEsSzU+iBGk8CCWdVgPYhnlPIgRqf0gRqLSB7EsovkglhUlD2Kk0PAgRp9nP4gR zXe/j6rFcyQoc7+PSsVzVyle8dzYH5pLwjGjbi7px3h2B323FPquFM+hOd/RcHpln+5tFzZGmmTh BEsh8U1LIRIqSyEJ6SyFJCVbCkmiaSlEQroUkgZ/KSRcZykkqdalECGNpZB0eEshoaylEEn1pRBF xUshiaguhaQoWgpRoWUpxM/zl0KkuUshsqpLIQpKlkLklZbCixTrObYz9mBqMXhfVQzg0FnFwO2e 22dPi950sV4JbYjLIjD17BdWf2XiJKaeyOuZeqKazNQTFdpMPYFgmnoCyTH1BEx0XQK81nVJkfpd m8liXFwvS7wbDnFhAjcu8KUrBF4zWeBEzWSBV2wmC2rCZrKg0NhMNgRuM1nMFmM0kw2yzmHAq72P hmP+zjvef4Y4uN5sfv/+A6N2Gzfr7tMUrqP+7SweHy8de3zcLxveu2Adt+QaHVY2x+tBDpyoBznw ij3IQU3YgxwUGnuQb47bg3xzrB7km6ACM/NqvjzbsQLz7fAYzDJswXT7lYNGYWdk0v3icR0d384e COFilQ67sGN4sBNuaePIakVOnGD1IF5t9SA10epBCk2rBxK81QNJxurRyborEa933OC0IlfZVd1d u0V/DYuur3I/0zwLmk76yQ4sow7kJEYdyLcZdQChY9SBQkpGHSglNOpAiTajDiDERh2oITDqQFzJ qAOlmo06AFIx6kAdplEHojyjDiD1jTpAVG7UgSK6Rh2oKDPqAIUmow74vMCoA2h+MoYdlJMxrCjZ G3m1ZAx7TPZ+7wvUMh6qk5NZh6nqv2xWY95xwRCvW8TBmkM3HbQ1ZNdBTN3KqoNATlIHgbxeHQSq yeogUKGtDgIIZh0EkJw6CMBEdRDAa9VBFKnWOog0x21M274MAgZXvEO7j43LYHjb3NRh4UMODtvt i4T85uPcs2/f/BD6mVO7T5ygdp/4ptp9JFRq90lIp3afpGS1+yTRVLuPhLR2nzT4tfuE69Tuk1Rr 7T5CGrX7pMOr3SeUVbuPpHrtPoqKa/dJRLV2nxRFtfuo0FK7j5/n1+4jzW9H3c/K7ah7STkS8Wrt qPtjOdJ7N4r9sh0c3aa1mytHtwnNq99eCk25aYddZXWVuI6+c/w6Czc5UOXsE4Fzkn0i8KnJ5xsI bEQm9/mebh4oUp9vlBplPt9TdlJp8PkGApuiiXy+UWPi+3wjvvDqLJCNvaTOghTSRSHPpP29QBnC L96ffAPlntiTruVLUZR5Yk8tli/58wJP7Ilr+ZJZXU/sSWL5knm1QxTf8mVLx141s92W2vLFbuuZ 5cvtAap0h/ChhzngWL468c3PhqHnPEBlTvA+S7za+yypid5nSaHpfRYJ3vsskoz3WcRE+yngtfZT RarxdmGa+hBiH3e3CzjousGm+9zvyyDMgY9j4/XD63+PJTz8cM3uSdWmCkuh+o3bPAUfos3QXrya OX7xauEbilcLIS9ezUIqxatFSlK8WiQailcLISpezRrs4tWCs4pXCysoXs0K0uLVosIrXi0wv3g1 C3CLVzPNb3dlTt9LznczJ0dgFJOl1Bidt5Kb1G7Kdg93MvuUgaWLYd2nDLjoRrdPGcChNxNp4Bx+ rVmtX0lsiHAm5uaZ2TAby0rDRE6Shom8XhomqsnSMFGhLQ0TCGYaJpCcNEzAZCVsRsnT8SrVuI/Z 3Bw6jz54d3EBg36dHT4r3iLjMjideXDAYnoJjmLr6D0E3cHWMXZeYuvYUauLdltH4IIgnQb5Kdsx 4few/DqWP9Gt9kObvY5SRr//4+fyy/7b6wxf420Vf/RfQG19lAuDYu4g9lAHJM4Ucvt5vFD4vsK6 //nvr9e90P0gxlh38o3ExRS3Th9g4f36+X5rVo/iN2Z46FPZUbcP+Pyy/kDk+/4X/8F/wPv2o1NW 0Zr3WkcR+pec/EJNW/cTfDD/2y/w3QDujU7+4tnSyn7HXf4E0HwGTbu/72U7/I0vG/571xN8KVPc 3d96+zPtBh7lngC5mDzF3YHXP9K6f/IPlllFdmrdTIrUfuG3h0mOvp77ZNpxi2lzQ79PpsXROAzT dH//fR2dT6/A75IlvLkkDqauP1ZjxK1f2Zfg0SQMtOZLcOQWwSU48FOvliyBal6ULIEK4Tbz3vD7 EfrtfdT5EOlLvHxbP67z/f45D+Bl6qMJF9llzhFzj97+jD+7RxeJAIru/YDXuvcrUruQSeYQMvnr 2W2XYzeZZPfb5TWMfn/tR0Nv9//xti+F7c75ypUL33o69sWGDePMcHouHL9FdebbWlQTo9KiOis1 tqgmiNeiOqPcFtWZFraoJhFWi+pMvr9FNX2e3aKaaLbLBrC6LhsgKKsUm/UqxWa20/Nm4qG0eB3m aPZvEDg0/c9ndwfL8sVCczB4UqkWbUmGo58D4w2icJLSYuT1SotRTVZajAptpcVAMEuLgeSUFgMm 29cGnTeIm1RrhuMGU0U/hsrpGUe7zprxvhTmOjq9XQ3jiwlN3/XOdXUG8Ly4wF2rDRbQM5IcM8dP csx8S5IjERpJjllIJckxS4mSHLNES5IjEcIkx6zBTnLMuEqSY5ZqTHIkSCHJMeuwkhwzyklyJFI7 yZFEpUmOWUQzyTErSpIcSaEhyZE+z05yJJq5UyRWc6dIghJPaeSVPKUvUpyd4jJOQ71TtDD77195 aGh580UeNovp0qJo6L05HFz9JvChcZZ1w0Oc4IYHeb0bHlIT3fCQAvuGh+jlvY0NHm49QWH2jY0N MrZMpbFB1dmgYOFR/Rxgon0n8Fr7ziJV3acen1K/VL0N4jCMg4l+fzmUNrt0+8shHDr1bP/l53/9 9n38/mG+7juL5VMXja/OZX5Iy8Led/qB1YGncJLcMMUOPEVNlhvW2IEnE8zcME4HnozJcu2VOvDc pBrPZaMz87YLjX7wycRdaNDQ/P7Q8OX6NKZUpRtE322Wa1BgbZhSss2rUOHYq1DmXd9kkY3EpGGR jUKbjkU2SPlBZpGNErbJIhuJSWiRDRo5YZ9nkY2407HIRqnQapEN0CW1T2SRjTqB5QtK6MiyyAYy BWVfUBTF3aTMIhtE8DijaJGNil5kkQ0Ks2uwyMbPT2XF+hUbZL1+252rrmO0hJxMTZeWVZXAbgi/ 5hNcdOUOvNaVe5GqdnjHNez6Ne3XMT9V65jb1qFax9y2vZ0uV2zZ8EoxHK8UV7abp4Uja+Kk/WdO 0LWKeLWuVaQm6lpFCk1dq5Dgda1CktG1CjHJDQPySjcMF6nWK/c+bd3mx2V/5Y6j8xpA+v7KPY/G rntHeKTL9s7YcHiRWnt2VQzMAJ5n6omcJJsUeb1sUlSTZZOiQls2KRDMbFIgOdmkgMnWDa/2VFuk Wl0LZ+sX16V9Uzca7Q3suO9NPa+j/Vnv0V2iVchvtn6INtr6Gg7WEEF33tGEyOgUTpygUzjxs9Y1 HKqlXnINRwpDkyvNmPt3//J/f11/XV9+WX/AVrAEyf0QbjbjI5MZfJm4dBTNH4Xd4F2UXcfwX/wo 0QoF1p3Agf4X3KU8Rhe7/7t/Xl/Xrx/m4z+g/AeM25NfhGXM4X7/j7j9GbfsjwqeR1kXbuS18sTG B124ne8OIZ+/oN1dYPLGhbjfKMatM3vbdxo6y814GOohDK42Y/RzivycymHAK532UCfO8kOd+KgV 6qS2SEKdFNaWUEcCX6p+/fT9H//3HrsboOSvB6GG7AK/5svnu9T58gf6Gh/8PhIDf9/LC/4FL5+/ 5K/o1w8QjV++fvhxLeH6Hz6AvwUnP4xLWucFvgXsZQRj/UH+NcKSkEVeKWQvUvtHsgfp0JcvaRe0 YfDbMs33QbuF5Nb9LSUNNQftUBkL+zglwfochp4TtMQJgpZ4taAlNVHQkkJT0CLBDVpkW4OWmOcE LUlzgxZh0ToLvNY6W6S0gtZ0wS37oM1DzUFbvyoEmySJkGscOUFLnCBoiVcLWlITBS0pNAUtEtyg RbY1aIl5TtCSNDdoERa9k6/YX1znnbxI6QXtOvs6aHGoOWjt4Sp17rmVFPB1LotnBS1ykqBFXi9o UU0WtKjQFrRAsIMW2OagReZJQYvS7KAFWBS0wGsFbZFSC9o+pbUKWhpqDlp3rIiOXLsA+Dq3jXWm JU4StMjrBS2qyYIWFdqCFgh20ALbHLTIPCloUZodtACLghZ4raAtUnpBu0RXBy0ONQetP5g9T7Mg 9XMMjhW0yEmCFnm9oEU1WdCiQlvQAsEOWmCbgxaZJwUtSrODFmBR0AKvFbRFSi1o4Qcfq6Cloeag DQ8KjQXtC6KNrIso5CQXUcjrXUShmuwiChXaLqKAYF9EAdt8EYXMky6iUJp9EQWwzBjARjVjgCyl FrRmmWwVtDTUHLR1W4U5DQN/e+zWcWIFLXKSoEVeL2hRTRa0qNAWtECwg3bFe7PGoEXmSUGL0uyg XfEGTxC0wGsFbZFSC1q7uPoiioaag/Zo6eEW/jutT2ZjbY+Rk2yPkdfbHqOabHuMCm3bYyDY22Ng m7fHyDxpe4zS7O0xwKLtMfBa2+MipRa0zg6mCloaag1aX/VHDsmPbP9tCPnUs860xElWWuT1VlpU k620qNC20gLBXmmBbV5pkXnSSovS7JUWYNFKC7zWSluk9II2zmMdtDjUHLR9lVxhsX6FHbShmy0n aIkTBC3xakFLaqKgJYWmoEWCG7TItgYtMc8JWpLmBi3CkqBFXiloL1JqQevjvFVBS0PNQVtZQ8LY tPLfaUM3dbygBU4UtMArBi2oCYMWFBqDdur4QTt17UELzLOCFqT5QTt1sqCdOrWgzVJqQRvsUF9E 0VBz0NYZUTbMThK0kXV7TJwoaKPi7TGpCYM2Nt4eI8EP2th8e0zMs4I28m+PEZYFbdS6Pb5IqQVt 7NapCloaag7afUaUd7abZsGZdh5YGVHESc60yOudaVFNdqZFhbYzLRDsMy2wzWdaZJ50pkVp9pl2 HkQZUchrnWmLlF7Q+jnUQYtDzUHr6ttj50NivtNiSy9OlU/h2EFbeKWgLWqCoC0KDUGbCV7QZrYt aAvzjKAt0rygzbCsP51Slc9NSi9ol3gIWhxqDlp/CFoX+ckVw+BWXmkecKLSPOAVS/NATViaBwqN pXlu5ZfmubW9NA+YZ5XmgTS/NM+tstI8t6qV5mUptaBNydRVPjTUHLT7jChnU+9H7kqLXWVH1u0x cZLtMfJ622NUk22PUaFtewwEe3sMbPP2GJknbY9Rmr09BljWInnUa5E86t4ej+PaV0FLQ81BG+uL qLjSLw83aFdWcgVxoqBdFZMrSE0YtGtjcgUS/KBdm5MriHlW0K785AqEZUG7aiVXXKTUgnYKfZ17 TEPNQZvq7s1pC4KmoBPnIqpw/DNt5rXOtFlNcqbNCi1nWiKYZ1piG8+0mXnKmTZLM8+0BMvcpZQu om5SakE7h7VOY6Sh1qCtrAkxjXFYyHeMt9JujlUET5xkpUVeb6VFNdlKiwptKy0Q7JUW2OaVFpkn rbQozV5pARattMBrrbRFSi9oIUTroMWh5qDtj3ZwmyBofeJtj5GTBC3yekGLarKgRYW2oAWCHbTA NgctMk8KWpRmBy3AoqAFXitoi5Ra0C7LXF9E0VBz0FYeUXY128wvGAjdxApa4mQZUYpBS2rSjKjG oEVCkBHVHLTEPC0jih+0CAszorSC9iKlFrSrDfVFFA01B22VEQWbeBhkZ0S51bKcK4gTleZZRecK UhOW5tlG5wok+KV5ttm5gphnleZZvnMFwrLSPKtmoFqk9II29qkOWhxqDtrKIyq6fk3cdjG4oxhm 3vYYONH2GHjF7TGoCbfHoNC4PUYTA+72eJjbt8fAPGt7DNL87fEwy7bHw6y2Pc5SakG7maUOWhpq Dtqq/dlsgpvYPZ78EGaGhWrhJBlRyOtlRKGaLCMKFdoyooBgZ0QB25wRhcyTMqJQmp0RFWaBhWrm 1bqyzaoWqrYbfPXkk4eag7byiFrH2XpBckXseb7HyInsZnpN32NUE9rN9K2+x0Dw7Wb6dt9jZJ5l N9MLfI8BltnN9Gq+x0VKL2jDstVBi0PNQVtlRHWTweMRO2hHn1hBi5wkaJHXC1pUkwUtKrQFLRDs oAW2OWiReVLQojQ7aAGWNQXxWv2BL1JqQdv3sbo9zkPNQRsPPbLcINgem4XhXFE4yfYYeb3tMarJ tseo0LY9BoK9PQa2eXuMzJO2xyjN3h4DLNoeA6+1PS5SekG7TnMdtDjUHLQHjyjKPWcXDBjOmbZw koIBo3emLWqyggHTdqbNBLtgwLSeaQvzpIIBwz7TZlhUMGCUzrQ3KbWgHfwapn3Q0lBr0MauasCV hj4IOgxMnud7jJzE2A15PWM3VJMZu6FCm7EbEGxjN2Cbjd2QeZKxG0qzjd0AFhm7Aa9l7Fak1ILW 2FRfRNFQc9AeMqKwTyO7YMAvC8tClThZWxBFC1VSk7YFabRQRULQFqTZQpWYp7UF4VuoIixsC6Jl oXqR0gvaUPfyyUPNQVt7RPWrZ5uVW9hRbIZTMECcoGCAeLWCAVITFQyQQlPBABLcggFkWwsGiHlO wQBJcwsGEJYUDCCvVDBwkVILWhuHqi1IHmoOWnPsMNDzkyv85FhnWuJE22OneKYlNeH22DWeaZHg b49d85mWmGdtjx3/TIuwbHvstM60Fym9oJ2G+vaYhpqDts6IMvNouGmM1oZxYxTBF05Smoe8Xmke qslK81ChrTQPCHZpHrDNpXnIPKk0D6XZpXkAi0rzgNcqzStSakHrbFe/09JQc9C6utXl0o/cMy3Q 68o40xaOH7SZ1wrarCYJ2qzQErREMIOW2MagzcxTgjZLM4OWYEHQEq8TtFcptaD1JlUWqnmoOWgP HlFYnMctgveDnxm5x4WTvNMir/dOi2qyd1pUaHunBYL9Tgts8zstMk96p0Vp9jstwKJ3WuC13mmL lF7QLlOqgxaHmoM2VNtjUGG3urS+W+LACVriBEFLvFrQkpooaEmhKWiR4AYtsq1BS8xzgpakuUGL sCRokVcK2ouUWtCGVNfT5qHmoI3HBlyWX5rn08B78kFO1IBr0HzyQTVhA66h9ckHCH4DrqH9yQeZ ZzXgGgRPPmmQPfkAr9aAa9DdHsfeVr188lBz0B4youw48M3K/TKyco+JE73Tjoq5x6QmfKcdG3OP keC/047NucfEPOudduTnHiMse6cdtXKPL1J6QettqIMWh1qDNnWHtiBrYt8ew47CsMzKiRNtj42i WTmpCbfHptGsHAn+9tg0m5UT86ztseGblSMs2x4bLbPyi5Ra0CZXW6jmoeag3WdEOTf2MQgyovqN 1cuHOMlKi7zeSotqspUWFdpWWiDYKy2wzSstMk9aaVGavdICLFppgddaaYuUXtAuW10wQEPNQTvU Tz7z3PMLBjw2FOIELXKSoEVeL2hRTRa01FipKWiBYActsM1Bi8yTghal2UELsChogdcK2iKlFrSj C/X2mIaag3afEeXW6N0isJsJkddUGjlJPS3yevW0qCarp0WFtnpaINj1tMA219Mi86R6WpRm19MC LKqnBV6rnrZI6QXtZuvtMQ01B23VNa/3k18Ft8ejYzz5FE6y0iKvt9KimmylRYW2lRYI9koLbPNK i8yTVlqUZq+0AItWWuC1VtoipRa0U+or3+M81By0VUZUDDZZ7jstGtjx+tMSJ3Jj1OxPS2pCN8bW /rRI8N0Y2/vTEvMsN0ZBf1qEZW6Mav1pL1JqQTu7tX7yoaHmoN1nRFkXsdUge6UdBt6TD3Girnma Tz6kJuya1/rkgwS/a177kw8xz+qaJ3jyQVjWNU/tyecipRa0i5vrNEYaag7acMg97rygl88cOlbB AHKSggHk9QoGUE1WMIAKbQUDQLALBoBtLhhA5kkFAyjNLhgAWFQwALxWwUCRUgva1QVTBS0NNQft PiPKw2odVv5KG/qO12EAOUmHAeT1OgygmqzDACq0dRgAgt1hANjmDgPIPKnDAEqzOwwALOowALxW h4EipRa0m52mKmhp6DRo/+VfriHrKGStjc5VXoxpXfA7xr6GmqdkOsY1FHKD4BoKeX8WIsujqyYg LPxLv3769vL14+f5l5cNfiTTOP+S6eM4KC3diRDMNv/cvr1s8KOcvi0vX9f54/jhU1Z68B/wHzWd SIXup7/+559f/vKnP/77H/7tP17+8Oe/ZplqEMPXnUik7qcfP39+Xcuvev7fCIyPbsyAcPCN+PH6 8euv9A99+fDpywXdD6JG6E80Imq8wDfr8/zyt/HrclG4G8Jv4clP38He7uf1++v69x+vn5cVvlPf 1q8/yrTx4D+A1Hjy03Bz/mn8E7DbjwD/hL8+22PIhwwtH76uMCl9/Pz5l1+/3OjdMMr45URnvurg h3cKOIAz1dk/Ycto/Xdf/1Izn3zFYd7//s0jxmqt8+i/gqoPj0XRRvkHYPTx3T+qHsWfxUmIwa/W T/mj25f84ZfXz/Cd/Ppblnr83zA+zMkvSppqxUdKOEHNJ/Exwq/aK4x/yCD9T/yRTiff3YmiA6bk 9fVl/rrefrGvI/i3dSd/G67Dv6xf86fH8gt5N4Bfazz5EcCh56ftw8cVlpWXv38dv3xZv+5kHv83 /FpO/jGiC17gtS54i9R+MUzpsBjmL2i3FK4RX2LHu6VwCCYs/Xq/FNLQ4N9eCtG3iZZCk+wh92Gc Rn4RzuAX7zZO5dzifS+pnAPevX8pLMQmXgqzUDAKS2GR8oKlsEjE9y+FhdgkS2HWiIG5FBY87ZbC wxJI37KHVYPAwozx/dMX+D7nv+pH+Q3Yj6HCcfXMCjhj50/nmbS/FyhD+MX7k29gmkXLRhYZe71l oygG/rKRFSb/3mWjfH4uE/+37+P33cSfB2iePWHxAIbnm+2evRugg9hjVFZruSg1HL5K7aZ454bD FF++ov0cn0bjxv0c343D1u/n+G403Xwyx3/55ed/ReUP83Wqt3mqd7Yz9uhsEAVd1cZuw9/Ydgd6 4IzEgR74oOdA3+Uervx3PFRY2hzou20KJU5Q3N+FCf0ZSBsfHDmQnEuU4AftLUboj7jPfrBFR0xm 9a52I3CR2kVIOMQHfTV30TE6M2+70Ojh/3XzLjR6GnvP9R3WVtAGKPl+qPoyrBP8G/kboOAW/Nm2 d0ACLgk2QMgvSnfupDYPgjv3rGDaOiABAV/Blw9f8Lw7lv3D9Y8YE/ZhxyC3LBav7vBgcANvfwYy nnyRS8rBdMMufwJofthryS2y/kRuUetPlKV2oWQOoYRfzG6dGZc+xHkfTKnf1mUfTDC0tQQT3YWn Qyb1Yvnddl0aPStVBDlJqgjyeqkiqCZLFUGFtlQRINipIsA2p4og86RUEZRmp4oALFr5gNda+YoU 8y68j0vVA7ALdb+EPPSuB6xb0JquKjT0g+0it6+9tWHpGQ9YhZO8OiOv9+qMarJXZ1Roe3UGgv3q DGzzqzMyT3p1Rmn2qzPAoldn4LVenYuUWtD2dq1SRfJQc9D2h5V2suzORGbZPMfxKnMSmzrk9Wzq UE1mU4cKbTZ1QLBt6oBttqlD5kk2dSjNtqkDWGRTB7yWTV2R0gva0Nd97WmoOWiHQ2ci69nW675b JkahYeFEJf2TWqFhUROW9E9NhYaZ4Jf0T42FhoV5Vkn/xC00zLCspH/SKTS8SakF7QDn0Spoaag5 aE2V37WZbRRsj+eZ0wMwc6KkzFmvB2BWEyZlzm09AIngJ2XOrT0AM/OspMyZ3QOQYFlS5qzUA/Aq pRe0qzN10OJQc9Daw0q7ScofjGElZRIn6gFoFJMySU3YA9A0JmUiwe8BaJqTMol5Vg9Aw0/KRFjW A9BoPcFcpNSC1gymMoTNQ81Buy80DMYNIfFL+kM3MPolFE6SSY28XiY1qskyqVGhLZMaCHYmNbDN mdTIPCmTGqXZmdQAizKpgdfKpC5SakFr+1T1S8hDzUHra5u62AvMc1wIvCcf5EQ+HEHzyQfVhD4c ofXJBwi+D0dof/JB5lk+HEHw5AOwzIcjqD35FCm9oA2+3h7TUHPQ7gsg/ObjPPJ7AIZ55NweZ050 ph31bo+zmvBMO7bdHhPBP9OOrbfHmXnWmXZk3x4TLDvTjkq3x1cptaB1g6kcr/JQc9DGQ/7esHKT K+DrnCzn9jhzkqBFXi9oUU0WtKjQFrRAsIMW2OagReZJQYvS7KAFWBS0wGsFbZHSC9qlq99paag5 aFNVarh13cK+PTbLtnBK+jMneqdd9Er6s5rwnXZpK+kngv9Ou7SW9GfmWe+0C7ukn2DZO+2iVNJ/ lVILWm+W+p2WhlqDtt9nRLmxG53l3x67zbPMc4iTnGmR1zvToprsTIsKbWdaINhnWmCbz7TIPOlM i9LsMy3AojMt8Fpn2iKlFrTBTfU7LQ01B21/DNrITuSHHUVibo+TcHucVLfHSbw9Ts3b4yTYHifG 9jg9b3ucJNvjJNweJ73tcdLdHocYfB20ONQctIeMKDNtgpq02DGanBROstIir7fSoppspUWFtpUW CPZKC2zzSovMk1ZalGavtACLVlrgtVbaIqUWtHFIdcEADTUHramefBzsuvlpjEMwnIyozEm67SKv 120X1WTddlGhrdsuEOxuu8G0ZkRl5knddlGa3W0XYFlRnlHKiLpK6QWt82sdtDjUHLSHjCgzC1yc h8GwGncSJzKENYqNO0lNaAhrGht3IsE3hDXNjTuJeZYhrOE37kRYZghrtBp3XqT0gjYFVwctDjUH bW29jskVbHces6yWU5qXOcntMfJ6t8eoJrs9RoW222Mg2LfHwDbfHiPzpNtjlGbfHgMsuj0GXuv2 uEipBW3qXH17TEPNQesPK21kW2oB7aeFdaZFTnKmRV7vTItqsjMtKrSdaYFgn2mBbT7TIvOkMy1K s8+0AIvOtMBrnWmLlF7QbmmqgxaHmoO2sl73/ep7buNOvHDzjM5EhRPdHnu1zkRFTXh77Js6E2WC f3vsGzsTFeZZt8ee25kow7LbY6/TmegmpRa0Y1/3S8hDzUFbZUTFbh5nwUo7GkYaY+EkKy3yeist qslWWlRoW2mBYK+0wDavtMg8aaVFafZKC7DMaM3opDHepNSCdhr6umCAhpqDdp8R5e1qtiRIrgiB dRFFnKxgQPEiitSkBQONF1FICAoGmi+iiHlawQD/IgphYcGA1kXURUovaOe6yUkeag3aofaIwn+o oNvutjHSGAsnyoja1NIYi5owI2prSmPMBD8jamtMYyzMszKiNm4aY4ZlGVGbThrjTUotaGd/SK6g oeag7Y8XUZMgaGfHu4hCThK0yOsF7eykF1Go0Ba06PbADVpgm4MWmScFLUqzg3aWmKdmXito5wfm qaKgXYe6CJ6GmoN2ONrNRP6Z1ruelRFFnKRFNvJ6LbJRTdYiGxXaWmQDwW6RDWxzi2xkntQiG6XZ LbIBFnVQAV6rg0qRUgvaZdnqMy0NNQdtnRFl53lluzH6bp4ZzhWFkxi7Ia9n7IZqMmM3VGgzdgOC bewGbLOxGzJPMnZDabaxG8AiYzfgtYzdipRa0K7Gd1XQ0lBz0No6aGG+T/wi+HHlddtFTvLkg7ze kw+qyZ58UKHtyQcI9pMPsM1PPsg86ckHpdlPPgCLnnyA13ryKVJqQbsNW/1OS0PNQeuOt8cLt8oH G5T2K3ypjG67mBci8IhC3mptj0ktSbbHpDC2ddvt19lyu+0CO1PQUoe1Xz/9+n39xzV278YwUo7t orLCoxC+E3r4n0DPnnz5gn67vaDOJ/Nq/XZ73TqfbYzDZPdhi0OhNWz3OVHe2xQtN70CH7eC4b3U Aid6qQVe8aU2GOlLbTCtL7XB8F9qg2l/qQXmWS+1IM1/qQ1G9lIbjNpLbZbSClrbBVdV1Oah5rW2 yomKwSZ2cR6mkcwMa7fCiXKiZjVrt6ImzImam6zdMsHPiZobrd0K86ycqJlr7ZZhWU7UrNTp8Sql F7Rb5+ugxaHmoN3nRDmbej8JKmrXyFtpkZOstMjrrbSoJltpUaFtpQWCvdIC27zSIvOklRal2Sst wKKVFnitlbZIqQVtP07V/XEeag7aqm/ebIKT+DEuXc9rwQWcqAUX8IotuEBN2IILFBpbcHU9vwVX 17e34ALmWS24QJrfgqvrZS24ul6tBVeW0gvatXd10OJQa9Ca7tBjoB/Zjz42jBvDJapwovvjTc0l qqgJ74+3JpeoTPDvj7dGl6jCPOv+eOO6RGVYdn+86bhE3aTUgnYwtg5aGmoO2kNOlHUb148R65lW VmMQ4kTFeatiYxBSExbnrY2NQZDgF+etzY1BiHlWcd7KbwyCsKw4b1XrzV6k9IJ2i0sdtDjUHLRH l6jUCW6P18Qoziuc6Eyb1IrziprwTJuaivMywT/TpsbivMI860ybuMV5GZadaZNOcd5NSi1obWer 9Io81By05rDSpsQ/0w5h5eREZU6wPSZebXtMaqLtMSk0bY+R4G6PkW3dHhPznO0xSXO3xwhLtsfI K22PL1JqQet6U1XU5qHmoK1zovCynv3kY9bOcy6iMicwnCFezXCG1ESGM6TQZDiDBNdwBtlWwxli nmM4Q9JcwxmEJYYzyCsZzlyk9IJ2msc6aHGoOWhd1WPAzynwz7SDGVlPPsSJml2Oik8+pCZsdjk2 PvkgwW92OTY/+RDzrGaXI//JB2FZs8tR68nnIqUWtN6Fyo8xDzUH7T4jynWTiT0/kdFtK68MHjlR Re2qWQaPasKK2rW1DB4IfkXt2l4Gj8yzKmpXQRn8tsrK4IFXq6hdVcvgbbChasGVh5qDNhzfaRd2 Cy6zbAMjaAsn6uYzqAVtURN28xmagjYT/G4+Q2PQFuZZ3XwGbtBmWNbNZ9AJ2puUXtDGum9eHmoO 2tolylobBYYzaeB180FOstIir7fSoppspUWFtpUWCPZKC2zzSovMk1ZalGavtACLVlrgtVbaIqUX tOOU6qDFoeagTYdml73ArtzPM+tMS5ykDB55vTJ4VJOVwaNCWxk8EOwyeGCby+CReVIZPEqzy+AB FpXBA69VBl+k1IIWk2OroKWh1qC1h4woOzozsoM2dqyVljhJ0CKvF7SoJgtaVGgLWiDYQQtsc9Ai 86SgRWl20AIsClrgtYK2SOkFbdzmOmhxqDlo+8NK23XsNEaYIyPnnTZzoiefqPdOm9WETz6x7Z2W CP6TT2x9p83Ms558IvudlmDZk09Ueqe9SqkFbYqHggEaag7afUaUtSFtPTe5Aian1LMyooiTrLTI 6620qCZbaVGhbaUFgr3SAtu80iLzpJUWpdkrLcCilRZ4rZW2SKkF7Wi2OveYhpqD9pgRhV1M+PW0 E8ParXCietpJzdqtqAnraacma7dM8Otpp0Zrt8I8q5524lq7ZVhWTzvpWLvdpPSCdp2nOmhxqDlo Dy5RMa78d9rB8ExUiRMlV2iaqJKaMLmi1UQVCX5yRbuJKjHPSq4QmKgiLEuuUDNRvUipBe3c9bEK WhpqDlpXn2nn1AmSK9LE6FBbONGTz6TWobaoCZ98pqYOtZngP/lMjR1qC/OsJ5+J26E2w7Inn0mn Q+1NSi9o51inMdJQc9Ae+uY5O/KdK/ycGHYzhRM9+SQ1u5miJnzySU12M5ngP/mkRruZwjzrySdx 7WYyLHvySTp2MzcptaBdhqk+09LQadD+y79cQ3bIIWti6KprKPg3hkGyObYpsjbHaCcn2RwDfwnZ j7/ckR9/IeaBAyExEFZoVvjp568QFeha+Lfx65Lx4ziGzLY9VkoJfv1/vJL1YfldL3/C79XJl5y2 7qcfP6/fX7YPrx++lS/4bgBQuzxGR/guzyP8SODv+PzLryUm7kcowMMJDd+rH9u3l6NCPYqR9XAv DyJLFqnxK2jdyV8/+e6nVxj/kCH6nzi3rP3J52EC+GX9+vry7fsI/5epuwGK35MfMc5lOFVs9+zd AM1pj1HZJt1qPTxdpHZTh3PDYeooX9Fu5pj7bXDjeDdzDGEeRmvuZw4a8tvJzPHpl+3z63f8e2+L vqUZxMJROlRp0G7oR8PNqMQGSp0xHacbGHCDYKeOvD9bZJeHHb86Y+Ff+vXTt5evNEds8JOZxkv8 H8dBaXnwW0pCMPn9E2Jng5/o9G15+brOH8cPn7LSg/+A/6jpRCp0P/31P//88pc//fHf//Bv//Hy hz//NctUgzg/uBOJhPPS59f142VKwv+NwPiwcVlnHHwjfrx+/Por/UNfPnz6ckH3g6gRHkQ5aeCk 9PoC36zP8908vBvCb+HJT9+N3U8wdb6uf//x+nlZ4Tv1bf36o2w8HvwHkBpPfhpuzj+NfwJ2+xHg n/DX58EKgJAPGVo+fF1hW3M/Kx6GUcY/mNlJZ77q4Id3CjiAM/LZP2HLaP133xaE+eQrDvP+94+W gKXWefRfQdU/mD9RFK3Y373KjCchltxlodu+5A+/vH6G7+TX3+6XvPq/YXyYk1+UNNWKj5RwgppP 4mMcH65f/XTy3Z0oOmBTt76+zF/X2y/2dQT/tu7kb8OdPC12+Ond6pcH8GuNJz+CBdbo7cPHFVaX l79/Hb98Wb/uZB7/N/xaTv4xsj57nZoRXJHab6dTOqyJ+QvaLYlmwePdbkns/Dr0y25JxKFhftcJ ONBi2Hd9CMekR5iP+Q9E8xpTz3ggQs4IHoiQDy3b6cIs8u10VsINasN2ukArYzud0annbacLHUTb 6SIyt2+nMzm7926ny+enMqHMf4M15W4+oT8DOZ98gx/NJncaD/8TTQaP5UQvV8BrvVwVqd1c0h+3 1/Tl7GaSEVOa9zNJn/ptHXczCQ5t3clM8uEVNu3wF17mkpiP5j6lumNndNQjgL+xtngZwthYW5gK JBtr4G3bxhqIVWVjnW8jVDbWIOWEG2uQCG0b63xDIdtYw6roBRtrwONuY33YUNO37GGLYWBhUvv+ 6Qt8n/Nf9aP8BuzHUOHxXtym0e4VcJI5auAoqtiHO2NQmS4qsNu6Xkfej+A38ORngOsK/oz/+Jc/ /eG//rT/ud/G6Kf4WOBd+z/bPWzxbBN//2eT8v5PdieCvNr+78GdyLv2fybEyXUphN0WEEftMvqw 3PfKuI6u5uxu5NvP26fv19k7XXaCts7vg53gMLFThcyywfGYVWgKnKjQFPimi9XCKFysZqXGi9UC cS5WM8q9WC207GK1iDAuVjP5/ovV8nnWxWpmWRerGZUVvBqdh9SbFOti1Q5uqS5WOzMus9ufImFo iW8+ycAGK+/7XIo2HlIfRna+kvVDCFP7vi9z/H1f4Rv2fYWQ7/uykMq+r0hJ9n1FomHfVwjRvi9r sPd9BWft+wor2PdlBem+r6jw9n0F5u/7sgB335dp3r4vs4r7viwo6UWKvFIv0osU595vXjtT3fuF NAa/P63jUBhOZuyPHz59+P7tOmn3JfVl6LpwTH1JgsP6aKd+47RHs9PQS9qjAe+aDutIbBqHdRAy RuewjlJedlhHidh0WEdiEx7WQQOf77iHdcSTzisYSk2tr2AAYd2h+BUMdSbeKxiiK+sVDEg/ab+C gWjYpK9gIIJnOMVXMFQcRa9goIAns3e/guHnt7Kerf9Y58uv5N0ARfPJvxazy/BscY/e/oz/yod9 +azMEAt4tRaB9rhuueOylb+eu2Vr6NMydHPvJnu3cuFovy0hxflu8bqOrmfr17e/Xdeuy6MVLBeH R6txmrnuk7hE92Fsf7TKHP/RKvNBJ22zqM2CqqassDRUNRGBT0kUEyju72KE/ozbxfhwt9+HuZzH 8YP2FiH0R5wtj/cdhMk2dr1S+dBVahcg4RAf9NXch4dZhi2YLo33R3Eadd4Oo78Pj8vo1L/nNi/0 XUlysl0/VAatW9cbwfZuHqb/19y3LEuS41bu9RXay6bN+Sa1kFmbpheyGdnI2sa0TfNndU5lZd7O zKp+fP0AJD0inB7eGQTQnaXWooLp59wbN4IgAAIHlDLJjOO0IyC+s0wyY0TKJJGpu0wyg2hlkgil l0lmNLdMMpOQyiQR2VMmmZ8nlkkillgmiVDWETsLydfdqUjZPL35tc3mOfAkm5oQTLBe3eTez1ZV 40KsUmn6mLxOy0gffKL1rCmXuIjjXOJmfNclbkZIXOIikdAlbqbiXeJmiq5L3IxgXuIiB+MSN8OJ l7gZy7rERQb+JW5moV7iZjDnEhcJ6Je4iKZe4iJW9BIXCVmF7YCXKmyvVP3JPHCftV3HaA/5vLwK JxlEi49u3766hiu3717MF1S5wrXgJyZ7zul5qoIhCkWYRIiLCo4eFxW8VFxU2DhxUWHoiYsyghQX ZWR/XJRhrDtKwEvdUVaqzrhIhWHC4bvuEBfl1SEplx7jotvqeOXaHOMiXe8qVYqhkdOOWHLC0At0 U6DERRnH6fhEfGdclDEicREydcdFGUSLixBKj4symhsXZRJSXITInrgoP0+MixBLjIsQyuo8dULC LHcqUlzk3ZZOVQ6b9WtT5bCB+b0wHn98PF61qserS42qd56fEeg5Fa/cQgmNEMcJjTK+KzTKCInQ CImEQqNMxQuNMkVXaJQRzNAIORihUYYTQ6OMZYVGyMAPjTILNTTKYE5ohAT00AjR1NAIsaKhERKy jLYSE+bZqSh1DtZC/NP0N42LMsc6B1yy6TWHL+wxURjMSUxLefoohqBWT3H4Mo7h8GV8p8OXMSIO HzJ1O3wZRHP4EEp3+DKa6/BlEpLDh8gehy8/T3T4EEt0+BDKsR2IF7IdOxXJ4XNxmVqHb5umbTga jw3+xC/VxAe9J1RCaoSqvVvhe8W4aMbLrP6a+IKj18QXfF9NfMFI1MRnpt6a+AIi1cRnKLkmvqCZ NfGFhFITn5EdNfHleVpNfMbSauIzlHUPD3ipe/hKRauJj9P5Fk2NLjS3aGr047dr4s29vDKq8/27 ol6joSp4ioSa+Ixj1MQXfE9NfEEI1MRnIpma+ELFqokvFD018QXBq4nPHPSa+AKn1cQXLKcmPjOw a+ILC7EmvoAZNfGZgFwTn9HEmviMlayJz4S8GQdJyGTfqCix4jIG19bEz3b0SyMPZeG5b9Y9mKIK pZSKOrR1D8p6hn5r3Ga8AeoumEJcYhRMIX6RuTsrbLOm351VBtOl34oIeAdv79+w5nus++32Ejeb faZ3CrjFogwkbos78P4akPHiTS6puEF32P4KQPOzVnGce8MpIwK8VBlRpTrsJnPaS/hmHu/bwOrP RoUY58f7NlidtiVO8TF8uq2mywvpQwRl0n7flkITQW2rdiNnqF6k3bchjqWwGvvv2xAjc98GTP33 bQgi3rcBlHHfhmj2fRuS0O7bANl134bPU+/bAEu9b5sj774N8GJKr5F+32a9mdsIarZhiscICpbm K6HXD8v86eP2/ofdfNhhz96qprXYpaTXjTp0yNqwmCVQ5KkQx5GnQnzoCKMyIip+GFWInEQYVagi J4wqFGNHGJURqCjGCKMKR6KGUQU+CXSpVaqlfBqPscxfHyMZvTwbAgNItKqHfq6G4biOu/2Z/hYS MWKiTECOiTKaGBNlrGRMlAlZml6Lkbk/u1P1x0Rem8Vqd1QIxEUT0+If2632xfm1dhJb01k62ti2 k6ybs9SyKQwAR4IbV3GsiY1jpxtXMSLiMMjULQ6TQTRxGITSxWEymisOk0lI4jCI7BGHyc8TxWEQ SxSHQSgvqzIKNaTdqGi3aKNqVbeHaVnCfHTjYCleqm4fjYetLSVeD+cZzc7Tp2y4MdF60RDH6UVD fG8vGmJketGAqb8XDUHEXjSAMnrREM3uRUMSWi8aILt60fB5ai/amMi9aGNitnsnuXbvRO9F8+Nw qrlMkz8o05Wl646Go/HwJQJM1riT8XDekz0PCEydIXgeBceaYAn4Ps+jYCQ8j8zU63kUEMnzyFCy 51HQTM+jkFA8j4zs8DzK8zTPI2NpnkeG8iZpOhlt8zsVLYE0udMV/Bo2tx2NByxdihw1xmPPPkej wkmZbk3UQfNgelY1kDwPxHE8D8T3eh6IkfE8gKnf80AQ0fMAKMPzQDTb80ASmucByC7PA5+neh6A pXoeAGV5HoCX8jwqFcl4qEWPrfHQGnUQDsYDlublJePh/M14NK1ifh39EBnDAZ0jFP9VHKtVzHUW /1WMUKuY6y3+qyBqq5gjF/9VNL9VzJGK/wqyr1XMEYv/CpbcKuYYlSQFL9Yq5ujFf+B6na6uwFJo czIexr9kPLzZK4e9Tm3bgXEzdciYNSlsBM+j4hgJ04zvTJhmjEjCFJm6E6YZREuYIpSeMM1obsI0 k5ASpojsSZjm54kJU8QSE6YI5SRMES+UMN2paDkPM6jGeCg1Ye34YZIKLKXXPA+/tx3YoZlxio2m YWO0HRg1k9oOEMdpO0B8b9sBYmTaDoCpv+0AQcS2A4Ay2g4QzW47QBJa2wEgu9oO8Hlq2wFgqW0H AGW1HQBequ2gUtE8D69d63mMM7yBpuFxxqqRV4xHMHvY4r0/NTyujm48hmUl3LZUHMN4ZHyn8cgY EeOBTN3GI4NoxgOhdOOR0VzjkUlIxgORPcYjP080HoglGg+EcowH4oWMx05F8zzAbTl5Htpr23ge 4KK8aDxilQ3VWpnTbcu6MtT/TCJOV0/M6eqJMF09iU1XT5Tp6ok+XT2xpqsnienqiTxdPXVOV0+M 6eqJPl09MaerJ7np6ok+XR27mk8Nj3EJTc4DluJrOY+4F+tGZU7yc3YbyDkPj1PkaWGLZRiPiu8O W6yI8ShMhLDF0oxHgXLCFss1HpWEGLbYDuNRnyeHLZZmPAqUF7YIjSG8U9HCliW1OY8hQLzcGI8w j8OLxqMWiZnojWquaoHXMRqFoiHptGQc57YF8b23LYiRuW0Bpv7bFgQRb1sAyrhtQTT7tgVJaLct gOy6bcHnqbctgKXetgCUddsCeKnblkpFMx5hOlWYwtfeNsYDltyLxqNOnrbaJtOUp8+jXRjTDtZE SJhWHKvOI3UmTCtGqM4j9SZMK4ha55HICdOK5td5JFLCtCD76jwSMWFasOQ6j8RrUwa8WJ1HoidM 7WDMqcswxrkpT4el5bXy9LRPPLA2tZ4Hji9llKevfqMZD8CxjAfgu40HYISMh98IxgNAVOPhN47x ADTfeAAJ0Xj4rc94+I1uPPxGNh5+4xkPv4kZj0JFMh5B+1Odh1t8qxCHqrqvKcSlfRSZSq6ZfeyH abbkqelea6NoCnGAYynEAb5bIQ4wQgpxRhEU4gBEVYgziqMQB2i+QhyQEBXijOpTiIPnyQpxRpEV 4oziKcQZJaYQV6hoYYve7KlIzG1TM2cJlpaXhhHEoV7V+uRsU54+KWfJRWJoJKeR5nkAjuV5AL7b 8wCMkOcxjQTPA0BUz2MaOZ4HoPmeB5AQPY9p7PM84Hmy54H3yUTPA0eZczyPaRTzPAoVURxl207G A8J11xiPOW7xNePhbmFLOA1BXYaNIVi2TATPo+I4xgPxvcYDMTLGA5j6jQeCiMYDoAzjgWi28UAS mvEAZJfxwOepxgOwVOMBUJbxALyU8ahUxJb88ySTFCb46h+7agNEHK8Zj1A9Dzs0Lfm5PH1kNMal jdaSjziO8UB8r/FAjIzxAKZ+44EgovEAKMN4IJptPJCEZjwA2WU88Hmq8QAs1XgAlGU8AC9lPCoV UdjansKWSdmpuW2Bpfml25ao7K5t7dRwSpgmTzce3gWS55FxrMa40O15ZIxQY1zo9zwyiNoYFxie R0bzG+MC0fNAZF9jXCB7HoglN8YFlueBeLHGuED3POwK+NNsaetcO1va+tfCFhX3xrhBtYKwPq7k 2xbrzbYSblsqjpEwzfjOhGnGiCRMkak7YZpBtIQpQukJ04zmJkwzCSlhisiehGl+npgwRSwxYYpQ TsIU8UIJ052K2NsytcZDwRbcdFOeDnv+SqD9aDzq9FZrUjInz2Oht+QDGt4lpas24xhdtRnf2VWb MSJdtcjU3VWbQbSuWoTSu2ozmttVm0lIXbWI7Omqzc8Tu2oRS+yqRShr/jPgpeY/Vyqa52GWU53H Og5jc1ULS9NLV7VR670xzhrdTnjAlAg5YRpgm9ImAQ4sMaCM754EOAiJASETYRLgQBUDQihnEuDA FwPKJMRJgEOXGFB+njwJcKCKASGUNwlwkBID2qlonsegWjEgpYehGeaFS5fDvBrjUROmejCDb3tb zMSaG78RxYA2phjQRhAD2sTEgDaKGNBGFwPaWGJAm4QY0EYWA9o6xYA2hhjQRhcD2phiQJucGNBG FwPyLrRjRNWQVDpWmOLS+FKFadRxLxID1+NcYarIjXE2LGElaZgijqNhivheDVPEyGiYAlO/himC iBqmAGVomCKarWGKJDQNU0B2aZji81QNU8BSNUwBypu/EFax+QuFiioGdEqYbktKjZ4HLI2XLfk/ Lu8/77ajzqXzPoVGCwg78oeVXiPmYyC1tmQcqy8u5NaWVyfgFETS7Ak4lcgLTMCpVIkxAadSTK9P wCkINET0CTiVYyROwKnwmTJItGBxXAx1kGhliLxBopVlK5/df/6v//kfvz9+crcl/BM+bXUM86yq hcyb9WAi6wr+cPW0iS+w+nAQL9bEF859OMr4k6HLb+nBzJnJpBGVJrZHS5dXI1aQ+Adjd1t1r0Va Zq+ox3jtlKZZLUdFQGmi+Jlmip9pgviZFhM/0xTxM00XP9Ms8TMtIX6myeJnulP8TDPEzzRd/Ewz xc+0nPiZpjtL8LueVARwjmCjIqBtuCpNuU3wjSbe8rttTZsN0SlqKw5Gk8s4ds8KrDjyrMCKD1IT fDPbzJngmxmWngm+iMDZx3k7ILl/2B75Ne7E+DSHscAvW3YHPmjveyO/xK3/NM+zsMrFES+Wg1jO 5eLhtC/yu3k2jXccn03jNeHZNF770jyEWMdp4tnqTmerRd+MfLb6gZKIKDjO2Yr43rMVMTJnKzD1 n60IIp6tAGWcrYhmn61IQjtbAdl1tuLz1LMVsNSzFaCssxXwUmdrpaLdny7DaR4CqpY0iQhYWi4T EeOP6810qF0Z0Jg2hzmPbqZLbAChs5GiDAi4kaMMCPi1Jw+BCBcF8hCZaBbJQ2SqjZWHQAqvevIQ GRF5eQjkCAM5D5HhWmASb6Wy5dP4K8DuHwG+wq/PU2VHAG0FBMHx+vHrwRyelpHGP1VsNC7aGw8+ fGDABTTFF79CDAXa/uz7STBfvONkmwnCaPuXlufZvwKrf6qLaFzX8TJebDGqeqRxfeqR8DxZPdI4 snqkcTz1SOPE1CMLFeF4URFHailz7ErCVb2tq4uPjUm31fRanY41u5OqmgrhXKezUTVoAR00TVIB cRwnFfG9TipiZJxUYOp3UhFEdFIBynBSEc12UpGE5qQCsstJxeepTipgqU4qQFlOKuClnNRKRazT Ma0ei1LLOsSmQnhZ1fSa8XB7b5K3/lwhvFDV74N120QSsM44TmMj4nsbGxEj09iIfY3djY0IIjY2 ApTR2IhodmMjktAaGwHZ1diIz1MbG7eJKmCNUFZjI7atCjU2VipahDuatk5nmJfTxD9Yem3iX7S1 yM8k04o5xWAT467dLY4mI4k4lqSC65eRRIyQpIIjyEgiiCqp4DgykojmSyo4qowkIPskFRxdRhKw ZEkFx5ORBLyYpIKjy0h6Y096LNZgd9rBeFiD481eMh7p3l5wGp1h3EDPrYPX4mhzd0bHm7sD+O65 O6OTmrszOsLcHQBR5+6MjjN3Z+xLfjyfuzOSkh8F2Td3ZyQmPwqWPHdnZCQ/Cl5s7s5ITX5gV7Q5 qd8vS1qarmhYWl/rinZqby9QWp1GZySyjGSwflsXUpUf4jhVfojvbS9AjEx7ATD1txcgiNheAFBG ewGi2e0FSEJrLwBkV3sBPk9tLwAstb0AoKyrfcBLXe1XKqL6/dh6HkrZodFjwaUX9VhcaWxUMWgb G/X7dXOergQX1EwyHhnHamycu41Hxgg1Ns79xiODqI2NM8N4ZDS/sXEmGg9E9jU2zmTjgVhyY+PM Mh6IF2tsnBnGI7p24t8wrXbUjZjTaqfXJBVc6S9Qyrh4bmw0iaF+H1Qg5TwQx8l5IL4354EYmZwH MPXnPBBEzHkAlJHzQDQ754EktJwHILtyHvg8NecBWGrOA6CsnAfgpXIelYoWttjxJKlg7ND2JsHS dW/S0Xj4W84jnEtudaDmPFBrVxEFrBVTwFoRBKyVmIC1oghYK7qAtWIJWCsJAWtFFrBWnQLWiiFg regC1oopYK3kBKwVXcDaB3cWc1rtlpqwZXXD/JrxCFUJzmmvToPK40jXYwFPhiZgHT1PwBrxvcYD MTLGA5j6jQeCiMYDoAzjgWi28UASmvEAZJfxwOepxgOwVOMBUJbxALyU8ahUtLAF9tUpYTqmsfE8 YGl60fNI1Xh43YQtzs1xXugyklp7gh5LxbHm7vhOPZaKEZq743v1WCqIOnfHk/VYKpo/d8eT9FgK sm/ujifqsRQsee6OZ+ixFLzY3B1P12Nxsw7nsGXWvg1bZpNeMh6+3rYE1Y7O8MZos9J7GfxKFLBe mQLWK0HAehUTsF4pAtYrXcB6ZQlYrxIC1itZwHrtFLBeGQLWK13AemUKWK9yAtYrQ8Dar/N5dIaO ITWjM3SMr4Utfh/aNZjYzN3xIeDEALLxWCJpdEbGcYwH4nuNB2JkjAcw9RsPBBGNB0AZxgPRbOOB JDTjAcgu44HPU40HYKnGA6As4wF4KeNRqWjGI82nHsppMkMz8Q+W9GsT/3wpT1dJDW3YkuIayGEL vE9koBgPxHGMB+J7jUd+tyLGA5j6jQeCiMYDoAzjgWi28UASmvEAZJfxwOepxgOwVOOBZQsc4wF4 KeNRqWhhC3xTTp5HCqkpEoOl8bUiMe9rY1yIyZ5HZ4zkCtNoJ4qAdcXRcx4F35fzKBiJnEdm6s15 FBAp55Gh5JxHQTNzHoWEkvPIyI6cR3melvPIWFrOI0MZOY+Ml8l53KhoV7VanytM57A0dR6wtL5W 5xF26RfYyO6sjmSojXGAngdCb0vFcTRoEd+rQYsYGQ1aYOrXoEUQUYMWoAwNWkSzNWiRhKZBC8gu DVp8nqpBC1iqBi1AWRq0gJfSoK1UJOMRjD8JWCu1RN9c1Sosc3zJeNSchwtRNwlT75ZIFrAGD2v2 lhS2II4TtiC+N2xBjEzYAkz9YQuCiGELQBlhC6LZYQuS0MIWQHaFLfg8NWwBLDVsASgrbAG8VNhS qYhFYtqdBpVPoanzgKV4WedRNRmDdSrdvA7rTqpRWjOU72ez9msyVhxZk7HihTQZKxtDk7EydGgy FgRFk7EguzUZC4x3rBoxafdK1anJCIflCOF8sysixL7rcVdErEz4lpoabAW91z5p24yTsV67ydJv IF20C0WrNOMYWqUZL6ZVmtlYWqWZoUurFBE0rVJEErRKEcYr7rFSPQk7Va9W6Ti7aTNRHbVKcXWZ 1WSmRxmofdVeBaw/vN8dzqTr1AOXfDo3U4eRPm5J6yFNhP2RcYz9kfFi+yOzsfZHZujaH4iY4R38 /NPXP//xEfawkF3IJ8r/GQu+6/Ip/1oFV1/k9/hMngwwC/y8d+/wB7z79PYAPa5dOMyZYC7bcn/+ vjP3Fdyc63MwS6cN8FI6bZXqsDmts6ftub+lw8mlVRqWaT7OUVzS1JxcsDRfld492ZghGNdsTBNH iJ6JRbsoJG6tHSjq/IDTHHV+wPvXZUALwg1sGdBKZAVkQCtVYMiAVor0ugxoQfiBIwNaOSJRBrTC RwEZ0Eo1d8mAFlAIXBnQyjMTZEArdOuXAS3IOIvKgBbScWDJgFYSBzH/z2AcH/EPC/iNuLAj1OEP tiPRUJ8nD3+wtERDgfKGP1iZRMOdipRo0LNvK7PiGP16rAnHpe2yMuthUlbStbYCO0vsWYBH06X/ tAmTnygK1YBbOArVYQqqT6EaEJOIQjUQbUIK1SGXtbEUqkOucOtRqAbExFWoDlMyDIVqgDshhWqg qqdM/s7fP4P8MluKp+LSIde6PX5YDf6wjO/k4s+P1pg4LyvDQzXm01/exi9fHox5WcAtoC4+R8ao LoTTR3UhmqfVHKTK2Haq/lFdwwjf4bip46guWNVbnNK4PY7q2lfnS6Xmg8Gt3b9e5bCgSe4OltrA Z82yGttvcCuObHALvsfgVgTf4FYiCYNbqFgGt1J0GNyKYBncwkE2uBUuYXArVbfBLTi2wa00VINb 4SSDW7Bkg1vgVINb0AzXNuNlXNsbVbfBjUrNMyrbHlxbWITv02AfM6L7oruaIj1//vRxN7bG7VIL 2qWmcjjNi6ZXDmsz5rx3v3cLuMTxbgG/iGVEgW3WrIwoMpiujCgi4B28vX9DgzJWW3V7iYGmfZZb BNxiMRuKU0rvwPtrQMaLN7mkEjXeYfsrAM3Pcq8A4nkxo9Q1w0512FTmtKXwzTzsKG3DOHkz2jg/ bKm8aoLR4N3c99R99Woi2m1gYDL+Ng3t1MmjB2UYgyb8ZAiX0wVHv5wueKnL6cLGuZwuDD2X0xlB upzOyP7L6QzjjfuajNi4r0LVewk3j25So/PHSzjsXFmj2+bHktH76pWo8fblLx/nfYfUoV/g4Ltm +Dh4h/O8MCQ6/DoGyjU14jjX1IgPPSkVREQlkFLJRE4kpZKpIiulkinGnpQKIpLipVQyRyKnVDJ8 EkmpZKqlc+gXgrASDh/Dortl/DregfsKJsafnMMInjA2eP/h3Yc/TdkW7PeZxzX86+mLH4/GcX/6 T5/ff4W/95cvP69Hmod/QHP5xAXJVFOhKj/04cNsFtG224uv0LJfX6HZePga40t0Yi7exRKriX/A 3V/jXz9eGA1WwQTgpQomKtXBVif3JPEN7+cxHZN0SGZVwT5mY2AxbGadHluDbotX9USfD5Za34cf t4V2y2IMY3RFULNeSTKO4E2zZBxnY7ssNSJWCUsNRFbLWGqkcjxLjRShy1IjYmVaauBwnm6pER5l LDVSjb2WGkDe8cczZp6RNJ4xQxfKeEZEhlF4PCOSYvU6azwjkmCGrjy6vZWHsfh8/PDup0/L++0v YKIK5d9+BvfLhcDljEf8Ef2MEQ3WfLFfxvHpNa6aLv7aUxlE+ult/fhu/rzev+i3lXymXOyTeT/N 8OlDvqss4HuNFx/JAt8QjPrhqIFje3x7Wz8faJ7/G/42F2+dKVQqNpylUh2LllI6nZDlDTVt95OK Q9N2r8NimrZ7WLqaCnc8HGvRkh3M0M51cmqdGRrH0djgKdW2gIucaluTy0X6lAbLtcDepvLni/aV Px/aV6an1aLAtHQrDZbkvozpiM8cbPwJPNMBDF2mA55nmA5Ak00HYGVNR+Sk3gterBj5Ser9e5mO W9rdK9Ok3bdhmBgKx3FZaaYDcCzTAfhu07GsUqYDmPpNx7L+vU0H/ASm6VjWPtOxrBzTsax007Gs wqZjYfT3FLyY6VjO/T3/ONPxeGFXp7JYp89TWeZtnhgXdjpo0oUd4jgXdoiXu7ADNuaFHTL0Xdgh gnJhBzjahR0Cuy/sAMRrPQharPWgUP06Luzc7sc738xnDSbYNJMv7AwwjGrrLzhCnFb0gqOMdz0F RxmxCRQcIZExIgVHmcqzCo4yRewpOMqIjVdwhBw5tUYrOMrwJFJwlKmmvuaDDMIcHbP5oPBMlOaD Al0JzQcZ6SfZ5oNMilV0nOaDTILFb40r+fET/CU//+Wpm1n/DfeHufii4GUb1b0sDHhx9pp7WZ/f qoO4/nmd96/kw0LezRe/LV7Q4DH1CL2/xt/ySc0UAjnlW4gXKt/aqY6dCWc/sLyfx7NriGYMzsfp 8YYmr1q1TO6xO+G26l+6o9kHXsCpan1bLzuYSD2+cKTYoimxZMYxYsmM74wlESMTS2am3lgSQX/f WDL/BFYsiQw9sSQ+T48lEU2NJRErGksiIW9O3yLl++5Uv4Y01K5Y75Ixx1J7N0xbilS9CGvDtMV+ 01FxZNNR8V2mo2AkTEdl6jMdBfT3NB31JzBMR2F43XSU56mmo6BppqNgBU1HIWTJzABeSmamUv0q TIfZvQ6v3aktEr7P5Bo+sEaky6+MY3gdGd/pdSBGxuvITL1eB4L+vl5H/gksrwMZerwOfJ7udSCa 6nUgVtTrQEKWdhvgpbTbKtWvIIMdq+VwUanTaGA7GrLetNchqP4MdsWRM9gVL5TBLmycDHZl6Mhg V0R3BrvgCBnsCuzLYBcQq6g+CA2PuVP9OjLYcZ+2rZQ7T9vWip4CcEFPJN03xHEK6hEvp/uGbDzd N2To030DBFH3DZAU3TeA8aq0tFRb+U7V2XJi7Gb95rbl0HKSV+fk/awe65hvq1cqHoeWk5j2lpPo TsqIdl6oym+YEizN3f13PICjN2UVfEfLSUEItJxUIomWk0rFaTmpFB0tJwXBazmpHNSWkwqXaDmp VH0tJwVEbDkpYE7LSWWQaDmpVKyWk8LR3XJSYYSWk4LkXWmIdaTvVL0tJ2A0wZg2LSe4GMM6uvBg qm+L344OIMg2t95A1Y7vDhbiKfooq3WaKfUtGceob8l4sfoWZOPVt2SGrvqWjCDUtyCOVN+Sgb31 LQjijZacpKrUd6pfQ3QAW8rVVF0wOK6lvSAcGarQSa/99S0FR69vqfiO+paK4Ne3FCKR+pZKxalv qRQd9S0VwapvKRzk+pYKl6hvqVR99S0FxK9vqTyU+pYKJdS3FKRwfUshZda3FBLJ+pbKyKhvKQyv 17fU50n1LQVLqG8pQFbgDnipwL1S9de3aDBDafTmWN+Cq9GOo3oUKLqtmqvppo8O4T7TwKQUz+ni ODNOr6A9xSHMOIZDmPFiDiGy8RzCzNDlEGYEwSFEHMkhzMBehxBBzISYl0uI+V+PQ6jjHmPl3dUU PLuV4xAuBIWiiuOkixEvly5GNl66GBn60sWAIKaLAUlJFwOMd+osMgpFd6rOdLHdhnkOjUARLk5m cMcMRF30V3vjIVesBmP26VLOtwKky7w48jAC+HuriZYrBhwrVwz4zlwxHNkyuWIgksoVAxU3VwwU nbliNfFzxcDByRUDXCpXDFT9uWI1MXLFauLmitUklisGKnauWE2kXDHAiLlixbjXK3ixXLE63+u9 kCtOIVgVNn3IFePiECGAfhwBeFu8EpL7fLDU/tZGrYfTvXccGcniZSS1UWccpwgN8b1FaIARKkJD pu4iNAD9nYvQ8CfwitCAoasIDZ5nFKEBmlyEBljZIjQgZKXFAS+VFq9U379+FfbeUCdOPVVpHRRD gWFKI0mBAXEcBQbE9yowAEZIgQGZuhUYAPR3VmDAn8BTYJhwQmdHChGeZygwAJqswABYWQUGIGSF iDgIQihErFTfyXT84evXtyWbDu3c7T5Nm0a+NnfNJIbu0ziMI2FsecYxxpZnfBRLnyDbwkqfIMPa lT4BxATv4MP2blx+QZMz3n3sZhFjo/GZrQAOzKTeH2/hGFY82ycIxPuwT2/vvnxdDuDjWr5KunjD +93BdgffXuYddvErs3Yo4KV2aKVqJhuoc3CAb6jZoGHZ9HGDWp8Oc17r0lVY8GyDuqRMk8DJgg4M dSWtfRxJGimAY2mkAF5QI8XH6X5l8PX9T+vydJce/gV/t6eyIkCGgy7nX47Pv/sy7WNVn/1TDnef aoMA37KPv1s/rPNh/F1ewB387KxG7D6a9RF6f40f7LORTADkaaN4mSavO9Uxxj7fFZT30+yjOKTH EDtas83TepjJmpe2q0To5T469XjZmEUiaJ0aw2J9/z6qOEanRsaLdWogm9A+qmRi+6jykfZRwRL2 UQFyOjQQL9ShsVPJ7CO7LXOzj3Dpch/9NP645m3koo1VRNuGodEXwjPTkQeaBSAcPOW6LeMY120Z L3bdltlY122Zoeu6DRG06zZEEq7bEMZSIAC8lAJBpertzvCLD/M0T8fuDFwdN/j7P3prt9XJ/e3b aBedSrUf0Oebt0MmZpjmyFK0C5TtkXGM7ZHxYtsjs7G2R2bo2h6IoG0PRBK2B8KY4nRBTpwuELZH nOM8WDsfdgcsTqtLcXpsXbotXnUu3Z0w+GXUsLfL2tTK+yyLmqjzh63X2o7RENplEecY7bKIT1JO WGZbhZwwJEv6aqeMzzpgEYHTuv/48/rz+u7H9Zf14+5oPS6hqxWfDYoC/JzqTiuPvtveHnbbbQ33 9rO53UiwHghOaLyTvHi3eJX5+HN+WD+un9/P51+g/gPu34vvxDKWbf/4S9xf4/3ws/laAGT19AJe qqe3Uh2LH/1w2vrlDZ1cxhDPKQx9TmGEjl3vtYvJtKGXUyO9o1frZaKlMADHSmEsk2QKY5kEUxhA JprCAD5yCmOZiCmMhTdVGvBiJ+jypEyAnMJwcxt64dILKYxmHz0ZJh3pMjXaWEvaR4hjzTcFvOB8 U2vl9hGSSe4j5KPuI8DS9hEAeXNNrZRoy04ltI+mKbX7CJYu91GN0ILLWQssiIwpqXi6K48DPZ/u Z0s6jDKO05OJeLmeTGAT20SZTHATZT7iJkIsaRMhkFV0AnipopNKRdhEanTpuInw245Tag5OHSxF /61NpGofixpUatMcEMcNEzULiJnOyVLGAmccYyxwxouNBc5srLHAmaFrLDAiaGOBEUkYC4wwXn58 Eur7v1H1pjnyrjgmOfIWmLbTrgAH6/muePvxh//x5ev49X0px3JwrNibEqkLzRGzQchEzXJg0ZnW lCRgxjGSgBkvlgTMbKwkYGboSgIigpYERCQhCYgwXqmillLp3ak6d8fozLwdtoZKS4T/e9waeekq 93faGlrdurXSaWtoZS09AWi25LC0rffgyLiZfnAgfhqkDo7M5oQSgJksyd3CIh/uhLxn3j7B1+7H bfnysI9ua7iXnuX3MkN6YGjAuNufHIYIW9ayBe+o/RV+I87tIRnEOZ4QL3Q87VSHDaiHc8Efvp3j Dhyndgdqt7Q7UPur5uO9TxKir3ozhcozjTy2m+OcyC5btOM6E+ofCo5e/1DwUvUPmU2q/qGQydU/ FD5a/UPGUuofMpCxgTJeZgPdqL5P3PPDrQAC9tFNKxoin3ZQqo2rpdYRoaxtHEjOHeI4zh3i5Zw7 ZOM5d8jQ59wBgujcAZLi3AGMJ4YcBzEx5ELVHfroaYxt6DOF6JpCVVgK6m+fLmoIqZwuejDa6lPz mp+pkpTw/jZwQilZNcRxsmqIl8uqAZtcVg3JJLNqyEfNqgGWllUDIGsLAV5qC1Wq73O6PFzxGB9v ioGx6eTyi1+joatZeDcrSjtGxjHaMTJerB0js7HaMTJDVzsGIrjtGMhBasfIQFY7BjJQ2jEQxzvj ZhmB8jvV927HeNigGHI1kp4WDs/IaLUcjaYcdBnHUXBCvJyCE7CJHXSZTPCgy3zEgw6xpIMOgby2 JiOVCNyppO5gN3u6g93sK7UMx32kTzdI2jFkm1ZKOqLiOPtolUtHFDa5fbRKpiMqH3UfraR0RAGy 9tEqlI64Uwm1Y0zWtO0YsPTNWgbvrKu1DNo1h5EbNpc2ck9TtOOQaDk9wLFyeoAXzOkNSTCnB2Si OT3gI+f0hkTM6Q2Jl9MbklhOr1B9n6jrltLzgw57NUN0qp2noVbwvukttkaTepoyjtO0gXi5pg1k 4zVtIENf04bRxJ4mRFKaNgxPYBPxYqVyTwQ2v53Sg/A+Di7q48SZvKpnDY7Ww+64r16VNdRTBrZH cnuxjzLhVHaqNvLNkUnBEBQ2K44hIZjxUsU+hY1T7FMYeop9MoJU7JOR/cU+GcYRbkO8kHDbTtW5 PXSYlxBWrw7bI6/6lFx61Ni8rY7fcsKCTrXaJyjlj5GM33xcJ3pjuQ6WVAuXcYxauIwXq4XLbKxa uMzQVQuHCFotHCIJtXAI403zEyq0vlNJ1cLZ9eRVQXT+Da8K98WuLOQGf9IzdNugVroomVEkYWbE cYSZES8nzIxsPGFmZOgTZgYEUZgZkBRhZoDxVLeMVBJ5p+o+NpS3KQyhOTZgNUa9jcvh2Kir01Vf 3O3YMErv28OY0yCb0QdqUxxO1naRcGwUHP3YKHipY6OwcY6NwtBzbGQE6djIyP5jI8N489id3Dx2 d5bqeUFIYYU/cVy27SikgKthhOj7UZfuvrpebI+P7z/+v3HfIU7tWgrg8R2zW2kafaRLjbgxLONE uWoB3MI4QAA/KbEDBNk8T5oOGEJJaR3hjyv4btyzuT+IXl4tB30udBeW2XeWgxbYMtVy0KYetMLC MzFLgPEuYsIidhFTqI7d2fYsPv3WVIRGbRZnoj96aTg00h29NFy6EoDcy3ZU1PUQcjjV4zRJ1mqq 5jSYk1lRRLEKjp5ALnipBHJmk0ogFzK5BHLhoyWQM5aSQM5A1okFeKkTq1J9nwTyfIt1rFNqP6qi 07qV/RkXxziqplmTjirEcY4qwAseVcjGO6qQgX5UIfqFrbI+m2wIWNqFJQB5sc8sdvFfqQhbZdTj 2GwVlQZl5qYJYVD2qgnh7tXBVtF2P3J8Oo1AMxN5XpM1yxbH/q1SceStUvBSW6WyMbZKZSBulYpm eHWFodurK7Bur67AOIlpxAslpncqilenxqH16uwS9NKUqS0BswV/M7WQrIn7hU3Q59TCQB47jUHi TMi8VRwrtTCLZd4qGzO1MHdl3gqCmlqY+zNvBcZLLQiVb96pvkd39t4A5+LeuOMgsmlu+aOZ00be Fd5sEyXhVnCMe5qMF7unyWyse5rM0HVPgwjaPQ0iCfc0COM1hU5CCbcb1Xe9p8F9YW9aHratxJxH H+kt2cOyUk6LgmNpeaxyp0VhY2p5rH2nRUZQtTxWwmmRYTwtj1XotLhR9Q7QXKfVjGMzQBMWldHz 8ChZelv8Rn4suVTlCqxz2jTTUdbo3UreG1aHjVL6UnAMTyrjxTypzMbypDJDlyeFCJonhUiCJ4Uw jieFeCFPaqfqrQwbBqs3u+hjZRiursrGQ+74tjpfhfMP7dAuxeEWayh1Oj30wLnGTJTql4JjxRpJ rvqlsDFjjdRX/ZIR1FgjEapfMowXaySh6pcbVX87dFhOXlWCI8IfvSpYUldyN/XcsGGoMvBexXQa 2uN04FxdpkBS6sw4TncL4uW6W4BNrrsFySS7W5CP2t0CWFqyGIC8CeZBSqlzp/ruYhsoYr1Pp3Px NL18UTM5WWzDEhLpdEEc53RBvNzpgmy80wUZ+k4XQBBPF0BSTheAsU4XwEudLpWq/3TBebHN6bJZ r49iarhkrmpjbvndLLmZ9QVjMo1IAN4wkyuOUSluWGn6goBj6QsCXlBfcFi5+oLD2qsvOKxUfcFh JekLDsxRyIPcKOThPAr5H5vfNUPa7+AtHC2xvVi0o+ecFY4g7FxxnFoWxMvVsgCbXC0LkknWsiAf tZYFsLRaFgDyDhYnI+x8p/quws42uKrj5F2C/z+PAw5UHSd4h+MykRwuxHEcrrGMCBHaRMjGc7jG Mjqkw+ECBNHhAiTF4Ro50zcKXmpfjE+mb/yDjxZlsSu4VK9A/H+WO7fUBmFMdidFmx0KONbs0KTE jpbMJjg7FMhEZ4cCH3l2aFLE2aGJeZ+S5O5T0vk+5R9/tKRUNpEKyTbSZtgPts700YgQ9ZAUXzKO IxGIeDmJQGCTkwhEMkmJQOSjSgQCliYRCEBWiAN4qRCnUn3XTeSt2TdRDKbpxcfqyWDoI6Dgq0zT 2UQca3rNKKmzCWyC02tGWZ1N5CNPrxmJOpsA5E2vGcV0NivVdyrY//zpY3XovLvXvLjhXK8fOHNI h5wd7Z9DCjiGQ5fxYg4dss2aEeYUBtPVs48IeAdv79/Wd5/XcSmg20sMcuzToaVDwsGfy6ft/Yf1 Dry/BmS8eJNLKlvqDttfAWh+OmaUJaCU8WIjPp8IKJ13E76Zx35miKkmb0YLgc9DPzOuwvmhzaNK zH31Sgbj7S9f//Dpo/lNuu2s23i1qELTtLkNaqDffPo0kTphMo7RCYN4uU6YzMbqhMkM5E6YjCZ2 wiCWdgYBkHUGAV7qDKpUFOm+sJzOoOC8UsczCJcuB8X/vHzad0q8V+m3qmNmmRd63aUN00hob644 eiNMxos1whQ2TiNMYaA2whQ0pxEmM/Q3wmRYfyNMhvGKcEaZ9uY7FaERxnm9tY0w3s5rc1GKS5dl OI+bLO2bzKd02mRxosZLWS+BuMkcc5M50U3m2JvMsTaZY28yR9tkjrbJHHOTOblN5oQ32XLeZMs3 at1gi6W9AlQPqEfbbDH4apBngWqzjCT9M8Rx9M8QL6d/hmw8/TNk6NM/AwRR/wyQFP0zgLHCIsBL hUWVSujKaNRDm2MY9Tc6ByBc8vrm3bX65jZEbxgXqZOfo6UcPIDznIMH8GPFf/jxAfkhz5AYnppL n2v0cCzDTz98hi8t/Me7P4yfa7B9Xgcmsz09KIAJ7PW7Xz6+uw+x2F/h3+riLaMB+eWH9eu77f3H 91/qG35YwI3wtKQZoPBXnkf4SN5tbx8+ffrx5zfEjPD9+8P7D/UdXP87Ej89eMplxRH4jAzTFsPF r4Zn9kdYf1+A+T/xQ5guPoRp3kO+ryP87zHkywt5X15gl2oOtkfsw0I2fM+hvKNSKHt/pzoelU6f 54eUd3QyC+5kFpIJrVlINv5tsxCVroXhOCVuMKdLMOupTaiAjp5QX1FxHFFExMuJIiIbTxQRGfpE EQFBFEUEJEUUEWCsWm/AS9V6V6rv2oQK+0IPe8OE9W1ztjWOXOgdbBhGklhoxjH2RcaL7YvMxtoX maFrXyCCti8QSdgXCOPsC8QL7YudqrfRTi8BnKQwHxvtcDWENDv92Gh3W/1GjAXbw1aNaWttbKUQ sQqYkSscHa3LDnGssjwn2WWHbMyyPNfbZQcIalmeI3XZAYxXlufEuuwqVa+W7rK5bZ2cOWrp4uo8 r9P8mEm/r15tj7usFOwQH26D5ZsSvTw4UTMGkE7RkO6dEMe5dwK84L0TsvHunZCBfu+EaJZYKDIQ xEIRRhALRRjrygrwUldWlYqS6AvD1Cb6Np22RlZq0+NwJSs1/8u/1D0WzK4W6m1oe72XGRvGyZk+ 703whEwf4iIj04f4uSunkTFRC+Q0CtPSl9PIoGQoOY0C9Xvq4Zx4uKUdzPCkpDWj4W/1y/bl3Zmh XcVt9SwxCSQ5IwOPt/Ab0D6rqM3I8WnWQ6/PEpnw/ARfSPhrfnpbP76bP6/1A3pcydv3Ao3SBdm6 4NOHnElZwJ8cL37TBT5WLD+BXf3uT5/Ht7f184Hm+b9l4/mckJVcBbxUcrVSHWyRTelki8obOhij xbsYh4MxCkGl6TiKFZfmF4yRH3ZjBJhTJsVzKiEnE/1KOfBNDAPnwAe8vXJBl6eHOiDgN/3805d3 n7O52eADmcbdlJzXgWl5etqZiDbtr7ANN/ggpy/Lu8/r/GF8/1NhevIP+Es9KxJBKjf803//n/96 9/vf/fv//u1//Oe73/7XfxeaZhFNzbNaR6TAmcc/fPq4ftitG/43AsanhzwgVtzsHz7/nH/Rd+9/ etuhx0XkCM/6MIEDrSOYB/hjfZofTPphCf+EF59+AosBVvjj+qdfPn5a1gJ+WMh/smeFpogF0/r1 pzf4O5cf9Uv9BhzXkGG8+AAxXf34NNqYMweuIot96qQBy7SzgH0cvx4Yygr+AS8+AxysjZ/xv//+ d7/9v787fu73tfwpPid4yWLb4al7aCLNYhesoMUuhDzv0cjIzN2pKBZ7tmZqLbbVSzom+XBpvKq6 /fMPN5MdbSle1/Bf1p47dBV9IpCPK+GmuOI4feuIl+tbRzZe3zoy9PWtA4LYtx7X/pviAmNtDcBL bY1K1Z38NotybfI7mME3N8W4dDVUfp7V24efv+xbI+xCc+FURGEDbBf6CFKvwtQfWlUcObSq+K7Q qmAkQqvK1BdaFRAptKpQYmhV0bzQqpBQQquKfDm0Ks9TQ6uCJh/UgL06qK+P6atDGsh47WVB6hpu pzqmeJ4e0QdDBB/bGNaDIbLrrMNxGExeurqFezyg026FjLat3NKcNk8W88MmVEr3f8GxpGCTXPd/ YWNKwaa+7v+MoErBJkL3f4YxW5eFuv9vVN0HtFVpbQ/oNQTfiPnBUrhMNRwO6DTcOsZcOFfrrzO5 ytGsw+QJuc+Co+c+C74v95kxIrnPwtSZ+8wgWu6zQKm5z4Jm5j4zCSn3WZCv5z7z8+TcZ0YTc58Z K3RAFzLOaA/EC4322Kl6D+hZD0N7QA/KB3M8oHHpamToPee5y1F7HW10p3Ft88SoHlvRPBKqxxDH iBIyvjNKQIxMlJCZeqMEBBGjhAwlRwkZzY0SkIQWJWRkR5SAz9OjBERTowTEiqbzkJBVsAd4qYK9 SkVJ561+UW06Txs7Ho1RXqoF7sv0r/pff1ym36wfv66ff/M2fnw//9s/w7f17evPn9d//rRt//T/ AbrnD02OsBQA --=-=-=-- From nobody Sun Aug 20 15:45:41 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTKhP5Fl3z4qM2G for ; Sun, 20 Aug 2023 15:45:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-24.consmr.mail.gq1.yahoo.com (sonic304-24.consmr.mail.gq1.yahoo.com [98.137.68.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTKhN1mH8z4PXF for ; Sun, 20 Aug 2023 15:45:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=g3XJBFVX; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692546354; bh=4DkC1yGtsLctA8uIo/4y2x+UG7zGW5neb7zotdHw7og=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=g3XJBFVXNKqCZUhPnhpfVklF2ov4pttR9YFNEojvFRyNOPD+H/JxZXvNpXDThji/Et0olxIKpqvsKMQSekkcigICtOjGD80VX9SLO9kPZiSGj+6Sx5leX4n8Sd0BEjT4F3ZaOQwCHSrctytlIaKxy3ClXD4KFQtF1fPz/85pUg453dLKlg9EwhFdtcSAtoPNgAo15/EaFqPIz9IGVDUUKdiE+zRBIjc0MEbE8PUOK+9MvkSZ8yj6bJrOb+4X/JI6bYfPC+tmmuImxUsZc3JmlLnhX0AUS5Dh44/oTP8Cg7bYjHrEgKQ/5X49T2+JU3oR3ukrC0FZc0lNafJii+awkA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692546354; bh=ivk7gt93NuvHINbFW4ocvBWgYHbtw/frEiO2xKbNHz1=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=KuMEeE+HXMNkIpgWJmYg1El8eBOqUWfDel2BFZ5rr3oUK1MmqCH+wTI9JYxTGO+urhhMk+df6zNiQ2wp4o78lFCGoQuNVz/D+uXuZ9Mhv4FCV5Zi1qqyXYwK74isBXXNGiuQMDAo2H57x5goyv5mYJZ2dX0yTm0kXXhpWnpDF0TgV8mh7uSvrM03ajVeb79q7n3gnyEVWLr7JuzSKtdaGkGOZEhZ2N5IhPq/WaBBBPBiqj0ZEDi/JeTR6iyU+DSdfd2nfqoJmloOuy02/XgVN8UyMIcwAOqFvR63TSGlSZwe9r4YUI1qQizJ8oQLgprV0lZabfAJQHg8bgRqqSjfQg== X-YMail-OSG: q1BcZEYVM1lN8BW5SQjpFg_z_2j_tdRMrcm9sTURxKl3Qvv1WHPVoljFLQYk1J8 jjuwZpSpTSXoAGuXPqmDQzTuuzKYqUIp.eLH5turGg2_K4jzDFSITm0WfwlFdgKPw9sBd02KqA8V yhGW.eLN3E3UzusmHO5i9Hce62UA1h7EJs1fCoa4e7aOWY54UiVNsn0bGl.XS8c.wDt_TfQ3uLvc PdNRaCU8PKrXg9ggb718wTuDMWAkSED3gGmv6aKsp6Ql8jAdyAOrdZIzvm.YEneq2usnsdfST1Yo OzVpp3ToqBooDu1wqH7orFPnmY.sDT_OBOdVi_Bgl5sRbrsDNrhNRHVuw2lH0iuDa7EydbfDjunv u_vJrxWaEeq_BWIxPQFgSFY4Sw.Yr1xybhi52nik5ypPENNGjjNEWNeppR6GT_0o4yNs7.NOJmyy Dp76T.NozT0t8f2tcanKrLniFAHZnbdc4rX2pcotg1Af73gNxWBshCTX4y8cn6JGtfSVKM.MPQZN O.LeE2YLkmWI2.l8.p.cOK_RwWMWzFhrIsKmI4qyi_K40a2aVrea4xbannZTt3yq.7eU.Syg32aI SuIXMG9BKXAYvgYmjkgirDcX607tCF2W9_zA_WkzYftPBNj9UlcwszYjSS2O9RISnZMDAvOy89b7 dqVFdtcKa2kLpfHFGmJGsCIRTJVwcEwCdNdSAlKUg5poMDOz0J4fYoFURYDz7G9wbIWh3BByf7m. lGCYQAcMCOGXhrPTqAUfxTxsFcbONvhCt8yKqk5dNWUVQZ3OFfOwvtlwhNZBSJ4sH1hebxWRuDhW cD_BjMF.AR0IYR6hl7X_0NTmLvZiun3u2IoJFXN0LsSLqpODrlBGRpsOAohxVLhiBoYEomK6SCPf dM3PrmFoCwClONNPh4Kk2ThKHKfZjEfTFntlp0GE1AfRpHRFUk7TyGQ_ZmtQUyGyN94aJY8Sx5jS VoO_l.X8o8eTEiBKvtLSs8ztrBaSRn.ojONFtxjT2u_8M0EGXT7hcOyvvJ8rQtJaetTD8OZKU3Iq Fe3ezguDpwRx0aUeTQs9BE36iMQLXuGUViDL5cIK_28QGcltCiim7svaZWqTY6dGC71hJ1peluRX G8CNvOEgycaqQGkd9JFas2d.kqlLEEBazlpzYsudXVh3Mp2M30gaAuW7K4o2s_NzFZhCBqNQKFvW kqmHBEZFHeiCVdy7YcMEjJA9wnTS1ycBSn81AhqtlKAnzG0tDWzasHrfL6rHbkX4RW_4xx2n2.sK NJQtd4vwKZTBROnb7dj13bS_v16LwlywfuMome3SiywrliwnnNREuLhP8sD3cIQ.UEGl1_CStxmc 4Y_J5l_juk9IVILtwz6jZ6OwO3QVlYKqEyBLkrX.nUtewh.0jsgnCg6fN.STOA5f5uyu6JP1lEm4 4bDNKNVkZ7t5L3jM51Wor.PoBgv0CrHsDLmowrPmtlR6hZW3IuKZHdxCQMfBmPkCg0at3Yzf83DL _5zF0vL4ORqqsbEMSyYTFFHCv0o1VoTy.xpBIn7dweX2vsJhAIZzmyljGgy3ny6JW9M6eiK0Z.Db kLVPqm3Pv4.UeK4LPfymFGwcXu8t7sAHDZ4B0_N1QrxCBS0FcrhfNIKM9cmQdMAqOZPiDqJIkJ9A 2SPPf4639LwxPg5tg_nuGg4vrs7wiLR_aA3D1u4J02UsF4KISnZ4dRB0j567mg.LDUJ.heWWQ6By JdMSLuo.Sz84LgbWZIMcIfchbjKOqLgM4j2PD5h7LUKJGiz.hwvmmfX1OfLvrvSHxqoI6XJgUAW. RwX8MqpVUjZmwbNnrssgqInzdTUy1hDi5TER..sDIm4FfZxyvmpbHNxcRPb9sJIS_VUlnxqwvBkA vR7ODCWluCio27nAgSTrRYaVj5JgKVn8WIJovSKh7k.mBPHzgPDJHD5xRaHhH_YnqTDYbMPgw_.D cex2SfHb0agFBvG8WGw4G_UJoD5iOM5XBhtlfCcgchsk6D1F3kOgI0.DxUxAc_OyFcraGFxu0KHf WIU_QOWOlNriByGpxeli72asYxr34p1ESs7zZHNcqA_PBY2tV1cUe8bfZoXaF2x2lPln7a1VIndD oxzCLFtaYTTOsM8sW7B1IKb4Sx2_fcAWSof3B.58d1iJIuNGPIWWy60ThTKjIRDdbcTpMi7_V4Wo XqXbGVyzbpAbdcmrHq5raCY_ZHjr09kHtBJw6UIaBWjbvyjFQFaDK3uneHkL8kTJkCV8k3y2BNOr F2fWHdo9CZehwVWv6x3fcgdw.yJ7Th7.QSX.07XZvepVyP0tg75VxA8Ol9SIeQ7c0yP9XFqs9Dok Ih8976cXgGvepbQ-- X-Sonic-MF: X-Sonic-ID: 1773c615-7ec5-470a-a9c3-e9c9f9984be8 Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Sun, 20 Aug 2023 15:45:54 +0000 Received: by hermes--production-ne1-7b767b77cc-84nnk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID fcc795d7f84c6c11e9d6972fb07b7f81; Sun, 20 Aug 2023 15:45:53 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 Message-Id: Date: Sun, 20 Aug 2023 08:45:41 -0700 Cc: Current FreeBSD To: des@freebsd.org, "mav@freebsd.org" , Graham Perrin , kevin.bowling@kev009.com, pi@freebsd.org, Cy Schubert , Mateusz Guzik X-Mailer: Apple Mail (2.3731.700.6) References: X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.205:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.205:from]; FREEMAIL_FROM(0.00)[yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_SEVEN(0.00)[8]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[freebsd.org,FreeBSD.org,kev009.com,cschubert.com,gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RTKhN1mH8z4PXF Dag-Erling_Sm=C3=B8rgrav wrote on Date: Sun, 20 Aug 2023 13:00:27 UTC : > Alexander Motin writes: > > Unfortunately I think the current code in main should still suffer > > from this specific deadlock. cd25b0f740 fixes some deadlocks in this > > area, may be that is why you are getting issues less often, but I > > don't believe it fixes this specific one, may be you was just lucky. >=20 > Yep, it took a while but I got a deadlock with unmodified 4f14d4b6b7. Are any of you testing based on USE_TMPFS=3Dno and such, so that ZFS handles all file system activity, while allowing multiple builders to be active as well? In my case, I also have: ALLOW_MAKE_JOBS=3Dyes In my testing, pkg builds and then the other builders report "Builder starting" but end up stuck waiting in a loop with timeouts involved ("vlruwk") before ever reporting any more. Some progress is made in the original builder before I eventually stop poudriere-devel. Am I the only one that gets such behavior for this kind of context? =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sun Aug 20 16:29:40 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTLgH2P4Pz4qQC7 for ; Sun, 20 Aug 2023 16:30:03 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTLgG38nDz4SWS for ; Sun, 20 Aug 2023 16:30:02 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=leidinger.net header.s=outgoing-alex header.b=i+PrjwIZ; spf=pass (mx1.freebsd.org: domain of Alexander@Leidinger.net designates 2a00:1828:2000:313::1:5 as permitted sender) smtp.mailfrom=Alexander@Leidinger.net; dmarc=pass (policy=quarantine) header.from=leidinger.net Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id 12B4D92 for ; Sun, 20 Aug 2023 18:29:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1692548983; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=YIvOHgxdk7Rg56kVER8aLiATIgrii5oVmBCzjXnCwFE=; b=i+PrjwIZZ0YzD/WRTTITnOdYXc9eOHhyO1wMJwADCAoCU46HiI8QbbIDAqdwWHPp7sPCQO JfmAuFxtv2W2vzvpZhlFYcdb/7bqs+bTb89xNLmRZjzpU94v0G+0ISMQd3J7fZxbl1RJo6 tXIoEED/S6pB+YyRWaYrtk9Y1JHFBGV6QpfpJEQ9j1qUuhSwZBlXRamtgO6l7yMLGc0vb4 G3vFxStktiP+KXKRp6Kxw2QeHXvdqZ04y0U/o1Q1a6j60NW6PKMmPEnqG/ibLUlIKMn5Oy 1O/IG1C9MCWa1oPe2m7lC8RfcGH5nga495xOQ9SlSzBNL16Ta6ygFcqJnqxt1A== List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Sun, 20 Aug 2023 18:29:40 +0200 From: Alexander Leidinger To: current@freebsd.org Subject: 100% CPU time for sysctl command, not killable Message-ID: <67616bb5b074851832053a0286c3c2fa@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RTLgG38nDz4SWS Hi, sysctl kern.maxvnodes=1048576000 results in 100% CPU and a non-killable sysctl program. This is somewhat unexpected... Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Sun Aug 20 16:48:30 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTM4g3mMnz4qRKK for ; Sun, 20 Aug 2023 16:48:35 +0000 (UTC) (envelope-from tezeka@hotmail.com) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11olkn2082f.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eab::82f]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTM4g068Bz4Wd2 for ; Sun, 20 Aug 2023 16:48:34 +0000 (UTC) (envelope-from tezeka@hotmail.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nQIs9tensjHes/5i8V66HMBDiCWbvuLAzXt97SHqm6ang6nIGnkKvzUA6JFeBtmiNvD0tThtKWTbmbA6OsPnuAEsrVAF9DA3OY4fyEf8y6VV5CNElDYR81SbKwDKDagT914fwJEoBhC6lPYUHjHv3RsQ7zLEQuH3uOg+Sr800OHexJAjVA+QsGzxAYSm50uXAmCpuNATSp6782uk9kSN8/9B6L9fQPV2aUxAWmAQNEZdQualmTNnkXCOiT2/lrvcG8J7H9REkZJKT6DkKUaEVrn+vN2QEGT4oLHVZt2BlH58Lqv19rFjc9ttvBkIkzTpzw4BGsQ3ELxYoGncYv0fcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=k0cpEESkRu/SEWLyMT7Hn+KuAiI/YHKXh4yOUQKTcro=; b=j7xZUlA0j3vquMKPz+A99SLesq0LQFGJecOuj3hEQ1TgCbAghA9b9mdekGbsChrP3XfZdMC65t/Ayg6geoj2CIKuJIcDgrRxvR00u2MAh9/RJhYKONf6t7I64FnX+HkYlf3ZAsDBJPyXtEhLy3Y77b97AuzSNKvUBFVxourK9orMoLIC8k5EN+m8D/ib0bppoiRrvke6RklFI6avtqgr2bQdmKOzEZ4p8YCgmdqDkPCPYrrXnfhvNRcJ+qgaYh15j7EYT6+OxSjyOSPwG+Rh1KUS6xs/gN5yH62/yxNM+OP1dp2KjiBW6dsYipRjndFkWnQu3IKnM9UAB4Ae7uZhBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k0cpEESkRu/SEWLyMT7Hn+KuAiI/YHKXh4yOUQKTcro=; b=CO6E/VceohLVGUA/MB9loqWFyTNHqNqVTmiieg+XDVpkhpK2k21OZ6NHX3aCF2u0mcOf7e1YCk9o3am8MWEaS1O5FV65wo8uU/XdRaW5sKIZjqcuhy4ICsZuMCb1NwKGQhEgFVRIzN4XJnKHNKiDhwzNzD/pqgCUZRCsNP7yCsTKYqQDCo86duyNb2UDdUx5WvR6IFkBWno8/GZKggjcsvwCSf3i9Ec6WZihvVfDRfD4IvLvBG3SF1PgJXzXoNivdePppJSWmA9yJl52Mhhm6Fl0pP5DEHqOFwAAB+yEDJePTxiolLnE46dkSl01NvEMdhUEcdrw1CHdoVzqEc9VRw== Received: from CPWP152MB7921.LAMP152.PROD.OUTLOOK.COM (2603:10d6:103:2f5::11) by CPWP152MB5337.LAMP152.PROD.OUTLOOK.COM (2603:10d6:103:171::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.20; Sun, 20 Aug 2023 16:48:30 +0000 Received: from CPWP152MB7921.LAMP152.PROD.OUTLOOK.COM ([fe80::1dfb:7cc6:7b33:c9d8]) by CPWP152MB7921.LAMP152.PROD.OUTLOOK.COM ([fe80::1dfb:7cc6:7b33:c9d8%6]) with mapi id 15.20.6699.020; Sun, 20 Aug 2023 16:48:30 +0000 From: Ivan Quitschal To: Alexander Leidinger , "current@freebsd.org" Subject: Re: 100% CPU time for sysctl command, not killable Thread-Topic: 100% CPU time for sysctl command, not killable Thread-Index: AQHZ04OW8DMfRbhpUkCh/F0a3U8jA6/zZRiL Date: Sun, 20 Aug 2023 16:48:30 +0000 Message-ID: References: <67616bb5b074851832053a0286c3c2fa@Leidinger.net> In-Reply-To: <67616bb5b074851832053a0286c3c2fa@Leidinger.net> Accept-Language: pt-BR, en-US Content-Language: pt-BR X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [iVtdBbBznPYCYG+zf5yPjjPXa0oo+qB1/Wlg/PeWWYU=] x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CPWP152MB7921:EE_|CPWP152MB5337:EE_ x-ms-office365-filtering-correlation-id: 312bda53-1cb6-4e6d-a2cd-08dba19d48bc x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: h3+dSpSfsrAH4gIIi/S/ETLHhRqtD6DuxeYPyK5R1mm0vfQPs6Mv2FOZC0W8j20O4nOTJkQVAxJynUroiaSvuQ5XkRrTZHR2Had5DcqR9rYZxW4Dr9ZzA92xKemWAX+c6J62XgWsyTDAG2UFPjKcQC30Sf5mbis36BBoNfVMyBHpRTltCvTOZOwuDcV568lcrZqTLKKjDEXK+hmJiXJI6pR7KYy+mAyQc+ZId4ujUYaPqYxxvEGIwTL+Uv+P/MP8pZ6ElcdR96jRqdlHsLXwSM7u9hTI30iqAEUccHcYjJIqqcTSqFP1EbtxoXzvXxMcUaxgyt0YB+ATfJvtaUWeL2p9aN1URNuR9czWau/Ip4YoXRDEBKpyKIlZNo+0vQCckXYTdUFaseIqhjFVwXM+VuAhc/JJBni4J1MQPRy0+bso7CWu2oE6TFU+DLHn/WqZV6zYowUagiWjAqgRhqMeGIEtzj6+JfXKPJapmItXhne7kBHPFwv5RHQ+KdkcO4iHJiRh73xfWTlEy+POhb/ikImK3BgCSc0jvqi5xTTrQl+MC+cq47vs3QnRfJ2d/LGjO4mx9u36a1oahv1heJr7/OOyV2/AtHjmkIJhf4rECcE= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?UBUkDo9mjkzbB8PPhxyatuEMcti3eUAv8GrAGR2PZ0Fo4qZ+zG76IF9MheC/?= =?us-ascii?Q?pdy9xh/WGdpRLklsOYlY5iR3LeMPqTejrefxrMwZRYAFgJpY8/Is2ifBiL0F?= =?us-ascii?Q?J170MBREwApnLyQ89dq0dmUG6ffuCtbRA/sMbZ+u+WDWFf2COfD727cjkrkF?= =?us-ascii?Q?/jjPIz/E7cHscwmLjq82LUna9kwBI00Uf0Utf122TN/h3SP8UcqMcnJ115l8?= =?us-ascii?Q?S0rbWfHHWLzUiYcHOrhwpDVnNPyGxdHb7llfhNYXm29dj9Hj+UutzYEKsC2t?= =?us-ascii?Q?6wfOdocnY6E3QBlrqp0+9Jv1Da1Hd/bmO6pcHXCYYj9ro2ah3CFDw0w0aoL4?= =?us-ascii?Q?6tMo6KprzvYMUctHqgqmNBZ4fEgMs5b1GT2S+5SCFASTy0OBX05Onz/OXBHa?= =?us-ascii?Q?YfvWpnS7lMPKxsD62y0Qbb1CRp3drvQW4CeG6WQApUOQ1RV6sX5OyAhWLP4N?= =?us-ascii?Q?Fer1GeiuFasB0vNcx+QTm3Th/yfyV33e9NCi6ebK2ou3acHDAKAY0vzzl/9J?= =?us-ascii?Q?BPgkNUceFOIoPI59Tk/7xHUtX85XvoelpUWla+wqamk6nB6Fsc3QyfROLcKZ?= =?us-ascii?Q?gsWBEivQ/EeGH5cJFX0AhFN3m2iozAWoFvaFe1O7VoI465JfogRgs1kFo3NE?= =?us-ascii?Q?bDPirESpk1kXfT6GatKoJx/8iufEvfZ+fDZVoYvs/jOE80DUMZnIF5nC5ZLC?= =?us-ascii?Q?7QXgUC1Z2kQED/Fk2Ewz1cHbhwninbYoUWOxat/Z56D34+1Khzi+WvMXDS0z?= =?us-ascii?Q?AePFbwOlm9Tk5sg5qXkxVtJHFyB4OkCzFDYQVPcr0sqmJ7BSp4uKI74gc2xW?= =?us-ascii?Q?lOWYbTazAVV0neLBR4lV30QC8IGmcwATgxJ6Ga9rCpH/D2mnvkmdMWIpKgUy?= =?us-ascii?Q?gnoGbewtnFeD1KP+PCjgeDIDBHupiLN+jpGbsqVoSg/0Rj5sEHRqDGYoVBkw?= =?us-ascii?Q?V79PRGv8yhOjDfy3eSs+bptUeNA9h7yhRDfWDI0/bq7hT6ZK/b5hU1eK4tI2?= =?us-ascii?Q?oF4yiBaXB3Rhx7yRjrfTuQ3BLOT5xpnQBoUFpcAxVwLIVW50SrjRWgrYZmUl?= =?us-ascii?Q?qR59vxMBZ4SB4GZViDh/u6P+okXDo82Kp0d0QnITQzm9WDAKfGOPmm/F/zfb?= =?us-ascii?Q?/0kM6Z7PzRrZu2oooIwwvHMtXuB9gvdGxjwKB+Va4wy8A46ZNKOhlZxOfKLt?= =?us-ascii?Q?W5EAVCgiqBSWHigJ5FykCEw8BM+x3TZpebS4EruXscKYPtN3EGVBrHhi8Ao?= =?us-ascii?Q?=3D?= Content-Type: multipart/alternative; boundary="_000_CPWP152MB7921AD045F2047C54BE78A5DCB19ACPWP152MB7921LAMP_" List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-bf824.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CPWP152MB7921.LAMP152.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 312bda53-1cb6-4e6d-a2cd-08dba19d48bc X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Aug 2023 16:48:30.5067 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CPWP152MB5337 X-Rspamd-Queue-Id: 4RTM4g068Bz4Wd2 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] --_000_CPWP152MB7921AD045F2047C54BE78A5DCB19ACPWP152MB7921LAMP_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Sounds like a fork bomb discovered Obter o Outlook para Android ________________________________ From: owner-freebsd-current@freebsd.org = on behalf of Alexander Leidinger Sent: Sunday, August 20, 2023 1:29:40 PM To: current@freebsd.org Subject: 100% CPU time for sysctl command, not killable Hi, sysctl kern.maxvnodes=3D1048576000 results in 100% CPU and a non-killable sysctl program. This is somewhat unexpected... Bye, Alexander. -- https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.leidi= nger.net%2F&data=3D05%7C01%7C%7C8154f1ba1c56451df44f08dba19ab6fb%7C84df9e7f= e9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638281458099404081%7CUnknown%7CTWFpbGZsb= 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300= 0%7C%7C%7C&sdata=3D1w9Qc6PlRiJU%2FYrGjSiyLGIbWfoekhX2XbRE%2FMr2MB8%3D&reser= ved=3D0 Alexander@Leidinger.net: PGP 0x8F31830F9= F2772BF https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fwww.freeb= sd.org%2F&data=3D05%7C01%7C%7C8154f1ba1c56451df44f08dba19ab6fb%7C84df9e7fe9= f640afb435aaaaaaaaaaaa%7C1%7C0%7C638281458099404081%7CUnknown%7CTWFpbGZsb3d= 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%= 7C%7C%7C&sdata=3DXONqhX5V5duH8oyRVwnAiruzHzcnIGCcjbWVDLscILU%3D&reserved=3D= 0 netchild@FreeBSD.org : PGP 0x8F31830F9F2772B= F --_000_CPWP152MB7921AD045F2047C54BE78A5DCB19ACPWP152MB7921LAMP_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Sounds  like a fork bomb discovered 

From: owner-freebsd-current= @freebsd.org <owner-freebsd-current@freebsd.org> on behalf of Alexand= er Leidinger <Alexander@Leidinger.net>
Sent: Sunday, August 20, 2023 1:29:40 PM
To: current@freebsd.org <current@freebsd.org>
Subject: 100% CPU time for sysctl command, not killable
 
Hi,

sysctl kern.maxvnodes=3D1048576000 results in 100% CPU and a non-killable <= br> sysctl program. This is somewhat unexpected...

Bye,
Alexander.

--
https://na01.safelinks.protection.out= look.com/?url=3Dhttp%3A%2F%2Fwww.leidinger.net%2F&data=3D05%7C01%7C%7C8= 154f1ba1c56451df44f08dba19ab6fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%= 7C638281458099404081%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2= luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3D1w9Qc6PlRi= JU%2FYrGjSiyLGIbWfoekhX2XbRE%2FMr2MB8%3D&reserved=3D0 Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
https://na01.safelinks.protection.outlo= ok.com/?url=3Dhttp%3A%2F%2Fwww.freebsd.org%2F&data=3D05%7C01%7C%7C8154f= 1ba1c56451df44f08dba19ab6fb%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C63= 8281458099404081%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz= IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DXONqhX5V5duH8o= yRVwnAiruzHzcnIGCcjbWVDLscILU%3D&reserved=3D0    netchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF

--_000_CPWP152MB7921AD045F2047C54BE78A5DCB19ACPWP152MB7921LAMP_-- From nobody Sun Aug 20 16:55:02 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTMDJ2mj5z4qRm6 for ; Sun, 20 Aug 2023 16:55:12 +0000 (UTC) (envelope-from freebsd@igalic.co) Received: from mail-4018.proton.ch (mail-4018.proton.ch [185.70.40.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTMDJ07DTz4XZR for ; Sun, 20 Aug 2023 16:55:11 +0000 (UTC) (envelope-from freebsd@igalic.co) Authentication-Results: mx1.freebsd.org; none Date: Sun, 20 Aug 2023 16:55:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=igalic.co; s=protonmail2; t=1692550507; x=1692809707; bh=DanR9sQoilB/g6SI65KVyd6uBeto+QeXEe/leleUa/c=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=VnkBAfy8/UTMAwOtZSt9XVo6XrGzTXRNX1ZuNtjigkKhC583DF6RvTQr/X706TYut zd6Wj2JdjXVzQ0pO7VIMmdumh7DUXBBcMIHBjBksVc72GnLzvjChAFeQey6zNFNgTp XnVfg2k8GLAn8f9uYZY9pyIsr5TDkSJAP0ugkEegf/kjC1ByaF2GCaJuw/lt28Bnpf sAm+M0RFcRK31Nmtcqf5o7Po8zFxlJjul3r+Vekk+p+Ru4zteJQ6ejw0yZF0G0SnfB 8cIgzv6rlJyADO+annHL+bLCi38zsZrj34pW2abEC7VOAo2LK/6Qg5b3SvDaMgeole 2i3Tt0VUfxPhQ== To: Alexander@Leidinger.net, current@freebsd.org From: =?utf-8?Q?Mina_Gali=C4=87?= Subject: Re: 100% CPU time for sysctl command, not killable Message-ID: <-2rv9P1fc0I_nDQomzn-3shvEyCOkCA84lNmp7B3RdDOr9qBwG5QXZNT0R4ZPq0Z1S4-PyGC1xbVoNjPNHNmajxFq-ygNfDS4AqGzB0ylRU=@igalic.co> In-Reply-To: <67616bb5b074851832053a0286c3c2fa@Leidinger.net> References: <67616bb5b074851832053a0286c3c2fa@Leidinger.net> Feedback-ID: 66573723:user:proton List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_eOT26PIU28odRghHE2P4xUQygDewpCS7RqYVD03hHEQ" X-Rspamd-Queue-Id: 4RTMDJ07DTz4XZR X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH] This is a multi-part message in MIME format. --b1_eOT26PIU28odRghHE2P4xUQygDewpCS7RqYVD03hHEQ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 cHJvY3N0YXQoMSkga3N0YWNrIGNvdWxkIGJlIGhlbHBmdWwgaGVyZS4KCi0tLS0tLS0tIE9yaWdp bmFsIE1lc3NhZ2UgLS0tLS0tLS0KT24gMjAgQXVnIDIwMjMsIDE3OjI5LCBBbGV4YW5kZXIgTGVp ZGluZ2VyIHdyb3RlOgoKPiBIaSwgc3lzY3RsIGtlcm4ubWF4dm5vZGVzPTEwNDg1NzYwMDAgcmVz dWx0cyBpbiAxMDAlIENQVSBhbmQgYSBub24ta2lsbGFibGUgc3lzY3RsIHByb2dyYW0uIFRoaXMg aXMgc29tZXdoYXQgdW5leHBlY3RlZC4uLiBCeWUsIEFsZXhhbmRlci4gLS0gaHR0cDovL3d3dy5M ZWlkaW5nZXIubmV0IEFsZXhhbmRlckBMZWlkaW5nZXIubmV0OiBQR1AgMHg4RjMxODMwRjlGMjc3 MkJGIGh0dHA6Ly93d3cuRnJlZUJTRC5vcmcgbmV0Y2hpbGRARnJlZUJTRC5vcmcgOiBQR1AgMHg4 RjMxODMwRjlGMjc3MkJG --b1_eOT26PIU28odRghHE2P4xUQygDewpCS7RqYVD03hHEQ Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 cHJvY3N0YXQoMSkga3N0YWNrIGNvdWxkIGJlIGhlbHBmdWwgaGVyZS48YnI+PGJyPi0tLS0tLS0t IE9yaWdpbmFsIE1lc3NhZ2UgLS0tLS0tLS08YnI+T24gMjAgQXVnIDIwMjMsIDE3OjI5LCBBbGV4 YW5kZXIgTGVpZGluZ2VyIDwgQWxleGFuZGVyQExlaWRpbmdlci5uZXQ+IHdyb3RlOjxibG9ja3F1 b3RlIGNsYXNzPSJwcm90b25tYWlsX3F1b3RlIj48YnI+SGksDQoNCnN5c2N0bCBrZXJuLm1heHZu b2Rlcz0xMDQ4NTc2MDAwIHJlc3VsdHMgaW4gMTAwJSBDUFUgYW5kIGEgbm9uLWtpbGxhYmxlDQpz eXNjdGwgcHJvZ3JhbS4gVGhpcyBpcyBzb21ld2hhdCB1bmV4cGVjdGVkLi4uDQoNCkJ5ZSwNCkFs ZXhhbmRlci4NCg0KLS0NCmh0dHA6Ly93d3cuTGVpZGluZ2VyLm5ldCBBbGV4YW5kZXJATGVpZGlu Z2VyLm5ldDogUEdQIDB4OEYzMTgzMEY5RjI3NzJCRg0KaHR0cDovL3d3dy5GcmVlQlNELm9yZyAg ICBuZXRjaGlsZEBGcmVlQlNELm9yZyAgOiBQR1AgMHg4RjMxODMwRjlGMjc3MkJGDQoNCjwvZGl2 Pg== --b1_eOT26PIU28odRghHE2P4xUQygDewpCS7RqYVD03hHEQ-- From nobody Sun Aug 20 17:10:12 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTMYg72xJz4qT04 for ; Sun, 20 Aug 2023 17:10:15 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x33.google.com (mail-oa1-x33.google.com [IPv6:2001:4860:4864:20::33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTMYf4h4wz4ZDc for ; Sun, 20 Aug 2023 17:10:14 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=Z2CC8mO8; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::33 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-1bbb4bde76dso1607269fac.2 for ; Sun, 20 Aug 2023 10:10:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692551413; x=1693156213; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MLe0zFUf6jIWOW517J+hi3t4+oeXrxr4nei2bzYFuME=; b=Z2CC8mO84nZm/YKdlsRa4+u3GUVPW5lL/Z2jqGSfdONGYVByqBijr1kTEiTtbeqWyu K6bTwj5ubbpAADxb7GOAHREeQ7es26QewbLjrFI0StbOgHqECz8YxTpYNgF5czydILJg g4YGVLIOly7i8VWKoIPpNw6ugZDPmvBWaNDSBWyXgu5AgtvC1MMDzLdY1mX5drZdzrbi z4njWz3AiHzeBmWJlwyY8Bw1Kfd2AYV1dnzZtYMCRDU2n9vPdFllKNe6nGHibJn0nDjY recwIJO6/abZro36Tvy7AzJzGLttNpGUqyJJNTfAHf3CfUvSM6vsqxoqxt659lw0+YKB hZfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692551413; x=1693156213; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MLe0zFUf6jIWOW517J+hi3t4+oeXrxr4nei2bzYFuME=; b=ik3KSnE8EzgHka6PA1mI/NUvzgBWsrOQc1iqhFL5Eb3OFfvuBDeW5DNBsnMF6IPZbZ IP8f0aZkhZ+wMPB3/ugeQGBj0Vhi1x3BjgPMaR+VrkfsCe7r9lUjJpkj5HKDBRjf8O9h P/jQdPLdlVeCPe9jCNZtrLsYWQpLpSVivJXmfK2kj8rtz9OlWXfnqD1YsywXBTW6qt8F 1kQTuI9+8vdAAyzoekD/k8kNZ6S8OkaS2TjQwvHyvLmT947rYGd17xJevCTq6Lz2GSiQ KDRPsaNYurJAxUBXrSlwSazeC+Avb7lmPHv+cy0Rwc3wFcxGDscfyxgqJN6GsFk3Ynoj SByQ== X-Gm-Message-State: AOJu0YwhkrPWVWcKgxRDDGePLlm/KmveZO9bPiFa0YOm9tWdpRARUvvc qUnfGcf+XF7+5zFoM3GzGPMwZh5wha+vlzhVA4L1BW8cmJQ= X-Google-Smtp-Source: AGHT+IHXPFj3Z7DHJL+ctpEmvrDORGecu73x4CpTrOojDTLCrZhELS2m6LgnNP5KIoJuIttL8MZQ4AsrCNOqXDaX0uM= X-Received: by 2002:a05:6870:210f:b0:1ba:c622:3d5a with SMTP id f15-20020a056870210f00b001bac6223d5amr6181315oae.18.1692551412814; Sun, 20 Aug 2023 10:10:12 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:79d9:0:b0:4f0:1250:dd51 with HTTP; Sun, 20 Aug 2023 10:10:12 -0700 (PDT) In-Reply-To: <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> From: Mateusz Guzik Date: Sun, 20 Aug 2023 19:10:12 +0200 Message-ID: Subject: Re: Speed improvements in ZFS To: Alexander Leidinger Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[current@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::33:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RTMYf4h4wz4ZDc On 8/18/23, Alexander Leidinger wrote: > Am 2023-08-16 18:48, schrieb Alexander Leidinger: >> Am 2023-08-15 23:29, schrieb Mateusz Guzik: >>> On 8/15/23, Alexander Leidinger wrote: >>>> Am 2023-08-15 14:41, schrieb Mateusz Guzik: >>>> >>>>> With this in mind can you provide: sysctl kern.maxvnodes >>>>> vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes >>>>> vfs.recycles_free vfs.recycles >>>> >>>> After a reboot: >>>> kern.maxvnodes: 10485760 >>>> vfs.wantfreevnodes: 2621440 >>>> vfs.freevnodes: 24696 >>>> vfs.vnodes_created: 1658162 >>>> vfs.numvnodes: 173937 >>>> vfs.recycles_free: 0 >>>> vfs.recycles: 0 >> >> New values after one rund of periodic: >> kern.maxvnodes: 10485760 >> vfs.wantfreevnodes: 2621440 >> vfs.freevnodes: 356202 >> vfs.vnodes_created: 427696288 >> vfs.numvnodes: 532620 >> vfs.recycles_free: 20213257 >> vfs.recycles: 0 > > And after the second round which only took 7h this night: > kern.maxvnodes: 10485760 > vfs.wantfreevnodes: 2621440 > vfs.freevnodes: 3071754 > vfs.vnodes_created: 1275963316 > vfs.numvnodes: 3414906 > vfs.recycles_free: 58411371 > vfs.recycles: 0 > >>>>> Meanwhile if there is tons of recycles, you can damage control by >>>>> bumping kern.maxvnodes. >> >> What's the difference between recycles and recycles_free? Does the >> above count as bumping the maxvnodes? > > ^^^^^ > >>>> Looks like there are not much free directly after the reboot. I will >>>> check the values tomorrow after the periodic run again and maybe >>>> increase by 10 or 100 so see if it makes a difference. >>>> >>>>> If this is not the problem you can use dtrace to figure it out. >>>> >>>> dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or >>>> something else? >>>> >>> >>> I mean checking where find is spending time instead of speculating. >>> >>> There is no productized way to do it so to speak, but the following >>> crapper should be good enough: >> [script] >> >> I will let it run this night. > > I have a 51MB text file, compressed to about 1MB. Are you interested to > get it? > Your problem is not the vnode limit, but nullfs. https://people.freebsd.org/~mjg/netchild-periodic-find.svg First, some of the contention is notorious VI_LOCK in order to do anything. But more importantly the mind-boggling off-cpu time comes from exclusive locking which should not be there to begin with -- as in that xlock in stat should be a slock. Maybe I'm going to look into it later. -- Mateusz Guzik From nobody Sun Aug 20 19:23:35 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTQWt2F8sz4qc8v for ; Sun, 20 Aug 2023 19:23:54 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTQWs62tJz3K5p for ; Sun, 20 Aug 2023 19:23:53 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id 0091258D; Sun, 20 Aug 2023 21:23:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1692559419; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=5LoL+MhLwzFexRyJS1m4U6J+JRkeLqo8+XQGvfaakY4=; b=Sh9GkuBo5c4Mn1lr1QbTVEOp+/OR/vQlzdM8ksYJuQcOqTpSS36nv2K8cESokfCBXSBgG0 2wWsARhQGW2+3b+t/K2pk2FjDX9TA3EFlbgmGZpuLwpOLN2kNx/CGVI92WPlF709CfYZbP Vai+9X5VQriNukGxkme9z5rdsnRWJ+32iJa61khhQDw38oPLzLH5LSZ12wXD6N80RhgEwW 8Q3bdWRQWEYR8ADWU87pjWRv+TZCf9iM7/W82RPMfXJY3nPGtI7azsH3vWJ3ryrZ2s6IQC XSRlgO6P/yJBuF5/mcGVnU1eQ77IxZvBRbSG1ll3o8Uym2LVTYqtVPv70Mf+GQ== List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Sun, 20 Aug 2023 21:23:35 +0200 From: Alexander Leidinger To: =?UTF-8?Q?Mina_Gali=C4=87?= Cc: current@freebsd.org Subject: Re: 100% CPU time for sysctl command, not killable In-Reply-To: <-2rv9P1fc0I_nDQomzn-3shvEyCOkCA84lNmp7B3RdDOr9qBwG5QXZNT0R4ZPq0Z1S4-PyGC1xbVoNjPNHNmajxFq-ygNfDS4AqGzB0ylRU=@igalic.co> References: <67616bb5b074851832053a0286c3c2fa@Leidinger.net> <-2rv9P1fc0I_nDQomzn-3shvEyCOkCA84lNmp7B3RdDOr9qBwG5QXZNT0R4ZPq0Z1S4-PyGC1xbVoNjPNHNmajxFq-ygNfDS4AqGzB0ylRU=@igalic.co> Message-ID: X-Sender: Alexander@Leidinger.net Content-Type: multipart/alternative; boundary="=_fb8a0bfe27c23b524a871433d6194cfd" X-Rspamd-Queue-Id: 4RTQWs62tJz3K5p X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] --=_fb8a0bfe27c23b524a871433d6194cfd Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed Am 2023-08-20 18:55, schrieb Mina Galić: > procstat(1) kstack could be helpful here. > > -------- Original Message -------- > On 20 Aug 2023, 17:29, Alexander Leidinger Alexander@Leidinger.net> > wrote: > >> Hi, sysctl kern.maxvnodes=1048576000 results in 100% CPU and a >> non-killable sysctl program. This is somewhat unexpected... Bye, >> Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP >> 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP >> 0x8F31830F9F2772BF PID TID COMM TDNAME KSTACK 94391 118678 sysctl - sysctl_maxvnodes sysctl_root_handler_locked sysctl_root userland_sysctl sys___sysctl amd64_syscall fast_syscall_common Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_fb8a0bfe27c23b524a871433d6194cfd Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

Am 2023-08-20 18:55, schrieb Mina Gali=C4=87:

procstat(1) kstack could be helpful here.
 
-------- Original Message --------
On 20 Aug 2023, 17:29, Alexand= er Leidinger Alexander@Leidinger.net> wrote:

Hi, sysctl kern.maxvnodes=3D= 1048576000 results in 100% CPU and a non-killable sysctl program. This is s= omewhat unexpected... Bye, Alexander. -- http://www.Leidinger.net Alexander= @Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@Free= BSD.org : PGP 0x8F31830F9F2772BF

  PID    TID COMM           &nbs= p;    TDNAME              KSTA= CK                     &n= bsp; 
94391 118678 sysctl           &nbs= p;  -                   s= ysctl_maxvnodes sysctl_root_handler_locked sysctl_root userland_sysctl sys_= __sysctl amd64_syscall fast_syscall_common 

Bye,

Alexander.

--
= http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
http:/= /www.FreeBSD.org    n= etchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF
--=_fb8a0bfe27c23b524a871433d6194cfd-- From nobody Sun Aug 20 19:29:47 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTQgS5Tmmz4qcW0 for ; Sun, 20 Aug 2023 19:30:28 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (P-256)) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTQgS4WYCz3Lqh for ; Sun, 20 Aug 2023 19:30:28 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id DE0162C8; Sun, 20 Aug 2023 21:29:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1692559822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UxcJ8VacR93uwsBrYTrJn0PSDg8RaBx2oFp3Moz5W7Y=; b=u6h6CI6P+k1hdHWggHYFosSB6qUXc8RS2EXunC2kP4VKRmIEbH6xED/fthaf04OxMmXQhN HAbT3SpBjQyERNIZgDhef3bf2t6AJtucYq3GTdf8lgYuEuZTH8Z3hwhROBTeDs3XU4YmBk Xt/nLlQ7uVW2qoJxCa3AdpdpedDCvXpo7K0ZNktkr8I/mB9rEU1dv3mUE/mpC3Y+5pqtCj 1+te8kF3rUnDxSXLsc5J2ytSZ18gPDIUnIGWBsM3AwWsRE7mV87fc6UChZVQVWLu7aLiCY o4SfuSzMgTdC1ofMRmfCjc4audXrTQvkVjG3vHctKzvBHQDmUxRRlACj3hFKlQ== List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Sun, 20 Aug 2023 21:29:47 +0200 From: Alexander Leidinger To: Mateusz Guzik Cc: current@freebsd.org Subject: Re: Speed improvements in ZFS In-Reply-To: References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> Message-ID: <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RTQgS4WYCz3Lqh X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] Am 2023-08-20 19:10, schrieb Mateusz Guzik: > On 8/18/23, Alexander Leidinger wrote: >> I have a 51MB text file, compressed to about 1MB. Are you interested >> to >> get it? >> > > Your problem is not the vnode limit, but nullfs. > > https://people.freebsd.org/~mjg/netchild-periodic-find.svg 122 nullfs mounts on this system. And every jail I setup has several null mounts. One basesystem mounted into every jail, and then shared ports (packages/distfiles/ccache) across all of them. > First, some of the contention is notorious VI_LOCK in order to do > anything. > > But more importantly the mind-boggling off-cpu time comes from > exclusive locking which should not be there to begin with -- as in > that xlock in stat should be a slock. > > Maybe I'm going to look into it later. That would be fantastic. Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Sun Aug 20 20:02:31 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTRNT5mrQz4qf8h for ; Sun, 20 Aug 2023 20:02:33 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTRNT3sDpz3P8l for ; Sun, 20 Aug 2023 20:02:33 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-56ce1bd7fc4so1938763eaf.2 for ; Sun, 20 Aug 2023 13:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692561752; x=1693166552; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YY1SR3y85g+ahr047W/A6oTYhluZvFVEymQ2LpichjI=; b=K97xQr+VohZgHKPkUXs0Bd1REOKZD7QwzEV01i+LPhBzVggCHvVt/tSAhcbM/f0ovt W9dU01Z6g8urILBue3OyL+lPr5M31UWR+Zux0l7QsE6VoAU0ub1u9ioat0V0uHwT58Wm 58Nd6XvYt6WTYBizRHeBpN5rkMzfQOv8ub1wiTFKsn+b+NL+5SllaeF4JzCPCnhfrDi9 lf5MdXiMujRW2v62CRKxUgxRuoXLKPgN5ZpG69dElTPLYFhBfGPM/fh90Z5JeTDyM/w2 xo0RU79syUYKCYBbKX/joqTav3jWDrXZbY1pLZw+SOZ4Bn/zG4YAPxS7CZM4YZ95qIBb Esig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692561752; x=1693166552; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YY1SR3y85g+ahr047W/A6oTYhluZvFVEymQ2LpichjI=; b=NBWcohykoz9AVGeIJzNpEOfP9wIlAuAI9llxWQpaseWptBGspbGe82MuNCDE+7hOU9 rAi1BedgIACLwf8UiNX404EzxXlAC4w3z7ItuztIgY47IMWKEaH9qwnpsF8rqe71l1uc Wbt1X1dwg0bNqUoUCd4Pde1eioDn4DYwZZ4cF/6esCVg8+/Fxdlrusihlj7d5Es4nowq UeusVsAsX5zvfMB/OlvCt0GvQL6qd2fgGgBkfCjLMC9CeKPbaj5w9cM60la7OuWKvji/ +D1PvLoPe0965maHvQsnP2GxsFspqBC3kmcI8t2HAQ4pADDjaiaIPkiKHObzY1/9Rfac GqTg== X-Gm-Message-State: AOJu0YxRZVqt61b2hiIEfZBsv8XUgQRmSE3QP5UAMVN5dQIhYa8zc4f3 MJs7lxAJtaLBrQJelKNWIw6rD8ahjVwpHQUjWHydKcCSHfM= X-Google-Smtp-Source: AGHT+IGeSkib8+8NiMPhi47CqQudsp7yU7TBZjGgQYfHWSZwy038XcKsd3zGIAHszeHxHdKgBIMrhTsFOx7rYHDkNS0= X-Received: by 2002:a4a:381d:0:b0:56d:2cbf:2315 with SMTP id c29-20020a4a381d000000b0056d2cbf2315mr4322451ooa.9.1692561752198; Sun, 20 Aug 2023 13:02:32 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:79d9:0:b0:4f0:1250:dd51 with HTTP; Sun, 20 Aug 2023 13:02:31 -0700 (PDT) In-Reply-To: <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> From: Mateusz Guzik Date: Sun, 20 Aug 2023 22:02:31 +0200 Message-ID: Subject: Re: Speed improvements in ZFS To: Alexander Leidinger Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RTRNT3sDpz3P8l X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On 8/20/23, Alexander Leidinger wrote: > Am 2023-08-20 19:10, schrieb Mateusz Guzik: >> On 8/18/23, Alexander Leidinger wrote: > >>> I have a 51MB text file, compressed to about 1MB. Are you interested >>> to >>> get it? >>> >> >> Your problem is not the vnode limit, but nullfs. >> >> https://people.freebsd.org/~mjg/netchild-periodic-find.svg > > 122 nullfs mounts on this system. And every jail I setup has several > null mounts. One basesystem mounted into every jail, and then shared > ports (packages/distfiles/ccache) across all of them. > >> First, some of the contention is notorious VI_LOCK in order to do >> anything. >> >> But more importantly the mind-boggling off-cpu time comes from >> exclusive locking which should not be there to begin with -- as in >> that xlock in stat should be a slock. >> >> Maybe I'm going to look into it later. > > That would be fantastic. > I did a quick test, things are shared locked as expected. However, I found the following: if ((xmp->nullm_flags & NULLM_CACHE) != 0) { mp->mnt_kern_flag |= lowerrootvp->v_mount->mnt_kern_flag & (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | MNTK_EXTENDED_SHARED); } are you using the "nocache" option? it has a side effect of xlocking -- Mateusz Guzik From nobody Sun Aug 20 20:13:43 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTRf83zqrz4qg56 for ; Sun, 20 Aug 2023 20:14:24 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (P-256)) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTRf826qYz3QPr for ; Sun, 20 Aug 2023 20:14:24 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id D199970C; Sun, 20 Aug 2023 22:13:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1692562457; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ce2enUYmpHis8DffVIeivdB8QDS0Zqt+4890lyO5yxg=; b=KNdZaNFVkYaSN+PUxX4JSRwWft4e2MOah9AejuDOsi1PSypgeClGjepHJWr37Z9aKRBoVM MBSomyN/noCKly8V8pQvVPVUB1RwRpTYZpxcH33rMF3c2Iqr+NV0i3YmJmqxO9UXEhSBFE gYelPCd5U54Z0yNcKIRyMfNtS4wlujEQy2qa6aVyD3ZHvskBWmsrLd9iSct1WAgZb8iUMF xDlt+KL33qlvd8pqZ0SIoeDjl8kXJEe7QXezN36pdFOa3vcCznDQcSDvMLj77Fx5AKMO9X azPxUJZeuExETG8XeP3CD6IzMd1A0u1Le4hTiHuxhjuSV90zUEp20Me9YQRDFQ== List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Sun, 20 Aug 2023 22:13:43 +0200 From: Alexander Leidinger To: Mateusz Guzik Cc: current@freebsd.org Subject: Re: Speed improvements in ZFS In-Reply-To: References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> Message-ID: <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RTRf826qYz3QPr X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] Am 2023-08-20 22:02, schrieb Mateusz Guzik: > On 8/20/23, Alexander Leidinger wrote: >> Am 2023-08-20 19:10, schrieb Mateusz Guzik: >>> On 8/18/23, Alexander Leidinger wrote: >> >>>> I have a 51MB text file, compressed to about 1MB. Are you interested >>>> to >>>> get it? >>>> >>> >>> Your problem is not the vnode limit, but nullfs. >>> >>> https://people.freebsd.org/~mjg/netchild-periodic-find.svg >> >> 122 nullfs mounts on this system. And every jail I setup has several >> null mounts. One basesystem mounted into every jail, and then shared >> ports (packages/distfiles/ccache) across all of them. >> >>> First, some of the contention is notorious VI_LOCK in order to do >>> anything. >>> >>> But more importantly the mind-boggling off-cpu time comes from >>> exclusive locking which should not be there to begin with -- as in >>> that xlock in stat should be a slock. >>> >>> Maybe I'm going to look into it later. >> >> That would be fantastic. >> > > I did a quick test, things are shared locked as expected. > > However, I found the following: > if ((xmp->nullm_flags & NULLM_CACHE) != 0) { > mp->mnt_kern_flag |= > lowerrootvp->v_mount->mnt_kern_flag & > (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | > MNTK_EXTENDED_SHARED); > } > > are you using the "nocache" option? it has a side effect of xlocking I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache. Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Sun Aug 20 21:07:08 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTSq24jkQz4qjcb for ; Sun, 20 Aug 2023 21:07:10 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc2f.google.com (mail-oo1-xc2f.google.com [IPv6:2607:f8b0:4864:20::c2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTSq22jRfz3gh7 for ; Sun, 20 Aug 2023 21:07:10 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc2f.google.com with SMTP id 006d021491bc7-570c856e946so522965eaf.2 for ; Sun, 20 Aug 2023 14:07:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692565629; x=1693170429; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=FftWY4B7CNJaRTG+rbcDQdkdqI2+efcueYIVqGwe0Ss=; b=MANI4Bqvjbu6ap4GlK1NcRz+MOQONoTAZ/oar1qhn5CsBUWD8CEki8hC6J6OAS41Zo 0MKQKvlsaTepbClCTIdtJv3pXOqZdpjh4QeeBl/TQqrIw9keSmLY0n9gCf1m97IHso4v c0GhzMtS0pay9g9yX5AX6kk38b7zqvHFKpxjADMcqpB1fPIrDcf+6opMcrBcSRKLB54j pzD+5XK0+bzVkowPf4EQL+AOe8D3NJkpAuIrHHmcaCRcsaKWGmCkmsLrcRHP2TdVrtNi xFD285ectO6W03wQB2BWpzob3sM6FY+gi4LWtb0HuNIrPPcBtCb5hMycOFZXzVVHkFvj cygw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692565629; x=1693170429; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FftWY4B7CNJaRTG+rbcDQdkdqI2+efcueYIVqGwe0Ss=; b=BIwgY/yzjxV9ULyyjlMnlcNDDkLgiPf1Ue2RXW/Te3E6uEdGhlHEzMzZm4vORisLkX CR27Fh1covyx9QUjx+xzumBzQffMut6cxxEfsfbLW7ARYAkMy7qfZC4429OGMqDZ+E4v it4oQtuEF8kCONzqse174dNacNuWLp5fH/gUTFUPMVZMoTEiD313Y8GNwFOFLhWwhVjM UQVKMTOcEjTurMvbAHRF3XjgQ/x9IH8g5IXu9B8att86yrv9nfz0ISy2/oDuW8lIOnYq knAEe0iy1wtpFB2AMOiqT8Y4DTfGUiKIKAimCOeJ+RRPzELZwSgF29s3uRd9lSiSDq3L 0lZw== X-Gm-Message-State: AOJu0YwGOeAafadxlUphDeReQ0AMmKniY1dgfzhcGuegfjHvjPVxrl2/ 6HMxrPxOXgMQ+PXa3QY5ShkgfTWKw/eIFebuyac= X-Google-Smtp-Source: AGHT+IFkuu+mtxGlp9IaV30Ovl+7mRwPKtDgpyvTP+WjkqnRDeZAgO0+Comh5ei1tUzyMOTNOtbGsi+F11LIn+9QL1M= X-Received: by 2002:a4a:654e:0:b0:566:ed69:422d with SMTP id z14-20020a4a654e000000b00566ed69422dmr5556239oog.7.1692565629181; Sun, 20 Aug 2023 14:07:09 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:79d9:0:b0:4f0:1250:dd51 with HTTP; Sun, 20 Aug 2023 14:07:08 -0700 (PDT) In-Reply-To: <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> From: Mateusz Guzik Date: Sun, 20 Aug 2023 23:07:08 +0200 Message-ID: Subject: Re: Speed improvements in ZFS To: Alexander Leidinger Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RTSq22jRfz3gh7 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On 8/20/23, Alexander Leidinger wrote: > Am 2023-08-20 22:02, schrieb Mateusz Guzik: >> On 8/20/23, Alexander Leidinger wrote: >>> Am 2023-08-20 19:10, schrieb Mateusz Guzik: >>>> On 8/18/23, Alexander Leidinger wrote: >>> >>>>> I have a 51MB text file, compressed to about 1MB. Are you interested >>>>> to >>>>> get it? >>>>> >>>> >>>> Your problem is not the vnode limit, but nullfs. >>>> >>>> https://people.freebsd.org/~mjg/netchild-periodic-find.svg >>> >>> 122 nullfs mounts on this system. And every jail I setup has several >>> null mounts. One basesystem mounted into every jail, and then shared >>> ports (packages/distfiles/ccache) across all of them. >>> >>>> First, some of the contention is notorious VI_LOCK in order to do >>>> anything. >>>> >>>> But more importantly the mind-boggling off-cpu time comes from >>>> exclusive locking which should not be there to begin with -- as in >>>> that xlock in stat should be a slock. >>>> >>>> Maybe I'm going to look into it later. >>> >>> That would be fantastic. >>> >> >> I did a quick test, things are shared locked as expected. >> >> However, I found the following: >> if ((xmp->nullm_flags & NULLM_CACHE) != 0) { >> mp->mnt_kern_flag |= >> lowerrootvp->v_mount->mnt_kern_flag & >> (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | >> MNTK_EXTENDED_SHARED); >> } >> >> are you using the "nocache" option? it has a side effect of xlocking > > I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache. > If you don't have "nocache" on null mounts, then I don't see how this could happen. -- Mateusz Guzik From nobody Sun Aug 20 21:17:19 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTT2v6Zpzz4qkJd for ; Sun, 20 Aug 2023 21:17:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTT2v4sPxz4JVn for ; Sun, 20 Aug 2023 21:17:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 37KLHJY6073848 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 21 Aug 2023 00:17:23 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 37KLHJY6073848 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 37KLHJoG073847; Mon, 21 Aug 2023 00:17:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 21 Aug 2023 00:17:19 +0300 From: Konstantin Belousov To: Mateusz Guzik Cc: Alexander Leidinger , current@freebsd.org Subject: Re: Speed improvements in ZFS Message-ID: References: <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4RTT2v4sPxz4JVn X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] On Sun, Aug 20, 2023 at 11:07:08PM +0200, Mateusz Guzik wrote: > On 8/20/23, Alexander Leidinger wrote: > > Am 2023-08-20 22:02, schrieb Mateusz Guzik: > >> On 8/20/23, Alexander Leidinger wrote: > >>> Am 2023-08-20 19:10, schrieb Mateusz Guzik: > >>>> On 8/18/23, Alexander Leidinger wrote: > >>> > >>>>> I have a 51MB text file, compressed to about 1MB. Are you interested > >>>>> to > >>>>> get it? > >>>>> > >>>> > >>>> Your problem is not the vnode limit, but nullfs. > >>>> > >>>> https://people.freebsd.org/~mjg/netchild-periodic-find.svg > >>> > >>> 122 nullfs mounts on this system. And every jail I setup has several > >>> null mounts. One basesystem mounted into every jail, and then shared > >>> ports (packages/distfiles/ccache) across all of them. > >>> > >>>> First, some of the contention is notorious VI_LOCK in order to do > >>>> anything. > >>>> > >>>> But more importantly the mind-boggling off-cpu time comes from > >>>> exclusive locking which should not be there to begin with -- as in > >>>> that xlock in stat should be a slock. > >>>> > >>>> Maybe I'm going to look into it later. > >>> > >>> That would be fantastic. > >>> > >> > >> I did a quick test, things are shared locked as expected. > >> > >> However, I found the following: > >> if ((xmp->nullm_flags & NULLM_CACHE) != 0) { > >> mp->mnt_kern_flag |= > >> lowerrootvp->v_mount->mnt_kern_flag & > >> (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | > >> MNTK_EXTENDED_SHARED); > >> } > >> > >> are you using the "nocache" option? it has a side effect of xlocking > > > > I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache. > > > > If you don't have "nocache" on null mounts, then I don't see how this > could happen. There is also MNTK_NULL_NOCACHE on lower fs, which is currently set for fuse and nfs at least. From nobody Sun Aug 20 21:27:38 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTTGm04gsz4ql0k for ; Sun, 20 Aug 2023 21:27:44 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTTGk39pJz4LM3 for ; Sun, 20 Aug 2023 21:27:42 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=XhrH2qhf; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::633 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-9a1869f2c06so193491866b.0 for ; Sun, 20 Aug 2023 14:27:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692566860; x=1693171660; h=in-reply-to:content-language:references:to:from:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=N2c8wwCuTb8bnaGjE/OwXHLFBHe6TO2vHGEzY6RawVU=; b=XhrH2qhfZCnsg0iN18Z+PCC26plQciph8tL+4zikf/nT0DUEXgG22IPJ502NQQ/9bs H6icTzFp88/fQ4huHTlQxPNF0pSkVB7u4tH6eU0ozD36HeUHkAf64IJilP1UP0jg3Vlh sMsYj2wG3/ekPodCoG65TaP3PtN3vz7pztLvg4ZRGIRt712v8Q58EMQjlOQv74RNOhT0 aY/CsSu4bxq3RRUTllPsML5vIFTWnJFXd8AUGawl0/rtqPx2DkpUZmhYCupMhCXWu4K4 RbIGO1/a5lFswHmdpCAyJx5jXDW5tJeEzMDF67yQXpP88Z3l/5GYGJWBY5t92mcgN/J0 cfhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692566860; x=1693171660; h=in-reply-to:content-language:references:to:from:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=N2c8wwCuTb8bnaGjE/OwXHLFBHe6TO2vHGEzY6RawVU=; b=UwZnI2cxkVjweU9MoyAfqGfO59q9ftTxA36cZhivurklLh0bcod30dLuLMPCaPvwao gapOY5QR05DRsIap3muDZvMkPCKnZe8p4WhqKOs4aFjJwqru+nFPSocQQCk+loHnHfFl UrqZpF7T5EWK6S1li2oB7vugpB+84obQaHuF1uxQNMxIq1cuCbmh004tIco4Jhqdlikw 6hOuUHuDcL0bkOpzn5RZ3Z9L3qc5PeNyBbRKdwE/0POv8EHwcfN/thUaamIj5RO6dJq0 I1lVJdcO+K+w4LDau6mHmenpBH1dSCKd9/zu8Ihf3Kqi+eiD3jt0yby1ue3OirtDEh2d Kg0Q== X-Gm-Message-State: AOJu0YxplWPzonvovMyr7eicYMXcVBwRL0qdXnIFrf3RBR3z2taH4k65 q3gK6ii04jcjodXztvyNuFAI4SpOfos= X-Google-Smtp-Source: AGHT+IFPRhg54xJEQDER0JKrRTsJkwoi7D5d6AuIgPwyjp+QuwhGmMNhg+X/2hQQS7N6di/gcmIMmw== X-Received: by 2002:a17:907:16ac:b0:992:13c7:560 with SMTP id hc44-20020a17090716ac00b0099213c70560mr4567022ejc.38.1692566859663; Sun, 20 Aug 2023 14:27:39 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id k17-20020a1709062a5100b0099bc2d1429csm5317123eje.72.2023.08.20.14.27.39 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 20 Aug 2023 14:27:39 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------KzlIEyIMmbxm0W3nKMF3bwCF" Message-ID: <0a9eeef7-edda-8352-184d-f4b10a1ed290@gmail.com> Date: Sun, 20 Aug 2023 22:27:38 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Boot failures with 14.0-ALPHA1 and 14.0-ALPHA2 From: Graham Perrin To: FreeBSD CURRENT References: <3ae7e409-e30c-9b05-9f82-d286cbc8fa21@gmail.com> Content-Language: en-US In-Reply-To: <3ae7e409-e30c-9b05-9f82-d286cbc8fa21@gmail.com> X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::633:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; REDIRECTOR_URL(0.00)[goo.gl]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RTTGk39pJz4LM3 This is a multi-part message in MIME format. --------------KzlIEyIMmbxm0W3nKMF3bwCF Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 18/08/2023 13:21, Graham Perrin wrote: > I'll update to ALPHA2. Another failure. In this case, removal of a USB device was unnecessary; the next start succeeded. % bectl list -c creation | grep edacf4b4824a n264868-edacf4b4824a-a NR     /          460G  2023-08-19 03:54 n264868-edacf4b4824a-b -      -          52.1M 2023-08-20 20:20 % uname -aKU FreeBSD mowa219-gjp4-8570p-freebsd 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400094 #4 main-n264868-edacf4b4824a-dirty: Fri Aug 18 23:46:09 BST 2023 grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.a md64/sys/GENERIC amd64 1400094 1400094 % > In the meantime, briefly: > > – no visible progress beyond the four lines of EFI framebuffer > information. > > In each case, I worked around by removing a USB flash drive that's > used for L2ARC. > > The most recent incident was preceded by a power button shutdown, in > response to a drm-510-kmod blackout (GPU lockup etc.) whilst typing in > Firefox (comparable to > for drm-515-kmod > 5.15.25). > > The prior incident was also preceded by a power button shutdown (in > response to a blackout at Plasma log out time). This was remarkable in > that there was an unexpected restart, when I expected the computer to > power off. > --------------KzlIEyIMmbxm0W3nKMF3bwCF Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 18/08/2023 13:21, Graham Perrin wrote:
I'll update to ALPHA2.

Another failure.

In this case, removal of a USB device was unnecessary; the next start succeeded.

% bectl list -c creation | grep edacf4b4824a
n264868-edacf4b4824a-a NR     /          460G  2023-08-19 03:54
n264868-edacf4b4824a-b -      -          52.1M 2023-08-20 20:20
% uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400094 #4 main-n264868-edacf4b4824a-dirty: Fri Aug 18 23:46:09 BST 2023     grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.a
md64/sys/GENERIC amd64 1400094 1400094
%

In the meantime, briefly: <https://photos.app.goo.gl/jcXxDuxWPyXTa59Z8>

– no visible progress beyond the four lines of EFI framebuffer information.

In each case, I worked around by removing a USB flash drive that's used for L2ARC.

The most recent incident was preceded by a power button shutdown, in response to a drm-510-kmod blackout (GPU lockup etc.) whilst typing in Firefox (comparable to <https://github.com/freebsd/drm-kmod/issues/239> for drm-515-kmod 5.15.25).

The prior incident was also preceded by a power button shutdown (in response to a blackout at Plasma log out time). This was remarkable in that there was an unexpected restart, when I expected the computer to power off.

--------------KzlIEyIMmbxm0W3nKMF3bwCF-- From nobody Sun Aug 20 22:06:04 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTV783Dxcz4qnL1 for ; Sun, 20 Aug 2023 22:06:12 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTV761h5sz4Qq0 for ; Sun, 20 Aug 2023 22:06:10 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=J+pwIfNx; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::62f as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-99c3d3c3db9so357212766b.3 for ; Sun, 20 Aug 2023 15:06:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692569168; x=1693173968; h=subject:from:content-language:to:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=cIZlbVsTf7v+ADpeOf1rwLlJcFz1+4ebvWVbX+B74Ds=; b=J+pwIfNxsuPSfy9PNfHrrocD3CGNNQ9T94wcHVctJv7N51epZ2tpylnql8yKAWTApF c6lpyHsjwfp6hJ27UwDoHgI1x93xtlUrtulq1s8Y3cuu7yrI2mIB6BvD1pahQ5TICXGM newor9rYWkZQpYIcvxiC+zITo7UbjveVAs88CV8eng0B4yIgvDCaDqcKHF0H1jhZ5QVu X0gRVjpkJrZJ1aq8KWnTUeHw0Cggyjz70UA4JQXoHeSeCl8uMKpwemQWVw2IGvyXO+pf SnFZOOYOYiO0dLn8G2EN2/EnU2aKgSOOKOLXanKHa4UK3+uj5J4C9djMMKZJcesxq1fd Frhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692569168; x=1693173968; h=subject:from:content-language:to:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cIZlbVsTf7v+ADpeOf1rwLlJcFz1+4ebvWVbX+B74Ds=; b=jgxuem3rRKBULZj7bB6I+4YN/fgwTQq8+KR/7Owe9UKAzmH+srGQunzmjfEo6q+ksH 4cupEV1SCCjgXe9QYpflPdkfcK6/5Af8LQQpAoVXCjMvunvOKf6CtPj+fUT+2yW6Se7w C0oSw1Yfe95xCudzgjA4qvqCicrVsuQM/6LQXjVy+kqmfRrudyOgHCaNdVA6fpt3AIFp 57hWn69Wre6Kn9mawzrXRcUXwHLQ6+zH6RCNbUE1TpWnqwRBTzDtyxd6eZvekeRithT9 u0TyWkR1iNUFj/377wg36ucQxihNk9hTi0eahCATo9gBxkD8gvj7NRK7zE/c81qS/ZGA WLKg== X-Gm-Message-State: AOJu0YzPgT0ntoJN/9BBRB8njFDKyeMKEJA/cLoOSEk9BooPO07riXRR bqnCwJg6BSFV6P55Al67UA8DpNQA420= X-Google-Smtp-Source: AGHT+IGb0UR0Qc/CO4iFeSn4aza8L83xxM+i/PbeLSFzDAEnN7ZBVh2s/ff6qrHc7Fe6C3bOiTwjJg== X-Received: by 2002:a17:906:30d6:b0:99d:ddae:f5d4 with SMTP id b22-20020a17090630d600b0099dddaef5d4mr3603956ejb.44.1692569168046; Sun, 20 Aug 2023 15:06:08 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id l16-20020a170906415000b0098cf565d98asm5304545ejk.22.2023.08.20.15.06.07 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 20 Aug 2023 15:06:07 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------Vf76SGcP7m8MHVJbmEbQauKv" Message-ID: <7431da82-72dd-ca57-e294-48704adc5067@gmail.com> Date: Sun, 20 Aug 2023 23:06:04 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 To: FreeBSD CURRENT Content-Language: en-US From: Graham Perrin Subject: GPU firmware: could not load firmware image, error 2 X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; URI_COUNT_ODD(1.00)[3]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62f:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4RTV761h5sz4Qq0 This is a multi-part message in MIME format. --------------Vf76SGcP7m8MHVJbmEbQauKv Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Some relevant messages at. Should I be concerned by error 2 in these contexts? % pkg iinfo drm-510-kmod drm-510-kmod-5.10.163_7 % uname -aKU FreeBSD mowa219-gjp4-8570p-freebsd 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400094 #4 main-n264868-edacf4b4824a-dirty: Fri Aug 18 23:46:09 BST 2023grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1400094 1400094 --------------Vf76SGcP7m8MHVJbmEbQauKv Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit 
Some relevant messages at <https://pastebin.com/raw/wxWi8YS7>.

Should I be concerned by error 2 in these contexts?

% pkg iinfo drm-510-kmod
drm-510-kmod-5.10.163_7
% uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400094 #4 main-n264868-edacf4b4824a-dirty: Fri Aug 18 23:46:09 BST 2023     grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1400094 1400094

--------------Vf76SGcP7m8MHVJbmEbQauKv-- From nobody Sun Aug 20 22:17:45 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTVNZ6ZSJz4qnqB for ; Sun, 20 Aug 2023 22:17:50 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTVNY2cqxz4S97 for ; Sun, 20 Aug 2023 22:17:49 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=pUsug4a3; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::430 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-31c3726cc45so647456f8f.0 for ; Sun, 20 Aug 2023 15:17:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692569867; x=1693174667; h=content-transfer-encoding:in-reply-to:references:to:from :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=wDSY725LWWlaixgZ27svnwe5LaJ7Z3TegddjwdbdLiA=; b=pUsug4a3kxx66V8HBKcJZBeGULJTW992wYj6XK5FxsfShh+BAfyhk7tt5oW4GKFta/ O34+tyOKgF0T36dc/ua1JtO3WhzlUvfOQkYJDEls74MFBJ2cA3E4lRMhbi2GklQgM3eP HQBloxUZT12DPzkIz9764mAU2xVtcXLJ4GWSznjLW9BYY3cWdQTMtlZtIQ581kmhCWYn biLn2sHlXQfyPbzhiJZ4Dmep/DW66NOKBZS9dcnXMAmlem8Dd+yQouEvg/WM+Y1PEyKk tURxk/EJVfvOHT4PgzkxefsGRrOjBbvZp4SzqmrHLhznXjyj+5heBblliRLe49hTP5pt Vw4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692569867; x=1693174667; h=content-transfer-encoding:in-reply-to:references:to:from :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wDSY725LWWlaixgZ27svnwe5LaJ7Z3TegddjwdbdLiA=; b=W01aJyEiihMn5tn1jmvX4YKKdxc8tn+NYD+PFFHA7BmNU4QK/PjF4/7jt3ESowyDxY a5/ULnLFNFr53GLyMpptHHtvDJOux8e7RzQuiOLsVG3UvmBjy0iQbknnZWeAghGf5FYi ulVEmylfjeV3IDHLr6E67HcXTGcBHhSUIi2e6JenoyWmHSG1VHGhQF+IWX/Pm/wJOjsP jlkd2rjVPZK7I5CnvrpW86H0SVJQrgQ49BGtxBQRZATZmr/7SqJO9BlrX3Kfh3iCpfnW iyWCNRt2BQuL3ti5gr8G5qHtlYtXaSOlZx855vHR3pgXwRithkNLgWVUTg4//G/xgDzR lb+A== X-Gm-Message-State: AOJu0Yx6fB/WNuwv4CNhT3l5HtpeE+GMj6idvHc9wq7qNwVfnT1oY/t5 bVWCP4d3dLOpRCwkKtEGVaiVKZ++zj0= X-Google-Smtp-Source: AGHT+IE1dWlga8HtBNSxF8UkmNWYbHzORT+75H88EX8h8RGV4DqyM04e+WApkeEbv8gdJe8s0NjkoA== X-Received: by 2002:a5d:6949:0:b0:314:1e47:8bc2 with SMTP id r9-20020a5d6949000000b003141e478bc2mr3527889wrw.0.1692569867265; Sun, 20 Aug 2023 15:17:47 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id k19-20020aa7d8d3000000b005257f90c976sm4912656eds.3.2023.08.20.15.17.46 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 20 Aug 2023 15:17:46 -0700 (PDT) Message-ID: Date: Sun, 20 Aug 2023 23:17:45 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: GPU firmware: could not load firmware image, error 2 Content-Language: en-US From: Graham Perrin To: FreeBSD CURRENT References: <7431da82-72dd-ca57-e294-48704adc5067@gmail.com> In-Reply-To: <7431da82-72dd-ca57-e294-48704adc5067@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::430:from]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RTVNY2cqxz4S97 On 20/08/2023 23:06, Graham Perrin wrote: > Some relevant messages at. > > Should I be concerned by error 2 in these contexts? > > % pkg iinfo drm-510-kmod > drm-510-kmod-5.10.163_7 > % uname -aKU > FreeBSD mowa219-gjp4-8570p-freebsd 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400094 #4 main-n264868-edacf4b4824a-dirty: Fri Aug 18 23:46:09 BST 2023grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1400094 1400094 Off-list, someone wrote: > I believe you have to install drm-515-kmod It's not a requirement. I did try it (and encountered the errors) before reverting to drm-510-kmod. OT: These errors aside, drm-515-kmod is, for me, so unreliable that I can't use it; the bugs are in GitHub. From nobody Mon Aug 21 06:19:28 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTj4Z09RNz4qHJW for ; Mon, 21 Aug 2023 06:19:42 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTj4Y4Cpbz3KRZ for ; Mon, 21 Aug 2023 06:19:41 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id EE305482; Mon, 21 Aug 2023 08:19:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1692598772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l2CLNhO1fMwXtWngLhUeZQpgrbYDp2FSJ7OxNqnXa9c=; b=Hjo/Cp4mxgnTSCBPhNR4TiLE3exznGmL35GRYGD/p9wTcbjVIaxI9fIW/64STNpYyWRx6X DP5fqZ2DumaZUYyeL1gRXM2bYrg3IDKDoBDjZCJdn1ETx20t3Y2o8qqvIc66VKeibdrhLl zw6ORs7qM4XX2IoXBndQz3Pfb+JJaNKxgdi1NlAzcLE1+u7VRQbEl9W6DxKjmkQ6dHiEfF zguGGtBpC2gB5uoKR21xIXHtlbBDD8RZKa97Iyd5t3+RBNU8GLlEeqXXLUWSePLHHfauOy ef6gfvQpP7oXzPlBOGWa9TXufKcUkDgmp/P6xNpPqxe6dRiAOiHwQ+bwcEwHYg== List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Mon, 21 Aug 2023 08:19:28 +0200 From: Alexander Leidinger To: Konstantin Belousov Cc: Mateusz Guzik , current@freebsd.org Subject: Re: Speed improvements in ZFS In-Reply-To: References: <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> Message-ID: <58ac6211235c52d744666e8ae2ec7568@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RTj4Y4Cpbz3KRZ X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] Am 2023-08-20 23:17, schrieb Konstantin Belousov: > On Sun, Aug 20, 2023 at 11:07:08PM +0200, Mateusz Guzik wrote: >> On 8/20/23, Alexander Leidinger wrote: >> > Am 2023-08-20 22:02, schrieb Mateusz Guzik: >> >> On 8/20/23, Alexander Leidinger wrote: >> >>> Am 2023-08-20 19:10, schrieb Mateusz Guzik: >> >>>> On 8/18/23, Alexander Leidinger wrote: >> >>> >> >>>>> I have a 51MB text file, compressed to about 1MB. Are you interested >> >>>>> to >> >>>>> get it? >> >>>>> >> >>>> >> >>>> Your problem is not the vnode limit, but nullfs. >> >>>> >> >>>> https://people.freebsd.org/~mjg/netchild-periodic-find.svg >> >>> >> >>> 122 nullfs mounts on this system. And every jail I setup has several >> >>> null mounts. One basesystem mounted into every jail, and then shared >> >>> ports (packages/distfiles/ccache) across all of them. >> >>> >> >>>> First, some of the contention is notorious VI_LOCK in order to do >> >>>> anything. >> >>>> >> >>>> But more importantly the mind-boggling off-cpu time comes from >> >>>> exclusive locking which should not be there to begin with -- as in >> >>>> that xlock in stat should be a slock. >> >>>> >> >>>> Maybe I'm going to look into it later. >> >>> >> >>> That would be fantastic. >> >>> >> >> >> >> I did a quick test, things are shared locked as expected. >> >> >> >> However, I found the following: >> >> if ((xmp->nullm_flags & NULLM_CACHE) != 0) { >> >> mp->mnt_kern_flag |= >> >> lowerrootvp->v_mount->mnt_kern_flag & >> >> (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | >> >> MNTK_EXTENDED_SHARED); >> >> } >> >> >> >> are you using the "nocache" option? it has a side effect of xlocking >> > >> > I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache. >> > >> >> If you don't have "nocache" on null mounts, then I don't see how this >> could happen. > > There is also MNTK_NULL_NOCACHE on lower fs, which is currently set for > fuse and nfs at least. 11 of those 122 nullfs mounts are ZFS datasets which are also NFS exported. 6 of those nullfs mounts are also exported via Samba. The NFS exports shouldn't be needed anymore, I will remove them. Shouldn't this implicit nocache propagate to the mount of the upper fs to give the user feedback about the effective state? Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Mon Aug 21 06:38:09 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTjTz3v61z4qJsc for ; Mon, 21 Aug 2023 06:38:15 +0000 (UTC) (envelope-from kiri@truefc.org) Received: from kx.truefc.org (1.212.52.36.ap.yournet.ne.jp [36.52.212.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp", Issuer "smtp" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTjTx6BkNz3Q8F for ; Mon, 21 Aug 2023 06:38:13 +0000 (UTC) (envelope-from kiri@truefc.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of kiri@truefc.org designates 36.52.212.1 as permitted sender) smtp.mailfrom=kiri@truefc.org; dmarc=none Received: from kx.truefc.org (kx.truefc.org [36.52.212.1]) by kx.truefc.org (8.17.1/8.17.1) with ESMTP id 37L6c9PJ074335 for ; Mon, 21 Aug 2023 15:38:09 +0900 (JST) (envelope-from kiri@kx.truefc.org) Message-Id: <202308210638.37L6c9PJ074335@kx.truefc.org> Date: Mon, 21 Aug 2023 15:38:09 +0900 From: KIRIYAMA Kazuhiko To: current@freebsd.org Subject: Announce /etc/rc.d/sendmail does not work in default should be made at use time. User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.8 MULE XEmacs/21.4 (patch 24) (Standard C) (amd64--freebsd) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Spamd-Result: default: False [-2.82 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.72)[-0.723]; R_SPF_ALLOW(-0.20)[+ip4:36.52.212.0/29]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; ONCE_RECEIVED(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; FREEFALL_USER(0.00)[kiri]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[truefc.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:10013, ipnet:36.52.208.0/21, country:JP]; ARC_NA(0.00)[] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4RTjTx6BkNz3Q8F Hi, all I found /etc/rc.d/sendmail does not work in default. To make use of sendmail, I had to be done with `cp /usr/share/examples/sendmail/mailer.conf /etc/mail'. I saw why default MTA changed to dma: commit a67b925ff3e58b072a60b633e442ee1d33e47f7f Author: Baptiste Daroussin Date: Thu Oct 13 11:37:21 2022 +0200 mail: make The Dragonfly Mail Agent (dma) the default mta. dma accepts mail from a local Mail User Agent (MUA) and delivers it locally or to a smarthost for delivery. dma does not accept inbound mail (i.e., it does not listen on port 25) and is not intended to provide the same functionality as a full MTA like postfix or sendmail. It is intended for use cases such as delivering cron(8) mail. which is the default configuration and usage of sendmail in the default setup of the base system. In order to switch the default from sendmail to dma, we teach mailwrapper to fallback on dma directly if the mailer.conf file cannot be opened. We install by default a mailer.conf file which points at dma We install a mailer.conf file for sendmail in the examples. Relnotes: yes Differential Revision: https://reviews.freebsd.org/D37035 But there are nothing pointers to be changed default MTA changed to dma from sendmail ;-( Especially people who daily use sendmail with `service sendmail start', "Starting sendmail" message was shown but nothing done ;-) There should be announced that default MTA is dma when sendmail start ? --- Kazuhiko Kiriyama From nobody Mon Aug 21 08:53:48 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTmVY1rQsz4qTLZ for ; Mon, 21 Aug 2023 08:53:57 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTmVX5p8Wz4G2g for ; Mon, 21 Aug 2023 08:53:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 37L8rmLt045439 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 21 Aug 2023 11:53:51 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 37L8rmLt045439 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 37L8rmhc045438; Mon, 21 Aug 2023 11:53:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 21 Aug 2023 11:53:48 +0300 From: Konstantin Belousov To: Alexander Leidinger Cc: Mateusz Guzik , current@freebsd.org Subject: Re: Speed improvements in ZFS Message-ID: References: <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> <58ac6211235c52d744666e8ae2ec7568@Leidinger.net> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <58ac6211235c52d744666e8ae2ec7568@Leidinger.net> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4RTmVX5p8Wz4G2g X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] On Mon, Aug 21, 2023 at 08:19:28AM +0200, Alexander Leidinger wrote: > Am 2023-08-20 23:17, schrieb Konstantin Belousov: > > On Sun, Aug 20, 2023 at 11:07:08PM +0200, Mateusz Guzik wrote: > > > On 8/20/23, Alexander Leidinger wrote: > > > > Am 2023-08-20 22:02, schrieb Mateusz Guzik: > > > >> On 8/20/23, Alexander Leidinger wrote: > > > >>> Am 2023-08-20 19:10, schrieb Mateusz Guzik: > > > >>>> On 8/18/23, Alexander Leidinger wrote: > > > >>> > > > >>>>> I have a 51MB text file, compressed to about 1MB. Are you interested > > > >>>>> to > > > >>>>> get it? > > > >>>>> > > > >>>> > > > >>>> Your problem is not the vnode limit, but nullfs. > > > >>>> > > > >>>> https://people.freebsd.org/~mjg/netchild-periodic-find.svg > > > >>> > > > >>> 122 nullfs mounts on this system. And every jail I setup has several > > > >>> null mounts. One basesystem mounted into every jail, and then shared > > > >>> ports (packages/distfiles/ccache) across all of them. > > > >>> > > > >>>> First, some of the contention is notorious VI_LOCK in order to do > > > >>>> anything. > > > >>>> > > > >>>> But more importantly the mind-boggling off-cpu time comes from > > > >>>> exclusive locking which should not be there to begin with -- as in > > > >>>> that xlock in stat should be a slock. > > > >>>> > > > >>>> Maybe I'm going to look into it later. > > > >>> > > > >>> That would be fantastic. > > > >>> > > > >> > > > >> I did a quick test, things are shared locked as expected. > > > >> > > > >> However, I found the following: > > > >> if ((xmp->nullm_flags & NULLM_CACHE) != 0) { > > > >> mp->mnt_kern_flag |= > > > >> lowerrootvp->v_mount->mnt_kern_flag & > > > >> (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | > > > >> MNTK_EXTENDED_SHARED); > > > >> } > > > >> > > > >> are you using the "nocache" option? it has a side effect of xlocking > > > > > > > > I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache. > > > > > > > > > > If you don't have "nocache" on null mounts, then I don't see how this > > > could happen. > > > > There is also MNTK_NULL_NOCACHE on lower fs, which is currently set for > > fuse and nfs at least. > > 11 of those 122 nullfs mounts are ZFS datasets which are also NFS exported. > 6 of those nullfs mounts are also exported via Samba. The NFS exports > shouldn't be needed anymore, I will remove them. By nfs I meant nfs client, not nfs exports. > > Shouldn't this implicit nocache propagate to the mount of the upper fs to > give the user feedback about the effective state? > > Bye, > Alexander. > > -- > http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF > http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Mon Aug 21 11:25:22 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTqsK5glgz4qdhD for ; Mon, 21 Aug 2023 11:25:25 +0000 (UTC) (envelope-from cmt@burggraben.net) Received: from smtp.burggraben.net (smtp.burggraben.net [IPv6:2a01:4f8:140:510a::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.burggraben.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTqsK2Zy6z4Slf for ; Mon, 21 Aug 2023 11:25:25 +0000 (UTC) (envelope-from cmt@burggraben.net) Authentication-Results: mx1.freebsd.org; none Received: from elch.exwg.net (elch.exwg.net [IPv6:2001:470:7120:1:127b:44ff:fe4f:148d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "elch.exwg.net", Issuer "R3" (not verified)) by smtp.burggraben.net (Postfix) with ESMTPS id 51A5DC00313; Mon, 21 Aug 2023 13:25:22 +0200 (CEST) Received: by elch.exwg.net (Postfix, from userid 1000) id 06FF13AB04; Mon, 21 Aug 2023 13:25:22 +0200 (CEST) Date: Mon, 21 Aug 2023 13:25:22 +0200 From: Christoph Moench-Tegeder To: KIRIYAMA Kazuhiko Cc: current@freebsd.org Subject: Re: Announce /etc/rc.d/sendmail does not work in default should be made at use time. Message-ID: References: <202308210638.37L6c9PJ074335@kx.truefc.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <202308210638.37L6c9PJ074335@kx.truefc.org> User-Agent: Mutt/2.2.11 (2023-08-18) X-Rspamd-Queue-Id: 4RTqsK2Zy6z4Slf X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] ## KIRIYAMA Kazuhiko (kiri@truefc.org): > But there are nothing pointers to be changed default MTA > changed to dma from sendmail ;-( There is: UPDATING 20221205. Regards, Christoph -- Spare Space From nobody Mon Aug 21 15:41:49 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTxYJ4Zs2z4qwZB for ; Mon, 21 Aug 2023 15:41:56 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTxYJ0Gq8z4LhS for ; Mon, 21 Aug 2023 15:41:56 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692632516; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=1o15Nw1bl/obJ/4yE6Wm07vurejwi4IpV86VukTKjCg=; b=fDNTeaxlcpB3E6bOjBKecTHcWq7sG1sx/ZcAt0R6iyYWNUuqmKuVDJIc996XHqZ5vGRi3C OyYrga6iPIb5+Ch8ThbPGZ1GYotIVIsAmO7iQnF/c6QSxZcUweleCjT570JcfEOg285Rxa MsmK4+owfMGEYNVx64FVoOwZOo/DoV4DnKZEfxBaH15IFkoNhQ/zHfh4LIdZ1DgOKkXV2z pj6gZwwMogEYNnEIDpnnXJGaDX0rbEJLzUEyPNrl4DxAbEHDxT+yly9TlmuXzuWETwUsKy 9Gp0hdJMjaL0ZpksRP/lL8VYMOCvHB6tHF9i/TyClBeSAOYhSmoruVvaKVxCHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692632516; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=1o15Nw1bl/obJ/4yE6Wm07vurejwi4IpV86VukTKjCg=; b=XdNWhDM52tBP7Al/QMeJ4bYh5zcaJdAIYniDa08XNr7dBf749pWtGbyCe741Qrz4DoTKJA Kz3rbUXalF2xEahrpL8hLLQ5ata+JAAdUbVeg+prQs2eViF5wGY2LcnTNMsHErZzhxHGcA rvDHItcu6hxbYhTNNdzO9VtHcFPPZOKPc1TvpkRYEdzzABlpV0zhN6FTH1ski9MqdXd3bE hl/jX/rxdu+KmGMfjsOvNHHl6NSQjTKmXdKZV02REngbfHuGWNeT86g8mBQS1JGJmXS4BL 68HLUwWXt4StUaanmXVZOc0oOrWfJxaEWNhSUXEbb9EPf98QyUNGuGS2uW1Vfg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692632516; a=rsa-sha256; cv=none; b=Bcfka9o3flf1qZaqu+sh9vJtqQ+/ESdZSPoG8CV5IAwVA2tFj1QF0I1xHZrH89m0yrdTow gmP++7FO3hy+jEPb6tQ0bdUjaI6HtTzQtnGTvOM056ORnv69QxEVd2PcvSjg3qJshHO0WJ RjtSU73hsM5nU2jX5+1ZtQdAenJ8vD4/xIrWvrnY0XdfM0Wy3xmXCrYssocHbJ7AXm5R5p iOpNe/ukL6Vpg99aiVG/hE0MRYR1QhAiPu88p+eRd8iMGgyN5wMTOe0YrQEaj3DLMyIwUJ yCW+pRgzxfeeXAwtRgUF1nJhAPeWqKdZTww83taOF9XMeLePLvpTvd32HWx6oQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RTxYH235sz9Bx for ; Mon, 21 Aug 2023 15:41:54 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Question about KBI change / new feature Message-Id: Date: Mon, 21 Aug 2023 23:41:49 +0800 To: FreeBSD CURRENT X-Mailer: Apple Mail (2.3696.120.41.1.4) Hi, The https://www.freebsd.org/releases/14.0R/schedule/ says CURRENT/14 's = KBI is froze and new features should be avoided. I'm working on https://reviews.freebsd.org/D39638 (sysctl(9): Enable = vnet sysctl variables be loader tunable) and I think it is new feature, but not quite sure whether the KBI = changed. So, 1. Is it a KBI change ? 2. It is a simple change ( while so far as I know currently only tested = by myself on x86 and qemu riscv ), can it catch up with 14 ? Best regards, Zhenlei From nobody Mon Aug 21 17:44:28 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV0Gw2wgBz4r6Gd for ; Mon, 21 Aug 2023 17:44:40 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV0Gv1bLmz3Z8k for ; Mon, 21 Aug 2023 17:44:39 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nomadlogic.org header.s=04242021 header.b=tCP6UOrS; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 66.165.241.226 as permitted sender) smtp.mailfrom=pete@nomadlogic.org; dmarc=pass (policy=quarantine) header.from=nomadlogic.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1692639871; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=a/jMVh3SYwmvS6CUF35BD56mHXiPZKI2bTwU5Ga7s18=; b=tCP6UOrSxf5FpWRhRqeraFQXAfH9paRX1nW4fH7oWMtG1FP9IufjOXjriOuzz+pnjtan6T LkfydG/TIMpJxPRzeBP+hciLWFGo+wmd7OAC6+H6pm+jUOrNc2mgO/xHmIUNdCu/7UIcpj UgWHY8XZ3K0nXl4/6GAra4vrmHSpeiU= Received: from [192.168.1.160] (cpe-24-24-168-214.socal.res.rr.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 9188d3c4 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Mon, 21 Aug 2023 17:44:31 +0000 (UTC) Message-ID: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> Date: Mon, 21 Aug 2023 10:44:28 -0700 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Content-Language: en-US To: FreeBSD Current From: Pete Wright Subject: kabylake + drm-515-kmod/drm-510-kmod hangs Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[nomadlogic.org,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[nomadlogic.org:s=04242021]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[nomadlogic.org:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RV0Gv1bLmz3Z8k hey there, i've got a kabylake laptop that i've been using with drm-kmod for several years without much hassle. after upgrading to a new CURRENT this weekend I've found that when loading either the 510 or 515 drm-kmod kernel modules my system will hang. unfortunately i am not getting a panic or crash, the screen stops updating and i am unable to ping or SSH into the system. interestingly the capslock LED still toggles but doing a CTL+ALT+DEL does not seem to do anything useful and i have to manually power cycle. any tips for finding out what's going on? i've booted the system with verbose dmesg output, and loaded the module with "kldload -v" but do not get any useful output. here's the uname: FreeBSD colony 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #0 main-n264924-e2340276fc73: Sun Aug 20 21:28:44 PDT 2023 pete@colony:/usr/obj/usr/home/pete/git/freebsd/amd64.amd64/sys/GENERIC amd64 these are the log messages i see before the system locks up: Aug 21 10:40:34 colony kernel: iic0: on iicbus0 Aug 21 10:40:35 colony kernel: drmn0: on vgapci0 Aug 21 10:40:35 colony kernel: vgapci0: child drmn0 requested pci_enable_io Aug 21 10:40:35 colony syslogd: last message repeated 1 times Aug 21 10:40:35 colony kernel: [drm] Unable to create a private tmpfs mount, hugepage support will be disabled(-19). Aug 21 10:40:35 colony kernel: [drm] Got stolen memory base 0x4b800000, size 0x4000000 Aug 21 10:40:35 colony kernel: lkpi_iic0: on drmn0 Aug 21 10:40:35 colony kernel: iicbus1: on lkpi_iic0 Aug 21 10:40:35 colony kernel: iic1: on iicbus1 Aug 21 10:40:35 colony kernel: lkpi_iic1: on drmn0 Aug 21 10:40:35 colony kernel: iicbus2: on lkpi_iic1 Aug 21 10:40:35 colony kernel: iic2: on iicbus2 Aug 21 10:40:35 colony kernel: lkpi_iic2: on drmn0 Aug 21 10:40:35 colony kernel: iicbus3: on lkpi_iic2 Aug 21 10:40:35 colony kernel: iic3: on iicbus3 Aug 21 10:40:35 colony kernel: lkpi_iic3: on drmn0 Aug 21 10:40:35 colony kernel: iicbus4: on lkpi_iic3 Aug 21 10:40:35 colony kernel: iic4: on iicbus4 cheers, -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From nobody Mon Aug 21 17:49:43 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV0Nx3Gmfz4r6Tq for ; Mon, 21 Aug 2023 17:49:53 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV0Nx0rR6z3cCp for ; Mon, 21 Aug 2023 17:49:52 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Authentication-Results: mx1.freebsd.org; none Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 837A98928E; Mon, 21 Aug 2023 17:49:45 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.17.1/8.16.1) with ESMTPS id 37LHnjWc083313 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 21 Aug 2023 17:49:45 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.17.1/8.16.1/Submit) id 37LHnhKt083312; Mon, 21 Aug 2023 17:49:43 GMT (envelope-from phk) Message-Id: <202308211749.37LHnhKt083312@critter.freebsd.dk> To: Pete Wright cc: FreeBSD Current Subject: Re: kabylake + drm-515-kmod/drm-510-kmod hangs In-reply-to: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> From: "Poul-Henning Kamp" References: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <83310.1692640183.1@critter.freebsd.dk> Date: Mon, 21 Aug 2023 17:49:43 +0000 X-Rspamd-Queue-Id: 4RV0Nx0rR6z3cCp X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU] -------- Pete Wright writes: > i've got a kabylake laptop that i've been using with drm-kmod for > several years without much hassle. after upgrading to a new CURRENT > this weekend I've found that when loading either the 510 or 515 drm-kmod > kernel modules my system will hang. Does it make any difference if you load the module from single-user mode ? I've seen similar problems on my T14s if I try to load it from multi-user. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nobody Mon Aug 21 17:53:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV0T75ryvz4r6ld for ; Mon, 21 Aug 2023 17:53:31 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta001.cacentral1.a.cloudfilter.net (omta001.cacentral1.a.cloudfilter.net [3.97.99.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV0T75KRFz3f4W for ; Mon, 21 Aug 2023 17:53:31 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; none Received: from shw-obgw-4004a.ext.cloudfilter.net ([10.228.9.227]) by cmsmtp with ESMTP id Y87TqAuJyLAoIY95mqt2gv; Mon, 21 Aug 2023 17:53:30 +0000 Received: from spqr.komquats.com ([70.66.152.170]) by cmsmtp with ESMTPA id Y95lqds913fOSY95mqAbk5; Mon, 21 Aug 2023 17:53:30 +0000 X-Authority-Analysis: v=2.4 cv=J8G5USrS c=1 sm=1 tr=0 ts=64e3a49a a=y8EK/9tc/U6QY+pUhnbtgQ==:117 a=y8EK/9tc/U6QY+pUhnbtgQ==:17 a=kj9zAlcOel0A:10 a=UttIx32zK-AA:10 a=H2chl1sJAAAA:8 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=EkcXrb_YAAAA:8 a=qq-yAt-NlqehHX3f-JsA:9 a=CjuIK1q_8ugA:10 a=1LKL4F7LEIE-fDNXaEJ9:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id 30B9FE43; Mon, 21 Aug 2023 10:53:29 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id 2211337D; Mon, 21 Aug 2023 10:53:29 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Pete Wright cc: FreeBSD Current Subject: Re: kabylake + drm-515-kmod/drm-510-kmod hangs In-reply-to: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> References: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> Comments: In-reply-to Pete Wright message dated "Mon, 21 Aug 2023 10:44:28 -0700." List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 21 Aug 2023 10:53:29 -0700 Message-Id: <20230821175329.2211337D@slippy.cwsent.com> X-CMAE-Envelope: MS4xfLCjW8f6gbLARCv1h4fywJ3KEWOp/xKTO6TX/UShP/MfQBrx2c/CiIydvzm+lKRtzWsDl5MHXCa4rJ6PbXWnSTt8/NLtk5rpwZXDY8fEDeaa+dBsp5ho NRUgocgGNAxQ4DIatHq4Rg/82kIbxipUuyxNB7f8Maz82QcCR4vQSE03j9D6v1UQbqvE4m6ugFtxEbd6jmazgaWN2qRcxzrN4WSzQjC3UV6Tk9vSHChzlcd6 pdsvieTyPmMN/SjaR0gPHQ== X-Rspamd-Queue-Id: 4RV0T75KRFz3f4W X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US] In message <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org>, Pete Wright w rites: > hey there, > i've got a kabylake laptop that i've been using with drm-kmod for > several years without much hassle. after upgrading to a new CURRENT > this weekend I've found that when loading either the 510 or 515 drm-kmod > kernel modules my system will hang. > > unfortunately i am not getting a panic or crash, the screen stops > updating and i am unable to ping or SSH into the system. interestingly > the capslock LED still toggles but doing a CTL+ALT+DEL does not seem to > do anything useful and i have to manually power cycle. > > any tips for finding out what's going on? i've booted the system with > verbose dmesg output, and loaded the module with "kldload -v" but do not > get any useful output. > > here's the uname: > FreeBSD colony 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #0 > main-n264924-e2340276fc73: Sun Aug 20 21:28:44 PDT 2023 > pete@colony:/usr/obj/usr/home/pete/git/freebsd/amd64.amd64/sys/GENERIC amd64 > > > these are the log messages i see before the system locks up: > Aug 21 10:40:34 colony kernel: iic0: on iicbus0 > Aug 21 10:40:35 colony kernel: drmn0: on vgapci0 > Aug 21 10:40:35 colony kernel: vgapci0: child drmn0 requested pci_enable_io > Aug 21 10:40:35 colony syslogd: last message repeated 1 times > Aug 21 10:40:35 colony kernel: [drm] Unable to create a private tmpfs > mount, hugepage support will be disabled(-19). > Aug 21 10:40:35 colony kernel: [drm] Got stolen memory base 0x4b800000, > size 0x4000000 > Aug 21 10:40:35 colony kernel: lkpi_iic0: on drmn0 > Aug 21 10:40:35 colony kernel: iicbus1: on lkpi_iic0 > Aug 21 10:40:35 colony kernel: iic1: on iicbus1 > Aug 21 10:40:35 colony kernel: lkpi_iic1: on drmn0 > Aug 21 10:40:35 colony kernel: iicbus2: on lkpi_iic1 > Aug 21 10:40:35 colony kernel: iic2: on iicbus2 > Aug 21 10:40:35 colony kernel: lkpi_iic2: on drmn0 > Aug 21 10:40:35 colony kernel: iicbus3: on lkpi_iic2 > Aug 21 10:40:35 colony kernel: iic3: on iicbus3 > Aug 21 10:40:35 colony kernel: lkpi_iic3: on drmn0 > Aug 21 10:40:35 colony kernel: iicbus4: on lkpi_iic3 > Aug 21 10:40:35 colony kernel: iic4: on iicbus4 > > > > cheers, > -pete > > -- > Pete Wright > pete@nomadlogic.org > @nomadlogicLA > Rebuilding drm-51[05]-kmod after an update to LinuxKPI affecting the ABI used by the drm modules is required. Typically I get a kernel panic on a page fault when this occurs. Depending on how memory is laid out on your system you may get a hang instead. You need to install thew new kernel and world first. Disable xdm, gdm, any other *dm, or simply not use startx. From a text console session rebuild the drm port and reinstall it. I use poudriere here. My procedure is to update the poudriere jail, rebuild the port (-C option) and pkg upgrade -f or pkg install -f. Use this approach if you use poudriere. -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0 From nobody Mon Aug 21 17:56:09 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV0XD1dcwz4r6tN for ; Mon, 21 Aug 2023 17:56:12 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV0XC6dH2z4DYy for ; Mon, 21 Aug 2023 17:56:11 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1692640570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Wdo9UnLGx/DoM2Aj4pcqEokpx35FzdFmv0+L+Txw2I=; b=UJiKdIXOvFJXmn7GiomMJlq3REvcdCTgPU6zBEaMEc7F1XLytPUVVxXYRC5/PE/adgJfOO FPr06D6kVZTrFC1UoyFCvAuI5ET2b/URwcbrvCHRfA7hVMobytV/GkR3THH1byoYjdTxX7 77jxKeGuQQFU9iehIGUduH2BUSobdK0= Received: from [192.168.1.160] (cpe-24-24-168-214.socal.res.rr.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 04092928 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 21 Aug 2023 17:56:10 +0000 (UTC) Message-ID: Date: Mon, 21 Aug 2023 10:56:09 -0700 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: kabylake + drm-515-kmod/drm-510-kmod hangs Content-Language: en-US To: Cy Schubert Cc: FreeBSD Current References: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> <20230821175329.2211337D@slippy.cwsent.com> From: Pete Wright In-Reply-To: <20230821175329.2211337D@slippy.cwsent.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RV0XC6dH2z4DYy X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US] On 8/21/23 10:53, Cy Schubert wrote: > In message <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org>, Pete > Wright w > rites: >> hey there, >> i've got a kabylake laptop that i've been using with drm-kmod for >> several years without much hassle. after upgrading to a new CURRENT >> this weekend I've found that when loading either the 510 or 515 drm-kmod >> kernel modules my system will hang. >> >> unfortunately i am not getting a panic or crash, the screen stops >> updating and i am unable to ping or SSH into the system. interestingly >> the capslock LED still toggles but doing a CTL+ALT+DEL does not seem to >> do anything useful and i have to manually power cycle. >> >> any tips for finding out what's going on? i've booted the system with >> verbose dmesg output, and loaded the module with "kldload -v" but do not >> get any useful output. >> >> here's the uname: >> FreeBSD colony 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #0 >> main-n264924-e2340276fc73: Sun Aug 20 21:28:44 PDT 2023 >> pete@colony:/usr/obj/usr/home/pete/git/freebsd/amd64.amd64/sys/GENERIC amd64 >> >> >> these are the log messages i see before the system locks up: >> Aug 21 10:40:34 colony kernel: iic0: on iicbus0 >> Aug 21 10:40:35 colony kernel: drmn0: on vgapci0 >> Aug 21 10:40:35 colony kernel: vgapci0: child drmn0 requested pci_enable_io >> Aug 21 10:40:35 colony syslogd: last message repeated 1 times >> Aug 21 10:40:35 colony kernel: [drm] Unable to create a private tmpfs >> mount, hugepage support will be disabled(-19). >> Aug 21 10:40:35 colony kernel: [drm] Got stolen memory base 0x4b800000, >> size 0x4000000 >> Aug 21 10:40:35 colony kernel: lkpi_iic0: on drmn0 >> Aug 21 10:40:35 colony kernel: iicbus1: on lkpi_iic0 >> Aug 21 10:40:35 colony kernel: iic1: on iicbus1 >> Aug 21 10:40:35 colony kernel: lkpi_iic1: on drmn0 >> Aug 21 10:40:35 colony kernel: iicbus2: on lkpi_iic1 >> Aug 21 10:40:35 colony kernel: iic2: on iicbus2 >> Aug 21 10:40:35 colony kernel: lkpi_iic2: on drmn0 >> Aug 21 10:40:35 colony kernel: iicbus3: on lkpi_iic2 >> Aug 21 10:40:35 colony kernel: iic3: on iicbus3 >> Aug 21 10:40:35 colony kernel: lkpi_iic3: on drmn0 >> Aug 21 10:40:35 colony kernel: iicbus4: on lkpi_iic3 >> Aug 21 10:40:35 colony kernel: iic4: on iicbus4 >> >> >> >> cheers, >> -pete >> >> -- >> Pete Wright >> pete@nomadlogic.org >> @nomadlogicLA >> > > Rebuilding drm-51[05]-kmod after an update to LinuxKPI affecting the ABI > used by the drm modules is required. Typically I get a kernel panic on a > page fault when this occurs. Depending on how memory is laid out on your > system you may get a hang instead. > > You need to install thew new kernel and world first. Disable xdm, gdm, any > other *dm, or simply not use startx. From a text console session rebuild > the drm port and reinstall it. > > I use poudriere here. My procedure is to update the poudriere jail, rebuild > the port (-C option) and pkg upgrade -f or pkg install -f. Use this > approach if you use poudriere. > > Thanks Cy, yes my local scripts ensure to update the ports tree, then rebuild the drm-kmod module i'm using as a package. then i remove the old pkg, install the freshly build one then reboot. this ensures my kernel and drm modules match. this is how i've been doing it for years on all my systems since we started the work on the drm-kmod. -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From nobody Mon Aug 21 17:59:02 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV0bY0y59z4r6wv for ; Mon, 21 Aug 2023 17:59:05 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV0bX6ZqLz4FNL for ; Mon, 21 Aug 2023 17:59:04 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1692640743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t+PVAFKaz0WadIcL7vlLJi7AsacS5NM+IxrcpodczzE=; b=EV05ziFQBrPyqcM0i/Pkrq1FZUXaFOMKP6NLLHP2LmRueeBw1bOoLLr9JRQf4xY8PIYGqc STbevOeo2hDGu1PGYVaINwC30QRUB355+W5yPN53hnXFQaj4OQyfV1MidwkGs9u3T7omq6 WQxWgBPLpUdyTb3ojfPYbmLt8TQYXQI= Received: from [192.168.1.160] (cpe-24-24-168-214.socal.res.rr.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 75b3fa19 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 21 Aug 2023 17:59:03 +0000 (UTC) Message-ID: Date: Mon, 21 Aug 2023 10:59:02 -0700 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: kabylake + drm-515-kmod/drm-510-kmod hangs Content-Language: en-US To: Poul-Henning Kamp Cc: FreeBSD Current References: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> <202308211749.37LHnhKt083312@critter.freebsd.dk> From: Pete Wright In-Reply-To: <202308211749.37LHnhKt083312@critter.freebsd.dk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RV0bX6ZqLz4FNL X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US] On 8/21/23 10:49, Poul-Henning Kamp wrote: > -------- > Pete Wright writes: > >> i've got a kabylake laptop that i've been using with drm-kmod for >> several years without much hassle. after upgrading to a new CURRENT >> this weekend I've found that when loading either the 510 or 515 drm-kmod >> kernel modules my system will hang. > > Does it make any difference if you load the module from single-user mode ? > > I've seen similar problems on my T14s if I try to load it from multi-user. > oh that's an interesting idea. i just tried it and unfortunately got the same results. -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From nobody Mon Aug 21 18:24:59 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV19X1MJmz4r8WP for ; Mon, 21 Aug 2023 18:25:04 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV19W6lnpz4HtF for ; Mon, 21 Aug 2023 18:25:03 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x82c.google.com with SMTP id d75a77b69052e-40ff82320a7so24296541cf.3 for ; Mon, 21 Aug 2023 11:25:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692642303; x=1693247103; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=uMivFOJmPVLLzDd+B1nKVC8SmP+VGCPFQzDrNwNQWnk=; b=WFfGG0yFB2zu2g2ge1XFlo7kH6fcQL7hPfQ3hQ7OGyBOCZeuMW4HEPg+qVAygQ7JBM ZLz7yIQDQn2Il7yIA9FUHY/i7WbD3hed/JyDujlYWAFJoL0uRngNySYrRWZh88YY5Bif fRyCll3V8EQwZwXzzPrSIsbhNZzny2/XAgv0od1LUAAo8RBrp/XSBOPatt7BBk0GJh+i 7NPlxMgXwC5wIJS1n5obBBa4Kg4q/pX2Z5SJJnUaeuPAihjtmXWCjDDQjQKhHG4T/O6Q ZIBzrD6Jty3MURaUaiRRxXFmLe278zKYLo/X7mUgE+mVgPRtKFWBBYHoso302Xvk/CfR qPYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692642303; x=1693247103; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uMivFOJmPVLLzDd+B1nKVC8SmP+VGCPFQzDrNwNQWnk=; b=bAtuX2sJpj6kO+gFv+G0rUqNZFjjGoLHrej+fuTKql59Uls+zO+qetQzOCRe7AE7xc yVogemhC6Hc4VgmTWAF8tcpNlql2sxXN9rIgXA6ZLleS7YROIwkccQYxyOka1vr95nXg C3Y64Z2F7uuWwjvtEbV0gWKpU7i0NZgBmtCXe06VqaPrYDe4LUwxeWBSvmDZB/1pLFkA 9/tKhargRy2ZggGZoXyAsP1wf5bVtsuJgqa3GR8J+HuBDFeGWvl6QiKFSIqv4Y+pbUy2 gzAsLjO2Mkj4keW12kMxv7rzusL3J/WSH9c6KAzBPm/Xejoju9jLtrz4gMOKWicEtoMF +etQ== X-Gm-Message-State: AOJu0Yya6d7+0fdFnUzEeau3z0Io16oSt5TA8AxhM8Eh/CWxRvJHsgIf OKRD73I8vwgvn+IbmoH/HXa8j5ps8S8= X-Google-Smtp-Source: AGHT+IF8emZPmB2Kx86ivVDnFoeVPbPncMI8HAOZROFwDMw0HNiAWAFyIAKsDJPAyGkJgRAhX0IiOA== X-Received: by 2002:a05:622a:1a81:b0:403:affb:57ac with SMTP id s1-20020a05622a1a8100b00403affb57acmr9689665qtc.19.1692642303006; Mon, 21 Aug 2023 11:25:03 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id y10-20020ac83e8a000000b00403ff38d855sm2547983qtf.4.2023.08.21.11.25.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 11:25:02 -0700 (PDT) Date: Mon, 21 Aug 2023 14:24:59 -0400 From: Mark Johnston To: Pete Wright Cc: FreeBSD Current Subject: Re: kabylake + drm-515-kmod/drm-510-kmod hangs Message-ID: References: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> X-Rspamd-Queue-Id: 4RV19W6lnpz4HtF X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Mon, Aug 21, 2023 at 10:44:28AM -0700, Pete Wright wrote: > hey there, > i've got a kabylake laptop that i've been using with drm-kmod for several > years without much hassle. after upgrading to a new CURRENT this weekend > I've found that when loading either the 510 or 515 drm-kmod kernel modules > my system will hang. > > unfortunately i am not getting a panic or crash, the screen stops updating > and i am unable to ping or SSH into the system. interestingly the capslock > LED still toggles but doing a CTL+ALT+DEL does not seem to do anything > useful and i have to manually power cycle. Does your system have the debug.debugger_on_panic sysctl set to 1? If so, does setting it to 0 allow the system to reboot following the hang? > any tips for finding out what's going on? i've booted the system with > verbose dmesg output, and loaded the module with "kldload -v" but do not get > any useful output. Commit cedc82c0466a in src changed the layout of a structure used by a stub in the GPU firmware kernel modules. If you rebuild the one(s) you need from ports, does the problem persist? FWIW I had to do this: $ cd /usr/ports/graphics/gpu-firmware-intel-kmod $ sudo make reinstall FLAVOR=kabylake From nobody Mon Aug 21 18:53:13 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV1p45GJ7z4rBVQ for ; Mon, 21 Aug 2023 18:53:16 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV1p43D1Sz4LdL; Mon, 21 Aug 2023 18:53:16 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1692643993; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TZdcK5Ccam9OhbXp+BEdKxxCgcKt+5Pn49ebDTw1FnQ=; b=bsoLG/EXpDcv3oU18tAzzP9aVwQ+PLCd8+nAMveNacnT7tHNOi9mrjVNFd/1OInLhVIigf nJBLBil69HryiS+dFqMudCSgfzD7zHnUwvFAxwDjDl4N23lK7UGO2IiopGJv5GF6YncL+s n8j5aQ9qTQn+UXileOeoYPWhjsBJVVM= Received: from [192.168.1.160] (cpe-24-24-168-214.socal.res.rr.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 7f440c8a (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 21 Aug 2023 18:53:13 +0000 (UTC) Message-ID: <05e489c4-ed6b-46f1-8650-1a3fb4ea1223@nomadlogic.org> Date: Mon, 21 Aug 2023 11:53:13 -0700 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: kabylake + drm-515-kmod/drm-510-kmod hangs Content-Language: en-US To: Mark Johnston Cc: FreeBSD Current References: <76275772-a9c3-ed59-5fb3-47a13d2a69e6@nomadlogic.org> From: Pete Wright In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RV1p43D1Sz4LdL X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US] On 8/21/23 11:24, Mark Johnston wrote: > > Does your system have the debug.debugger_on_panic sysctl set to 1? If > so, does setting it to 0 allow the system to reboot following the hang? > oh fantastic, yea that works - thanks for the heads up! > > Commit cedc82c0466a in src changed the layout of a structure used by a > stub in the GPU firmware kernel modules. If you rebuild the one(s) you > need from ports, does the problem persist? > > FWIW I had to do this: > > $ cd /usr/ports/graphics/gpu-firmware-intel-kmod > $ sudo make reinstall FLAVOR=kabylake bingo that was it. i missed that change, thanks for pointing that out Mark. I'm able to load the i915kms module now. next step is figuring out why the nvidia-drm module is causing a panic (this is one of those funky intel + nvidia GPU laptops). but i get a core on that, and having the intel gpu load allows me to run X at least. thanks everyone! -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From nobody Mon Aug 21 23:03:31 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RV7M85mYQz4rPxl for ; Mon, 21 Aug 2023 23:03:48 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RV7M8188wz3Hlt for ; Mon, 21 Aug 2023 23:03:48 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id 0AD358C; Tue, 22 Aug 2023 01:03:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1692659015; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z2q1cptHbUqgPGL1qK0ToetEjDa9F+xJG7CnSxXjpgk=; b=e3kyAHmADvZIrhfqJ5T4gJvHkG0ENjqXtwxibIsOfptvZQAKCQZ1eCNpG6D7ETjDcXXIfx PalHulfPRLrf1hHRyVHrgzWPCe2Jgp8axJ+Ea5OLTxypgNfIQz9og6eFGmroVJlwrzX73s oUEeCKkqMbWBZVwKNTn7uZND5zNSCc/nyv2skUpG9NFext/S//xNOI7foAE9Et/cPt0YsE QOI+2JqCPMkWJ+0QSlXLIDlzax7Kmj7pbZ7y7Uc3HecmJCz+SshVGVnhcxI7CCNjDNmbiD BgIYg67FHRjCZfmulrBvO/GMBhgNBlld4tgvXrWkoYri7SKkOjNNCqhmac+hOA== List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Tue, 22 Aug 2023 01:03:31 +0200 From: Alexander Leidinger To: Konstantin Belousov Cc: Mateusz Guzik , current@freebsd.org Subject: Re: Speed improvements in ZFS In-Reply-To: References: <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> <58ac6211235c52d744666e8ae2ec7568@Leidinger.net> Message-ID: <444770b977b02b98985928bea450e4ce@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RV7M8188wz3Hlt X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] Am 2023-08-21 10:53, schrieb Konstantin Belousov: > On Mon, Aug 21, 2023 at 08:19:28AM +0200, Alexander Leidinger wrote: >> Am 2023-08-20 23:17, schrieb Konstantin Belousov: >> > On Sun, Aug 20, 2023 at 11:07:08PM +0200, Mateusz Guzik wrote: >> > > On 8/20/23, Alexander Leidinger wrote: >> > > > Am 2023-08-20 22:02, schrieb Mateusz Guzik: >> > > >> On 8/20/23, Alexander Leidinger wrote: >> > > >>> Am 2023-08-20 19:10, schrieb Mateusz Guzik: >> > > >>>> On 8/18/23, Alexander Leidinger wrote: >> > > >>> >> > > >>>>> I have a 51MB text file, compressed to about 1MB. Are you interested >> > > >>>>> to >> > > >>>>> get it? >> > > >>>>> >> > > >>>> >> > > >>>> Your problem is not the vnode limit, but nullfs. >> > > >>>> >> > > >>>> https://people.freebsd.org/~mjg/netchild-periodic-find.svg >> > > >>> >> > > >>> 122 nullfs mounts on this system. And every jail I setup has several >> > > >>> null mounts. One basesystem mounted into every jail, and then shared >> > > >>> ports (packages/distfiles/ccache) across all of them. >> > > >>> >> > > >>>> First, some of the contention is notorious VI_LOCK in order to do >> > > >>>> anything. >> > > >>>> >> > > >>>> But more importantly the mind-boggling off-cpu time comes from >> > > >>>> exclusive locking which should not be there to begin with -- as in >> > > >>>> that xlock in stat should be a slock. >> > > >>>> >> > > >>>> Maybe I'm going to look into it later. >> > > >>> >> > > >>> That would be fantastic. >> > > >>> >> > > >> >> > > >> I did a quick test, things are shared locked as expected. >> > > >> >> > > >> However, I found the following: >> > > >> if ((xmp->nullm_flags & NULLM_CACHE) != 0) { >> > > >> mp->mnt_kern_flag |= >> > > >> lowerrootvp->v_mount->mnt_kern_flag & >> > > >> (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | >> > > >> MNTK_EXTENDED_SHARED); >> > > >> } >> > > >> >> > > >> are you using the "nocache" option? it has a side effect of xlocking >> > > > >> > > > I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache. >> > > > >> > > >> > > If you don't have "nocache" on null mounts, then I don't see how this >> > > could happen. >> > >> > There is also MNTK_NULL_NOCACHE on lower fs, which is currently set for >> > fuse and nfs at least. >> >> 11 of those 122 nullfs mounts are ZFS datasets which are also NFS >> exported. >> 6 of those nullfs mounts are also exported via Samba. The NFS exports >> shouldn't be needed anymore, I will remove them. > By nfs I meant nfs client, not nfs exports. No NFS client mounts anywhere on this system. So where is this exclusive lock coming from then... This is a ZFS system. 2 pools: one for the root, one for anything I need space for. Both pools reside on the same disks. The root pool is a 3-way mirror, the "space-pool" is a 5-disk raidz2. All jails are on the space-pool. The jails are all basejail-style jails. Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Tue Aug 22 04:24:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVGTH6rmkz4qGyx for ; Tue, 22 Aug 2023 04:24:35 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com [IPv6:2a00:1450:4864:20::235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVGTH23t0z4Jbn for ; Tue, 22 Aug 2023 04:24:35 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=UkhKZJkT; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::235 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2bcb89b476bso32013531fa.1 for ; Mon, 21 Aug 2023 21:24:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692678273; x=1693283073; h=subject:from:content-language:to:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=sO3yNy7yBQouvEdAiwACS5fYc2YRx072SptcyBhrm+g=; b=UkhKZJkTLUQAeH0wA1ApPl11QZ+ri1l9mMyS73LqYNrUSyrheYodi6ekFk3sFsxa5Q jEQkyCs0HLEfLtY9bM3blVpgnQ+8onwcKZ/vV0PamI9/TBm5WWOrltQgMHDFmnZpgw+O agesG/RL1CTL7xvXZ7zuLpGWate4KPd8PXCvh/5r3gdlPgTagPlsjfQuUIjsH8T/Fsge dxw80FBbjfqF4zy5VAAzUS32LFOD8fRpJrA1/4kVta0V0bpwC32lI9kSeyKCIgyzSkzX ahwBsDb0mdbOqUyXJ6E8yjGNSjDHiMz34DkuACd34BzkpX+Ea8rHmCDPVg2r9OZQPdym vcJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692678273; x=1693283073; h=subject:from:content-language:to:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sO3yNy7yBQouvEdAiwACS5fYc2YRx072SptcyBhrm+g=; b=HpD7Ds/PHKiLNTqhub3/495JO2OC9pUcJjU70d/2be9NycRVdJbwQdRxtmlwMMaxOp fIoZhKBAlC2V0yS+KAfDMalamNuKC+gQ4ffSyUS6cBorP3Fuxk1I1puR4clEi6YCWsiA Mhga5fm188RHixk6lCkiTlkWmPd+032RDbQiv279VBllCGSrPC+8Zr42PjgIkXj7BLyD Bu5UZuucZ8eC+znqLkjDr3lUIcx2JkKTpWiRGFEQ7FFnVLjFjNkkZ273Osc+5yyw4QiV svhZkTE+ojARtQlZp5QPV/rQa1iHPNQcYd2eANaMxkq16Y2IFVNTap3fBmuERgKhsugB rXXw== X-Gm-Message-State: AOJu0YxYgHNVi8sIZSUk/rXBJPFGYuFrYlh1B1c+lwHQ1WkfXqH2PZXz 7jAe56q+AWdhUUpf6zpuHoDmCKuizzE= X-Google-Smtp-Source: AGHT+IFGHcB4qy6Fxp/6AKcHALrxfCiLKaMUowib6waWTiPZxeLC6emYvFGogkmdLoXkfs03R9vEHg== X-Received: by 2002:a2e:3615:0:b0:2b9:acad:b4b2 with SMTP id d21-20020a2e3615000000b002b9acadb4b2mr6230428lja.6.1692678272443; Mon, 21 Aug 2023 21:24:32 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id l18-20020a1709062a9200b00992ca779f42sm7529890eje.97.2023.08.21.21.24.31 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 21 Aug 2023 21:24:31 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------69wo9RbwHY9E8yi4oMVf0hwE" Message-ID: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> Date: Tue, 22 Aug 2023 05:24:29 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 To: FreeBSD CURRENT Content-Language: en-US From: Graham Perrin Subject: src.conf(5) to specify multiple flavours of a port X-Spamd-Result: default: False [-2.99 / 15.00]; URI_COUNT_ODD(1.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_HAM_LONG(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::235:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4RVGTH23t0z4Jbn This is a multi-part message in MIME format. --------------69wo9RbwHY9E8yi4oMVf0hwE Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit In a thread elsewhere, as an example that did /not/ involve src.conf, Mark Johnston wrote: > $ cd /usr/ports/graphics/gpu-firmware-intel-kmod > $ sudo make reinstall FLAVOR=kabylake How might I use /etc/src.conf to achieve much the same, with a different port? A recent edition of my file included these four lines, the third of which causes an error: PORTS_MODULES= graphics/drm-510-kmod # PORTS_MODULES= graphics/drm-515-kmod PORTS_MODULES+= graphics/gpu-firmware-radeon-kmod@btc graphics/gpu-firmware-radeon-kmod@sumo graphics/gpu-firmware-radeon-kmod@turks # PORTS_MODULES+= graphics/gpu-firmware-radeon-kmod This morning: I use the fourth line, instead. In future: I'd prefer to be without the build time that's associated with so many flavours, when only three are required. More than doubly time-consuming, because: KERNCONF=GENERIC GENERIC-NODEBUG # KERNCONF=GENERIC # KERNCONF=GENERIC-NODEBUG NO_INSTALLEXTRAKERNELS=no TIA --------------69wo9RbwHY9E8yi4oMVf0hwE Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit In a thread elsewhere, as an example that did not involve src.conf, Mark Johnston wrote: 

$ cd /usr/ports/graphics/gpu-firmware-intel-kmod
$ sudo make reinstall FLAVOR=kabylake

How might I use /etc/src.conf to achieve much the same, with a different port?

A recent edition of my file included these four lines, the third of which causes an error:

PORTS_MODULES= graphics/drm-510-kmod
# PORTS_MODULES= graphics/drm-515-kmod
PORTS_MODULES+= graphics/gpu-firmware-radeon-kmod@btc graphics/gpu-firmware-radeon-kmod@sumo graphics/gpu-firmware-radeon-kmod@turks
# PORTS_MODULES+= graphics/gpu-firmware-radeon-kmod

This morning: I use the fourth line, instead.

In future: I'd prefer to be without the build time that's associated with so many flavours, when only three are required.

<https://www.freshports.org/graphics/gpu-firmware-radeon-kmod/#flavors>


More than doubly time-consuming, because:

KERNCONF=GENERIC GENERIC-NODEBUG
# KERNCONF=GENERIC
# KERNCONF=GENERIC-NODEBUG

NO_INSTALLEXTRAKERNELS=no

TIA


--------------69wo9RbwHY9E8yi4oMVf0hwE-- From nobody Tue Aug 22 04:34:55 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVGjS3bqzz4qHpb for ; Tue, 22 Aug 2023 04:35:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVGjS1y18z4L5f for ; Tue, 22 Aug 2023 04:35:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-98377c5d53eso510538366b.0 for ; Mon, 21 Aug 2023 21:35:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1692678907; x=1693283707; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=TrB2C75+mebFwU9gZNg74lgmrsImrIi048iYruKSQoc=; b=mw8p5WY7F9eRnGuQYXLuhQnzmbMOW9hOp+dfHm0s2PzcfkUQTWFVvvEYCekFCbRISD Qpsk1AGER5LuWWfbxfHJhwY9x/ei2psINfJZ/PJieQZlwU39WWKfYnFnpyosVB8Kh8EM AFjTt/xiylb1rxXqqbG7DR/cQV/9mUKN9FwLW13K+ZIz2N68iGnXIDwCYjAPXzLz38hR UwxMskDZnWJzF3zjQr7I6nxbzbutX4dlZPMzGg60eeftoXRf1UiLxdh2fW19zg8oGX8Z waU2vXu1HauIKnuusAi6R/A5eZegxzV3lM28I9cWn3bWCU+OyQ5Om8omHVyE4lF+VvXn phVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692678907; x=1693283707; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TrB2C75+mebFwU9gZNg74lgmrsImrIi048iYruKSQoc=; b=AiLHqoXP1puU2yob9ZH/2eLnTt70y83eKi6+ikiI/hu4FqyfV+nwbHJmMLh0RfY/NL JwHUIgFwSACmNkcekDo+klNCaV5DICsLyxiHcFBxJdwkCCWTgnPQC/nESBpO7t4E5gHF ce3IFYjHNfAXIgdRFkNciZIKkLDl2nKYUoaNg7f/LrfAifnnr1tX5szYDRxw9vAORCAb ByI5eaHJrBYD27RF5bUM2NHP/JMCFK6PhRR1M6vPcSjJo6V68D5darEP72qQxNoN4N4g usqwV9wSjHLPK7HgZPjVrULcQewnvSLo4GPj+MXaMxQ4nosDwDaYy468N5+Up7inuUV6 82YQ== X-Gm-Message-State: AOJu0Yw3ia+vlW/9MUaqRdA7g8riwsya5m8B2KRLKBMhff3Ezaje6cz/ YZyzp8UveJEHGmdBp5nJbV49nQQjtzOXtMVF0pkevJmp4mWzng/R X-Google-Smtp-Source: AGHT+IFBE2bBSAL5f//UI6HAzqy4lrBca0hPZLTb4Vpa4vuCrYhgOFXuUb8qjglyGfzLwazoqEO5IKHnJRIb4ko1N6I= X-Received: by 2002:a17:906:5a6f:b0:9a1:872b:f693 with SMTP id my47-20020a1709065a6f00b009a1872bf693mr5242959ejc.63.1692678906596; Mon, 21 Aug 2023 21:35:06 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> In-Reply-To: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> From: Warner Losh Date: Mon, 21 Aug 2023 22:34:55 -0600 Message-ID: Subject: Re: src.conf(5) to specify multiple flavours of a port To: Graham Perrin Cc: FreeBSD CURRENT Content-Type: multipart/alternative; boundary="000000000000f2649306037b840b" X-Rspamd-Queue-Id: 4RVGjS1y18z4L5f X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] --000000000000f2649306037b840b Content-Type: text/plain; charset="UTF-8" On Mon, Aug 21, 2023, 10:24 PM Graham Perrin wrote: > In a thread elsewhere, as an example that did *not* involve src.conf, > Mark Johnston wrote: > > $ cd /usr/ports/graphics/gpu-firmware-intel-kmod > $ sudo make reinstall FLAVOR=kabylake > > > How might I use /etc/src.conf to achieve much the same, with a different > port? > I thought stuff like this went in ports.conf... Warner > A recent edition of my file included these four lines, the third of which > causes an error: > > PORTS_MODULES= graphics/drm-510-kmod > # PORTS_MODULES= graphics/drm-515-kmod > PORTS_MODULES+= graphics/gpu-firmware-radeon-kmod@btc > graphics/gpu-firmware-radeon-kmod@sumo > graphics/gpu-firmware-radeon-kmod@turks > # PORTS_MODULES+= graphics/gpu-firmware-radeon-kmod > > This morning: I use the fourth line, instead. > > In future: I'd prefer to be without the build time that's associated with > so many flavours, when only three are required. > > > > > > More than doubly time-consuming, because: > > KERNCONF=GENERIC GENERIC-NODEBUG > # KERNCONF=GENERIC > # KERNCONF=GENERIC-NODEBUG > > NO_INSTALLEXTRAKERNELS=no > > TIA > > > --000000000000f2649306037b840b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Mon, Aug 21, 2023, 10:24 PM Graham Perrin <grahamperrin@gmail.com> wrote:<= br>
=20 =20 =20
In a thread elsewhere, as an example that did not involve src.conf, Mark Johnston wrote: 

$ cd /usr/ports/graphics/gpu-firmware-intel-kmod
$ sudo make reinstall FLAVOR=3Dkabylake

How might I use /etc/src.conf to achieve much the same, with a different port?

<= /div>


I thought stuff like this went in ports.conf...

Warner=C2=A0

A recent edition of my file included these four lines, the third of which causes an error:

PORTS_MODULES=3D graphics/drm-510-kmod
# PORTS_MODULES=3D graphics/drm-515-kmod
PORTS_MODULES+=3D graphics/gpu-firmware-radeon-kmod@btc graphics/gpu-firmware-radeon-kmod@sumo graphics/gpu-firmware-radeon-kmod@turks
# PORTS_MODULES+=3D graphics/gpu-firmware-radeon-kmod

This morning: I use the fourth line, instead.

In future: I'd prefer to be without the build time that's associated with so many flavours, when only three are required.

<https://www.freshpor= ts.org/graphics/gpu-firmware-radeon-kmod/#flavors>


More than doubly time-consuming, because:

KERNCONF=3DGENERIC GENERIC-NODEBUG
# KERNCONF=3DGENERIC
# KERNCONF=3DGENERIC-NODEBUG

NO_INSTALLEXTRAKERNELS=3Dno

TIA


--000000000000f2649306037b840b-- From nobody Tue Aug 22 05:01:19 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVHHy0dpjz4qKp8 for ; Tue, 22 Aug 2023 05:01:34 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVHHx1bHPz4NcJ for ; Tue, 22 Aug 2023 05:01:32 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1bf078d5fb7so25894445ad.0 for ; Mon, 21 Aug 2023 22:01:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20221208.gappssmtp.com; s=20221208; t=1692680491; x=1693285291; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=PEonwMh3GvvZqmTUYYOZuuUtMzZtvETgYBFp+pAOkhU=; b=bN6bSLOiSYTwTHhWv1bLLcI4WfE8NPx8N0K9bVscrHs6SYn3u1S/y1kaw+Fw5o4Ahw fkNrNnt6fSAOYngEaySXiwK5CH2V/oarfwx0eX04jsLSUScStCUyeAtbmevFZdbjA+ff 1zHPdlgQJk0cMV6wfqT4fZVDRr5wvqbGh/4mvxzQUc9uvri4l/LuioRmyrLlu86i3HKF yGo74O5vrSHclKJYMJxZAsnk6JNGPzQOabL7uRmgtN+6yhaLNRkicFDapmo6ywThf54Z bWaLxSn1vrgGf5/UUQA7Ii4Cky9cWtd0gqhpyvjuSxDxVS1Cn0fBtpxbpjc2GcWyYfJH 4eow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692680491; x=1693285291; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PEonwMh3GvvZqmTUYYOZuuUtMzZtvETgYBFp+pAOkhU=; b=VCOdXtEBwuqWcAl3R5uc0682z5WcmnhYthr7B7w2UFAaZftg4zoq6o59TA3hifPZs1 6luZp3ADYB/wcOGDiFFjqC2NyfwkS+rbYVCG5SW8pfy6RMyuLgSyvmOWeVv9wSWPLe1J dyTYlYQU5+ZBeRENydEZ5KFiKXzTDc93wlKnPzGCEU2VYCxhf8+EFNoMe3AqcUIsSlrQ TievfqMORDWC69A1k4RPDI4/uj1EKX+w1qLS3F85oDXZQqRPw6UTwPfTGqYBGU3e2w5T 13BYzpjM+ZPttKmAYvrdkSVHLCAam74bRBXrxNFYTsduBDZqcxEcE2yDg/m6sKAOOaMv As2Q== X-Gm-Message-State: AOJu0YyOu6N1nn66/Bu4HdWTuAGkn0cM7nvG65rCNc7Aes8+isZO1pxe vM1ESnZ4FgaGePw5UmigQECbtg== X-Google-Smtp-Source: AGHT+IF3KoNQiNjmHvDGy8G1AavhkkB6FgJZjzuSiOh5eD2YBX2N+S0pCaA+xsYBXGnJaywOySeRtA== X-Received: by 2002:a17:903:1212:b0:1b0:3df7:5992 with SMTP id l18-20020a170903121200b001b03df75992mr8064210plh.32.1692680490876; Mon, 21 Aug 2023 22:01:30 -0700 (PDT) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id n9-20020a170902968900b001b03a1a3151sm7975290plp.70.2023.08.21.22.01.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Aug 2023 22:01:30 -0700 (PDT) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: src.conf(5) to specify multiple flavours of a port From: Bakul Shah In-Reply-To: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> Date: Mon, 21 Aug 2023 22:01:19 -0700 Cc: FreeBSD CURRENT Content-Transfer-Encoding: quoted-printable Message-Id: <813CD008-5C7D-4664-B605-3F61A2785213@iitbombay.org> References: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> To: Graham Perrin X-Mailer: Apple Mail (2.3731.600.7) X-Rspamd-Queue-Id: 4RVHHx1bHPz4NcJ X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Aug 21, 2023, at 9:24 PM, Graham Perrin = wrote: >=20 > In a thread elsewhere, as an example that did not involve src.conf, = Mark Johnston wrote:=20 >=20 >> $ cd /usr/ports/graphics/gpu-firmware-intel-kmod >> $ sudo make reinstall FLAVOR=3Dkabylake >=20 > How might I use /etc/src.conf to achieve much the same, with a = different port? Since /etc/src.conf is included from make, may be you can use some make = feature for conditional define?=20= From nobody Tue Aug 22 05:42:50 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVJCl43yZz4qN8m for ; Tue, 22 Aug 2023 05:42:59 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVJCj6dTRz4RNf for ; Tue, 22 Aug 2023 05:42:57 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=jOm4JG7g; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::229 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2b974031aeaso63474761fa.0 for ; Mon, 21 Aug 2023 22:42:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692682974; x=1693287774; h=in-reply-to:subject:from:references:to:content-language:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=M7XBR7sJU8fLaMaMAerlvve5OGBAqzs7vYk6VlGdvsk=; b=jOm4JG7gyXDxEov0/txL4/cZi3QI0EFHqWTi0FW9kq8KebTINMS3u+uYNihv1hGxEt 9WBp4+iBis4m1Nd0jiGzyS546w1fg++pFOCI1nPOpgA/ZkM+rhkBHfkAwb6amltfpLAK SP07bX8Lyc05WrwE+zwmErrDZbv+1S8fJnNB0YlEuO+wMr9AHFE3D5cP2208UhnU6ZsM G941PqlXRfP9+D+qYZ98ZibD3LyWyF9I1ZzWRCnv//sITvlqmpw4E7Ugq5T9wLSoXasg zHncloLWpeSuWvqATu/+3W1rUA0HR51Bbz/VQUugvXOxlA+aVreyLxyvdT521qWCRAof 1niA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692682974; x=1693287774; h=in-reply-to:subject:from:references:to:content-language:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=M7XBR7sJU8fLaMaMAerlvve5OGBAqzs7vYk6VlGdvsk=; b=eP2ug6Yb7G/j21rg895KP9cDXXblSKZ704WttGozdpIjTGAbKEycTvDLKK7x0wfQjU 5sMYHXfs6Q0at/wpIpXNS2ks6Sy7KTuIvZVcyIZFt3vim/oyADhvSbosNSBBiNuEf6o3 1XLgsxSPfhfjr+4ZoWh4EO72wDYv6WKjypxhxGfxdouzVgMTXtr9K0VSLQd+8IC5ADxz +EEV+vKdQkog3KyddeGx0cliXts8RpifW+xLUqSr9PTTfdsgmDrv/GCeJIVL8t2bKLlJ b9j8jB51LYxiJKUV4gwtG6TiPujYHRvy4Sxf7bdQXJz3c/LTBwnexeSiBNX22i6Gh500 zlXg== X-Gm-Message-State: AOJu0YzltRXp9CGSg7cA1hhpcxI+aaYLQexoDQB4KqhPBNrJuf6ZIpmS WArXcTMXAlMnM1Xh1iwhkM/0D89D6Yg= X-Google-Smtp-Source: AGHT+IFzQlhVZ0bet74OO1qIG+pNR7eds2yYhxx1vUQmAV7d1CzUzX86ACoqZQ/ZXk1Ub4pmxxC/tA== X-Received: by 2002:a2e:894b:0:b0:2ba:38c4:1624 with SMTP id b11-20020a2e894b000000b002ba38c41624mr6912455ljk.15.1692682973370; Mon, 21 Aug 2023 22:42:53 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id bs9-20020a170906d1c900b0099bcd1fa5b0sm7587560ejb.192.2023.08.21.22.42.51 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 21 Aug 2023 22:42:52 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------2bLO4bdgjOQdXxrBu3Fcq0Cj" Message-ID: Date: Tue, 22 Aug 2023 06:42:50 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Content-Language: en-US To: FreeBSD CURRENT References: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> From: Graham Perrin Subject: =?UTF-8?B?cG9ydHMtbWdtdC9wb3J0Y29uZiDigJMgbm90IHNyYy5jb25mKDUpIA==?= =?UTF-8?Q?=e2=80=93_to_specify_multiple_flavours_of_a_port_for_buildkernel_?= =?UTF-8?Q?purposes?= In-Reply-To: X-Spamd-Result: default: False [-2.99 / 15.00]; URI_COUNT_ODD(1.00)[7]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.989]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::229:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4RVJCj6dTRz4RNf This is a multi-part message in MIME format. --------------2bLO4bdgjOQdXxrBu3Fcq0Cj Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 22/08/2023 05:34, Warner Losh wrote: > > How might I use /etc/src.conf to achieve much the same, with a > different port? > > > > I thought stuff like this went in ports.conf... > > … Warner solves another mystery. Thanks! Honestly, I was oblivious to the possibility: % man -P cat 5 ports.conf No manual entry for ports.conf % apropos ports.conf apropos: nothing appropriate % rg -i -e 'ports\.conf' /usr/doc/website/content/en % rg -i -e 'ports\.conf' /usr/doc/documentation/content/en % – and so on. As far as I can tell, it's not documented in the usual places. Eventually, Google helped to remind me of a 2021 comment , where part of the previous person's comment had never sunk in. I wrote: > (I never used ports-mgmt/portconf, and so on.) So: % gh repo sync grahamperrin/freebsd-ports && git -C /usr/ports pull --ff-only --quiet && git -C /usr/ports pull --ff-only freebsd main ✓Synced the "grahamperrin:main" branch from "freebsd:main" Updating files: 100% (35/35), done. From https://git.freebsd.org/ports * branch                      main       -> FETCH_HEAD   15a5c70847f1..355374a1f6be  main       -> freebsd/main Already up to date. % sudo pkg install ports-mgmt/portconf grahamperrin's password: Updating FreeBSD repository catalogue... FreeBSD repository is up to date. Updating poudriere repository catalogue... Fetching meta.conf: 100%    163 B   0.2kB/s    00:01 Fetching packagesite.pkg: 100%    2 KiB   2.2kB/s    00:01 The provides database is up-to-date. Processing entries: 100% poudriere repository update completed. 7 packages processed. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): New packages to be INSTALLED:        portconf: 1.6_1 [FreeBSD] Number of packages to be installed: 1 2 KiB to be downloaded. Proceed with this action? [y/N]: y [1/1] Fetching portconf-1.6_1.pkg: 100%    2 KiB   2.3kB/s    00:01 Checking integrity... done (0 conflicting) [1/1] Installing portconf-1.6_1... [1/1] Extracting portconf-1.6_1: 100% Spamming /etc/make.conf... Done. ===== Message from portconf-1.6_1: -- To set port-specific make variables, create the /usr/local/etc/ports.conf configuration file with the following syntax: # this is a comment *: NOPORTDOCS editors/openoffice-3: WITH_CCACHE|LOCALIZED_LANG=it print/ghostscript-* print/lpr-wrapper: A4 sysutils/fusefs-kmod*: !KERNCONF | !NOPORTDOCS www/firefox-i18n: WITHOUT_SWITCHER | FIREFOX_I18N=fr it x11/fakeport: CONFIGURE_ARGS=--with-modules="aaa bbb ccc" Global port directory patterns and blanks around the pipe "|" symbol are allowed. Values shouldn't be quoted even if they contain spaces. Lines beginning with a '#' are comments. % apropos ports.conf apropos: nothing appropriate % rg --count -e 'ports\.conf' /usr/ports /usr/ports/net-mgmt/ocsinventory-ocsreports/files/pkg-message.in:1 /usr/ports/ports-mgmt/portconf/files/pkg-message.in:1 /usr/ports/ports-mgmt/portconf/files/portconf.sh.in:1 ^C % sudo nano /usr/local/etc/ports.conf grahamperrin's password: % cat /usr/local/etc/ports.conf graphics/gpu-firmware-radeon-kmod: FLAVORS=btc sumo turks % Still, there's guesswork. I have /no/ idea whether the FLAVORS part of that last line is valid :-) Time will tell. --------------2bLO4bdgjOQdXxrBu3Fcq0Cj Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 22/08/2023 05:34, Warner Losh wrote:

How might I use /etc/src.conf to achieve much the same, with a different port?



I thought stuff like this went in ports.conf...

…

Warner solves another mystery. Thanks!

Honestly, I was oblivious to the possibility:

% man -P cat 5 ports.conf
No manual entry for ports.conf
% apropos ports.conf
apropos: nothing appropriate
% rg -i -e 'ports\.conf' /usr/doc/website/content/en
% rg -i -e 'ports\.conf' /usr/doc/documentation/content/en
%

– and so on. As far as I can tell, it's not documented in the usual places.

Eventually, Google helped to remind me of a 2021 comment <https://github.com/freebsd/poudriere/issues/835#issuecomment-962728743>, where part of the previous person's comment had never sunk in. I wrote:

(I never used ports-mgmt/portconf, and so on.)


So:


% gh repo sync grahamperrin/freebsd-ports && git -C /usr/ports pull --ff-only --quiet && git -C /usr/ports pull --ff-only freebsd main
✓ Synced the "grahamperrin:main" branch from "freebsd:main"
Updating files: 100% (35/35), done.
From https://git.freebsd.org/ports
* branch                      main       -> FETCH_HEAD
  15a5c70847f1..355374a1f6be  main       -> freebsd/main
Already up to date.
% sudo pkg install ports-mgmt/portconf
grahamperrin's password:
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating poudriere repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01     
Fetching packagesite.pkg: 100%    2 KiB   2.2kB/s    00:01     
The provides database is up-to-date.
Processing entries: 100%
poudriere repository update completed. 7 packages processed.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
       portconf: 1.6_1 [FreeBSD]

Number of packages to be installed: 1

2 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching portconf-1.6_1.pkg: 100%    2 KiB   2.3kB/s    00:01     
Checking integrity... done (0 conflicting)
[1/1] Installing portconf-1.6_1...
[1/1] Extracting portconf-1.6_1: 100%
Spamming /etc/make.conf... Done.
=====
Message from portconf-1.6_1:

--
To set port-specific make variables, create the
/usr/local/etc/ports.conf configuration file
with the following syntax:


# this is a comment
*: NOPORTDOCS
editors/openoffice-3: WITH_CCACHE|LOCALIZED_LANG=it
print/ghostscript-* print/lpr-wrapper: A4
sysutils/fusefs-kmod*: !KERNCONF | !NOPORTDOCS
www/firefox-i18n: WITHOUT_SWITCHER | FIREFOX_I18N=fr it
x11/fakeport: CONFIGURE_ARGS=--with-modules="aaa bbb ccc"


Global port directory patterns and blanks around the
pipe "|" symbol are allowed.
Values shouldn't be quoted even if they contain spaces.
Lines beginning with a '#' are comments.
% apropos ports.conf
apropos: nothing appropriate
% rg --count -e 'ports\.conf' /usr/ports
/usr/ports/net-mgmt/ocsinventory-ocsreports/files/pkg-message.in:1
/usr/ports/ports-mgmt/portconf/files/pkg-message.in:1
/usr/ports/ports-mgmt/portconf/files/portconf.sh.in:1
^C
% sudo nano /usr/local/etc/ports.conf
grahamperrin's password:
% cat /usr/local/etc/ports.conf
graphics/gpu-firmware-radeon-kmod: FLAVORS=btc sumo turks
%


Still, there's guesswork. I have no idea whether the FLAVORS part of that last line is valid :-)

Time will tell.

--------------2bLO4bdgjOQdXxrBu3Fcq0Cj-- From nobody Tue Aug 22 07:39:54 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVLpl2hVvz4qWCW for ; Tue, 22 Aug 2023 07:39:59 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVLpk1DXDz4kCW for ; Tue, 22 Aug 2023 07:39:58 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=lo0ktkVX; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::633 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-99c47ef365cso563955466b.0 for ; Tue, 22 Aug 2023 00:39:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692689996; x=1693294796; h=in-reply-to:subject:references:to:from:content-language:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=+30qXKjvHnSGu80yXX8QyJCVQUDNEPccxoBxBDn6OTo=; b=lo0ktkVXg5qK4NenUPC8bUngV2M5/lDpcWDG1BLFZIBvgA1EsvDw8DErYT0kH6XZZO K9ZxwKWvOS0n2VlxcB1HXZeOh/hT6Yze027hZw/yCGRln+zN3VDOM0+pYM464M91+vpe oC7OAWulbtvAJDp7hvtgORfWnA3b7O8huiquaCvG4qGedJI4Veh4mVFktAGCfPRNW8Zh Fe2abEXUMy89h35Koko5EuzEkc6XovYaPZmq5T6GLIuwi9OOR0pfDwQo/+ZNSkIypWU3 4nf4T3JPOWrrpTsienFw+jXQqipyR7RyRFunFbSchAqYerb7UbXM0OWuSiEG2yCQcb2z xCiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692689996; x=1693294796; h=in-reply-to:subject:references:to:from:content-language:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=+30qXKjvHnSGu80yXX8QyJCVQUDNEPccxoBxBDn6OTo=; b=TFunbHwtxSPHSD5vWU0ABHARfqIH7keNEprtE3O9Fgat8PISZIuYUUzdgQKfZjCqEL r04mASzwfOsnFCGTmzNwMUJWxi0yo+VGVXuY8zO0ZecMeefYz5NW+/Ce42es0KGvR7hZ QMEXbt1sDpxBhGIMj2C5MqlLvTtOgjxrOcbQYJPVdsqqt16Y92CNApY9k2CvW/m4uhNQ bQWWkCpmbQ5qIgjOSoapfgLNRfLn4I9EiCd18g0XLFUU7F1cS3xmUS0R8Tstz9ZvtVl9 VLkvk1aL85rBBy4LvFfFxUKs5WBxpWAvMlPZ+rmjZy2TBy26JShPTknw5w+xWGHDhfcC Gifw== X-Gm-Message-State: AOJu0YysHNe4mzIPPkHHm31aVy3fGXmsZNKZDMZLs1ur8GUW3TToCvnO nfDa8cD0OV5rKjIK/4rEs497eZZT3us= X-Google-Smtp-Source: AGHT+IE1RqIKtdzznkgSIkMoDhKI3hHa2ByuuW23yl5EamDLm25860X7GeT9QEqoc4E3Q60Xp31zCw== X-Received: by 2002:a17:906:1019:b0:9a1:be5b:f4a5 with SMTP id 25-20020a170906101900b009a1be5bf4a5mr474382ejm.35.1692689996391; Tue, 22 Aug 2023 00:39:56 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id mb12-20020a170906eb0c00b0099bd5b72d93sm7799587ejb.43.2023.08.22.00.39.55 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Aug 2023 00:39:55 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------Qrz98UOFXSTgtv3Hz4K3I0MC" Message-ID: Date: Tue, 22 Aug 2023 08:39:54 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Content-Language: en-US From: Graham Perrin To: FreeBSD CURRENT References: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> Subject: =?UTF-8?Q?Re=3a_ports-mgmt/portconf_=e2=80=93_not_src=2econf=285=29?= =?UTF-8?Q?_=e2=80=93_to_specify_multiple_flavours_of_a_port_for_buildkernel?= =?UTF-8?Q?_purposes?= In-Reply-To: X-Spamd-Result: default: False [-3.00 / 15.00]; URI_COUNT_ODD(1.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::633:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4RVLpk1DXDz4kCW This is a multi-part message in MIME format. --------------Qrz98UOFXSTgtv3Hz4K3I0MC Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 22/08/2023 06:42, Graham Perrin wrote: > > … > % cat /usr/local/etc/ports.conf > graphics/gpu-firmware-radeon-kmod: FLAVORS=btc sumo turks > % > > > Still, there's guesswork. I have /no/ idea whether the FLAVORS part of > that last line is valid :-) > > Time will tell. > /me tails /var/log/buildkernel.log, rocks with laughter at innumerable warnings, cancels, … make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5393: warning: duplicate script for target "sumo" ignored make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using previous script for "sumo" defined here make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5402: warning: duplicate script for target "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/graphics/gpu-firmware-radeon-kmod/work-"btc" ignored make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using previous script for "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/graphics/gpu-firmware-radeon-kmod/work-"btc" defined here make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5402: warning: duplicate script for target "sumo" ignored make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using previous script for "sumo" defined here env: sumo: No such file or directory ^C % , begins a new build that excludes GPU firmware whilst ignoring the relevance of , intends to build the modules with poudriere-devel after the OS is updated, rebelliously  begins a paragraph with a comma and makes a whitespace error that should be barely visible in the HTML version of this email that will not be seen in an archive :-) --------------Qrz98UOFXSTgtv3Hz4K3I0MC Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit


On 22/08/2023 06:42, Graham Perrin wrote:

…
% cat /usr/local/etc/ports.conf
graphics/gpu-firmware-radeon-kmod: FLAVORS=btc sumo turks
%


Still, there's guesswork. I have no idea whether the FLAVORS part of that last line is valid :-)

Time will tell.

/me tails /var/log/buildkernel.log, rocks with laughter at innumerable warnings, cancels,

…
make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5393: warning: duplicate script for target "sumo" ignored
make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using previous script for "sumo" defined here
make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5402: warning: duplicate script for target "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/graphics/gpu-firmware-radeon-kmod/work-"btc" ignored
make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using previous script for "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/usr/ports/graphics/gpu-firmware-radeon-kmod/work-"btc" defined here
make[4028]: "/usr/ports/Mk/bsd.port.mk" line 5402: warning: duplicate script for target "sumo" ignored
make[4028]: "/usr/ports/Mk/Uses/kmod.mk" line 74: warning: using previous script for "sumo" defined here
env: sumo: No such file or directory
^C
%


, begins a new build that excludes GPU firmware whilst ignoring the relevance of <https://github.com/freebsd/freebsd-src/commit/cedc82c0466a>, intends to build the modules with poudriere-devel after the OS is updated, rebelliously  begins a paragraph with a comma and makes a whitespace error that should be barely visible in the HTML version of this email that will not be seen in an archive :-)

--------------Qrz98UOFXSTgtv3Hz4K3I0MC-- From nobody Tue Aug 22 08:26:22 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVMrS0C5cz4qZ2s for ; Tue, 22 Aug 2023 08:26:32 +0000 (UTC) (envelope-from mm@freebsd.org) Received: from www541.your-server.de (www541.your-server.de [213.133.107.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVMrQ5wbhz4pxj; Tue, 22 Aug 2023 08:26:30 +0000 (UTC) (envelope-from mm@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 213.133.107.7 is neither permitted nor denied by domain of mm@freebsd.org) smtp.mailfrom=mm@freebsd.org; dmarc=none Received: from sslproxy03.your-server.de ([88.198.220.132]) by www541.your-server.de with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qYMiU-0002Qt-HZ; Tue, 22 Aug 2023 10:26:23 +0200 Received: from [192.168.0.32] (helo=webmail.your-server.de) by sslproxy03.your-server.de with esmtpsa (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256) (Exim 4.92) (envelope-from ) id 1qYMiU-000BRP-Px; Tue, 22 Aug 2023 10:26:22 +0200 Received: from [145.243.168.67] ([145.243.168.67]) by webmail.your-server.de (Horde Framework) with HTTPS; Tue, 22 Aug 2023 10:26:22 +0200 Date: Tue, 22 Aug 2023 10:26:22 +0200 Message-ID: <20230822102622.Horde.tUEwNRTMDn0AjOBiO1UB7OX@webmail.your-server.de> From: Martin Matuska To: Alexander Motin Cc: Dag-Erling =?utf-8?b?U23DuHJncmF2?= , current@freebsd.org, Mateusz Guzik Subject: Re: ZFS deadlock in 14 References: <86leeltqcb.fsf@ltc.des.no> <86h6p4s64h.fsf@ltc.des.no> <86a5utrafp.fsf@ltc.des.no> <86350kqokl.fsf@ltc.des.no> <86y1icp95t.fsf@ltc.des.no> <86ttt0p8wv.fsf@ltc.des.no> <197ead1e-210a-6be6-7e24-5c56b14bb777@FreeBSD.org> <86350hpiby.fsf@ltc.des.no> <0fd64567-59cd-9276-cc0c-17a09611f11a@FreeBSD.org> In-Reply-To: <0fd64567-59cd-9276-cc0c-17a09611f11a@FreeBSD.org> User-Agent: Horde Application Framework 5 Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Authenticated-Sender: martin@matuska.de X-Virus-Scanned: Clear (ClamAV 0.103.8/27007/Mon Aug 21 09:40:10 2023) X-Spamd-Result: default: False [-3.09 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_THREE(0.00)[4]; FREEFALL_USER(0.00)[mm]; MLMMJ_DEST(0.00)[current@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; HAS_X_AS(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_SOME(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:213.133.96.0/19, country:DE]; RCVD_TLS_ALL(0.00)[] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RVMrQ5wbhz4pxj Hi Alexander, as 15107 is a prerequisite for 15122, would it be possible to have https://github.com/openzfs/zfs/pull/15107 merged into the OpenZFS zfs-2.2-release branch (and of course later 15122)? If the patches help I can cherry-pick them into main. Cheers, mm Alexander Motin wrote: > On 17.08.2023 15:41, Dag-Erling Smørgrav wrote: >> Alexander Motin writes: >>> Trying to run your test (so far without reproduction) I see it >>> producing a substantial amount of ZIL writes. The range of commits >>> you reduced the scope to so far includes my ZIL locking refactoring, >>> where I know for sure are some deadlocks. I am already waiting for 3 >>> weeks now for reviews and tests for PR that should fix it: >>> https://github.com/openzfs/zfs/pull/15122 . It would be good if you >>> could test it, though it seems to depend on few more earlier patches >>> not merged to FreeBSD yet. >> >> Do you have a FreeBSD branch with your patch applied? > > I don't have a FreeBSD branch, but these two patches apply clean and > build on top of today's FreeBSD main branch: > > https://github.com/openzfs/zfs/pull/15107 > https://github.com/openzfs/zfs/pull/15122 > > And if you still experience the issue, please show all stacks, or at > least include ZFS sync threads. > > -- > Alexander Motin From nobody Tue Aug 22 12:08:31 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVSmf0vxPz4qpKX for ; Tue, 22 Aug 2023 12:08:34 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVSmf0Qykz3SVV; Tue, 22 Aug 2023 12:08:34 +0000 (UTC) (envelope-from ronald@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692706114; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6Pt/JDLYl0viZXWEqAaLHBM9fUjnf6C0ZCg9UCAqIE=; b=cEx+Tt322c0s2dWmJofwmum9+dvmvXRuFTiRprMeNpd4aQiVNzRRFjcrJGEHvRw1J5ywdG PTcsmPIsWNO8+erjlFcISgiDUbPRwOuyGgph1hLNNPc6HF2zLxck+aIbLc1Jkxw+uoNs1M XZ6Mu8TK8iRBNe/cvIYhyFpcoaaGtOK++Y5cL8s7gmWgOrH/499wj/Eda5dRTq45qJueJR 0335Ppj83UHsH7QAcr3PjG5S1oiAlkhVrC6xdF4lJNqwveMf6wBtKW2LNIf+nNeFh0umH+ AYR9V29PZRdZ7HGeT+6MUlC4GYdN5Jc6Og7TbWAqhI9n6cGYdI9BvO/gLz3iyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692706114; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6Pt/JDLYl0viZXWEqAaLHBM9fUjnf6C0ZCg9UCAqIE=; b=RLT23SNrYMAnoUunSbpr6lTkFawgfAhNXgCLA++mz5hsUxTJLicFTWb9t3VjC42yE+t/yW IhFlfXQhGh00aUGpCaOd/b1Q7bEOf009jWlpyYfCG1YIQAwE5bBzfsSatcHtN6bosVS15c 8YfwjIG1DB7lM2lhetPH79waFL+blTYZlLZ3wMyeC3Fdac9/yBj1an9YmL2iODARoZPyL6 KG5kjqbWwcymd3xG9m9+bmvRlAwZDUoB2e8tV7TxB47Rip06cEYhuu3kBP8CRwTb2RnIm3 06MY5opX03L0vvyUH9dmHs34e+I9uvcHpppbQHtdQ7G4cI/G6aGwroVzC7HScg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692706114; a=rsa-sha256; cv=none; b=hEaITy/4/TlbSPyYsdHefNwheCSpPyPLPdiUOXPr7K0B5OqSOEn8zYu6qctTkDHje/t5cD o70eINbVtmTCHHvduJ8hN0p8z/rSAfradWrKi/TLCS7y21UPM3XUySLVrXbos8Ib6omStH 4sz745WQXQoy0U6bYmSgIA/VQ0aLzvv7Sc+ejnuWB3fhq2nNFY+EWf440Ld5uZbZm1WjRT L3DsABYC+6GORTXuWhhoXIrXZMW1VvCazDyGAF3tblPglGy3T+Av88nnuZzQrZMCV8amKk Osk9dMpHIK/rsFfSTaGkp+6Lc1go9oEws97jEH6dTmTS8yFrx1CXfZwJQVlxZA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [IPV6:2001:1c00:2709:2010:749d:807f:4b36:6fb] (2001-1c00-2709-2010-749d-807f-4b36-06fb.cable.dynamic.v6.ziggo.nl [IPv6:2001:1c00:2709:2010:749d:807f:4b36:6fb]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: ronald/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RVSmd4Ycpz121G; Tue, 22 Aug 2023 12:08:33 +0000 (UTC) (envelope-from ronald@FreeBSD.org) Message-ID: Date: Tue, 22 Aug 2023 14:08:31 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: building with llvm16 pkg fails in tests From: Ronald Klop To: Brooks Davis Cc: freebsd-current@freebsd.org References: <76ecade1-3f42-a314-825d-6f1ba0b2dcaa@FreeBSD.org> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 8/19/23 22:28, Ronald Klop wrote: > On 8/17/23 21:33, Brooks Davis wrote: >> On Thu, Aug 17, 2023 at 12:45:06PM +0200, Ronald Klop wrote: >>> Hi, >>> >>> To save time on my Raspberry Pi I would like to build FreeBSD using a llvm pkg instead of llvm in the tree. >>> >>> My /etc/make.conf: >>> WITHOUT_TOOLCHAIN=yes >>> LD=/usr/local/llvm16/bin/ld.lld >>> CC=/usr/local/llvm16/bin/clang >>> CXX=/usr/local/llvm16/bin/clang++ >>> CPP=/usr/local/llvm16/bin/clang-cpp >>> OBJCOPY=/usr/local/llvm16/bin/llvm-objcopy >>> >>> #WITHOUT_CLEAN=yes >>> >>> >>> This fails in: >>> >>> /usr/local/llvm16/bin/clang++ -O2 -pipe -fno-common -fPIE -Wno-format-zero-length -nobuiltininc -idirafter /usr/local/llvm16/lib/clang/16/include -fstack-protector-strong -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Wdate-time -Wmissing-variable-declarations -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-error=unused-but-set-parameter -O0 -g0 -Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/usr/include/private -I/usr/src/contrib/googletest/googlemock/include -I/usr/src/contrib/googletest/googlemock -I/usr/src/contrib/googletest/googletest/include -I/usr/src/contrib/googletest/googletest -I/usr/obj/usr/src/amd64.amd64/tmp/usr/include/private -DGTEST_HAS_POSIX_RE=1 -DGTEST_HAS_PTHREAD=1 -DGTEST_HAS_STREAM_REDIRECTION=1 -frtti -Wno-deprecated-copy -Wno-signed-unsigned-wchar -DGTEST_HAS_POSIX_RE=1 >>> -DGTEST_HAS_PTHREAD=1 -DGTEST_HAS_STREAM_REDIRECTION=1 -frtti -Wno-deprecated-copy -Wno-signed-unsigned-wchar -fPIE -std=c++14 -Wno-deprecated-copy -Wno-error=inconsistent-missing-override -Wno-error=missing-variable-declarations -Wno-error=sign-compare -Wno-error=unused-parameter -Wno-c++11-extensions  -Wl,-zrelro -pie  --ld-path=/usr/local/llvm16/bin/ld.lld -o gmock-actions_test  gmock-actions_test.o -lprivategmock_main -lprivategmock -lprivategtest >>> ld.lld: error: undefined symbol: testing::internal::DeathTest::Create(char const*, testing::Matcher, std::__1::allocator> const&>, char const*, int, testing::internal::DeathTest**) >>>>>> referenced by gmock-actions_test.cc >>>>>>                gmock-actions_test.o:(testing::(anonymous namespace)::BuiltInDefaultValueDeathTest_IsUndefinedForReferences_Test::TestBody()) >>>>>> referenced by gmock-actions_test.cc >>>>>>                gmock-actions_test.o:(testing::(anonymous namespace)::BuiltInDefaultValueDeathTest_IsUndefinedForReferences_Test::TestBody()) >>>>>> referenced by gmock-actions_test.cc >>>>>>                gmock-actions_test.o:(testing::(anonymous namespace)::BuiltInDefaultValueDeathTest_IsUndefinedForNonDefaultConstructibleType_Test::TestBody()) >>>>>> referenced 4 more times >>> >>> ld.lld: error: undefined symbol: testing::Expectation::Expectation(std::__1::shared_ptr const&) >>> >>> >>> Any thoughts on how to fix this? >>> Compiling with the in tree llvm does work properly. >> >> Did it work with exactly this git revision?  I suspect an issue with the >> recent google test update rather than an llvm16 issue.  Note that for >> every sync to github we build the tree with the llvm16 port (all be it >> on amd64 by default). >> >> It's worth noting one difference between your configuration and the >> CI one:  We don't set CC and friends directly.  Instead we use >> CROSS_TOOLCHAIN=llvm16. >> >> -- Brooks > > > Hi, > > What I would like to accomplish is this: > > CROSS_TOOLCHAIN=llvm16 > WITHOUT_TOOLCHAIN=yes > > yes | make delete-old delete-old-libs > make buildworld buildkernel > > So I can run a system with only external toolchain. > But doing this quickly errors out because /usr/bin/cc does not exist and the build uses that even though CROSS_TOOLCHAIN is set. To circumvent that I set CC, etc. instead of CROSS_TOOLCHAIN. > > Regards, > Ronald. > Hi, I found what was going on. Passing CC=/usr/local/llvm16/bin/clang does not pass the -target, --sysroot and -B option. If I set those in CFLAGS everything compiles fine. What I do now is this: pkg -j ${JAIL_NAME} install -y ${CROSS_TOOLCHAIN} byacc jexec ${JAIL_NAME} sh -c "yes | /usr/bin/make CC=${LLVM_DIR}/bin/clang LD=${LLVM_DIR}/bin/ld.lld -C /usr/src delete-old delete-old-libs" cd ${JAIL_PATH}/usr/bin && ln -fs ../local/llvm16/bin/clang cc cd ${JAIL_PATH}/usr/bin && ln -fs ../local/llvm16/bin/clang CC cd ${JAIL_PATH}/usr/bin && ln -fs ../local/llvm16/bin/clang++ c++ cd ${JAIL_PATH}/usr/bin && ln -fs ../local/llvm16/bin/clang-cpp cpp cd ${JAIL_PATH}/usr/bin && ln -fs ../local/llvm16/bin/llvm-objcopy objcopy cd ${JAIL_PATH}/usr/bin && ln -fs ../local/llvm16/bin/ld cd ${JAIL_PATH}/usr/bin && ln -fs ../local/bin/yacc jexec ${JAIL_NAME} /usr/bin/make -C /usr/src -j${NUM_CPUS} CROSS_TOOLCHAIN=llvm16 WITHOUT_TOOLCHAIN=yes buildworld buildkernel This builds fine also. Because CC is not set SYSROOT is just the default. Any advice on how to make set CC and use the proper sysroot instead of the (ugly) symlinking? And why do parts of buildworld use CROSS_TOOLCHAIN and other parts don't? Regards, Ronald. From nobody Tue Aug 22 14:30:39 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVWwg6Ctlz4qxmp for ; Tue, 22 Aug 2023 14:30:43 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVWwf50JVz4Ghd for ; Tue, 22 Aug 2023 14:30:42 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=JRuhcNh3; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::633 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-99cdb0fd093so621137966b.1 for ; Tue, 22 Aug 2023 07:30:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692714641; x=1693319441; h=content-transfer-encoding:subject:from:content-language:to :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=G9+Htx3PeJU1L5r+/zN19YYwrX7GeqU8MHwUxiq6+s8=; b=JRuhcNh38IX741i4x/7refRB5hOn+lGLJwjFnMIdMkHkK2TUlCFUbnHWWDHZBcdaUW gpEaQJT/09Nyu3zAwIqvqgPxFGMFossSX8oEMOka+tpvmpwqR4GxhLfrDxAZpdNN5Kwu INEql/KY2vR/1MbgUW/sBJXqFJD2Hu2zy8Q8e/WmeDB/whcqfBApX/bMjXx2TbSouNZC +sMGRkyGxOecw6BpIfJhDgZ3XpQjFy0M++EJ2WzX1Zsh5q7bEeX3UZxsfGu0b6LmT90+ Qvdil2h50hZ5RV+j2ny7b76z8mvsFTAxSoNNMHuWb0mECqWjdTxsQLkp0UkL4LbXvJg1 Fojw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692714641; x=1693319441; h=content-transfer-encoding:subject:from:content-language:to :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=G9+Htx3PeJU1L5r+/zN19YYwrX7GeqU8MHwUxiq6+s8=; b=TUueZHakjm7TM+cEDyPezo2ADHjCZKjJcY+8AmfZXDjZt1IPM+7KEUaclMYoCkvH51 mo1e2dRJejHS+95DkAaq1PHnntfJ9kfLHBhq2f0p5Jrxe14lIadg3GB9yumh5L03t1PF uc1dKLrnE5LTw4M0y7wCnftzMfC5Jo2qCm6+xOnVYokRPjYWebcSLlXx8E9iopWzroD+ F7AZsiKjfp0F3R3TNFmbOLZ75nLuSHAeZNFlPM7Q4tA/0/klpt/NWrFZyy0N6R9Rd90E 8cxdnFR7C7lpIED623dZcmeO/n6DEWeBI0POqJtgV4h4XikPtxQKBu8gVKUmY84xkmcJ BZ7A== X-Gm-Message-State: AOJu0YxcwszRirOlpaHUTTaXveH6DHpNXZWvm1I5508zghWaKdtX2m7u 0jZxcNdcSYP7joh06w4iWbOwgl5q/vM= X-Google-Smtp-Source: AGHT+IE2jdqNahaHRZrYuemliAEH+dwyehCXxcL9N+I8shsl+M5XBZYlkM1mjgy+3lC4HgnpFVbyww== X-Received: by 2002:a17:907:7612:b0:977:ecff:3367 with SMTP id jx18-20020a170907761200b00977ecff3367mr7577695ejc.40.1692714640641; Tue, 22 Aug 2023 07:30:40 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id c25-20020a170906529900b0099cf840527csm8220452ejm.153.2023.08.22.07.30.39 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Aug 2023 07:30:40 -0700 (PDT) Message-ID: <5d7149c5-0351-542b-c868-a24b055880a0@gmail.com> Date: Tue, 22 Aug 2023 15:30:39 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 To: FreeBSD CURRENT Content-Language: en-US From: Graham Perrin Subject: 7addfafe73e0 early boot kernel panics Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.90)[-0.903]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::633:from]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RVWwf50JVz4Ghd I could not get 7addfafe73e0 to boot. (2023-08-21, 21 hours ago). I reverted to a edacf4b4824a boot environment, which is fine. Should I simply update to ? Assuming a known issue with 7addfafe73e0 From nobody Tue Aug 22 14:52:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVXPV4QxFz4qyxk for ; Tue, 22 Aug 2023 14:52:14 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVXPV3c3Gz4L43; Tue, 22 Aug 2023 14:52:14 +0000 (UTC) (envelope-from otis@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692715934; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8v1BCVUXl89cPVSa0B8LlKOX6niyHafANcxeFEPHFFM=; b=Y5LWBfpdhw79pHgdI8QE0F9gyhBasGsVJ4mB8wbMLV481ap75c891/tDpm88l92GDrIs4J kGtosT4SRu2NJYHba0b3cbdZUQzIt6rRiy1aVBWVpRuPvaheRhjv+IjiwxKKAP73C3YqMQ tvbaqp4Nf6qFALax7xU5pUFd3/BghLimrWdEejR18twcOoZeFAHoN/inBUJdmEMhncdeBP /Eiuax/fR/mYH/wFmTRyz+4zw0q+CnSv4LH/S6mIz2nT3dJpT8ohwVfDW+yAQyro4JtN7v 3gbm64Z0s7vkcoQkK+xYjhFxZ3JaYp/T3Nc2oNuB9J24sjUD+NBHEQknCiGmiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692715934; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8v1BCVUXl89cPVSa0B8LlKOX6niyHafANcxeFEPHFFM=; b=vzHuq0BcqqYC0TfX+j2Y8M0o69Aqq4gbQa/gPp5NbqWV6wKUt2BLB4BwuZDwTjZsCbHMtu U6dy92XeqSKSOdYbWPsKKbL79x43ZsHu8Mq1rchFsj4IB9PHBAWTgwdmS7IMzQyAIlksL8 IMFwmkCF5PIbKnRblJGwT9GEgB1TNs+oHXQb5KQ9beVmDRE1AamfXw8z33PYe4oB8dED8I dNJc1dRyaVyq2tbQf7XYrdVzLWFKXMFbDkTQdG8WsJXEJKXVCwlgQ1Xz/qtbnPsv9sofEc gCarHphj9FJ8g4Kq8q92DejkIAfRdJPhOulKlnFwJdC0b6B7nz0q3LPz1KmJYw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692715934; a=rsa-sha256; cv=none; b=Z8YfmdnT8p7vY7csdyfpLxkaf40PiRjET7Cj1shMeZGXc1Lc9tu9SDyYaU4TnsS+T+jEF4 DreWOnLrDmUV8ccMKrdntMaDoZoJPhUPKHd1ThnrlE6mClOFTjIqdaT/gnraRQv46EjFRD 6zL5U1cXVKLT9dtu6YRm47UYeX42j/Se1fJUWXZ2n9l11xYR424M0fu58KtzPsUEX1P5Jr 3pOFz9TYlsVpZ0+zvAm0IMsJ5MduHhDZnR+Ic5X42WlVYC16pFT/x1oK/NNOJF1ONd7obr JSfdFsvIXpcJMnAt8ageaHQcIPSbdn5m+TIjMmWot+yhpE8HxwagimWnhr2UCg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from ns2.wilbury.net (ns2.wilbury.net [92.60.51.55]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) (Authenticated sender: otis) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RVXPV1z97z15w5; Tue, 22 Aug 2023 14:52:14 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtpclient.apple (unknown [217.73.28.193]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id 6736461FAD; Tue, 22 Aug 2023 16:52:11 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: 7addfafe73e0 early boot kernel panics From: Juraj Lutter In-Reply-To: <5d7149c5-0351-542b-c868-a24b055880a0@gmail.com> Date: Tue, 22 Aug 2023 16:52:00 +0200 Cc: FreeBSD CURRENT Content-Transfer-Encoding: quoted-printable Message-Id: References: <5d7149c5-0351-542b-c868-a24b055880a0@gmail.com> To: Graham Perrin X-Mailer: Apple Mail (2.3731.700.6) > On 22 Aug 2023, at 16:30, Graham Perrin = wrote: >=20 > I could not get 7addfafe73e0 to boot. >=20 > = (2023-08-21, 21 hours ago). >=20 > I reverted to a edacf4b4824a boot environment, which is fine. >=20 > Should I simply update to ? = Assuming a known issue with 7addfafe73e0 Recent c941b82e1c31 boots and runs fine (as a bhyve guest) otis =E2=80=94 Juraj Lutter otis@FreeBSD.org From nobody Tue Aug 22 15:21:25 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVY3P5gCVz4r260 for ; Tue, 22 Aug 2023 15:21:37 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVY3P5Dwvz4Nlb; Tue, 22 Aug 2023 15:21:37 +0000 (UTC) (envelope-from lwhsu@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692717697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xBfxIiLYQxXw9860H5N+WhmvYOVlpXeIwBjnMjNPwJw=; b=E/59WOXy0bGPDEvlk0hibjbaQMnnHUzHVs5GHRKLB7Wf1kiR3rNjwMFiC7xA/ECpJMmeHV 2kxgfBs/TcJnvbFk+3QHloxIq1Hl/p/XDmV+mCdJfcyglSteunM2m86zEX5rf0rxoIh2uS kj3EuxaUNkC+dAyrOCmbSzao59sYMowqSEhKowlhp+Oe4eCrJkofxN/7XZnEjY/0DgkuFN ev3g7fTGrBT1ypdCTX93AgcsGH1Nl50C6jewitdANNVopUBZvVnsEybaQhUPJdsYry6/j5 PdBXoxRHJ5a2Kidvbrdt0cyrbWhq2uGf/ATSg19k3rLVPOxVfebaIv9iQTdm+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692717697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xBfxIiLYQxXw9860H5N+WhmvYOVlpXeIwBjnMjNPwJw=; b=dCOn38Qgp1oqYR6hBdeqFoIJB34fQr0YGK/fPIF8WVHByZv+hXeQ+bp/x7XNktOfq7auFC n1zFy8YRwzcXTsZcnlApyPjrIn6ZSO0nt26hln9NCIEpqguaCUVIAR+muNrDP+wQJNAKzs TDxFhaw6RzP2qWOCHDVSX7/Ad54d43iw+xf4OLOqA6vH5nMHPlfKDNEymcxfeO03YWRxvV ytAx2Ogkp5traZqqq10Z36GCQ345nH2Iu9I2sJ0dkg7NmTxxAB5d+kFdQQHk5yVukVbrT0 Dn9W1lsO0RsYzjuQ0qTs4AWZZ/QeDiXRXrQKOZkrz13nB/UQplxyiVAsLYx8vw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692717697; a=rsa-sha256; cv=none; b=EM0XCSLLlSj97WecNOIsGSNQ8C/Bvr8Stei8mfGduwhSWwMhzr6F3zCfptf6fE65MU1+nq uL0LaPdzwvFQiFy5AeY78PdIUTVoyuig5NT1vQOponNAeBaXTnDPiV/4Da7qCZukmMLB5K +AQ750scMvMGyeIKmuxKa4ETl2j9k3IVG7T/GbTzwdmMYfOnPhzekznuVjBBgJ9LTBYtFD aA1Ce3TQETEBqSVWp9KRSgUDzmcn3dhr1t8J3IYANtyaYCb9j26JQP3ZkBtClzCAwvH6Hi JkcHiYhuH0MmjAqpLlaDUSDU+WcwLIENKQvOFeMXTtUxubH9wnUlD3x8Txxezw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: lwhsu/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RVY3P4CcZz16Dt; Tue, 22 Aug 2023 15:21:37 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-26d4e1ba2dbso2469036a91.1; Tue, 22 Aug 2023 08:21:37 -0700 (PDT) X-Gm-Message-State: AOJu0YxTQvezOkj/u/xyWjISVa2luxJ4TbDEnj4cZq3HwAoI2yFzhtda L6ayMp371Rvw5qeZ0GYfz8rvufn3eV/Pe47U3kU= X-Google-Smtp-Source: AGHT+IExJcRb+vj9JHh10EUsix10hacuhPHeoZQN/GtjlKHFpAQ+giNLes1habMJYVlBzMIFPRN9TwxJpcxNiMqwcAA= X-Received: by 2002:a17:90b:30c6:b0:268:5575:93d9 with SMTP id hi6-20020a17090b30c600b00268557593d9mr6704356pjb.10.1692717696561; Tue, 22 Aug 2023 08:21:36 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <5d7149c5-0351-542b-c868-a24b055880a0@gmail.com> In-Reply-To: From: Li-Wen Hsu Date: Tue, 22 Aug 2023 23:21:25 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: 7addfafe73e0 early boot kernel panics To: Juraj Lutter Cc: Graham Perrin , FreeBSD CURRENT Content-Type: multipart/alternative; boundary="000000000000023a5a0603848d54" --000000000000023a5a0603848d54 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Aug 22, 2023 at 10:52=E2=80=AFPM Juraj Lutter wr= ote: > > > > On 22 Aug 2023, at 16:30, Graham Perrin wrote: > > > > I could not get 7addfafe73e0 to boot. > > > > > (2023-08-21, 21 hours ago). > > > > I reverted to a edacf4b4824a boot environment, which is fine. > > > > Should I simply update to ? > Assuming a known issue with 7addfafe73e0 > > Recent c941b82e1c31 boots and runs fine (as a bhyve guest) > > otis > You can also check the test status in our CI https://ci.freebsd.org/job/FreeBSD-main-amd64-test/ There might be some failing test cases, or panic on running some test case, but at least you can check if one revision boots or not, in a reference system. Best, Li-Wen --000000000000023a5a0603848d54 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Tue, Aug 22, 2023 at 10:52=E2=80=AFPM Juraj Lutter <otis@freebsd.org> wrote:


> On 22 Aug 2023, at 16:30, Graham Perrin <grahamperrin@gmail.com> wrote:
>
> I could not get 7addfafe73e0 to boot.
>
> <https://cgit.freebsd.org/s= rc/log/?qt=3Drange&q=3D7addfafe73e0> (2023-08-21, 21 hours ago).=
>
> I reverted to a edacf4b4824a boot environment, which is fine.
>
> Should I simply update to <c941b82e1c31a67a025c43cc7bd31f269fa62588= >? Assuming a known issue with 7addfafe73e0

Recent c941b82e1c31 boots and runs fine (as a bhyve guest)

otis

You can also check the test status= in our CI=C2=A0https://ci.freebsd.org/job/FreeBSD-main-amd64-test/

=
There might be some failing test cases, or panic on running some= test case, but at least you can check if one revision boots or not, in a r= eference system.

Best,
Li-Wen
--000000000000023a5a0603848d54-- From nobody Tue Aug 22 16:18:12 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVZKJ0Jmpz4r5Jd for ; Tue, 22 Aug 2023 16:18:44 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-x112f.google.com (mail-yw1-x112f.google.com [IPv6:2607:f8b0:4864:20::112f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVZKH5kxbz4VpX; Tue, 22 Aug 2023 16:18:43 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-58d70c441d5so52065097b3.2; Tue, 22 Aug 2023 09:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692721122; x=1693325922; h=content-transfer-encoding:in-reply-to:subject:from:references:cc:to :content-language:user-agent:mime-version:date:message-id:sender :from:to:cc:subject:date:message-id:reply-to; bh=l5EurCr6pG0MIN1QE7aSHthGy31lapUBJ0Li5/kg9Ic=; b=J7EpGBfHSlV4oxEW7WYyMFpEc9RwVR/uUpVH+rflx7yTho7IJv9SL4vE4z8uph6dIg /joZ586hKpndoEQIhtEPN+1NjYZTM5MJV2K1rtoiet3q526/jaUIsWMrS8eZ69QQQP3k 42RhSpkg0+ouTjUNtk+lu+rvQVfUQVQW+QASXoNlJJn5d+NlnW0urRZhwXtJLTpB53k2 3+egLTab/F1yyVunuK4DwQDUWNzYXwDDSbtOgr5aU7+PbZLdb7rj8G+Lg70iu3n0DO6S TAWrHACVa9Uvhc4Do3Dd0frj1DL7Cdh9/WKg+FRduIBdBAAMkNvB7xVkJ/IaQP5tcE3O iIWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692721122; x=1693325922; h=content-transfer-encoding:in-reply-to:subject:from:references:cc:to :content-language:user-agent:mime-version:date:message-id:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l5EurCr6pG0MIN1QE7aSHthGy31lapUBJ0Li5/kg9Ic=; b=PIutYLMdjop5CggqMmSvbL7oVbTpAkzQi6NJ4o7w8rrtCMLf0CS2u67BLWPFE+UnWt 2iPjIy7pNUPc1f4XbvqvGFfcTVZ8p5eKI3eU4uKWjf/j2gmEzD1D7GgnZAdV2rEa1y6t OZwHXjm657cav4OrW/465y1QihVzkmzqaIUJOtD+QG65jULSopJxZ+D4whzt1yAXXu19 lhzp0SpAm9rjQBXmn8aJu5c6K4hwfrjbsf+MLMTWgm/3zixXNemJHLmIUf0S4PlJwC3a nh4Ws8qbXY97l3TGDC7aZmc1PWs3X7biw5LsVktw0WJRM2ZlwQlCOT1v3jQ4zay7eQ8g LPEg== X-Gm-Message-State: AOJu0YyjUjTxjSUJeHlMSTsccah/dnZ1KAk0+e+r94m1yAuthgAin/Fn uLMgAsucuUGFvaZ7yQx1G+AGhnKsvj4= X-Google-Smtp-Source: AGHT+IE5bNJEsqXtQxOB8P1vHXwUo0WjXeZwSok4ttaGmqMYXHDDkkpDgspRlyHw0t+lAE96pExFwQ== X-Received: by 2002:a0d:e889:0:b0:586:c1fc:f307 with SMTP id r131-20020a0de889000000b00586c1fcf307mr11212493ywe.0.1692721122610; Tue, 22 Aug 2023 09:18:42 -0700 (PDT) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id i194-20020a816dcb000000b0057042405e2csm2858921ywc.71.2023.08.22.09.18.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Aug 2023 09:18:41 -0700 (PDT) Message-ID: Date: Tue, 22 Aug 2023 12:18:12 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Martin Matuska Cc: =?UTF-8?Q?Dag-Erling_Sm=c3=b8rgrav?= , current@freebsd.org, Mateusz Guzik References: <86leeltqcb.fsf@ltc.des.no> <86h6p4s64h.fsf@ltc.des.no> <86a5utrafp.fsf@ltc.des.no> <86350kqokl.fsf@ltc.des.no> <86y1icp95t.fsf@ltc.des.no> <86ttt0p8wv.fsf@ltc.des.no> <197ead1e-210a-6be6-7e24-5c56b14bb777@FreeBSD.org> <86350hpiby.fsf@ltc.des.no> <0fd64567-59cd-9276-cc0c-17a09611f11a@FreeBSD.org> <20230822102622.Horde.tUEwNRTMDn0AjOBiO1UB7OX@webmail.your-server.de> From: Alexander Motin Subject: Re: ZFS deadlock in 14 In-Reply-To: <20230822102622.Horde.tUEwNRTMDn0AjOBiO1UB7OX@webmail.your-server.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4RVZKH5kxbz4VpX X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] Hi Martin, I am waiting for final test results from George Wilson and then will request quick merge of both to zfs-2.2-release branch. Unfortunately there are still not many reviewers for the PR, since the code is not trivial, but at least with the test reports Brian Behlendorf and Mark Maybee seem to be OK to merge the two PRs into 2.2. If somebody else have tested and/or reviewed the PR, you may comment on it. On 22.08.2023 04:26, Martin Matuska wrote: > as 15107 is a prerequisite for 15122, > would it be possible to have https://github.com/openzfs/zfs/pull/15107 > merged into the OpenZFS zfs-2.2-release branch (and of course later 15122)? > > If the patches help I can cherry-pick them into main. > > Alexander Motin wrote: > >> On 17.08.2023 15:41, Dag-Erling Smørgrav wrote: >>> Alexander Motin writes: >>>> Trying to run your test (so far without reproduction) I see it >>>> producing a substantial amount of ZIL writes.  The range of commits >>>> you reduced the scope to so far includes my ZIL locking refactoring, >>>> where I know for sure are some deadlocks.  I am already waiting for 3 >>>> weeks now for reviews and tests for PR that should fix it: >>>> https://github.com/openzfs/zfs/pull/15122 .  It would be good if you >>>> could test it, though it seems to depend on few more earlier patches >>>> not merged to FreeBSD yet. >>> >>> Do you have a FreeBSD branch with your patch applied? >> >> I don't have a FreeBSD branch, but these two patches apply clean and >> build on top of today's FreeBSD main branch: >> >> https://github.com/openzfs/zfs/pull/15107 >> https://github.com/openzfs/zfs/pull/15122 >> >> And if you still experience the issue, please show all stacks, or at >> least include ZFS sync threads. -- Alexander Motin From nobody Tue Aug 22 16:40:31 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVZpj0qbJz4r6tN for ; Tue, 22 Aug 2023 16:40:45 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVZph661mz4ZCb for ; Tue, 22 Aug 2023 16:40:44 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-68a3f1d8be2so2144900b3a.3 for ; Tue, 22 Aug 2023 09:40:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692722443; x=1693327243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=LFk2hW5vE9FBDCgoiF1QdYZEZGy09lrz8P6R4foWs68=; b=AqRoxN9j6zUIsgWOQvQQIo+6uZ+54A89rNOTnlAR6BTkvMJm78L1nHeMeFAVLU7PKR t10+DW5/4HMyfp9n0vR+N/3xHdGIEeNexBLVLHhKckLJe10WxiL0ajf0HA2pEsB0Xd3m rEwHlDwvLlZjnycOyQgJfHqvmBiID3iTke0o8/wg8QjFFeEhgX+YAlCzQA8QcxanSmcF 3gRqhihpW9I2tvpZBOxC2+zT3mtegjWG6rB2Tffj4B9Ipe4Jp0fmxEHoaryHtPTr7E3O TjhnVF7HIujnO4ZjCmZZRYbcnnNFb0M6aHWCMdsW8up0LImesLywqExJF9bOqiltBS1I ikMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692722443; x=1693327243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LFk2hW5vE9FBDCgoiF1QdYZEZGy09lrz8P6R4foWs68=; b=ZdnBT0rOageqdQNTa/nn1iiHvVftBa0V2//M2UWl/7rK+ljpnG/BRvkUU5f9M+UGo1 AkZdGbVIHKD6aAUiCQqePzQkFhJy2LtsiFnf1rW7zQC8F/gf9iv1769ehtea4Fe6zXP3 YJSqhMO5JGgiwyASVMz2scMjww+vhA3qyP5VIk3ANmdhQcLN39wB1ytMmBcAYV8GX91G 2MWtSxjATLl++j82MpqeEjXgZ57BW2Z7yTCzMX57nRbgQOmNSYmGoWFVbORGovHW38NR WiMrGarhIfE0bB6hUY7lWZCvyeOnGp7IGJU+9tE16rVfbz+EkknrUWMbswSOaFwdSPiE dk5w== X-Gm-Message-State: AOJu0YwonysuegO0gihVc1NDq0gpSOBAU4wheeZUrEh8L1akG1LKzRPr gzNW8BMsC9gdbTZ319it1T+ncD/HbWY= X-Google-Smtp-Source: AGHT+IFdLCA9PTlDi/T9M+c07mkn8440u/zhu9BphsjVm1xSQ+8+6N4muse1Fa/R2vHuvUsIAmPt0Q== X-Received: by 2002:a05:6a20:8e01:b0:135:4527:efe4 with SMTP id y1-20020a056a208e0100b001354527efe4mr13956276pzj.10.1692722443419; Tue, 22 Aug 2023 09:40:43 -0700 (PDT) Received: from smtpclient.apple (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id 10-20020aa7914a000000b00689f1ce7dacsm3980382pfi.23.2023.08.22.09.40.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Aug 2023 09:40:42 -0700 (PDT) From: Enji Cooper Message-Id: <2083B385-D726-473F-A2C0-3574F154AFF4@gmail.com> Content-Type: multipart/signed; boundary="Apple-Mail=_3245D8FE-4AF9-4DE1-8E72-D5C73F585FA3"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: src.conf(5) to specify multiple flavours of a port Date: Tue, 22 Aug 2023 09:40:31 -0700 In-Reply-To: <813CD008-5C7D-4664-B605-3F61A2785213@iitbombay.org> Cc: Graham Perrin , FreeBSD CURRENT To: Bakul Shah References: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> <813CD008-5C7D-4664-B605-3F61A2785213@iitbombay.org> X-Mailer: Apple Mail (2.3731.600.7) X-Rspamd-Queue-Id: 4RVZph661mz4ZCb X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --Apple-Mail=_3245D8FE-4AF9-4DE1-8E72-D5C73F585FA3 Content-Type: multipart/alternative; boundary="Apple-Mail=_D843FC4A-9C45-4D82-8579-A4EC357496CC" --Apple-Mail=_D843FC4A-9C45-4D82-8579-A4EC357496CC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Aug 21, 2023, at 10:01 PM, Bakul Shah wrote: >=20 > On Aug 21, 2023, at 9:24 PM, Graham Perrin = wrote: >>=20 >> In a thread elsewhere, as an example that did not involve src.conf, = Mark Johnston wrote: >>=20 >>> $ cd /usr/ports/graphics/gpu-firmware-intel-kmod >>> $ sudo make reinstall FLAVOR=3Dkabylake >>=20 >> How might I use /etc/src.conf to achieve much the same, with a = different port? >=20 > Since /etc/src.conf is included from make, may be you can use some = make feature > for conditional define? I think there=E2=80=99s some confusion=E2=80=A6 /etc/make.conf is always = included by bmake; /etc/src.conf is only included when building the base = system. FreeBSD ports doesn=E2=80=99t have a special systemwide config file like = the base system for toggling build/install behavior, but it does have = "${PORTSDIR}/Mk/bsd.local.mk=E2=80=9D . -Enji $ grep -r /etc/src.conf share/mk/ share/mk/src.opts.mk:# Users define WITH_FOO and WITHOUT_FOO on the = command line or in /etc/src.conf share/mk/src.opts.mk:# to set via WITH_*/WITHOUT_* in /etc/src.conf and = override in the share/mk/bsd.port.mk:# Needed to keep bsd.own.mk from reading in = /etc/src.conf share/mk/bsd.opts.mk:# Users define WITH_FOO and WITHOUT_FOO on the = command line or in /etc/src.conf share/mk/bsd.opts.mk:# to set via WITH_*/WITHOUT_* in /etc/src.conf and = override in the share/mk/src.sys.mk:SRCCONF?=3D /etc/src.conf share/mk/src.sys.mk: (exists(${SRCCONF}) || ${SRCCONF} !=3D = "/etc/src.conf") && \ --Apple-Mail=_D843FC4A-9C45-4D82-8579-A4EC357496CC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Aug 21, = 2023, at 10:01 PM, Bakul Shah <bakul@iitbombay.org> = wrote:

On Aug 21, = 2023, at 9:24 PM, Graham Perrin <grahamperrin@gmail.com> = wrote:

In a thread elsewhere, as an = example that did not involve src.conf, Mark Johnston wrote: =

$ cd = /usr/ports/graphics/gpu-firmware-intel-kmod
$ sudo make reinstall = FLAVOR=3Dkabylake

How might I use /etc/src.conf to = achieve much the same, with a different port?

Since = /etc/src.conf is included from make, may be you can use some make = feature
for conditional define? =

I think there=E2=80=99s some = confusion=E2=80=A6 /etc/make.conf is always included by bmake; = /etc/src.conf is only included when building the base = system.
FreeBSD ports doesn=E2=80=99t have a special = systemwide config file like the base system for toggling build/install = behavior, but it does have "${PORTSDIR}/Mk/bsd.local.mk=E2=80=9D = .
-Enji

$ grep -r = /etc/src.conf share/mk/

share/mk/src.opts.mk:# Users define WITH_FOO and = WITHOUT_FOO on the command line or in /etc/src.conf

share/mk/src.opts.mk:# to set via WITH_*/WITHOUT_* = in /etc/src.conf and override in the

share/mk/bsd.port.mk:# Needed to keep bsd.own.mk = from reading in /etc/src.conf

share/mk/bsd.opts.mk:# Users define WITH_FOO and = WITHOUT_FOO on the command line or in /etc/src.conf

share/mk/bsd.opts.mk:# to set via WITH_*/WITHOUT_* = in /etc/src.conf and override in the

share/mk/src.sys.mk:SRCCONF?=3D   = /etc/src.conf

share/mk/src.sys.mk:    = (exists(${SRCCONF}) || ${SRCCONF} !=3D "/etc/src.conf") && = \

= --Apple-Mail=_D843FC4A-9C45-4D82-8579-A4EC357496CC-- --Apple-Mail=_3245D8FE-4AF9-4DE1-8E72-D5C73F585FA3 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEtvtxN6kOllEF3nmX5JFNMZeDGN4FAmTk5P8ACgkQ5JFNMZeD GN6uCQ//cUdCEfiHw0obKevARyOVubnH1liA5QLwTRpZSwZKbo0pa6MeemzfUs4j Jx+DxC6DIZyxpOX94Zxd6QxrkIVmd30RAJ8UubdTQYKJFYCIkBVHFb/d2P2ceLPR ZvQja00qDLfZQiz3tBqKR1/zGdAIVVCWNjBbjhV7Meswy6e1cMBph9mso2b60VSI wzARb8WZwg0rn5KQO61TBKAgLfd53lt7u0O2J6phkr+AiJ5Ov7tOFIJIlmhlhhoy 9TsiL5+kc4CX3xk4F4WotTuc+2aaCJFNYy1dzaJkOjhcGYQkMWB7i0E5QWWHxdrL Mb9OQr9gkuyx2oh8Pmo1Unxvje2CNUXx0mKVAUz62XADQSxQUQMQKIzF2biGlLqe UvJgH6NZKqva7cJjhBKmLtz0l/2lJ/rgg1ImNOW36LLSIlHkSvPx9JMM/2eanV2v E9OjoTbBF1ZJmkoVUjJ0rtUABTFdJfOy9z8jl+xYBIR9poo4GH0QA/PWFO9J32M6 NjxYePMCPVyPvPYW+GfyaSP3mqonoNap4uEaJopNBeXGR5YwdoC+Fd+BaaJh8Jsj pZ14YtmnQbG9/5gN9qxeYjYcybRmvfxElMw3cggyqPl7KjhfrJ8OmCfC8avEYP7Q CCbUhQ6JnGMST3b8mGN9KS3li7+ENKbSslo00/MwlhMBXiu/V3Q= =NlAV -----END PGP SIGNATURE----- --Apple-Mail=_3245D8FE-4AF9-4DE1-8E72-D5C73F585FA3-- From nobody Tue Aug 22 16:42:34 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVZs34cyYz4r7VN for ; Tue, 22 Aug 2023 16:42:47 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVZs32V6qz4bNb for ; Tue, 22 Aug 2023 16:42:47 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1bba48b0bd2so30611025ad.3 for ; Tue, 22 Aug 2023 09:42:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692722566; x=1693327366; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=/lQGuFhIUoIBGiaXvZxABITdHl/itycYqk2o6cwXOGg=; b=kWTIyJwbojr0uLTu6mAd9B7KlCc3zG2H7TMnMB+lTg47Nw5cltj7bNmoGbHm3xdme1 mrvO0AOHGAm7B/zNG4pV9vUTzQdzo1mncc4LkVRiE5v2Y6hu6XPRZpeNrHSJgULm1To3 CwJ7jRob81EGzleImPQxl75yRf14GbwIpWqRxsJZQmPK7qDF4vhEmFNjzWz8WhWd+abB eRPieUCc4RR+OzHzVPmKq0REPIr+tgsw1T5dt0ycf0Qnsw25NsmUgjzAaPrpKPaKj6Wi w0zNpnFc9RCvqdJ3wq8nJ+diMtV7EvBMctfLDBtL5lh+tKC6NjJEGr3Yy9xpo5zeAk5v I3Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692722566; x=1693327366; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/lQGuFhIUoIBGiaXvZxABITdHl/itycYqk2o6cwXOGg=; b=PayPLSLWaU0LrR7/TGxsIU2LY15llQKzKqsoCiLoHDeClCnOjkffIYYNQwW6lKhCyc AoYr6d8rkRFm/dk1ZdM1JdsDNOCPeaUj82o9zv1wAluOg4mW/9gwkONIoVZL59RLdy1K oWQrY8OgPAoZGweusvLqGPrCdh61wtIFDYUTJ7VTjpDsFVpDbWMeoq5OBsu3tw2yttAZ a0SKzWed34V2dZ+wk5KwX7oEZJPhUer7pkB9Wg54770Tur1BrcwQWskG4PndQqgOhQH8 4f62l/H/yZEMivpIefoa1sfGPHCxWUwFqGq6id73QVveaOOWP3q5YbtBmzmdi+DwObCj jNwg== X-Gm-Message-State: AOJu0Yzd5bmp7SRJ1OKS+3DGdtM/L02CMZW8QgFlu6ozeYvuomNI0qYb 6sL9b6rMQKsplexNEm4XI2U= X-Google-Smtp-Source: AGHT+IFM1lrhUSb6s8t1MPM9hZxIluUzLU6x1jtY6cIg5i8NxV39dV7VoXyf7ulGiOwMjnNFphzT3Q== X-Received: by 2002:a17:902:d4cb:b0:1bd:c931:8c32 with SMTP id o11-20020a170902d4cb00b001bdc9318c32mr7836022plg.62.1692722565660; Tue, 22 Aug 2023 09:42:45 -0700 (PDT) Received: from smtpclient.apple (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id x21-20020a170902ea9500b001bc39aa63ebsm9262160plb.121.2023.08.22.09.42.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Aug 2023 09:42:45 -0700 (PDT) From: Enji Cooper Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_795EB426-128A-4774-A910-72684951F896"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: src.conf(5) to specify multiple flavours of a port Date: Tue, 22 Aug 2023 09:42:34 -0700 In-Reply-To: Cc: Graham Perrin , FreeBSD CURRENT To: Warner Losh References: <31c2b00f-a3c4-5608-9776-fa20ea005a45@gmail.com> X-Mailer: Apple Mail (2.3731.600.7) X-Rspamd-Queue-Id: 4RVZs32V6qz4bNb X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --Apple-Mail=_795EB426-128A-4774-A910-72684951F896 Content-Type: multipart/alternative; boundary="Apple-Mail=_10ACAEDB-B500-4650-92F3-347FCF338BFC" --Apple-Mail=_10ACAEDB-B500-4650-92F3-347FCF338BFC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Aug 21, 2023, at 9:34 PM, Warner Losh wrote: >=20 > On Mon, Aug 21, 2023, 10:24 PM Graham Perrin > wrote: >> In a thread elsewhere, as an example that did not involve src.conf, = Mark Johnston wrote: >>=20 >>> $ cd /usr/ports/graphics/gpu-firmware-intel-kmod >>> $ sudo make reinstall FLAVOR=3Dkabylake >> How might I use /etc/src.conf to achieve much the same, with a = different port? >>=20 >=20 >=20 >=20 > I thought stuff like this went in ports.conf=E2=80=A6 ports.conf? ports-mgmt/portconf/files/portconf.sh.in:_conf=3D%%PREFIX%%/etc/ports.conf= ports-mgmt/portconf/files/pkg-message.in:%%PREFIX%%/etc/ports.conf = configuration file Huh=E2=80=A6 TIL. -Enji --Apple-Mail=_10ACAEDB-B500-4650-92F3-347FCF338BFC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Aug 21, = 2023, at 9:34 PM, Warner Losh <imp@bsdimp.com> = wrote:

On Mon, Aug = 21, 2023, 10:24 PM Graham Perrin <grahamperrin@gmail.com> = wrote:
=20 =20 =20
In a thread elsewhere, as an example that did not involve src.conf, Mark Johnston wrote: 

$ cd /usr/ports/graphics/gpu-firmware-intel-kmod
$ sudo make reinstall FLAVOR=3Dkabylake

How might I = use /etc/src.conf to achieve much the same, with a different = port?



I thought stuff like this went = in = ports.conf=E2=80=A6

ports.con= f?

ports-mgmt/portconf/files/portconf.sh.in:_conf=3D%%PR= EFIX%%/etc/ports.conf

ports-mgmt/portconf/files/pkg-message.in:%%PREFIX%%/e= tc/ports.conf configuration file


Huh=E2=80=A6= TIL.
-Enji
= --Apple-Mail=_10ACAEDB-B500-4650-92F3-347FCF338BFC-- --Apple-Mail=_795EB426-128A-4774-A910-72684951F896 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEtvtxN6kOllEF3nmX5JFNMZeDGN4FAmTk5XoACgkQ5JFNMZeD GN6OhA//fHaF7kLDQ06kP/qqeO7cSpiKxrcDFqJTDkoJ2AiIBJEHNvSWdt6d4JFU l6zssjxyxCKkxGAyL2AVjB1cT/CPm6cBM/Cwvs87ZR0lTmc7DWjkr7Q0jJn+r0iN Zw2lfEKB4MhntZyJYiysgmcYWhc4Lk6rXKsj1nTFZjMdDQA0xGNoqEzklA3pBhA2 lrIzuZm4Dfpe0JqjYQH2Mfbh9Me1GUmmuiHnzihqi3NRiNY45u7Zl0oMAj/BoelB QuwApfHcOn0TdbamyN47ZE7hXwGLKzXYRwkkHaJZ1By5DGOhebOymmyeIq8pnfVe UKINatDJZiJdj7R5Ttj8EV0VCdzD70mnC20hxjlec6obyIfcJ0ayAUZsq2aGklgl V3eDBujkKCLHJpQcTSpFvtjWMnjwDXUEQrITOgz+nXksacvWi0t6XhwkSbxEU74x V7n/lgBtcfW6PBHiCkEcyNZOvKdjklo0eLrUNAotxMKH8dN4XtpofbrndtOMYl5Q CdhyIhdFnzokxK0jvm+DXIVcGuI6OqXy98kCYmFmKDSJ6mpf4FxP6daIFIokhHF3 C+qlJcx1tIjYmGWA2IiUlU+b67qeKbAZxKQ/YN7dd/snNJcjOYSYbKpYmVz3YfP8 uWyMQqLm5PL6tDI6svWi8nnGElpBUBvbuJ1X9iMTqZ/WDKrBshk= =/Mr4 -----END PGP SIGNATURE----- --Apple-Mail=_795EB426-128A-4774-A910-72684951F896-- From nobody Tue Aug 22 16:59:36 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVbDV3LjGz4r8NW for ; Tue, 22 Aug 2023 16:59:38 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVbDV1C5wz4fNp for ; Tue, 22 Aug 2023 16:59:38 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc31.google.com with SMTP id 006d021491bc7-56c4c4e822eso3026981eaf.3 for ; Tue, 22 Aug 2023 09:59:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692723577; x=1693328377; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Qb6BIDwBDJmjuSUBP221zOmLxC3/lFUm8fRqUTtcEjY=; b=VvO9ekBeZTakjDm8dJCY0Q1gHnLDRq5TIp7rg56NE6W0H0ObINQ+GWr81v8IdiHYFc mfdIxJ4pBGPDr2vmrkRtVSndLq4iPje2q7taYQUlabFUUD7EUR6qO10Ct1teSwGnRKJ8 yLk7FnhQLnqT/pbnBD0ZQzxuyf7m3PtSAMhtxFwo2NnZm/YMyNNkj0Me9/cAgXdEw/ns Hpr6nvlWJtwiMBW+7v+bmSLXY9az1HfJNuwOBuevYDy9/PNknavPIKNmLGZlyGdNKyc/ cir6B2UuywKf4MJc3Sa0futudD/+rJ3CDj2Q8//ioBm9xemUf7Wsr5GCwIx6ZL6IMuND 8gFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692723577; x=1693328377; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Qb6BIDwBDJmjuSUBP221zOmLxC3/lFUm8fRqUTtcEjY=; b=Z7X+N6Iek7WG+oGH9VrCxpnzq1RsOXJl1d2fegYa3ssDehzjFgbovV5sTUYnwzfBO0 J7/YIGt+3QUoUjCnikgnHSn+0CgdrOnITowuZR8IVtH1FnBzEx37XltOo8XvzCKyfgxL zFrCw3/rNMVHUT5rmN4WMqSbnMu488awAOS/3WCTe0kTYkemafT5hFgljIX5RXJE3nhk 6jjRJ5TRRIFqIDPCQaSZa5Wy74Y0PFSA47c/iL1NMeXLWPQJrR1rlUV0UvQgZQhSL8LH 95WgCfOT0DQ9aiHs+11I1pLEwWt9GYWaqPzEP3UEXKbDA4UgPoUBdNh50aq6w7H+mxRE 7fAQ== X-Gm-Message-State: AOJu0YwGhgWJKbUj02NzSXRJCjKCeEFs2cabldQ7FkYYTEG9o0McaCTh xh3nf8zP3bBPTpzGb0PBWzb4ezLGKAlBzQN6e4M= X-Google-Smtp-Source: AGHT+IH6Aj+0uA/KSuiItzJfwIB+6kc83a3VUqwbcKn1moUzdd9cshzlD4gU0C0fIT+gBqV7qJCYGjWzjWJmqYLKN4M= X-Received: by 2002:a4a:d1d7:0:b0:56e:975a:1290 with SMTP id a23-20020a4ad1d7000000b0056e975a1290mr11258644oos.0.1692723576859; Tue, 22 Aug 2023 09:59:36 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:5797:0:b0:4f0:1250:dd51 with HTTP; Tue, 22 Aug 2023 09:59:36 -0700 (PDT) In-Reply-To: <444770b977b02b98985928bea450e4ce@Leidinger.net> References: <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> <73f7c9d3db8f117deb077fb17b1e352a@Leidinger.net> <58493b568dbe9fb52cc55de86e01f5e2@Leidinger.net> <58ac6211235c52d744666e8ae2ec7568@Leidinger.net> <444770b977b02b98985928bea450e4ce@Leidinger.net> From: Mateusz Guzik Date: Tue, 22 Aug 2023 18:59:36 +0200 Message-ID: Subject: Re: Speed improvements in ZFS To: Alexander Leidinger Cc: Konstantin Belousov , current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RVbDV1C5wz4fNp X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On 8/22/23, Alexander Leidinger wrote: > Am 2023-08-21 10:53, schrieb Konstantin Belousov: >> On Mon, Aug 21, 2023 at 08:19:28AM +0200, Alexander Leidinger wrote: >>> Am 2023-08-20 23:17, schrieb Konstantin Belousov: >>> > On Sun, Aug 20, 2023 at 11:07:08PM +0200, Mateusz Guzik wrote: >>> > > On 8/20/23, Alexander Leidinger wrote: >>> > > > Am 2023-08-20 22:02, schrieb Mateusz Guzik: >>> > > >> On 8/20/23, Alexander Leidinger wrote: >>> > > >>> Am 2023-08-20 19:10, schrieb Mateusz Guzik: >>> > > >>>> On 8/18/23, Alexander Leidinger >>> > > >>>> wrote: >>> > > >>> >>> > > >>>>> I have a 51MB text file, compressed to about 1MB. Are you >>> > > >>>>> interested >>> > > >>>>> to >>> > > >>>>> get it? >>> > > >>>>> >>> > > >>>> >>> > > >>>> Your problem is not the vnode limit, but nullfs. >>> > > >>>> >>> > > >>>> https://people.freebsd.org/~mjg/netchild-periodic-find.svg >>> > > >>> >>> > > >>> 122 nullfs mounts on this system. And every jail I setup has >>> > > >>> several >>> > > >>> null mounts. One basesystem mounted into every jail, and then >>> > > >>> shared >>> > > >>> ports (packages/distfiles/ccache) across all of them. >>> > > >>> >>> > > >>>> First, some of the contention is notorious VI_LOCK in order to >>> > > >>>> do >>> > > >>>> anything. >>> > > >>>> >>> > > >>>> But more importantly the mind-boggling off-cpu time comes from >>> > > >>>> exclusive locking which should not be there to begin with -- as >>> > > >>>> in >>> > > >>>> that xlock in stat should be a slock. >>> > > >>>> >>> > > >>>> Maybe I'm going to look into it later. >>> > > >>> >>> > > >>> That would be fantastic. >>> > > >>> >>> > > >> >>> > > >> I did a quick test, things are shared locked as expected. >>> > > >> >>> > > >> However, I found the following: >>> > > >> if ((xmp->nullm_flags & NULLM_CACHE) != 0) { >>> > > >> mp->mnt_kern_flag |= >>> > > >> lowerrootvp->v_mount->mnt_kern_flag & >>> > > >> (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED | >>> > > >> MNTK_EXTENDED_SHARED); >>> > > >> } >>> > > >> >>> > > >> are you using the "nocache" option? it has a side effect of >>> > > >> xlocking >>> > > > >>> > > > I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache. >>> > > > >>> > > >>> > > If you don't have "nocache" on null mounts, then I don't see how >>> > > this >>> > > could happen. >>> > >>> > There is also MNTK_NULL_NOCACHE on lower fs, which is currently set >>> > for >>> > fuse and nfs at least. >>> >>> 11 of those 122 nullfs mounts are ZFS datasets which are also NFS >>> exported. >>> 6 of those nullfs mounts are also exported via Samba. The NFS exports >>> shouldn't be needed anymore, I will remove them. >> By nfs I meant nfs client, not nfs exports. > > No NFS client mounts anywhere on this system. So where is this exclusive > lock coming from then... > This is a ZFS system. 2 pools: one for the root, one for anything I need > space for. Both pools reside on the same disks. The root pool is a 3-way > mirror, the "space-pool" is a 5-disk raidz2. All jails are on the > space-pool. The jails are all basejail-style jails. > While I don't see why xlocking happens, you should be able to dtrace or printf your way into finding out. -- Mateusz Guzik From nobody Tue Aug 22 17:06:34 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVbNp3GPqz4r8fq for ; Tue, 22 Aug 2023 17:06:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVbNm25Bfz3DRt for ; Tue, 22 Aug 2023 17:06:48 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20221208.gappssmtp.com header.s=20221208 header.b="1T4/XQus"; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::531) smtp.mailfrom=wlosh@bsdimp.com; dmarc=none Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-52a23227567so234091a12.0 for ; Tue, 22 Aug 2023 10:06:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1692724006; x=1693328806; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=OhG0ySnsFbsRKCntDeL5BtQMtrJos56rFwz0KmgoEEU=; b=1T4/XQusLX+yWtMoaI+BJHyUgpILmt2FrfQMLUnYzGzsUvYlAdyKqDr9LimMEkPO96 Gs7bZU0Xi+3gvBfHlSJvwo2NfyN0Ae9QH69Dva4VKzwpfc4zRQi59dWlRbu2WphgPXxr nWdTVFdJHl6ju2pQ21J41eNMD+EnS7NQjfleo37r60TtXhXb9/79LI3/eE/sPWL54Rix 4S0K00Of2FWCrZoHO98rZ3WoCAhRVvRNHHmZNLuKKRwLy8IguC0HHI7CgqtmeB0qk36X 47l14C3oVnJl83NIQuKFpy6vVtpfINzqwgeIPTrkwbcGADgjLDK43B8Fd6BVoekPB5uV PLPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692724006; x=1693328806; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OhG0ySnsFbsRKCntDeL5BtQMtrJos56rFwz0KmgoEEU=; b=c1rtF74zeZ42U/LSpn1hPnHVODb2KdW63TGyDm46z+iEi/fdNGjpR82tpqJPLLseY4 wyxdvFBBMzdfQ7oc2m7X74/hwtkV/5WKSTxPr908C2MPco17Gun8/lgAGFfBZU6Cur/5 nBY4aHT4IWgReDC9zrQsdxH7zvDx3x7g6d+T24EuzusxxATZ85ErQmpX9fSqjmJOMmkQ I+1EMwzCWiulWVwl2a6rLKfQ9WWfo7XNGIRQKEBOvXRnxCaBk62WDwJNL3JP2IiqF0aM quq18+bdPyqUe82qbyHqjtGw9M0zULqk6yDVfRxyZOrFQnbAzBSR7wt3gnou5EDeVgT/ Tl1Q== X-Gm-Message-State: AOJu0YyRcVe+8PBEJpjpCMx6buslveyQx7M6mTtJxRZL+DCwNVQ4Sn0Q YKFXhNwEgcQ2lScVY3yqBo4ovF50IcQRJChUUvE20+iMDfpMKABi X-Google-Smtp-Source: AGHT+IFWprnoaJZnHQ9Mc8n8IH92t9iEt/f0ImsYzssHOm3wBQlQ8ka0wpLOaC30yjyhIORpaPtYj51rNefjEe/+Eyk= X-Received: by 2002:a17:907:2722:b0:982:45ca:ac06 with SMTP id d2-20020a170907272200b0098245caac06mr7706946ejl.60.1692724005894; Tue, 22 Aug 2023 10:06:45 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Tue, 22 Aug 2023 11:06:34 -0600 Message-ID: Subject: Re: Question about KBI change / new feature To: Zhenlei Huang Cc: FreeBSD CURRENT Content-Type: multipart/alternative; boundary="000000000000130bde06038605aa" X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20221208.gappssmtp.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::531:from]; DKIM_TRACE(0.00)[bsdimp-com.20221208.gappssmtp.com:+]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[bsdimp.com]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4RVbNm25Bfz3DRt --000000000000130bde06038605aa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Aug 21, 2023 at 9:42=E2=80=AFAM Zhenlei Huang wr= ote: > Hi, > > The https://www.freebsd.org/releases/14.0R/schedule/ says CURRENT/14 's > KBI is froze > and new features should be avoided. > > I'm working on https://reviews.freebsd.org/D39638 (sysctl(9): Enable vnet > sysctl variables be loader tunable) > and I think it is new feature, but not quite sure whether the KBI changed= . > > So, > > 1. Is it a KBI change ? > IMHO, It's a KPI change, not a KBI breakage. So from that perspective, it's OK. > 2. It is a simple change ( while so far as I know currently only tested b= y > myself on x86 and qemu riscv ), can > it catch up with 14 ? That I'm less sure of. I think it's good, but I'm gun shy about approving / committing vnet things. The review suggests, though, there's at least some consensus for having this in the tree. > Best regards, > Zhenlei > > > --000000000000130bde06038605aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Aug 21, 2023 at 9:42=E2=80=AF= AM Zhenlei Huang <zlei@freebsd.org> wrote:
Hi= ,

The https://www.freebsd.org/releases/14.0R/schedule/= says CURRENT/14 's KBI is froze
and new features should be avoided.

I'm working on https://reviews.freebsd.org/D39638 (sysctl(= 9): Enable vnet sysctl variables be loader tunable)
and I think it is new feature, but not quite sure whether the KBI changed.<= br>
So,

1. Is it a KBI change ?

IMHO, It's = a KPI change, not a KBI breakage. So from that perspective, it's OK.
=C2=A0
2. It is a simple change ( while so far as I know currently only tested by = myself on x86 and qemu riscv ), can
it catch up with 14 ?

That I'm less sur= e of. I think it's good, but I'm gun shy=C2=A0about approving / com= mitting vnet things. The review suggests,
though, there's at = least some consensus for having this in the tree.
=C2=A0
Best regards,
Zhenlei


--000000000000130bde06038605aa-- From nobody Tue Aug 22 18:24:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVd6G3kMZz4rDrX for ; Tue, 22 Aug 2023 18:24:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVd6D4Sgqz3QkY for ; Tue, 22 Aug 2023 18:24:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=pK0OCuEi; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692728658; bh=6dLtdxIx1FV+bYeMYsCQIZSgbgzUJPb4BrLA/Z6ny9w=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=pK0OCuEi/lZgrRDg5u6VyeJPjjstTyc0Y3hSKrwmZKHDeghewWB9TnlGRRtcMiHHFKRJ/nh7EzXF2WbeuUD7cQc2TfGYqSaGfcngTAIG8rQ1dJ3OhROONeKd+9GPkqot+DMhW5uRu0IBDPuepQRBOXAzYkYKxncM1/ARoT5MyDGvKHJtFg9uSWpNyuzkfhZzqbHzqd90HZ+B9mVWAVOA6xmJUoVxnulcNjEtrt+KXYW99cYvNrrnXvAzcrKrS8ck+wUzEgFYCfXGf+R/4OvM9Zr8He6VkzGPYBK6godGeNtKTpm3deZLhB+bFmU+OQxmu+qnJ9aagsdHu+aO9jnySA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692728658; bh=1a3RdOUFXCuJRGajKYpNOKo5cvgLZlX0eaQM8Iw0Iz5=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=SeYpbS2DQkqWjJomYBa3X3mauh4EoRLVp2dxkZoUI0N5aNNY+WIPr7fUpPWKg+jPaMAGZFd+xjAVbzcBZhb6cy/dPosicjGraQd1sV4P4TKjaSUZ38VjQknbDeWNtth8mQZE9fEu9OKejCsQ63ClcpQleEiFp8bPekeLRT6fdzBgyU3Yl8biMbFD0LK+LguJBFVKmBbmSwAJEUwNu9cqGOT1JVoMS5W0FOweEKqdf/v7+HE2eMLAKGglV7xoRNqNNSDJaUMxSgX3XovR8amwti3GhDB80l4QUhcYMuTpGb/0R2oRj3Q5vp0j5Z9apI5Ey7zyIWQrWS2W45YWv63WeA== X-YMail-OSG: B02N1HwVM1mh1cvRVMLMdOfmWs5gwkG8fGhjWL3mIyiodVODJyQTCnuoPTKIQ66 4o1C4VZojLR4bmz1EVft8tmU.v6XbwDHMv0pKo6kp1TeoJq6tHYIp28TqyO5HapH_1v5tSKeRJjJ I2BcOl81vaJomjQDXQw9w8P.U2XgzMCHTK9T_0TgxppkPLgsoUmFr3KrH1XeVURIXOjfHRvPB32N iU2vqkvButyQtqu1KATWqljEqolapVOUfLpTpO3O9_AsSTmvltfJpnB9UlTDBulEKdJ4K64.oE96 LyEikKMJ63rYiOnRbhSwq1HjLzZs4JC2Lt7bqBVb_eitaqKv.gpEnCzFRLBE65G2ZDslyb48S8Ke a8W8cdjqNz_AVJll53bFfr8a5b8K9TZz2hziQZm_DM_nDq6q1S0ZSdddrbLbIIPdZJKZ7YfGRWwc uRE7f.EvITPq_DH29J4vPQE6HGj5ygOvUOy57wsrcZvBzUN8blIjQaOcfhGE0OBZNt2jdkzq9TN. 8_Hk60Q6ka2I9xoGfo5clsXHTtIF4JhYZE4YvUNjbrpHgxfy_BfQxYXwlsrtGsZ9wV1ZNY4gOwMG sPi_aMWUCiZhNL47yyeMHPcn7y6bvC2oTMImLiMMXDCJ.44ccm6FXDKHq1riOrHSjF.bA4RwVvg6 iS6vq2Eb_ONEid_jbgeZK9tfAE3YcdrkHq4FMmPdWPf7axIbWXYHFEyPd852In6HUjSNFW3XRkAW Gk6UH3T0mEIJkjyXhfqm2qd0zprVQfvZ56HK5zAlBsAHQramzpNEFKXgqH4kgMftIzbmawOTsxK6 cH4zDmmwwsBiQDg2ACIhi9krAYbVguiUrvQguu959WeRG.ZCO9DVlK3SH1St0k3A1xZcpvXpqUz_ sJmQm9GrkA8Tki5qc_osU78_gtG1EzGRN9GFcD1h2wOZWXEPOR4Lc9fz6ZVe60QF9pqfGLlsn1fh tL5UuxbRL_jV6JvQuXIjnNwxfi97xnAQi1vNyCGVIRQrfsnX1tgBPH1jwTWUqKtRqrkmTZr1lZLP YDqqtyRgGBQ0u0Sh87Ss7JOw9X5L2h8YSp3MjC0CGJTttTnNEZHrAkiwrLu3HocNCXZ7Oqbgf4kE lSuF13AxHCtJNQKQQowwAej18q6YPZ6ytgtWk3LOjcokJ5aK9sn6Fz6j7bWUrMGX_DjqkU9TaTGb PX.y7ftd9J7vrqnrykV_mxXPVD.rxuCSUomCwtqPRJGBvwNn2ho8gohzEf2_v.RxmoWU0MnOgUGK jdUhUuz4XCJZjN4bkneYJ9dc9XIGpcmeCKEvzJz_8fHSc6uunrV1S9sVrdo5CZK29ihfa9yzxyKM JJtRGkxFAQDmJdkeq8o5VolHibM3giGZeY3LOezVyDUVtO1qUxRaI75ccglXYkXHZ7W2YZCwQIGm iZkKsooO85zbJ4ncSOLrfRbe0CwvXIsIlIMhk1b6_IGN.b3BOhlAMG95Xlc7p9KAM03OHhW2p01m 6JNyPTsObVSX9Ot9K6455XIKJeEoxDWqDZx8TRLpz_o1dyvvJ7RqSIzex7ekbzsjMDNRJwrZpMDl a88AAL2ugNVwbS9b.YEMZvjD_SrYtiIV6OMk9rssow_rz0YNU_I3tlhnDWHW0sSxaIL6up6prLSW mkuGLTyzadbw4MgOp98uTUG7hlnZWA_BuNfsJoLBYAyZBA8K8QqI1KReW5.e9XCrG_nsNJtFahHG F40kpyKkWoP8u26I4iLnBtWt_I2QCl.0K6prPp14acHneA8Qt.d7UO6bcaVTaPYE3479C6XjJSCK DlXJHB.U2eruAzBq_de4bo9DKmpywnbL04vgrzofVMp7GL0cgbXrA9g_jDXrEpK30nQmvOu7jHfo sJlSjBhzrmOafvT9gHbpDVSH4E7DciFdOic2PcxDJU040HRzsh2rHxhsvTVQ5z4Fcs9sQsdNvuM6 Xkj9xg9YmjHbNn.1hGMJvcsR49Qi8OYobQCxN5_wXcJwp2Y3D16acPMTaP.Gu7XcAdo2txRQ8VNA fJcS7RcwkWrdrzKXNSByl3sofvYxlf47Kf.izcF0jPAVyyN4WxwAfW2TMAeJ.B0g1lyKHvgFbq9E NH0mCCuWh5CWUptW66t24xyPpxT7xqUtenR5dcMKS7uAD1mRf6Df91O5A.pi5Kze7yzlALPaStFE 7SNxD0DJ_kJNpw7oiZUY.FixbaUgSr2TpQEsfhc9uuCaWJavDEWCvPUaQdUp0I8rxag_ekQpBR9D ptmaezBj0nCHh__5V1m0G.KfVLkLGmaQ.hrS94kVcICVOLY7DYMbQUa4ALV819RS7q1DIpD4- X-Sonic-MF: X-Sonic-ID: d7414c24-6a7b-494c-9860-274f8e25d276 Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Tue, 22 Aug 2023 18:24:18 +0000 Received: by hermes--production-bf1-865889d799-7x4p2 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 142457281a2c78487630c2b1feea3719; Tue, 22 Aug 2023 18:24:12 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 Message-Id: <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> Date: Tue, 22 Aug 2023 11:24:00 -0700 To: Alexander Motin , Current FreeBSD X-Mailer: Apple Mail (2.3731.700.6) References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RVd6D4Sgqz3QkY Alexander Motin wrote on Date: Tue, 22 Aug 2023 16:18:12 UTC : > I am waiting for final test results from George Wilson and then will=20= > request quick merge of both to zfs-2.2-release branch. Unfortunately=20= > there are still not many reviewers for the PR, since the code is not=20= > trivial, but at least with the test reports Brian Behlendorf and Mark=20= > Maybee seem to be OK to merge the two PRs into 2.2. If somebody else=20= > have tested and/or reviewed the PR, you may comment on it. I had written to the list that when I tried to test the system doing poudriere builds (initially with your patches) using USE_TMPFS=3Dno so that zfs had to deal with all the file I/O, I instead got only one builder that ended up active, the others never reaching "Builder started": [00:01:34] [01] [00:00:00] Builder starting [00:01:57] [01] [00:00:23] Builder started [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: Success [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 [00:03:21] [02] [00:00:00] Builder starting [00:03:21] [03] [00:00:00] Builder starting [00:03:21] [04] [00:00:00] Builder starting [00:03:21] [05] [00:00:00] Builder starting [00:03:21] [06] [00:00:00] Builder starting [00:03:21] [07] [00:00:00] Builder starting [00:03:22] [08] [00:00:00] Builder starting [00:03:22] [09] [00:00:00] Builder starting [00:03:22] [10] [00:00:00] Builder starting [00:03:22] [11] [00:00:00] Builder starting [00:03:22] [12] [00:00:00] Builder starting [00:03:22] [13] [00:00:00] Builder starting [00:03:22] [14] [00:00:00] Builder starting [00:03:22] [15] [00:00:00] Builder starting [00:03:22] [16] [00:00:00] Builder starting [00:03:22] [17] [00:00:00] Builder starting [00:03:22] [18] [00:00:00] Builder starting [00:03:22] [19] [00:00:00] Builder starting [00:03:22] [20] [00:00:00] Builder starting [00:03:22] [21] [00:00:00] Builder starting [00:03:22] [22] [00:00:00] Builder starting [00:03:22] [23] [00:00:00] Builder starting [00:03:22] [24] [00:00:00] Builder starting [00:03:22] [25] [00:00:00] Builder starting [00:03:22] [26] [00:00:00] Builder starting [00:03:22] [27] [00:00:00] Builder starting [00:03:22] [28] [00:00:00] Builder starting [00:03:22] [29] [00:00:00] Builder starting [00:03:22] [30] [00:00:00] Builder starting [00:03:22] [31] [00:00:00] Builder starting [00:03:22] [32] [00:00:00] Builder starting [00:03:30] [01] [00:00:09] Finished print/indexinfo | indexinfo-0.3.1: = Success [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 . . . Top was showing lots of "vlruwk" for the cpdup's. For example: . . . 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 . . . But those processes did show CPU?? on occasion, as well as *vnode less often. None of the cpdup's was stuck in Removing your patches did not change the behavior. So far I've not seen any similar reports to these resuls that I got the ThreadRipper 1950X that I have access to. I normally use USE_TMPFS=3Dall but that hides the problem and is why I've no clue when the behavior would have started if I'd been using USE_TMPFS=3Dno instead. I never got so far as testing for the kinds of reports I've seen about the deadlock issue. No one has commented one what I reported or if they have done any USE_TMPFS=3Dno style of testing. (I also use ALLOW_MAKE_JOBS=3Dyes .) The ZFS context is a simple single partition context. I use ZFS in order to use bectl BE's, not other reasons. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Tue Aug 22 19:38:23 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVfln5f4Lz4qLPC for ; Tue, 22 Aug 2023 19:38:29 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVfln0nNFz3Zf9 for ; Tue, 22 Aug 2023 19:38:29 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=eFJSuFLG; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::529 as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x529.google.com with SMTP id 4fb4d7f45d1cf-52557cc5e7bso6063922a12.0 for ; Tue, 22 Aug 2023 12:38:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692733105; x=1693337905; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=4aM0uq3KiSrjJHJmGtBzTAZNfpAakhj1o0bbyg2Xqog=; b=eFJSuFLGOXLFPSyRySapKVDVuApQYQ5mptZtb+GHuyJb5fzKVsLo1P5ks1yRcCwQI5 E+3d/OMDarWFU++FGmrJA07ka3E8xk6c0p+DF5pIgXebn8gqQHbtDuGrqz0fIToRpEwi PP2Ga2ZZ8M/9W1dwsxT7OZDi8ys7g87iIo4R4fS1F8RI/YHx+aE2Xron/dGulF8z6sNk 2Ksz6QDoEcXlKenRVUSNTuDXDbh5JQrjwYZAajVZQt0vxEBAWwqWUlGDbhDTfcIaSB+e ppEo3qJjIdjfHW+WNM7D/dKHnIkGzYoL0ZKqDAUkoP05Bu1jXZ3dzPno9KeGqgczwpGx 4vzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692733105; x=1693337905; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4aM0uq3KiSrjJHJmGtBzTAZNfpAakhj1o0bbyg2Xqog=; b=bzuZubz97BzcNulv9zBtegcDMJ8gai3DyTdUsnoiiEl3uF4aDzPVSrktvGbxLuZIT6 WnihHa0x7pjMhl1owUxSVBacnrj5Dc5WdSJs5X1pup5SqSPXuWrOBu9ZFfJ1afjlwkRC kWHug1hNq3zY6K0eHbgtT8Fz5M8UWvUbQ0PQR8gboB7bikOZvYNVUKEO4HmQEvM+l/ns /1wNyDSqz7PQvR79YnrZKxwKt0PH7P+Z8/TXlgNBzQgW+6+VIC1pLz48iLqwq0OOVXjQ 8jRvNk5sISllfkua2Z+dKp2t73hpIVbDdEGEb4Y4bTCINz6djhD+nWXPl9S5z1+Ndwwr feLw== X-Gm-Message-State: AOJu0YytjPX7cu9NAWeOWN/AbYZS43fQVZ8s7PiBVU0l2E2MBXyUyac5 StE4Vj9jKb+1YjfOE8VEv2P2SyT311I= X-Google-Smtp-Source: AGHT+IFxIkpNttvhi1pMAHOpK0Dqwra9hFygFKPm95uOzi1w5XhBjIgcTbp4L04D081pBY33iVeK7A== X-Received: by 2002:aa7:d3d2:0:b0:522:ab77:36b7 with SMTP id o18-20020aa7d3d2000000b00522ab7736b7mr9184456edr.4.1692733105187; Tue, 22 Aug 2023 12:38:25 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id v13-20020aa7d9cd000000b0052568bf9411sm8009565eds.68.2023.08.22.12.38.24 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Aug 2023 12:38:24 -0700 (PDT) Message-ID: <28617034-8228-fee6-153b-a5a1da31ffcd@gmail.com> Date: Tue, 22 Aug 2023 20:38:23 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: =?UTF-8?Q?273297_=e2=80=93_BOOT_time_panic_after_preloading_TSLOG_d?= =?UTF-8?Q?ata_=28was=3a_7addfafe73e0_early_boot_kernel_panics=29?= Content-Language: en-US To: FreeBSD CURRENT References: <5d7149c5-0351-542b-c868-a24b055880a0@gmail.com> From: Graham Perrin In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::529:from]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RVfln0nNFz3Zf9 Thanks, people. I couldn't attach a photograph to the opening post (entire emails, not just attachments, disappear in the ether). Instead, a photo here: From nobody Tue Aug 22 19:57:50 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVgB66qxTz4qMm5 for ; Tue, 22 Aug 2023 19:57:50 +0000 (UTC) (envelope-from pstef@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVgB66HqDz3csc; Tue, 22 Aug 2023 19:57:50 +0000 (UTC) (envelope-from pstef@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692734270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xGgMoZW3DQvZIgO0LeDd2JrFmuFsWuT2Obf7amQbpbo=; b=ACZzzyZUYxLcDDypltuLwstMpuXE5ajjKVyD+7n9W/iicCts8HxlMF0y6W9kqbklXLSUS/ 0rHkY5K3GnjF1ltHLxww1TaxITWfIOuiGYvaGRRk6OhTLXjy/1ORzqO3aJPhn3fUXy2UKI ZiKnKPfmh4Znr4VafKfoT7FOri4Hnssrz7I0KvLyLawqLtnLXJIFJD+xBgf6hyBUpADTYa OHjPfnbLKWZ6SGWFbv7Wnhij+40c0tq4Zfwy6LYDhCLmQGBLOauAhcKYidroBpUNPT2LUj tlAdZwD05xlJN1WkldAOOilEhEvp2usA4vSJpjXyKPgtAwT3g4EJvd+Cin311A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692734270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xGgMoZW3DQvZIgO0LeDd2JrFmuFsWuT2Obf7amQbpbo=; b=bJf3+s/x2b2tia0AqqR3tUhXjwto9OdXtU8NJhcnm2JbiVQ8oIiEmlAyf+i/zoBx5PRdTD CwNBqGRgV/uwD5ALNss/B3VnFjLDgIonS2BFSLrzMd+fuoJVcJp5aJoiY85Qd3wz4LAAzm dfF/F9ppJ9DeTXy1BZk5i+AuvJ2r/d07sypOC9Qi38GSJgQYZe/kd57RzFcCQHdS/rfpRo JtB4rY1eWiKBcmtJTJyyfGRmVx2Nfzv1GX38hjiBIJqwiKnrAi+Dc3ChSK0IAJN50DoSIf 15hJ9pDZBjJilcTJ7kCJADmBiL5MyH154lJph94eLo295d5evmrwqy4ck7dj+Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692734270; a=rsa-sha256; cv=none; b=QBCmj0CqXsyHCP9cNkXhPXqzkVdWluZXxxdXi6Qz30Fl6sadibYUaLcXrq/lFcncIYzokU qHuqAi/7ougK3axPzoZqUUY2X+NBbqAvddnLGZGzBiuH4qzNbJT1HYLA5xJU9E9mNy1Ixd Cvao0sFHo97aP4C+kKNaKlB4tsP/gDXZwiCmN23vzpYrVFdzIpt/r14UdboYJVHkUO43Is rdyU3oEXZ9Mdmf8L4P435QnukZ1pouBaeu7UsKhG38l1k50EACrqzlzOHlnywH/8vXm7W2 Y4vkO9cLfVLzX/vMoAtYSuBf+NB7bg0VllvwIfhSocBGAMPi6VDEJf99p74JPw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: by freefall.freebsd.org (Postfix, from userid 1403) id C79DB1FC15; Tue, 22 Aug 2023 19:57:50 +0000 (UTC) Date: Tue, 22 Aug 2023 19:57:50 +0000 From: "Piotr P. Stefaniak" To: Jamie Landeg-Jones Cc: freebsd-current@freebsd.org Subject: Re: ps(1) bugs and problems Message-ID: References: <202307282307.36SN7b7v026284@donotpassgo.dyslexicfish.net> <202308111132.37BBW23A064898@donotpassgo.dyslexicfish.net> <202308151228.37FCS78i063976@donotpassgo.dyslexicfish.net> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <202308151228.37FCS78i063976@donotpassgo.dyslexicfish.net> On 2023-08-15 13:28:07, Jamie Landeg-Jones wrote: >> The old -d and the new -D'$^' would be the best in that -d would go back >> to what it was and -D would provide the much needed feature in two >> variants (possibly more in the future, if needed) while only taking one >> option-letter. The only problem is that it looks ugly. > >I see why you chose "$" and "^", but wouldn't it look more friendly if >you instead used "up" and "down" or "A" and "D" or "forwards" and "backwards", >for example? Thank you, in the phab review I went with -D up, -D down, and -D both if you don't want to type -D twice. I don't think we need -D none to clear the flags so it's currently not supported, but it'd be easy to implement if we wanted to. I also tried to update the manual page, but it may require some wordsmith. Piotr From nobody Wed Aug 23 02:37:53 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVr3v0LRGz4qpqB for ; Wed, 23 Aug 2023 02:38:03 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [IPv6:2001:19f0:7400:8808:123::1]) by mx1.freebsd.org (Postfix) with ESMTP id 4RVr3s5Tj0z3LVT for ; Wed, 23 Aug 2023 02:38:01 +0000 (UTC) (envelope-from jamie@catflap.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of jamie@catflap.org designates 2001:19f0:7400:8808:123::1 as permitted sender) smtp.mailfrom=jamie@catflap.org; dmarc=pass (policy=none) header.from=catflap.org X-Catflap-Envelope-From: X-Catflap-Envelope-To: freebsd-current@FreeBSD.org Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [209.250.224.51]) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5) with ESMTP id 37N2brQV029908; Wed, 23 Aug 2023 03:37:53 +0100 (BST) (envelope-from jamie@donotpassgo.dyslexicfish.net) Received: (from jamie@localhost) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5/Submit) id 37N2brSh029907; Wed, 23 Aug 2023 03:37:53 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <202308230237.37N2brSh029907@donotpassgo.dyslexicfish.net> Date: Wed, 23 Aug 2023 03:37:53 +0100 Organization: Dyslexic Fish To: grahamperrin@gmail.com, freebsd-current@FreeBSD.org Subject: Re: etcupdate -B, /.cshrc and /.profile References: In-Reply-To: User-Agent: Heirloom mailx 12.4 7/29/08 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (donotpassgo.dyslexicfish.net [209.250.224.51]); Wed, 23 Aug 2023 03:37:53 +0100 (BST) X-Spamd-Result: default: False [-3.08 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[catflap.org,none]; NEURAL_HAM_SHORT(-0.38)[-0.383]; R_SPF_ALLOW(-0.20)[+mx:dyslexicfish.net]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; FREEMAIL_TO(0.00)[gmail.com,FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@FreeBSD.org]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:7400::/38, country:US]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[jamie]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RVr3s5Tj0z3LVT Graham Perrin wrote: > If I recall correctly, a few hours ago etcupdate -B resulted in removal > of two files: > > /.cshrc > /.profile > > Is this degree of checking/removal a novelty? > > (I can't recall the files' contents, or when I created them. I guess > that I carelessly created them as dot files months ago without realising > that I wasn't at ~, I don't mourn their loss.) For as long as I can remember, (as far back as FreeBSD 2.2.7 in 1998) all FreeBSD installs have /.cshrc and /profile as hardlinks to /root/.cshrc and /root/.profile . Removing them both is one of the first things I do when I install a new system from install-media. If etcupdate is now removing them, maybe there has been an update to the src distribution / mntree so that this historical weirdness has finally been removed? If you have a /root/.cshrc and /root/.profile, try doing an ls -c on them to see if their changed-date is when you did the etcupdate. Jamie From nobody Wed Aug 23 03:06:49 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVrjN5lcDz4qr8v for ; Wed, 23 Aug 2023 03:07:04 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail2.karels.net (mail2.karels.net [3.19.118.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "freebsd", Issuer "freebsd" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVrjN3tptz3Q7S for ; Wed, 23 Aug 2023 03:07:04 +0000 (UTC) (envelope-from mike@karels.net) Authentication-Results: mx1.freebsd.org; none Received: from mail2.karels.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by mail2.karels.net (8.17.1/8.17.1) with ESMTP id 37N36oxg016736; Tue, 22 Aug 2023 22:06:50 -0500 (CDT) (envelope-from mike@karels.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=karels.net; s=mail2; t=1692760010; bh=FKxJIeJwJydt9v2kxFdkoAYK012TcUKxG5rjr/o3UJE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=qjQcU7oF8tiYsFXYikzyiNDJNwOxDtcOltl3VSgOWHL25d2SPm21wd2qFIbAI6THU pnoxxIwaxEPAgvAW3pyJOiY/RHxZHFQ0e9cPH3BX2RDyhLHi4LX/FzFPD7iHpEEk2g vyjjg9Epg/ALHwAjBmkhASZgqZxVmBtKVkoldtkvhzghQCqKOo3HECPDVCkhyDJ/f+ 2oOdkXxW8gxHFeHUT8SRVQmA83SCVOVbH8ZEQgEAkB1C1LSUr/APb1z6USCEwYmv6Q u6lIO1KRboD7QdFT0FSdDlls0mPGidw94cc+qLkh4d4NjL0PecjsN9E5MG3c4W12/H ZlntWlnIvcAAQ== Received: from [10.0.2.130] ([73.62.165.147]) by mail2.karels.net with ESMTPSA id Z65WEMp35WReQQAAs/W3XQ (envelope-from ); Tue, 22 Aug 2023 22:06:50 -0500 From: Mike Karels To: Jamie Landeg-Jones Cc: grahamperrin@gmail.com, freebsd-current@FreeBSD.org Subject: Re: etcupdate -B, /.cshrc and /.profile Date: Tue, 22 Aug 2023 22:06:49 -0500 X-Mailer: MailMate (1.14r5964) Message-ID: <563EEA5B-0624-4EB3-BCB3-DD8A8F57C3A9@karels.net> In-Reply-To: <202308230237.37N2brSh029907@donotpassgo.dyslexicfish.net> References: <202308230237.37N2brSh029907@donotpassgo.dyslexicfish.net> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Rspamd-Queue-Id: 4RVrjN3tptz3Q7S X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16509, ipnet:3.16.0.0/14, country:US] On 22 Aug 2023, at 21:37, Jamie Landeg-Jones wrote: > Graham Perrin wrote: > >> If I recall correctly, a few hours ago etcupdate -B resulted in removal >> of two files: >> >> /.cshrc >> /.profile >> >> Is this degree of checking/removal a novelty? >> >> (I can't recall the files' contents, or when I created them. I guess >> that I carelessly created them as dot files months ago without realising >> that I wasn't at ~, I don't mourn their loss.) > > For as long as I can remember, (as far back as FreeBSD 2.2.7 in 1998) all > FreeBSD installs have /.cshrc and /profile as hardlinks to /root/.cshrc and > /root/.profile . Both sets have been present since 4.3-Reno in 1990, although they were apparently not links. > Removing them both is one of the first things I do when I install a new > system from install-media. Why? > If etcupdate is now removing them, maybe there has been an update to the src > distribution / mntree so that this historical weirdness has finally been removed? It is not weird. /.profile and /.cshrc are used in single-user mode, the ones in /root are used for root logins. > If you have a /root/.cshrc and /root/.profile, try doing an ls -c on them to > see if their changed-date is when you did the etcupdate. They were modified by the removal of $FreeBSD$ a few days ago at the next etcupdate. Mike From nobody Wed Aug 23 03:44:17 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVsXL6yZJz4qtQk for ; Wed, 23 Aug 2023 03:44:18 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [IPv6:2001:19f0:7400:8808:123::1]) by mx1.freebsd.org (Postfix) with ESMTP id 4RVsXL5c0jz3T98 for ; Wed, 23 Aug 2023 03:44:18 +0000 (UTC) (envelope-from jamie@catflap.org) Authentication-Results: mx1.freebsd.org; none X-Catflap-Envelope-From: Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [209.250.224.51]) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5) with ESMTP id 37N3iHto032843; Wed, 23 Aug 2023 04:44:17 +0100 (BST) (envelope-from jamie@donotpassgo.dyslexicfish.net) Received: (from jamie@localhost) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5/Submit) id 37N3iHJF032842; Wed, 23 Aug 2023 04:44:17 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <202308230344.37N3iHJF032842@donotpassgo.dyslexicfish.net> Date: Wed, 23 Aug 2023 04:44:17 +0100 Organization: Dyslexic Fish To: mike@karels.net, jamie@catflap.org Cc: grahamperrin@gmail.com, freebsd-current@FreeBSD.org Subject: Re: etcupdate -B, /.cshrc and /.profile References: <202308230237.37N2brSh029907@donotpassgo.dyslexicfish.net> <563EEA5B-0624-4EB3-BCB3-DD8A8F57C3A9@karels.net> In-Reply-To: <563EEA5B-0624-4EB3-BCB3-DD8A8F57C3A9@karels.net> User-Agent: Heirloom mailx 12.4 7/29/08 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (donotpassgo.dyslexicfish.net [209.250.224.51]); Wed, 23 Aug 2023 04:44:18 +0100 (BST) X-Rspamd-Queue-Id: 4RVsXL5c0jz3T98 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:7400::/38, country:US] Mike Karels wrote: > Both sets have been present since 4.3-Reno in 1990, although they were > apparently not links. Well before my time! > > > Removing them both is one of the first things I do when I install a new > > system from install-media. > > Why? Because I don't use them > > If etcupdate is now removing them, maybe there has been an update to the src > > distribution / mntree so that this historical weirdness has finally been removed? > > It is not weird. /.profile and /.cshrc are used in single-user mode, the ones > in /root are used for root logins. Ah yeah, that would make sense, I guess. I remember my first unix systems, the default home for the root login was / so there was no /root duplication of those files. > > If you have a /root/.cshrc and /root/.profile, try doing an ls -c on them to > > see if their changed-date is when you did the etcupdate. > > They were modified by the removal of $FreeBSD$ a few days ago at the next > etcupdate. Ahhhh. That idea won't work then! Cheers From nobody Wed Aug 23 03:44:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVsXg60Lcz4qtDN for ; Wed, 23 Aug 2023 03:44:35 +0000 (UTC) (envelope-from madis555@hot.ee) Received: from SMTPOUT05.DKA.mailcore.net (smtpout05.dka.mailcore.net [81.7.169.178]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtpout05.dka.mailcore.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RVsXg2lNzz3V2x for ; Wed, 23 Aug 2023 03:44:35 +0000 (UTC) (envelope-from madis555@hot.ee) Authentication-Results: mx1.freebsd.org; none Received: from SMTP.DKA.mailcore.net (unknown [10.1.0.52]) by SMTPOUT05.DKA.mailcore.net (Postfix) with ESMTP id D8876E01F4; Wed, 23 Aug 2023 05:44:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=online.ee; s=mailcore; t=1692762270; bh=jfEFS3Au+0+B15Ix/rOGQfg7Q/Dwjoapdo1J3hu0ozM=; h=Date:From:To:CC:Subject:In-Reply-To:References:From; b=tZ4TppHSnw72skeCmVgZ70u5TXqwSQvRH68AyOoGDMegLOfytaOQ7nQwX0VqpKqyI TeAUEHT5EJBys27X0JBZ6nzdZ1ugI5lOLHu5mZSeVtOAnKQJdlYCMFFnSFP9PfIDu/ kp7jUsYe1pt0+WUR7vGBStl5wqMzCND3kKqD8lsVhe81pVkFOzuaobQcHIkTduC0kR jj6a++WmfsV9Rj1HE91H8THIeoZvW5h5h+Du7dw1qxRTHmW95tY8SeryB11uqC56xU hg0ez5PXo+I7ZWgi1++nS7gYFFGAmutqmnISrsrzg+7DOL4Tj9MsgGpdSFlTeoNypj 2XadwBUfvnP3Q== Received: from [127.0.0.1] (111-190-35-213.dyn.estpak.ee [213.35.190.111]) by SMTP.DKA.mailcore.net (Postfix) with ESMTPSA id EE60A400FE; Wed, 23 Aug 2023 05:44:28 +0200 (CEST) Date: Wed, 23 Aug 2023 06:44:29 +0300 From: Sulev-Madis Silber To: freebsd-current@freebsd.org, Mike Karels , Jamie Landeg-Jones CC: grahamperrin@gmail.com, freebsd-current@FreeBSD.org Subject: Re: etcupdate -B, /.cshrc and /.profile User-Agent: K-9 Mail for Android In-Reply-To: <563EEA5B-0624-4EB3-BCB3-DD8A8F57C3A9@karels.net> References: <202308230237.37N2brSh029907@donotpassgo.dyslexicfish.net> <563EEA5B-0624-4EB3-BCB3-DD8A8F57C3A9@karels.net> Message-ID: <37546B83-989D-48B7-A26C-B5B38DF9F5C6@hot.ee> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4RVsXg2lNzz3V2x X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:47292, ipnet:81.7.128.0/18, country:DK] on why removing those, i for example only use /etc/csh=2Ecshrc so i don't n= eed others From nobody Wed Aug 23 03:46:21 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RVsZj6nlFz4qwb4 for ; Wed, 23 Aug 2023 03:46:21 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [IPv6:2001:19f0:7400:8808:123::1]) by mx1.freebsd.org (Postfix) with ESMTP id 4RVsZj5ZHJz3Wqt for ; Wed, 23 Aug 2023 03:46:21 +0000 (UTC) (envelope-from jamie@catflap.org) Authentication-Results: mx1.freebsd.org; none X-Catflap-Envelope-From: Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [209.250.224.51]) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5) with ESMTP id 37N3kLJp032917; Wed, 23 Aug 2023 04:46:21 +0100 (BST) (envelope-from jamie@donotpassgo.dyslexicfish.net) Received: (from jamie@localhost) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5/Submit) id 37N3kLvA032916; Wed, 23 Aug 2023 04:46:21 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <202308230346.37N3kLvA032916@donotpassgo.dyslexicfish.net> Date: Wed, 23 Aug 2023 04:46:21 +0100 Organization: Dyslexic Fish To: mike@karels.net, madis555@hot.ee, jamie@catflap.org, freebsd-current@freebsd.org Cc: grahamperrin@gmail.com, freebsd-current@freebsd.org Subject: Re: etcupdate -B, /.cshrc and /.profile References: <202308230237.37N2brSh029907@donotpassgo.dyslexicfish.net> <563EEA5B-0624-4EB3-BCB3-DD8A8F57C3A9@karels.net> <37546B83-989D-48B7-A26C-B5B38DF9F5C6@hot.ee> In-Reply-To: <37546B83-989D-48B7-A26C-B5B38DF9F5C6@hot.ee> User-Agent: Heirloom mailx 12.4 7/29/08 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (donotpassgo.dyslexicfish.net [209.250.224.51]); Wed, 23 Aug 2023 04:46:21 +0100 (BST) X-Rspamd-Queue-Id: 4RVsZj5ZHJz3Wqt X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20473, ipnet:2001:19f0:7400::/38, country:US] Sulev-Madis Silber wrote: > on why removing those, i for example only use /etc/csh.cshrc so i don't need others Exactly the same here! From nobody Wed Aug 23 18:40:49 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWFRN4pTTz4rDjS for ; Wed, 23 Aug 2023 18:41:20 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWFRN0WpNz4kkL for ; Wed, 23 Aug 2023 18:41:20 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=crrKO+8I; spf=pass (mx1.freebsd.org: domain of mavbsd@gmail.com designates 2607:f8b0:4864:20::32e as permitted sender) smtp.mailfrom=mavbsd@gmail.com; dmarc=none Received: by mail-ot1-x32e.google.com with SMTP id 46e09a7af769-6bcd4b5ebbaso4409945a34.1 for ; Wed, 23 Aug 2023 11:41:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692816079; x=1693420879; h=content-transfer-encoding:in-reply-to:subject:from:references:to :content-language:user-agent:mime-version:date:message-id:sender :from:to:cc:subject:date:message-id:reply-to; bh=WEDuVo/nTnBb/NjtYKhAGXQdHgGMu8fauo6swjY36KY=; b=crrKO+8IlSoQYQB8TErKb5OHNdBq4oDtpiK2Dj0vZXeTGFXagP/bN5XTUV/sD733Ed JyQDqgVNgra+XJtCyGPmuTHN1nbWNntX0d3F03Q3ebE4Pnch/ppQ71x0p0wUAcB3M5LU /8I5293WdvkonHamojzguiJBj03NSvbIM0PDT50UU0Q95Qdd68EyAnCybo6sbU6Gq09c f3i5B6jZ+oOP5DeunR/XA8GQwTBM6Inqioqteu8WP36i1rmZMhAuSDjHUuTBm2jwcutz +pJByP4hXEvZ/+ryLharO/bDqxx99wrKXyJ3b+w1fEF0X7Vd55QdnmYf1H0ycPpfwMr1 TGGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692816079; x=1693420879; h=content-transfer-encoding:in-reply-to:subject:from:references:to :content-language:user-agent:mime-version:date:message-id:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WEDuVo/nTnBb/NjtYKhAGXQdHgGMu8fauo6swjY36KY=; b=WzhLTWUf7NdFgNHl69hbfaAWHuvqERB29YFxY8lBGlg52ZxxkWVIjfnxDVUw9gxhrB tfJ6ctJihWoJbtjYIDVcPAh9okvVeSsl+aZXDpUTX4FxNKXerO7+MfxfPTbtEu7YpDon lalSXCrhsRsAVzse+Ja1VN9t/p8gMZKBzqy4Ey6bsLhz6QgCrKBI97BMaxpQI324orHI Wwoq6atkdCTJ0ios4MXfFfkVbEUFwyXz4H0gVegRdHfD109nPWvgn3Pt0JDrrXoWgjLf 5ggeJRiyHCAcS/M+qqP2W3L12KA6oyYPYemNz3rUoyqBYmfvREB3nF5SgT3ip3c6+DZC /F6A== X-Gm-Message-State: AOJu0YxgoJrg01FbjxhNr5LInFZ5M2GsPgv19wE+05aqwwcZ0s4e5p+5 VvVN5rrXUcc4u+J03fy5boLE35lM7JA= X-Google-Smtp-Source: AGHT+IEYFtW8LeKCvMkcRdYqAKdkf5erwSGwUZ7T+mXClLlnw9Op64oVbDcFhfNtDnWLiChf4LtMDg== X-Received: by 2002:a05:6358:248b:b0:132:d42f:8e19 with SMTP id m11-20020a056358248b00b00132d42f8e19mr12863746rwc.31.1692816078751; Wed, 23 Aug 2023 11:41:18 -0700 (PDT) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id x137-20020a0dd58f000000b005837b48d16csm3471433ywd.84.2023.08.23.11.41.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 23 Aug 2023 11:41:18 -0700 (PDT) Message-ID: Date: Wed, 23 Aug 2023 14:40:49 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Mark Millard , Current FreeBSD References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> From: Alexander Motin Subject: Re: ZFS deadlock in 14 In-Reply-To: <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[mav@FreeBSD.org,mavbsd@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32e:from]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_NEQ_ENVFROM(0.00)[mav@FreeBSD.org,mavbsd@gmail.com]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FREEMAIL_TO(0.00)[yahoo.com,freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWFRN0WpNz4kkL On 22.08.2023 14:24, Mark Millard wrote: > Alexander Motin wrote on > Date: Tue, 22 Aug 2023 16:18:12 UTC : > >> I am waiting for final test results from George Wilson and then will >> request quick merge of both to zfs-2.2-release branch. Unfortunately >> there are still not many reviewers for the PR, since the code is not >> trivial, but at least with the test reports Brian Behlendorf and Mark >> Maybee seem to be OK to merge the two PRs into 2.2. If somebody else >> have tested and/or reviewed the PR, you may comment on it. > > I had written to the list that when I tried to test the system > doing poudriere builds (initially with your patches) using > USE_TMPFS=no so that zfs had to deal with all the file I/O, I > instead got only one builder that ended up active, the others > never reaching "Builder started": > Top was showing lots of "vlruwk" for the cpdup's. For example: > > . . . > 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 0.00% cpdup -i0 -o ref 32 > 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 0.01% cpdup -i0 -o ref 31 > 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 0.01% cpdup -i0 -o ref 30 > 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 0.01% cpdup -i0 -o ref 29 > 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 0.01% cpdup -i0 -o ref 28 > 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 0.00% cpdup -i0 -o ref 27 > . . . > > But those processes did show CPU?? on occasion, as well as > *vnode less often. None of the cpdup's was stuck in > > Removing your patches did not change the behavior. Mark, to me "vlruwk" looks like a limit on number of vnodes. I was not deep in that area at least recently, so somebody with more experience there could try to diagnose it. At very least it does not look related to the ZIL issue discussed in this thread, at least with the information provided, so I am not surprised that the mentioned patches do not affect it. -- Alexander Motin From nobody Wed Aug 23 20:37:09 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWJ1K3P1bz4rLdR for ; Wed, 23 Aug 2023 20:37:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-55.consmr.mail.gq1.yahoo.com (sonic307-55.consmr.mail.gq1.yahoo.com [98.137.64.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWJ1J4zy7z3Dvs for ; Wed, 23 Aug 2023 20:37:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692823043; bh=KH2isnGzwYUN8xH4wwYgL4cl9PnsMcCzBQdZjJ3OCbA=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=X1P7TVG2H1Gv4Hbf58aF3LUM41jaiEuWYdWX+WQcGkdwIdOHRblBz/BbGZS2419G7yTUo89wvM3PRXlm2fFjAUN47EIlUvOa/5a9iEpyM1YtxWafVhJOti1gtMBmsVB8WLDuBWrlofQ5EsAL/j1u6NdeqZdgt5GvIx1ouhJ8HXGxcZ/CF5b0kOaa90BTJjjJNgOQioYLGUSoJOagOiaDUI+5Ida3bv4Hf110Y93fXBrgGLw3MRfRi2TXAZ/wwYvAYiprzDXIXDmD1uJnXPmFTsN6Jt3vSgVxH77MtFt/h37BXJ7QwejoPOq50xJWwEPeFWeqeUtYbVW2cibK+KRiHA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692823043; bh=iSBBJEhn6q7sD5mlU+Ox/FypCuJW1Ycznvn/naYG5ay=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=EfXorG0gRtrm58fy87fPa2HKRJ0ciCkTl1AVmltfBeU0OQx43FGDeSY3Zyk+z05X4iaKwxQdMNS6IdXTKpSx9PXgxAFr1ymemzXDoG+QNe0Mfq2ulDmp9suNQJXrMClFdh5SPnoOHRQkGaPC4m4Os6tMB0Qr88Kh6vRU7Syl8c8vKJwpurBMnwBqoy6VCbZXNaSXhcklYwlGysoUGKx+ZhIK1LHqOcNktKUOQk1ZdezIeRYz1QVXIcCnGLHf1R624Ufwt8Wn+NJIUQr6bDOsuiTXCELq7h5eR0x31auDOF582hb8E50DaQeixvHa09muNjCIo028QvTq35lAcEt5Gg== X-YMail-OSG: 22w0o3MVM1lwK4FNacEsoqWbpW0GNjKKM2dwtYLc8tVHXbtBMUv1yqNdv9ZcKY7 A5iTN_zFog6SSbqu2bkyU4d3.MWobRFOp2c06VU.DhxWrYmG53OFSgaLUYTQW..Hof3SNwpgJ7ag Q8TF.VGU9T3re1ADMap1fzltbPLrjshH1h0E26OP.aq7VnWVG4Ms19sedptFxA7O7D3DrxBoqstu wA9VrUPOWcRI0dYzZDT0XBrVaZTFSIfKx1ryIal1ZoLic7waZuq6S5m0LfeoV0bCXBZY9zBUeObJ 8FIvRIUXAf00arjMsLo4RAq9M70pbT9rrpCNyNKwTmfaWficWVyfnzuv_901flfnxgAO934sAqWt MnFiTFM2BwfeJUmdewepgqj.wLnAt4yjUxMhiepKaL7GL0Q1vK8h4EnrR4N61uYUBhlvxx_A37h9 c4JpWwBXJVKNv3EOtyyZTuB.0dMznjbt0QADHEv4BuUb27G7XaNGtuzbY7QnfL9PzI.2sDphi.sB zee.T6JJUkupGCH9rxsRIUlC32YWcMgQC5LzFuqxhJeT5OTVmNF4QnqR_NCQ4wctigaBxLSlDy5r H1fsJXcp8YrrkI3kcbaxbRArLyvsiSWhqmB7pxn.D7vC.xjfWtz.RZ8D7nxfL_9YZYMsNLszHK_h JQfiozoRfxhEeQ2JjiikL7yGtZIOsvMqZ1J66HPN8aOo_emx3F9cwL_vCGtYlyc1Qp0Lzc5aFmv9 dvsHFknX5Sx7szJlhUt8osMJvyIs_GLHCnLQilxPWiZyUcsFOYc3Iw1oCLkNhHdAerV2lHAhyEes wP7uMn61flDm7hk8dVVMte3eL0Q3mzlqvvKXck94Soo_khiUluh0ka0I0JcLeJDZe1wXMSFn2AuT u0QNHyH3HWFT1tTBU8W6u9HnI75SJiH2.egEXh0NuI4AQ89TqcEMECO7gS8P6_iPfHRauHyBUk5u rzf_Cf3GXjsUeB2RoCIkDQWCfJ5c_5fpQiCNhbC2ovkc2rzHilQB.qMlms8GS31cY9LgF3DhpEo_ LgG62N3.TecuB8x4XPAwkTukWYSEl6hnr4ySGntPjIIG4P8sqjUHgpsiCSHQpTI6obe.Ehpke8dV 1mlkNLnGkeKUPnX_lF7z2uuP6mvz2Tz8TihpLlLh8GBlqqmjqRcxPrMPtSNMu8ZuiD9s16z831cQ CJRqVp7W_lSRAzppqoJeJ8sgMhVj4qE0y9i57CmwjSAR6dHko2L0C1BqgE9nJSekzBZHTd9J5HAA itxCIeuX4TteMpcSKLNP9EuUKWrRYKQLiCfXbwt99VZ1Ts1krUmXpHLRENqBovZnCjfchwPBdNw1 7byNkFQQkp49dDgzyu6wUsqbEKwsUYgMjWKJsw8qcTP9Ezsk4NHyNNOYDrxvX3mFoehXlYy8sARP zs1BuY3MXYjOoEYVHAHHKkjYuTTTeO1sifPK2SFI83YSxvTRB2tLLvuMKrSsJ_k5BC3WHqCx5Ryk sw0fOkVEDSAri9N3TgJ6XQehlPMNUp2N9mMKRvAepN3XzqC2fCWg1FUIGPkfwIUIWDPBYcYjoLJJ ySKqyPvqpcK5qmurcgpJJQtehvcSdLQn2ltMoRB0iUJe9w31MZcYK07BFq.tnARcsXtucFojvAOC C4Owl19p_cnkfPQiJaGL4W0ziAa6XZdjNnYisreJA2Yorc0jwmEpprA2ViOcZpSddN7bUvZsr6Z0 _4WzGGGHtJ.BtLzC_r0pOa3VnquByJaaiCpWvDwObt_OD.AzpGgTdoxE.ugPsZK4AwUyhRrTSIms E4spNYuSuyy7_NkGvz8nQMq_GYIvmnO8qUkB3EjqW_E6S4LK2Vl0sXIX0QFKxU_E2xrX5KRvTARh R.GW6D8_O.Plghp1JbWBTXPateAVRDQWZYGAcRpJFM1bLWJxavM6wjrxn0HFicwy0Lpu8onrBwGm jNL3DBTBC1vvXVRDnwxQKZ1Jv7psz_PLJ6xNZsY3p024acUjIegYxrBkKWzs6tF_TxL6SIQSoC63 peAl3e5LTeGtCTWECnjjpfVWAsgz2hGggxudmWofM3UO5JXlp72Pq.vj3rF85J.BwYJgxRZPuymi RjG16W2DQ1od0s3LnorjL2QkgfjkC8Zf5oc02OQpiPnrZjJBkzHtSQeXPK5HQU57yQsgfHtBw_.8 VKFD8VrZVPA2jwZaaFcu5nWSSsRFfdFGezOxYinMkvXeNQiL2G9H_iUezmUCVpvKMbGM.cQSj5xU jLUoa846cyDaFWf04OFyCqxL.S4o.OzYQ4gg8ReHZ3r.3D4VK1GMjAcIdfYfIHONm7X5aR1QlCg- - X-Sonic-MF: X-Sonic-ID: 831ed6a1-9c1e-48b3-a86a-21223f7f34e4 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Wed, 23 Aug 2023 20:37:23 +0000 Received: by hermes--production-gq1-6b7c87dcf5-m4lb7 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 313664c8eea31570ad9db9f264ff0949; Wed, 23 Aug 2023 20:37:19 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 From: Mark Millard In-Reply-To: Date: Wed, 23 Aug 2023 13:37:09 -0700 Cc: Current FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: <1AC87B79-6B65-402B-B65F-CCFFCC503861@yahoo.com> References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> To: Alexander Motin X-Mailer: Apple Mail (2.3731.700.6) X-Rspamd-Queue-Id: 4RWJ1J4zy7z3Dvs X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] On Aug 23, 2023, at 11:40, Alexander Motin wrote: > On 22.08.2023 14:24, Mark Millard wrote: >> Alexander Motin wrote on >> Date: Tue, 22 Aug 2023 16:18:12 UTC : >>> I am waiting for final test results from George Wilson and then will >>> request quick merge of both to zfs-2.2-release branch. Unfortunately >>> there are still not many reviewers for the PR, since the code is not >>> trivial, but at least with the test reports Brian Behlendorf and = Mark >>> Maybee seem to be OK to merge the two PRs into 2.2. If somebody else >>> have tested and/or reviewed the PR, you may comment on it. >> I had written to the list that when I tried to test the system >> doing poudriere builds (initially with your patches) using >> USE_TMPFS=3Dno so that zfs had to deal with all the file I/O, I >> instead got only one builder that ended up active, the others >> never reaching "Builder started": >=20 >> Top was showing lots of "vlruwk" for the cpdup's. For example: >> . . . >> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 >> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 >> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 >> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 >> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 >> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 >> . . . >> But those processes did show CPU?? on occasion, as well as >> *vnode less often. None of the cpdup's was stuck in >> Removing your patches did not change the behavior. >=20 > Mark, to me "vlruwk" looks like a limit on number of vnodes. I was = not deep in that area at least recently, so somebody with more = experience there could try to diagnose it. At very least it does not = look related to the ZIL issue discussed in this thread, at least with = the information provided, so I am not surprised that the mentioned = patches do not affect it. Thanks for the information. Good to know. I'll redirect this to be a = different discussion. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Aug 23 21:50:38 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWKfF0lCFz4rQ46 for ; Wed, 23 Aug 2023 21:51:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWKfC53f5z3KRd for ; Wed, 23 Aug 2023 21:50:59 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=KLlwOyJo; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692827456; bh=9TxvsSJCcnpQRDtpXhiDYMrEcMBYJGqVRW8SZl/L8cY=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=KLlwOyJoi8yCRoC9Bg268/gZ0issxUUg9GqpHoJkaJYNpIzLWzpx9KQOGzYpDXq2FQ5/8NqS9ZFJ0P1TMMv/54vJwfuMEaXXbDdpFp1jSzlZE1Qg2eiQMbZa5co8k2NvIRY6rD1ONcGHFKytYMXEutKq4CpRQNcEPLkMLzhU5Nh4DOB/EUDxelttE4XtrMc46zMr1s+lVCcT8NafgAGSJrf5j2dwuTvtfBmUv9tOGK9KUx6mbo+GGl3ccEBGvLk6IL2NgQCmAO6DuHMIaN3b9B8iO46b0Y9cwIavd+DhNMM8sCMp/XpBTi5Np0L9XMtTDwf/5U9CXE85r0KoDP5AKQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692827456; bh=LSvu7HawOK7BwVKjqo33GaYwcjYJ/zTeUbGzJU6wEs1=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=jKD51JI9x8/upMWVBX4WdbBCqWpHed4Ytmvz5ge0j3EEnviLZ6ZQUclJEMzI3k7OHXB1/5+wx2E2ilsU1ZnYRFivqKCDh7Xjf79osE57xMY9IOQsZoWwDmmPkBdFeOBYYcwogTVhFox0UAfPpuLFBfyf7KgMYopGEiyPf9rp4jwsQ1ZlY8uyqxWQDPcwHkrlgt7QzBgtcCeRa+ZftsGygubcumDh21KCUdIi7LBAP+zI/WL23gukV+ZSKfW1bVHk6Rb7FAODG+lwtviAhCH9LIC8oMRtLibVt968dJbpLXReRDqPHUzroQ3afmIaZMzQDwbO4JRsxRPgHw8vkCVf2g== X-YMail-OSG: WKVTivoVM1l.CYBGlIxHRamS6Q5QD5qurXl1Q8iNcytMfMFlAyFirag14u6.GUf Yyy5ZJliVDXpbiUPu6hJatii9UIRY02NVfbI7T9N99PnNaJ7vdDwUrvyLczEkkhIV.mKg545JxZK CaHHxabgpqnEg4pgNFlECccZqMgp0V0jYs1w3cI.wvMJxGQ.jtfwDm71VWjH1vQQrzGt4WACFUyr S5ShwyjW.gGFmD3729yKFouJj8S736pQvZnHAgfrVTpTczXBbMW6hn5eN2BVq4OucwzKiotGE3xx RjrLZRd.Xrf0PS2YyAi6qDw9DY1sDuaAqkw7yNT3dtcpch770ZDk2RIM6jdyPmimjdIL4mfGFqx1 okf8V2JVSrEpOLD19UD6.KZA6FBVro6b2M7QZosLomYCeTWo.B6E6Npw.ANoaWpWKisvM_6E_07p B3uBw5jZ823O9B40gSGBnpjoiUtOjV.MP69fhox0Wzashh_GlOkkl6b4YanPj1fUR_T.cRvNmB3u slKHrMm2t0BaYudvzWdCeIh9VXKYms.4sgiWtEN8RY5rbQL2zE6HQB1Q7N7aDygqbkaRjBR0H.jC vwtY.89PTRyD3cMghRe2Qz_nFEYkhFxJMQB9MpIR_kBED3GcVhFihs8YKfHFYgwPgiq_PLEF0ypP fPW_K9tT2sYNUOX4lvUk5UnepE8juZ7EJ9uf3DNrPvJ.z4C8VNpE1VFdMjWNz5ZSXyb7xvJb7VIY NWkK5RYyJn6q7uJaUqDsq7sbdEzSlNqOo8QcDMUvW1svO5eaR1MAZz19uSjWlZu_YTp6Pe1Ml8k8 TN10Y6uu9Lka2JPsxaD3rGXzUBYGVkAMA5YXt1yXMmFda9OA.khqyGVWGa4Xw194A_NRcYg02BXr Be4urdtD9UAsw6ZJbeutCFG87USOb4.V3HzZAVzD5y27xsjlaDw24tyf3jrTAJXVPlVvZh8iZoNJ mc77PvS__BzKC8zzjNrd7p415FNBVSj30db4kMT5HGnDFNi9QcAzgaAQTakn8eIoMVnhBejgSJZO ORjTdE_WuiTESxK8yKpZHN1bxYJFRTQQ8B9fysqlEUaJt..7ouFSDdDzXnfW.8mdhhVLW8CxGtmn vKlovlBUg1NVljOe40x1hPQCZdJrZW4W4SH1h9KbRfGJIx0fJ1MZqXTni0sA1fYT08ZGN5fH0WFZ UaTN_52inVKE0LHvCPcZJUvkgv5P97PQhMwQ7cnH8FGEBrUUxyUyduvM129lXwnrQ8Y2Iqlds9Q5 zuZ2UdhSsJigcXlMZMc8yb5wDCtRiRkRjY9fkPzLxzgFN1jzEIT_ecwdDm42ZIuDJtdz4zGX0ZZe txO9PCCUfuH1J9jCy_yczjwQzEfBB8QQcNuqSo_wpOF3AMjYD3I71V_H7dOIRXi4SbQ375qaibbx bBGta3AU8aNfkiE_ioTcoCmByKBTzRE6eCPLJQYlC6jpZxlqwohxcvHRf3CSSsUXCRVHiliQMDph kvpQg7B4hMr5bPu2JZ_zhBR93mZGPGJGE.GZAQa2pPxVc0RSOrEoqHUqH237L9ihwrNjFOGYvte3 31cj4awVjNf3hJ9XZ35fC8C3Se4x57QpMETT75qT6ecHkTS70TZXrHbFgwPjsJRIq0FdCKXSbH.X z9mLafmc05AOYnWAdJNLc0CJgBFSNpgX82mWJHOrKExEEOU3DZnURK1iu.wB4XB0OrlBSXuwLaWr 0E6GPtFUTsioFkk60PryoQLuHRtyw9xudV__ZGZKKHN8IMIsmpLgNTJ_TeZD_fzUC7SsRnNbiQDc 1_FkORLi_tiY0GOfSYSteqNWJQUMGkvw642eeVrYILLD70AWQ_PyJnPf2.m7kJJW2FTFPm3enWKV i9QLq2oHUIzp9vGX_FrqAUi8rvQl1.o_D1nMXd84SgjmRuYFA1O1N2puGNiJXg46xeoFyNnnWEVc kf7k4UwPzFupC1htpiaU15uSZGkp59pE8jpPmNF_KSxrZ3PH3KUC2EcQU1Jy4nKdA_oRWEQDfL3v ctu3bIMoomgAdllytPTasJd36.VEQ_UYh1xCyjpN5cgyAMM_HzxUU4XesRz4ET5Fnw10KqrZi0J4 9LZEupmJ80WS4mP7CYkQbojaJIGaN9cVUVYs973L66jYKaw8yrW00oaOn8IZF2pYO4eWkolom4TU SV8B7X.3FWiSmjrZ2._xswMBTIe7oM7dq_A3Ukb3DubIstKOD_pzW_I7W5qCIs5XnHoCT5dgtm_8 EmQXr9nUX_2zRgmdyCKBMm4Xsl9E7NtYPKV1tSVFHe68.ZXJkKxTYoUyvbcV5MQ-- X-Sonic-MF: X-Sonic-ID: 5e8e9c42-d77d-42db-b56a-da4929fd6255 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Wed, 23 Aug 2023 21:50:56 +0000 Received: by hermes--production-bf1-865889d799-2bcr6 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 410857c89b7726185f7ea400fc2966a1; Wed, 23 Aug 2023 21:50:51 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder Date: Wed, 23 Aug 2023 14:50:38 -0700 References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> To: Current FreeBSD In-Reply-To: Message-Id: <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.205:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWKfC53f5z3KRd [Forked off the ZFS deadlock 14 discussion, per feedback.] On Aug 23, 2023, at 11:40, Alexander Motin wrote: > On 22.08.2023 14:24, Mark Millard wrote: >> Alexander Motin wrote on >> Date: Tue, 22 Aug 2023 16:18:12 UTC : >>> I am waiting for final test results from George Wilson and then will >>> request quick merge of both to zfs-2.2-release branch. Unfortunately >>> there are still not many reviewers for the PR, since the code is not >>> trivial, but at least with the test reports Brian Behlendorf and = Mark >>> Maybee seem to be OK to merge the two PRs into 2.2. If somebody else >>> have tested and/or reviewed the PR, you may comment on it. >> I had written to the list that when I tried to test the system >> doing poudriere builds (initially with your patches) using >> USE_TMPFS=3Dno so that zfs had to deal with all the file I/O, I >> instead got only one builder that ended up active, the others >> never reaching "Builder started": >=20 >> Top was showing lots of "vlruwk" for the cpdup's. For example: >> . . . >> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 >> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 >> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 >> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 >> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 >> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 >> . . . >> But those processes did show CPU?? on occasion, as well as >> *vnode less often. None of the cpdup's was stuck in >> Removing your patches did not change the behavior. >=20 > Mark, to me "vlruwk" looks like a limit on number of vnodes. I was = not deep in that area at least recently, so somebody with more = experience there could try to diagnose it. At very least it does not = look related to the ZIL issue discussed in this thread, at least with = the information provided, so I am not surprised that the mentioned = patches do not affect it. I did the above intending to test the deadlock in my context but ended up not getting that far when I tried to make zfs handle all the file I/O (USE_TMPFS=3Dno and no other use of tmpfs or the like). The zfs context is a simple single partition on the boot media. I use ZFS for bectl BE use, not for other typical reasons. The media here is PCIe Optane 1.4T media. The machine is a ThreadRipper 1950X, so first generation. 128 GiBytes of RAM. 491520 MiBytes of swap, also on that Optane. # uname -apKU FreeBSD amd64-ZFS 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #112 = main-n264912-b1d3e2b77155-dirty: Sun Aug 20 10:01:48 PDT 2023 = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1400096 1400096 The GENERIC-DBG variant of the kernel did not report any issues in earlier testing. The alter referenced /usr/obj/DESTDIRs/main-amd64-poud-bulk_a was installed from the same build. # zfs list NAME USED AVAIL REFER = MOUNTPOINT zoptb 79.9G 765G 96K /zoptb zoptb/BUILDs 20.5G 765G 8.29M = /usr/obj/BUILDs zoptb/BUILDs/alt-main-amd64-dbg-clang-alt 1.86M 765G 1.86M = /usr/obj/BUILDs/alt-main-amd64-dbg-clang-alt zoptb/BUILDs/alt-main-amd64-nodbg-clang-alt 30.2M 765G 30.2M = /usr/obj/BUILDs/alt-main-amd64-nodbg-clang-alt zoptb/BUILDs/main-amd64-dbg-clang 9.96G 765G 9.96G = /usr/obj/BUILDs/main-amd64-dbg-clang zoptb/BUILDs/main-amd64-dbg-gccxtc 38.5M 765G 38.5M = /usr/obj/BUILDs/main-amd64-dbg-gccxtc zoptb/BUILDs/main-amd64-nodbg-clang 10.3G 765G 10.3G = /usr/obj/BUILDs/main-amd64-nodbg-clang zoptb/BUILDs/main-amd64-nodbg-clang-alt 37.2M 765G 37.2M = /usr/obj/BUILDs/main-amd64-nodbg-clang-alt zoptb/BUILDs/main-amd64-nodbg-gccxtc 94.6M 765G 94.6M = /usr/obj/BUILDs/main-amd64-nodbg-gccxtc zoptb/DESTDIRs 4.33G 765G 104K = /usr/obj/DESTDIRs zoptb/DESTDIRs/main-amd64-poud 2.16G 765G 2.16G = /usr/obj/DESTDIRs/main-amd64-poud zoptb/DESTDIRs/main-amd64-poud-bulk_a 2.16G 765G 2.16G = /usr/obj/DESTDIRs/main-amd64-poud-bulk_a zoptb/ROOT 13.1G 765G 96K none zoptb/ROOT/build_area_for-main-amd64 5.03G 765G 3.24G none zoptb/ROOT/main-amd64 8.04G 765G 3.23G none zoptb/poudriere 6.58G 765G 112K = /usr/local/poudriere zoptb/poudriere/data 6.58G 765G 128K = /usr/local/poudriere/data zoptb/poudriere/data/.m 112K 765G 112K = /usr/local/poudriere/data/.m zoptb/poudriere/data/cache 17.4M 765G 17.4M = /usr/local/poudriere/data/cache zoptb/poudriere/data/images 96K 765G 96K = /usr/local/poudriere/data/images zoptb/poudriere/data/logs 2.72G 765G 2.72G = /usr/local/poudriere/data/logs zoptb/poudriere/data/packages 3.84G 765G 3.84G = /usr/local/poudriere/data/packages zoptb/poudriere/data/wrkdirs 112K 765G 112K = /usr/local/poudriere/data/wrkdirs zoptb/poudriere/jails 96K 765G 96K = /usr/local/poudriere/jails zoptb/poudriere/ports 96K 765G 96K = /usr/local/poudriere/ports zoptb/tmp 68.5M 765G 68.5M /tmp zoptb/usr 35.1G 765G 96K /usr zoptb/usr/13_0R-src 2.64G 765G 2.64G = /usr/13_0R-src zoptb/usr/alt-main-src 96K 765G 96K = /usr/alt-main-src zoptb/usr/home 181M 765G 181M = /usr/home zoptb/usr/local 5.08G 765G 5.08G = /usr/local zoptb/usr/main-src 833M 765G 833M = /usr/main-src zoptb/usr/ports 26.4G 765G 26.4G = /usr/ports zoptb/usr/src 96K 765G 96K = /usr/src zoptb/var 52.6M 765G 96K /var zoptb/var/audit 356K 765G 356K = /var/audit zoptb/var/crash 128K 765G 128K = /var/crash zoptb/var/db 49.7M 765G 96K = /var/db zoptb/var/db/pkg 49.4M 765G 49.4M = /var/db/pkg zoptb/var/db/ports 164K 765G 164K = /var/db/ports zoptb/var/log 1.61M 765G 1.61M = /var/log zoptb/var/mail 632K 765G 632K = /var/mail zoptb/var/tmp 128K 765G 128K = /var/tmp # poudriere jail -jmain-amd64-bulk_a -i Jail name: main-amd64-bulk_a Jail version: 14.0-ALPHA2 Jail arch: amd64 Jail method: null Jail mount: /usr/obj/DESTDIRs/main-amd64-poud-bulk_a Jail fs: =20 Jail updated: 2021-12-04 14:55:22 Jail pkgbase: disabled So, setting up another test with some related information shown before, during, and after. sysctl output is from another ssh session than the bulk -a run. # sysctl -a | grep vnode kern.maxvnodes: 2213808 kern.ipc.umtx_vnode_persistent: 0 kern.minvnodes: 553452 vm.vnode_pbufs: 2048 vm.stats.vm.v_vnodepgsout: 0 vm.stats.vm.v_vnodepgsin: 272429 vm.stats.vm.v_vnodeout: 0 vm.stats.vm.v_vnodein: 12461 vfs.vnode_alloc_sleeps: 0 vfs.wantfreevnodes: 553452 vfs.freevnodes: 962766 vfs.vnodes_created: 2538980 vfs.numvnodes: 1056233 vfs.cache.debug.vnodes_cel_3_failures: 0 vfs.cache.stats.heldvnodes: 91878 debug.vnode_domainset: debug.sizeof.vnode: 448 debug.fail_point.status_fill_kinfo_vnode__random_path: off debug.fail_point.fill_kinfo_vnode__random_path: off # poudriere bulk -jmain-amd64-bulk_a -a . . . [00:01:34] Building 34042 packages using up to 32 builders [00:01:34] Hit CTRL+t at any time to see build progress and stats [00:01:34] [01] [00:00:00] Builder starting [00:01:57] [01] [00:00:23] Builder started [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: Success [00:03:22] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 [00:03:22] [02] [00:00:00] Builder starting [00:03:22] [03] [00:00:00] Builder starting . . . [00:03:22] [31] [00:00:00] Builder starting [00:03:22] [32] [00:00:00] Builder starting [00:03:31] [01] [00:00:09] Finished print/indexinfo | indexinfo-0.3.1: = Success [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 . . . Note that only [01] makes progress: no new "Builder started" notices occur. top shows 31 of the pattern: cpdup -i0 -o ref ?? Then during the 31 cpudup's showing vlruwk most of the time: # sysctl -a | grep vnode kern.maxvnodes: 2213808 kern.ipc.umtx_vnode_persistent: 0 kern.minvnodes: 553452 vm.vnode_pbufs: 2048 vm.stats.vm.v_vnodepgsout: 22844 vm.stats.vm.v_vnodepgsin: 582398 vm.stats.vm.v_vnodeout: 890 vm.stats.vm.v_vnodein: 34296 vfs.vnode_alloc_sleeps: 2994 vfs.wantfreevnodes: 553452 vfs.freevnodes: 2209662 vfs.vnodes_created: 12206299 vfs.numvnodes: 2214071 vfs.cache.debug.vnodes_cel_3_failures: 0 vfs.cache.stats.heldvnodes: 459 debug.vnode_domainset: debug.sizeof.vnode: 448 debug.fail_point.status_fill_kinfo_vnode__random_path: off debug.fail_point.fill_kinfo_vnode__random_path: off Wait a while but still the mostly cpdup vlruwk status: # sysctl -a | grep vnode kern.maxvnodes: 2213808 kern.ipc.umtx_vnode_persistent: 0 kern.minvnodes: 553452 vm.vnode_pbufs: 2048 vm.stats.vm.v_vnodepgsout: 22844 vm.stats.vm.v_vnodepgsin: 583527 vm.stats.vm.v_vnodeout: 890 vm.stats.vm.v_vnodein: 34396 vfs.vnode_alloc_sleeps: 8053 vfs.wantfreevnodes: 553452 vfs.freevnodes: 2210166 vfs.vnodes_created: 12212061 vfs.numvnodes: 2215106 vfs.cache.debug.vnodes_cel_3_failures: 0 vfs.cache.stats.heldvnodes: 497 debug.vnode_domainset: debug.sizeof.vnode: 448 debug.fail_point.status_fill_kinfo_vnode__random_path: off debug.fail_point.fill_kinfo_vnode__random_path: off ^C[00:14:55] Error: Signal SIGINT caught, cleaning up and exiting # sysctl -a | grep vnode kern.maxvnodes: 2213808 kern.ipc.umtx_vnode_persistent: 0 kern.minvnodes: 553452 vm.vnode_pbufs: 2048 vm.stats.vm.v_vnodepgsout: 22844 vm.stats.vm.v_vnodepgsin: 584474 vm.stats.vm.v_vnodeout: 890 vm.stats.vm.v_vnodein: 34591 vfs.vnode_alloc_sleeps: 17584 vfs.wantfreevnodes: 553452 vfs.freevnodes: 2210796 vfs.vnodes_created: 12222343 vfs.numvnodes: 2216564 vfs.cache.debug.vnodes_cel_3_failures: 0 vfs.cache.stats.heldvnodes: 539 debug.vnode_domainset: debug.sizeof.vnode: 448 debug.fail_point.status_fill_kinfo_vnode__random_path: off debug.fail_point.fill_kinfo_vnode__random_path: off [main-amd64-bulk_a-default] [2023-08-23_13h58m08s] [sigint:] Queued: = 34435 Built: 2 Failed: 0 Skipped: 35 Ignored: 358 Fetched: = 0 Tobuild: 34040 Time: 00:14:36 [00:16:13] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-23_1= 3h58m08s [00:16:49] Cleaning up load: 5.28 cmd: sh 77057 [vlruwk] 141.63r 0.00u 30.98s 28% 6932k #0 0xffffffff80b76ebb at mi_switch+0xbb #1 0xffffffff80bc960f at sleepq_timedwait+0x2f #2 0xffffffff80b76610 at _sleep+0x1d0 #3 0xffffffff80c5b2dc at vn_alloc_hard+0x2ac #4 0xffffffff80c50a12 at getnewvnode_reserve+0x92 #5 0xffffffff829afb12 at zfs_zget+0x22 #6 0xffffffff8299ca8d at zfs_dirent_lookup+0x16d #7 0xffffffff8299cb5f at zfs_dirlook+0x7f #8 0xffffffff829ac410 at zfs_lookup+0x350 #9 0xffffffff829a782a at zfs_freebsd_cachedlookup+0x6a #10 0xffffffff80c368ad at vfs_cache_lookup+0xad #11 0xffffffff80c3b6d8 at cache_fplookup_final_modifying+0x188 #12 0xffffffff80c38766 at cache_fplookup+0x356 #13 0xffffffff80c43fb2 at namei+0x112 #14 0xffffffff80c62e5b at kern_funlinkat+0x13b #15 0xffffffff80c62d18 at sys_unlink+0x28 #16 0xffffffff83b8e583 at filemon_wrapper_unlink+0x13 #17 0xffffffff81049a79 at amd64_syscall+0x109 [00:26:28] Unmounting file systems Exiting with status 1 # sysctl -a | grep vnode kern.maxvnodes: 2213808 kern.ipc.umtx_vnode_persistent: 0 kern.minvnodes: 553452 vm.vnode_pbufs: 2048 vm.stats.vm.v_vnodepgsout: 22844 vm.stats.vm.v_vnodepgsin: 585384 vm.stats.vm.v_vnodeout: 890 vm.stats.vm.v_vnodein: 34798 vfs.vnode_alloc_sleeps: 27578 vfs.wantfreevnodes: 553452 vfs.freevnodes: 61362 vfs.vnodes_created: 20135479 vfs.numvnodes: 59860 vfs.cache.debug.vnodes_cel_3_failures: 0 vfs.cache.stats.heldvnodes: 208 debug.vnode_domainset: debug.sizeof.vnode: 448 debug.fail_point.status_fill_kinfo_vnode__random_path: off debug.fail_point.fill_kinfo_vnode__random_path: off For reference (from after): # kldstat Id Refs Address Size Name 1 95 0xffffffff80200000 274b308 kernel 2 1 0xffffffff8294c000 5d5238 zfs.ko 3 1 0xffffffff82f22000 7718 cryptodev.ko 4 1 0xffffffff83b10000 3390 acpi_wmi.ko 5 1 0xffffffff83b14000 3220 intpm.ko 6 1 0xffffffff83b18000 2178 smbus.ko 7 1 0xffffffff83b1b000 2240 cpuctl.ko 8 1 0xffffffff83b1e000 3360 uhid.ko 9 1 0xffffffff83b22000 4364 ums.ko 10 1 0xffffffff83b27000 33c0 usbhid.ko 11 1 0xffffffff83b2b000 3380 hidbus.ko 12 1 0xffffffff83b2f000 4d20 ng_ubt.ko 13 6 0xffffffff83b34000 abb8 netgraph.ko 14 2 0xffffffff83b3f000 a250 ng_hci.ko 15 4 0xffffffff83b4a000 2670 ng_bluetooth.ko 16 1 0xffffffff83b4d000 83a0 uftdi.ko 17 1 0xffffffff83b56000 4e58 ucom.ko 18 1 0xffffffff83b5b000 3360 wmt.ko 19 1 0xffffffff83b5f000 e268 ng_l2cap.ko 20 1 0xffffffff83b6e000 1bf68 ng_btsocket.ko 21 1 0xffffffff83b8a000 38f8 ng_socket.ko 22 1 0xffffffff83b8e000 3250 filemon.ko 23 1 0xffffffff83b92000 4758 nullfs.ko 24 1 0xffffffff83b97000 73c0 linprocfs.ko 25 3 0xffffffff83b9f000 be70 linux_common.ko 26 1 0xffffffff83bab000 3558 fdescfs.ko 27 1 0xffffffff83baf000 31b20 linux.ko 28 1 0xffffffff83be1000 2ed40 linux64.ko Note that before the "Cleaning up" notice, the vfs.freevnodes shows as being around (for example) 2210796. But after "Exiting with status": 61362. vfs.vnodes_created has a similar staging of in the ball park of up to 12222343 but then the change to: 20135479. Similarly, vfs.numvnodes 2216564 -> 59860. Anything else I should gather and report as basic information? =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Aug 23 22:10:11 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWL4Q4XfDz4rQYB for ; Wed, 23 Aug 2023 22:10:14 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x2a.google.com (mail-oa1-x2a.google.com [IPv6:2001:4860:4864:20::2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWL4Q0p83z3MYh for ; Wed, 23 Aug 2023 22:10:14 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-1c504386374so4478230fac.3 for ; Wed, 23 Aug 2023 15:10:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692828612; x=1693433412; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=hqCkIYPAvkEGjjBySTDp12x27czkEt9UseJhJTzS0Cs=; b=EB/rrvYGU6Nd+6m18uszulFTt7g5WaAnKSIp62FnIYHkatskFKr6gN9FRrul7nN8XI GNM0pGstj+p6CZT5N9dNFzrYeinsWxvZrUfS63sQbWf7wDHjwjOg0md1et+LQEdD0wPu j2KYV1coPM1RjvArte8us8eTVmfsBd+rcDGu7rbsFFetthMv6Dj8WZ5cgA+v1DC/kdxp xPKGIn4h9QkFF/0Yo89FEJiEfLC+Y9bk2J7vqJKkK4mbr9Wcd9Vvz2uz8CruxsZ/64+z +ktRuVBWFXLy57L2IpBTw43xfNbf0IwSlU9NvY5Ikv9Cgo+2oWGiFyQ3jk9dM0fO4ALi EoTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692828612; x=1693433412; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hqCkIYPAvkEGjjBySTDp12x27czkEt9UseJhJTzS0Cs=; b=QOIu6dSHHkdZoaT0Y3KvqufivNwSA94QgJWmWHJb8jEyYg8j44MBUkZhswETm7eHuc VJtKl+P1y439EQd/rulFyRx5uimqBT4B/Xp4dYG+fDNjWcIS3VUixcR7BZBtYW5v0y0n YvJpDIf+LaKO/liHbbdLB4nILzla1b3KgS74dZFnshja9CGp7lUxStEUtQxVDVrwJXtI kDzGgdtvdHtCmOjOjUKTAX0F6IeqAvCvN+8gr+/x5Rztbc6j7enEWy2PQw1jD92AFGcx +SiTaNY0cWkij5giQO7INtL30Jbq4/485OLoF5SulyAzfpuv5vgFvBefANJ8p4EMPE0h V+Mw== X-Gm-Message-State: AOJu0YyPuMF+q8NrDW+bEIx1ZP5Cj9HqBv50h8OluPDx73Oy0V0aKqNy 5cGrLnGEIVk7nyU/1szodsS8C/bZUOCBwqTLPDhSsHGM X-Google-Smtp-Source: AGHT+IFLrYKMq+3BXjC5S8hOOA4CUQ+TI/neQP+3Mgr2pt9JhafEC4u3d+NpuCYkjHVAgGcZJ8YYSXSHzc/JX5LmfrU= X-Received: by 2002:a05:6870:7390:b0:1bf:4f42:de9d with SMTP id z16-20020a056870739000b001bf4f42de9dmr16098210oam.42.1692828612024; Wed, 23 Aug 2023 15:10:12 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:5797:0:b0:4f0:1250:dd51 with HTTP; Wed, 23 Aug 2023 15:10:11 -0700 (PDT) In-Reply-To: <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> From: Mateusz Guzik Date: Thu, 24 Aug 2023 00:10:11 +0200 Message-ID: Subject: Re: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder To: Mark Millard Cc: Current FreeBSD Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RWL4Q0p83z3MYh X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US] On 8/23/23, Mark Millard wrote: > [Forked off the ZFS deadlock 14 discussion, per feedback.] > > On Aug 23, 2023, at 11:40, Alexander Motin wrote: > >> On 22.08.2023 14:24, Mark Millard wrote: >>> Alexander Motin wrote on >>> Date: Tue, 22 Aug 2023 16:18:12 UTC : >>>> I am waiting for final test results from George Wilson and then will >>>> request quick merge of both to zfs-2.2-release branch. Unfortunately >>>> there are still not many reviewers for the PR, since the code is not >>>> trivial, but at least with the test reports Brian Behlendorf and Mark >>>> Maybee seem to be OK to merge the two PRs into 2.2. If somebody else >>>> have tested and/or reviewed the PR, you may comment on it. >>> I had written to the list that when I tried to test the system >>> doing poudriere builds (initially with your patches) using >>> USE_TMPFS=no so that zfs had to deal with all the file I/O, I >>> instead got only one builder that ended up active, the others >>> never reaching "Builder started": >> >>> Top was showing lots of "vlruwk" for the cpdup's. For example: >>> . . . >>> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 >>> 0.00% cpdup -i0 -o ref 32 >>> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 >>> 0.01% cpdup -i0 -o ref 31 >>> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 >>> 0.01% cpdup -i0 -o ref 30 >>> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 >>> 0.01% cpdup -i0 -o ref 29 >>> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 >>> 0.01% cpdup -i0 -o ref 28 >>> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 >>> 0.00% cpdup -i0 -o ref 27 >>> . . . >>> But those processes did show CPU?? on occasion, as well as >>> *vnode less often. None of the cpdup's was stuck in >>> Removing your patches did not change the behavior. >> >> Mark, to me "vlruwk" looks like a limit on number of vnodes. I was not >> deep in that area at least recently, so somebody with more experience >> there could try to diagnose it. At very least it does not look related to >> the ZIL issue discussed in this thread, at least with the information >> provided, so I am not surprised that the mentioned patches do not affect >> it. > > I did the above intending to test the deadlock in my context but > ended up not getting that far when I tried to make zfs handle all > the file I/O (USE_TMPFS=no and no other use of tmpfs or the like). > > The zfs context is a simple single partition on the boot media. I > use ZFS for bectl BE use, not for other typical reasons. The media > here is PCIe Optane 1.4T media. The machine is a ThreadRipper > 1950X, so first generation. 128 GiBytes of RAM. 491520 MiBytes of > swap, also on that Optane. > > # uname -apKU > FreeBSD amd64-ZFS 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #112 > main-n264912-b1d3e2b77155-dirty: Sun Aug 20 10:01:48 PDT 2023 > root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG > amd64 amd64 1400096 1400096 > > The GENERIC-DBG variant of the kernel did not report any issues in > earlier testing. > > The alter referenced /usr/obj/DESTDIRs/main-amd64-poud-bulk_a was > installed from the same build. > > # zfs list > NAME USED AVAIL REFER > MOUNTPOINT > zoptb 79.9G 765G 96K /zoptb > zoptb/BUILDs 20.5G 765G 8.29M > /usr/obj/BUILDs > zoptb/BUILDs/alt-main-amd64-dbg-clang-alt 1.86M 765G 1.86M > /usr/obj/BUILDs/alt-main-amd64-dbg-clang-alt > zoptb/BUILDs/alt-main-amd64-nodbg-clang-alt 30.2M 765G 30.2M > /usr/obj/BUILDs/alt-main-amd64-nodbg-clang-alt > zoptb/BUILDs/main-amd64-dbg-clang 9.96G 765G 9.96G > /usr/obj/BUILDs/main-amd64-dbg-clang > zoptb/BUILDs/main-amd64-dbg-gccxtc 38.5M 765G 38.5M > /usr/obj/BUILDs/main-amd64-dbg-gccxtc > zoptb/BUILDs/main-amd64-nodbg-clang 10.3G 765G 10.3G > /usr/obj/BUILDs/main-amd64-nodbg-clang > zoptb/BUILDs/main-amd64-nodbg-clang-alt 37.2M 765G 37.2M > /usr/obj/BUILDs/main-amd64-nodbg-clang-alt > zoptb/BUILDs/main-amd64-nodbg-gccxtc 94.6M 765G 94.6M > /usr/obj/BUILDs/main-amd64-nodbg-gccxtc > zoptb/DESTDIRs 4.33G 765G 104K > /usr/obj/DESTDIRs > zoptb/DESTDIRs/main-amd64-poud 2.16G 765G 2.16G > /usr/obj/DESTDIRs/main-amd64-poud > zoptb/DESTDIRs/main-amd64-poud-bulk_a 2.16G 765G 2.16G > /usr/obj/DESTDIRs/main-amd64-poud-bulk_a > zoptb/ROOT 13.1G 765G 96K none > zoptb/ROOT/build_area_for-main-amd64 5.03G 765G 3.24G none > zoptb/ROOT/main-amd64 8.04G 765G 3.23G none > zoptb/poudriere 6.58G 765G 112K > /usr/local/poudriere > zoptb/poudriere/data 6.58G 765G 128K > /usr/local/poudriere/data > zoptb/poudriere/data/.m 112K 765G 112K > /usr/local/poudriere/data/.m > zoptb/poudriere/data/cache 17.4M 765G 17.4M > /usr/local/poudriere/data/cache > zoptb/poudriere/data/images 96K 765G 96K > /usr/local/poudriere/data/images > zoptb/poudriere/data/logs 2.72G 765G 2.72G > /usr/local/poudriere/data/logs > zoptb/poudriere/data/packages 3.84G 765G 3.84G > /usr/local/poudriere/data/packages > zoptb/poudriere/data/wrkdirs 112K 765G 112K > /usr/local/poudriere/data/wrkdirs > zoptb/poudriere/jails 96K 765G 96K > /usr/local/poudriere/jails > zoptb/poudriere/ports 96K 765G 96K > /usr/local/poudriere/ports > zoptb/tmp 68.5M 765G 68.5M /tmp > zoptb/usr 35.1G 765G 96K /usr > zoptb/usr/13_0R-src 2.64G 765G 2.64G > /usr/13_0R-src > zoptb/usr/alt-main-src 96K 765G 96K > /usr/alt-main-src > zoptb/usr/home 181M 765G 181M /usr/home > zoptb/usr/local 5.08G 765G 5.08G > /usr/local > zoptb/usr/main-src 833M 765G 833M > /usr/main-src > zoptb/usr/ports 26.4G 765G 26.4G > /usr/ports > zoptb/usr/src 96K 765G 96K /usr/src > zoptb/var 52.6M 765G 96K /var > zoptb/var/audit 356K 765G 356K > /var/audit > zoptb/var/crash 128K 765G 128K > /var/crash > zoptb/var/db 49.7M 765G 96K /var/db > zoptb/var/db/pkg 49.4M 765G 49.4M > /var/db/pkg > zoptb/var/db/ports 164K 765G 164K > /var/db/ports > zoptb/var/log 1.61M 765G 1.61M /var/log > zoptb/var/mail 632K 765G 632K /var/mail > zoptb/var/tmp 128K 765G 128K /var/tmp > > # poudriere jail -jmain-amd64-bulk_a -i > Jail name: main-amd64-bulk_a > Jail version: 14.0-ALPHA2 > Jail arch: amd64 > Jail method: null > Jail mount: /usr/obj/DESTDIRs/main-amd64-poud-bulk_a > Jail fs: > Jail updated: 2021-12-04 14:55:22 > Jail pkgbase: disabled > > > > So, setting up another test with some related information > shown before, during, and after. sysctl output is from > another ssh session than the bulk -a run. > > # sysctl -a | grep vnode > kern.maxvnodes: 2213808 > kern.ipc.umtx_vnode_persistent: 0 > kern.minvnodes: 553452 > vm.vnode_pbufs: 2048 > vm.stats.vm.v_vnodepgsout: 0 > vm.stats.vm.v_vnodepgsin: 272429 > vm.stats.vm.v_vnodeout: 0 > vm.stats.vm.v_vnodein: 12461 > vfs.vnode_alloc_sleeps: 0 > vfs.wantfreevnodes: 553452 > vfs.freevnodes: 962766 > vfs.vnodes_created: 2538980 > vfs.numvnodes: 1056233 > vfs.cache.debug.vnodes_cel_3_failures: 0 > vfs.cache.stats.heldvnodes: 91878 > debug.vnode_domainset: > debug.sizeof.vnode: 448 > debug.fail_point.status_fill_kinfo_vnode__random_path: off > debug.fail_point.fill_kinfo_vnode__random_path: off > > # poudriere bulk -jmain-amd64-bulk_a -a > . . . > [00:01:34] Building 34042 packages using up to 32 builders > [00:01:34] Hit CTRL+t at any time to see build progress and stats > [00:01:34] [01] [00:00:00] Builder starting > [00:01:57] [01] [00:00:23] Builder started > [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 > [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: Success > [00:03:22] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 > [00:03:22] [02] [00:00:00] Builder starting > [00:03:22] [03] [00:00:00] Builder starting > . . . > [00:03:22] [31] [00:00:00] Builder starting > [00:03:22] [32] [00:00:00] Builder starting > [00:03:31] [01] [00:00:09] Finished print/indexinfo | indexinfo-0.3.1: > Success > [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | > gettext-runtime-0.22 > . . . > > Note that only [01] makes progress: no new "Builder started" > notices occur. top shows 31 of the pattern: > cpdup -i0 -o ref ?? > > Then during the 31 cpudup's showing vlruwk most of the time: > > # sysctl -a | grep vnode > kern.maxvnodes: 2213808 > kern.ipc.umtx_vnode_persistent: 0 > kern.minvnodes: 553452 > vm.vnode_pbufs: 2048 > vm.stats.vm.v_vnodepgsout: 22844 > vm.stats.vm.v_vnodepgsin: 582398 > vm.stats.vm.v_vnodeout: 890 > vm.stats.vm.v_vnodein: 34296 > vfs.vnode_alloc_sleeps: 2994 > vfs.wantfreevnodes: 553452 > vfs.freevnodes: 2209662 > vfs.vnodes_created: 12206299 > vfs.numvnodes: 2214071 > vfs.cache.debug.vnodes_cel_3_failures: 0 > vfs.cache.stats.heldvnodes: 459 > debug.vnode_domainset: > debug.sizeof.vnode: 448 > debug.fail_point.status_fill_kinfo_vnode__random_path: off > debug.fail_point.fill_kinfo_vnode__random_path: off > > Wait a while but still the mostly cpdup vlruwk status: > > # sysctl -a | grep vnode > kern.maxvnodes: 2213808 > kern.ipc.umtx_vnode_persistent: 0 > kern.minvnodes: 553452 > vm.vnode_pbufs: 2048 > vm.stats.vm.v_vnodepgsout: 22844 > vm.stats.vm.v_vnodepgsin: 583527 > vm.stats.vm.v_vnodeout: 890 > vm.stats.vm.v_vnodein: 34396 > vfs.vnode_alloc_sleeps: 8053 > vfs.wantfreevnodes: 553452 > vfs.freevnodes: 2210166 > vfs.vnodes_created: 12212061 > vfs.numvnodes: 2215106 > vfs.cache.debug.vnodes_cel_3_failures: 0 > vfs.cache.stats.heldvnodes: 497 > debug.vnode_domainset: > debug.sizeof.vnode: 448 > debug.fail_point.status_fill_kinfo_vnode__random_path: off > debug.fail_point.fill_kinfo_vnode__random_path: off > > ^C[00:14:55] Error: Signal SIGINT caught, cleaning up and exiting > > # sysctl -a | grep vnode > kern.maxvnodes: 2213808 > kern.ipc.umtx_vnode_persistent: 0 > kern.minvnodes: 553452 > vm.vnode_pbufs: 2048 > vm.stats.vm.v_vnodepgsout: 22844 > vm.stats.vm.v_vnodepgsin: 584474 > vm.stats.vm.v_vnodeout: 890 > vm.stats.vm.v_vnodein: 34591 > vfs.vnode_alloc_sleeps: 17584 > vfs.wantfreevnodes: 553452 > vfs.freevnodes: 2210796 > vfs.vnodes_created: 12222343 > vfs.numvnodes: 2216564 > vfs.cache.debug.vnodes_cel_3_failures: 0 > vfs.cache.stats.heldvnodes: 539 > debug.vnode_domainset: > debug.sizeof.vnode: 448 > debug.fail_point.status_fill_kinfo_vnode__random_path: off > debug.fail_point.fill_kinfo_vnode__random_path: off > > [main-amd64-bulk_a-default] [2023-08-23_13h58m08s] [sigint:] Queued: 34435 > Built: 2 Failed: 0 Skipped: 35 Ignored: 358 Fetched: 0 > Tobuild: 34040 Time: 00:14:36 > [00:16:13] Logs: > /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-23_13h58m08s > [00:16:49] Cleaning up > load: 5.28 cmd: sh 77057 [vlruwk] 141.63r 0.00u 30.98s 28% 6932k > #0 0xffffffff80b76ebb at mi_switch+0xbb > #1 0xffffffff80bc960f at sleepq_timedwait+0x2f > #2 0xffffffff80b76610 at _sleep+0x1d0 > #3 0xffffffff80c5b2dc at vn_alloc_hard+0x2ac > #4 0xffffffff80c50a12 at getnewvnode_reserve+0x92 > #5 0xffffffff829afb12 at zfs_zget+0x22 > #6 0xffffffff8299ca8d at zfs_dirent_lookup+0x16d > #7 0xffffffff8299cb5f at zfs_dirlook+0x7f > #8 0xffffffff829ac410 at zfs_lookup+0x350 > #9 0xffffffff829a782a at zfs_freebsd_cachedlookup+0x6a > #10 0xffffffff80c368ad at vfs_cache_lookup+0xad > #11 0xffffffff80c3b6d8 at cache_fplookup_final_modifying+0x188 > #12 0xffffffff80c38766 at cache_fplookup+0x356 > #13 0xffffffff80c43fb2 at namei+0x112 > #14 0xffffffff80c62e5b at kern_funlinkat+0x13b > #15 0xffffffff80c62d18 at sys_unlink+0x28 > #16 0xffffffff83b8e583 at filemon_wrapper_unlink+0x13 > #17 0xffffffff81049a79 at amd64_syscall+0x109 > > [00:26:28] Unmounting file systems > Exiting with status 1 > > # sysctl -a | grep vnode > kern.maxvnodes: 2213808 > kern.ipc.umtx_vnode_persistent: 0 > kern.minvnodes: 553452 > vm.vnode_pbufs: 2048 > vm.stats.vm.v_vnodepgsout: 22844 > vm.stats.vm.v_vnodepgsin: 585384 > vm.stats.vm.v_vnodeout: 890 > vm.stats.vm.v_vnodein: 34798 > vfs.vnode_alloc_sleeps: 27578 > vfs.wantfreevnodes: 553452 > vfs.freevnodes: 61362 > vfs.vnodes_created: 20135479 > vfs.numvnodes: 59860 > vfs.cache.debug.vnodes_cel_3_failures: 0 > vfs.cache.stats.heldvnodes: 208 > debug.vnode_domainset: > debug.sizeof.vnode: 448 > debug.fail_point.status_fill_kinfo_vnode__random_path: off > debug.fail_point.fill_kinfo_vnode__random_path: off > > > For reference (from after): > > # kldstat > Id Refs Address Size Name > 1 95 0xffffffff80200000 274b308 kernel > 2 1 0xffffffff8294c000 5d5238 zfs.ko > 3 1 0xffffffff82f22000 7718 cryptodev.ko > 4 1 0xffffffff83b10000 3390 acpi_wmi.ko > 5 1 0xffffffff83b14000 3220 intpm.ko > 6 1 0xffffffff83b18000 2178 smbus.ko > 7 1 0xffffffff83b1b000 2240 cpuctl.ko > 8 1 0xffffffff83b1e000 3360 uhid.ko > 9 1 0xffffffff83b22000 4364 ums.ko > 10 1 0xffffffff83b27000 33c0 usbhid.ko > 11 1 0xffffffff83b2b000 3380 hidbus.ko > 12 1 0xffffffff83b2f000 4d20 ng_ubt.ko > 13 6 0xffffffff83b34000 abb8 netgraph.ko > 14 2 0xffffffff83b3f000 a250 ng_hci.ko > 15 4 0xffffffff83b4a000 2670 ng_bluetooth.ko > 16 1 0xffffffff83b4d000 83a0 uftdi.ko > 17 1 0xffffffff83b56000 4e58 ucom.ko > 18 1 0xffffffff83b5b000 3360 wmt.ko > 19 1 0xffffffff83b5f000 e268 ng_l2cap.ko > 20 1 0xffffffff83b6e000 1bf68 ng_btsocket.ko > 21 1 0xffffffff83b8a000 38f8 ng_socket.ko > 22 1 0xffffffff83b8e000 3250 filemon.ko > 23 1 0xffffffff83b92000 4758 nullfs.ko > 24 1 0xffffffff83b97000 73c0 linprocfs.ko > 25 3 0xffffffff83b9f000 be70 linux_common.ko > 26 1 0xffffffff83bab000 3558 fdescfs.ko > 27 1 0xffffffff83baf000 31b20 linux.ko > 28 1 0xffffffff83be1000 2ed40 linux64.ko > > > Note that before the "Cleaning up" notice, the vfs.freevnodes > shows as being around (for example) 2210796. But after > "Exiting with status": 61362. vfs.vnodes_created has a > similar staging of in the ball park of up to 12222343 but > then the change to: 20135479. Similarly, vfs.numvnodes > 2216564 -> 59860. > > > > Anything else I should gather and report as basic information? > This is a known problem, but it is unclear if you should be running into it in this setup. Can you try again but this time *revert* 138a5dafba312ff39ce0eefdbe34de95519e600d, like so: git revert 138a5dafba312ff39ce0eefdbe34de95519e600d may want to switch to a different branch first, for example: git checkout -b vfstesting -- Mateusz Guzik From nobody Thu Aug 24 01:14:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWQ8n5ssjz4rbt0 for ; Thu, 24 Aug 2023 01:14:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWQ8n2dQWz3bTx for ; Thu, 24 Aug 2023 01:14:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692839654; bh=SgIxz7Vx8Q8osFVPJw6GkGlcqIOtsi9U3uUfh03HE04=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=RPWZqOZeH8dR4uKE8Wz/IYSxjeoFHBstGLzcbFYQqDb2tKb9tJ01q+PGc588Gp2ZPANp/SKq3NEKw4Rz5gz2VWSpKBSzLNjHGTIaDKqvZrE4qmxF9OQRI6SZqNWmgPTYphj2KPRk0axHWaKqH75b6gPiqxQ27UuLWGzvfqfRRsX0qVifhlPqPpU0agSZpHVVngaL/ltXQuNM0lhEVU8s1ePmH0c58Y/7U2KsS244VyFvwmD9Knuun6YD0+2/w0nY0VtzoJLbBV8Zswvqqv18iN3nxaHgd5AEsZrwoxLVjBoDZHYWCocs2CD3N8VrvC+Tvz4dzSPAqOESoBihaEcwFA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692839654; bh=yRVfc/lZ9hWPvG/5NXePySt8FZwbB/HG9QhKw5Sbh+7=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=qdYLss6+++6V6neqRdDkKm9aLE0N/DyYcP5v9aH/OB3h8SxGEu54j1KYx1y8x9UzcugaSTs4NaLY+TGFmL5WNokdqP92qvPQviLjdjJCkPcVuBWv6B5yYn/XYF6FX9GedLXInFu1l7vQvMyUxfhaZz2ndPXnVw6QVcNa1QmC1SXAvIvaxFBWRppERwzuBS/pD0+Kv7taxet2Qn3ItyYKdGCkZzfcLfm4UTCJkgYCSEA22TqJrR6kDJwE3auY4gjCb0BpVK3EVQaPvh88X2J0/UpDEIBPQouFrRkCGwTJCaIUJcta9QC2Uk1R+LWFTgA0s1Rd2Jnf4901333AqP/drg== X-YMail-OSG: xBA.jcwVM1nxXThuw5BmGS5PB5qPEeRNoMnHAwTXsFkcm19C9bpBjxJRt_CqWkt iQKRnAwQZJje7vAyQb3jN.AGGMWU0u.RbgPg7IB6XahQdNV50IZTGBYofjhHw3vfBUj8x6QMjsRi EvaT9WVw3IajNWO5EkmM8t76Ex_0IWcLHSHu_lyDgei1lESQxQ7ZCwgTLhzVkWEQN6IUsoSa9PZ0 jRzJgaMJv9c6LlPogPu2jfZa_aIoRS9C6Ro_urVZ4BTt7fkhapImxqOFucFpuQr92wmbJb7ldr1H xaCN0.vUlRprpYo7O_bq2I2jf7H2OATh8yF7YEVcVilfroS4YLasIQDtmHasip0yhqhPeaJcJd9C 7FfoBt_TNeOR9X4RPs5c8TglSQAP_d2Qi08PaeiM21GmnlpNbWhMAH7.m2mmSVJ.pnO6r8DpPv2S jJk4QtBNGr_uygPHXoK_B5PbpvGC74AK3Es0usFwR2NhMinFarbK4lbpoJrsZAxBMuj7uA9iz3Fj 6EkDR.NH37YLSfQOBfO1gGeJQE.8yIQOUZDH.groc8plp8jJOXldvcqgEtPFgd7c0PsziCsU37lN oYqdYf1doAKLKoRS8ncTE760YlKVjNzk85Nro808YnCNRAzxuLNw1cHfMea7_erWTllQRlJqcgDs wjT9ut8bIyq7a7m90B3e8gRcxW0Ve_TufdGISd4AJB7VWX7Hh1DAtEZ4jrEI2KG_yNCGU1.nIFfC pXj8JMtZZvW_0NyKs.TjhC1x0lJzt765LVOd0bw4Qisv8Lii3zpkS.cKyeTD1itFN1o78621BsU2 wtglXszYnLhj6iVLQ4XI0mvOaIVDfBQ7Ft6OJ444mgUxGNnfVj_eUFlTaFZEdNpojL9dpMvSGvqf sDkT_NjyHN9iR74CpjxKTnww0Bnp05xvzWO96aHQ2On7Va3VMzgYdCSl_XoshLoGxBQC0WnKV1pQ Qv.rcUt6j1J.GmzeX59ToQ3eCG.CARN3.pL1ah3kiTtL1igIfcw5IJjwvUH9RNBTO7deRBSZfJYQ u5dy_RB497pKKmheCTicWPMTjsmmLbSLe9x7y.D9AmKbIMvP6QGRAIPCjGZ1r5L5ryTJ_d.pH771 CWYmJ5VW6DoWtMrB8brH27cDNfMKsGaWUvTj1CAeBMdiGSYOUVmVn7LnU55X82K2hCDSXqgsRHrf ncQF5fMGnC3DR0CS.cvs.oYNJ.m55KVFjSogV9N92OcvePIzf72pdsa5BhdydzO04ZjPK_q9x7wg FeDgtsg8DA2XDV_917R5DqWbFoemOKjbQwtshoh79vtWWS3tWC2WBaaMmgkqPX9Ofd2_hURXavx8 sqxQCG.SzvdWnnIOaS8r9A0yJLEmPJAQmNpv7IGPaXwOvAy4Uer6A00MuoC1Ag2iBT3dSCkDy3uJ tqEP8uQcPUBWWFC85QxJlRSul3RLuboWJSwDxIqWQftgaHX14IIMFeABYbM1xGS1vWmSpH91jMDG qaJ47KnzhhGuNuEwE5a7FRGF94kdCLGXga1mYnTv2VBZebyvTdYFMuHAb8AgEPab_xeTSlmyLJzh 2X6V1GV2rmzpEI7p1Hws4ZUh5MP2J.zfj6xfrYYrD3BlgzsGr_kCo_ZgTZArGogIDC_d4a.Uk7Sa 6Yp.uR_71_ymYr1bq.c.adKTHKlIqkZOVu_OtvUYb8tEdK9EvALbHN5OP0DX5bPcJQJGQJTN7epG p4T2A5J0_zen38ZcXIm6BBkpV9rAFEBuYu5PIAc3YQ.KmDa24yWW8TB.2Clvjc4UH4QKCuWVh82F Wnp7umpNBiEwFN5.jDenY8sCoITLxQwYd.ceicPMr2fkfRiMiXQgiCaphpApJaZKv726Ahl49eKl f8qBQ7bpJgzIKNe2lMcxaX8b63Dcbh211FYD4y6y.Tx.6uMOR7VTmFuBCDYHVK5I3Wm6kkts76cZ DVf30EJnQIjLLcXGB74KBaefW2WVrT6nvp8Ac671dUwuGZ4Kuogq6wA8XNSWj.7taSp6aZPgrUvj R8WWFGaPYvPr2KZu2.5lGDjMStlDVVt3BnaBLWkoM98PMqiwHcMIjf_fxPmOO_WfJOIb9q8EGaAu K.vE5hIS8U3msWAec.c2mkjoC7k10OGaA911sQiFjNwmHkbLIx5yuE1AGpWn3PiS15au.Wf.XU5G mC_IF8YMFpt6QiaJUzLEc5EVkNZN.pKNQsdbd74.GoL9CgQr14erddwuUE9GXBE9TdhZfX1Ok_hQ UE1lwqkoiVhSfVCZx3zVpH93mNUl6O48ZUv3HSCuDL8GCF1KEDphwZKMrDTldKn2_z9g66SM7csa x X-Sonic-MF: X-Sonic-ID: f05c7ab8-387f-470d-aa26-958aa86912b7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Aug 2023 01:14:14 +0000 Received: by hermes--production-ne1-7b767b77cc-swgdr (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID edd85d3e5d7a59c7c1950f329de4c31b; Thu, 24 Aug 2023 01:14:12 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder From: Mark Millard In-Reply-To: Date: Wed, 23 Aug 2023 18:14:00 -0700 Cc: Current FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.700.6) X-Rspamd-Queue-Id: 4RWQ8n2dQWz3bTx X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] On Aug 23, 2023, at 15:10, Mateusz Guzik wrote: > On 8/23/23, Mark Millard wrote: >> [Forked off the ZFS deadlock 14 discussion, per feedback.] >>=20 >> On Aug 23, 2023, at 11:40, Alexander Motin wrote: >>=20 >>> On 22.08.2023 14:24, Mark Millard wrote: >>>> Alexander Motin wrote on >>>> Date: Tue, 22 Aug 2023 16:18:12 UTC : >>>>> I am waiting for final test results from George Wilson and then = will >>>>> request quick merge of both to zfs-2.2-release branch. = Unfortunately >>>>> there are still not many reviewers for the PR, since the code is = not >>>>> trivial, but at least with the test reports Brian Behlendorf and = Mark >>>>> Maybee seem to be OK to merge the two PRs into 2.2. If somebody = else >>>>> have tested and/or reviewed the PR, you may comment on it. >>>> I had written to the list that when I tried to test the system >>>> doing poudriere builds (initially with your patches) using >>>> USE_TMPFS=3Dno so that zfs had to deal with all the file I/O, I >>>> instead got only one builder that ended up active, the others >>>> never reaching "Builder started": >>>=20 >>>> Top was showing lots of "vlruwk" for the cpdup's. For example: >>>> . . . >>>> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 >>>> 0.00% cpdup -i0 -o ref 32 >>>> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 >>>> 0.01% cpdup -i0 -o ref 31 >>>> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 >>>> 0.01% cpdup -i0 -o ref 30 >>>> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 >>>> 0.01% cpdup -i0 -o ref 29 >>>> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 >>>> 0.01% cpdup -i0 -o ref 28 >>>> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 >>>> 0.00% cpdup -i0 -o ref 27 >>>> . . . >>>> But those processes did show CPU?? on occasion, as well as >>>> *vnode less often. None of the cpdup's was stuck in >>>> Removing your patches did not change the behavior. >>>=20 >>> Mark, to me "vlruwk" looks like a limit on number of vnodes. I was = not >>> deep in that area at least recently, so somebody with more = experience >>> there could try to diagnose it. At very least it does not look = related to >>> the ZIL issue discussed in this thread, at least with the = information >>> provided, so I am not surprised that the mentioned patches do not = affect >>> it. >>=20 >> I did the above intending to test the deadlock in my context but >> ended up not getting that far when I tried to make zfs handle all >> the file I/O (USE_TMPFS=3Dno and no other use of tmpfs or the like). >>=20 >> The zfs context is a simple single partition on the boot media. I >> use ZFS for bectl BE use, not for other typical reasons. The media >> here is PCIe Optane 1.4T media. The machine is a ThreadRipper >> 1950X, so first generation. 128 GiBytes of RAM. 491520 MiBytes of >> swap, also on that Optane. >>=20 >> # uname -apKU >> FreeBSD amd64-ZFS 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #112 >> main-n264912-b1d3e2b77155-dirty: Sun Aug 20 10:01:48 PDT 2023 >> = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG >> amd64 amd64 1400096 1400096 >>=20 >> The GENERIC-DBG variant of the kernel did not report any issues in >> earlier testing. >>=20 >> The alter referenced /usr/obj/DESTDIRs/main-amd64-poud-bulk_a was >> installed from the same build. >>=20 >> # zfs list >> NAME USED AVAIL REFER >> MOUNTPOINT >> zoptb 79.9G 765G 96K = /zoptb >> zoptb/BUILDs 20.5G 765G 8.29M >> /usr/obj/BUILDs >> zoptb/BUILDs/alt-main-amd64-dbg-clang-alt 1.86M 765G 1.86M >> /usr/obj/BUILDs/alt-main-amd64-dbg-clang-alt >> zoptb/BUILDs/alt-main-amd64-nodbg-clang-alt 30.2M 765G 30.2M >> /usr/obj/BUILDs/alt-main-amd64-nodbg-clang-alt >> zoptb/BUILDs/main-amd64-dbg-clang 9.96G 765G 9.96G >> /usr/obj/BUILDs/main-amd64-dbg-clang >> zoptb/BUILDs/main-amd64-dbg-gccxtc 38.5M 765G 38.5M >> /usr/obj/BUILDs/main-amd64-dbg-gccxtc >> zoptb/BUILDs/main-amd64-nodbg-clang 10.3G 765G 10.3G >> /usr/obj/BUILDs/main-amd64-nodbg-clang >> zoptb/BUILDs/main-amd64-nodbg-clang-alt 37.2M 765G 37.2M >> /usr/obj/BUILDs/main-amd64-nodbg-clang-alt >> zoptb/BUILDs/main-amd64-nodbg-gccxtc 94.6M 765G 94.6M >> /usr/obj/BUILDs/main-amd64-nodbg-gccxtc >> zoptb/DESTDIRs 4.33G 765G 104K >> /usr/obj/DESTDIRs >> zoptb/DESTDIRs/main-amd64-poud 2.16G 765G 2.16G >> /usr/obj/DESTDIRs/main-amd64-poud >> zoptb/DESTDIRs/main-amd64-poud-bulk_a 2.16G 765G 2.16G >> /usr/obj/DESTDIRs/main-amd64-poud-bulk_a >> zoptb/ROOT 13.1G 765G 96K = none >> zoptb/ROOT/build_area_for-main-amd64 5.03G 765G 3.24G = none >> zoptb/ROOT/main-amd64 8.04G 765G 3.23G = none >> zoptb/poudriere 6.58G 765G 112K >> /usr/local/poudriere >> zoptb/poudriere/data 6.58G 765G 128K >> /usr/local/poudriere/data >> zoptb/poudriere/data/.m 112K 765G 112K >> /usr/local/poudriere/data/.m >> zoptb/poudriere/data/cache 17.4M 765G 17.4M >> /usr/local/poudriere/data/cache >> zoptb/poudriere/data/images 96K 765G 96K >> /usr/local/poudriere/data/images >> zoptb/poudriere/data/logs 2.72G 765G 2.72G >> /usr/local/poudriere/data/logs >> zoptb/poudriere/data/packages 3.84G 765G 3.84G >> /usr/local/poudriere/data/packages >> zoptb/poudriere/data/wrkdirs 112K 765G 112K >> /usr/local/poudriere/data/wrkdirs >> zoptb/poudriere/jails 96K 765G 96K >> /usr/local/poudriere/jails >> zoptb/poudriere/ports 96K 765G 96K >> /usr/local/poudriere/ports >> zoptb/tmp 68.5M 765G 68.5M = /tmp >> zoptb/usr 35.1G 765G 96K = /usr >> zoptb/usr/13_0R-src 2.64G 765G 2.64G >> /usr/13_0R-src >> zoptb/usr/alt-main-src 96K 765G 96K >> /usr/alt-main-src >> zoptb/usr/home 181M 765G 181M = /usr/home >> zoptb/usr/local 5.08G 765G 5.08G >> /usr/local >> zoptb/usr/main-src 833M 765G 833M >> /usr/main-src >> zoptb/usr/ports 26.4G 765G 26.4G >> /usr/ports >> zoptb/usr/src 96K 765G 96K = /usr/src >> zoptb/var 52.6M 765G 96K = /var >> zoptb/var/audit 356K 765G 356K >> /var/audit >> zoptb/var/crash 128K 765G 128K >> /var/crash >> zoptb/var/db 49.7M 765G 96K = /var/db >> zoptb/var/db/pkg 49.4M 765G 49.4M >> /var/db/pkg >> zoptb/var/db/ports 164K 765G 164K >> /var/db/ports >> zoptb/var/log 1.61M 765G 1.61M = /var/log >> zoptb/var/mail 632K 765G 632K = /var/mail >> zoptb/var/tmp 128K 765G 128K = /var/tmp >>=20 >> # poudriere jail -jmain-amd64-bulk_a -i >> Jail name: main-amd64-bulk_a >> Jail version: 14.0-ALPHA2 >> Jail arch: amd64 >> Jail method: null >> Jail mount: /usr/obj/DESTDIRs/main-amd64-poud-bulk_a >> Jail fs: >> Jail updated: 2021-12-04 14:55:22 >> Jail pkgbase: disabled >>=20 >>=20 >>=20 >> So, setting up another test with some related information >> shown before, during, and after. sysctl output is from >> another ssh session than the bulk -a run. >>=20 >> # sysctl -a | grep vnode >> kern.maxvnodes: 2213808 >> kern.ipc.umtx_vnode_persistent: 0 >> kern.minvnodes: 553452 >> vm.vnode_pbufs: 2048 >> vm.stats.vm.v_vnodepgsout: 0 >> vm.stats.vm.v_vnodepgsin: 272429 >> vm.stats.vm.v_vnodeout: 0 >> vm.stats.vm.v_vnodein: 12461 >> vfs.vnode_alloc_sleeps: 0 >> vfs.wantfreevnodes: 553452 >> vfs.freevnodes: 962766 >> vfs.vnodes_created: 2538980 >> vfs.numvnodes: 1056233 >> vfs.cache.debug.vnodes_cel_3_failures: 0 >> vfs.cache.stats.heldvnodes: 91878 >> debug.vnode_domainset: >> debug.sizeof.vnode: 448 >> debug.fail_point.status_fill_kinfo_vnode__random_path: off >> debug.fail_point.fill_kinfo_vnode__random_path: off >>=20 >> # poudriere bulk -jmain-amd64-bulk_a -a >> . . . >> [00:01:34] Building 34042 packages using up to 32 builders >> [00:01:34] Hit CTRL+t at any time to see build progress and stats >> [00:01:34] [01] [00:00:00] Builder starting >> [00:01:57] [01] [00:00:23] Builder started >> [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 >> [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: = Success >> [00:03:22] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 >> [00:03:22] [02] [00:00:00] Builder starting >> [00:03:22] [03] [00:00:00] Builder starting >> . . . >> [00:03:22] [31] [00:00:00] Builder starting >> [00:03:22] [32] [00:00:00] Builder starting >> [00:03:31] [01] [00:00:09] Finished print/indexinfo | = indexinfo-0.3.1: >> Success >> [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | >> gettext-runtime-0.22 >> . . . >>=20 >> Note that only [01] makes progress: no new "Builder started" >> notices occur. top shows 31 of the pattern: >> cpdup -i0 -o ref ?? >>=20 >> Then during the 31 cpudup's showing vlruwk most of the time: >>=20 >> # sysctl -a | grep vnode >> kern.maxvnodes: 2213808 >> kern.ipc.umtx_vnode_persistent: 0 >> kern.minvnodes: 553452 >> vm.vnode_pbufs: 2048 >> vm.stats.vm.v_vnodepgsout: 22844 >> vm.stats.vm.v_vnodepgsin: 582398 >> vm.stats.vm.v_vnodeout: 890 >> vm.stats.vm.v_vnodein: 34296 >> vfs.vnode_alloc_sleeps: 2994 >> vfs.wantfreevnodes: 553452 >> vfs.freevnodes: 2209662 >> vfs.vnodes_created: 12206299 >> vfs.numvnodes: 2214071 >> vfs.cache.debug.vnodes_cel_3_failures: 0 >> vfs.cache.stats.heldvnodes: 459 >> debug.vnode_domainset: >> debug.sizeof.vnode: 448 >> debug.fail_point.status_fill_kinfo_vnode__random_path: off >> debug.fail_point.fill_kinfo_vnode__random_path: off >>=20 >> Wait a while but still the mostly cpdup vlruwk status: >>=20 >> # sysctl -a | grep vnode >> kern.maxvnodes: 2213808 >> kern.ipc.umtx_vnode_persistent: 0 >> kern.minvnodes: 553452 >> vm.vnode_pbufs: 2048 >> vm.stats.vm.v_vnodepgsout: 22844 >> vm.stats.vm.v_vnodepgsin: 583527 >> vm.stats.vm.v_vnodeout: 890 >> vm.stats.vm.v_vnodein: 34396 >> vfs.vnode_alloc_sleeps: 8053 >> vfs.wantfreevnodes: 553452 >> vfs.freevnodes: 2210166 >> vfs.vnodes_created: 12212061 >> vfs.numvnodes: 2215106 >> vfs.cache.debug.vnodes_cel_3_failures: 0 >> vfs.cache.stats.heldvnodes: 497 >> debug.vnode_domainset: >> debug.sizeof.vnode: 448 >> debug.fail_point.status_fill_kinfo_vnode__random_path: off >> debug.fail_point.fill_kinfo_vnode__random_path: off >>=20 >> ^C[00:14:55] Error: Signal SIGINT caught, cleaning up and exiting >>=20 >> # sysctl -a | grep vnode >> kern.maxvnodes: 2213808 >> kern.ipc.umtx_vnode_persistent: 0 >> kern.minvnodes: 553452 >> vm.vnode_pbufs: 2048 >> vm.stats.vm.v_vnodepgsout: 22844 >> vm.stats.vm.v_vnodepgsin: 584474 >> vm.stats.vm.v_vnodeout: 890 >> vm.stats.vm.v_vnodein: 34591 >> vfs.vnode_alloc_sleeps: 17584 >> vfs.wantfreevnodes: 553452 >> vfs.freevnodes: 2210796 >> vfs.vnodes_created: 12222343 >> vfs.numvnodes: 2216564 >> vfs.cache.debug.vnodes_cel_3_failures: 0 >> vfs.cache.stats.heldvnodes: 539 >> debug.vnode_domainset: >> debug.sizeof.vnode: 448 >> debug.fail_point.status_fill_kinfo_vnode__random_path: off >> debug.fail_point.fill_kinfo_vnode__random_path: off >>=20 >> [main-amd64-bulk_a-default] [2023-08-23_13h58m08s] [sigint:] Queued: = 34435 >> Built: 2 Failed: 0 Skipped: 35 Ignored: 358 Fetched: 0 >> Tobuild: 34040 Time: 00:14:36 >> [00:16:13] Logs: >> = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-23_1= 3h58m08s >> [00:16:49] Cleaning up >> load: 5.28 cmd: sh 77057 [vlruwk] 141.63r 0.00u 30.98s 28% 6932k >> #0 0xffffffff80b76ebb at mi_switch+0xbb >> #1 0xffffffff80bc960f at sleepq_timedwait+0x2f >> #2 0xffffffff80b76610 at _sleep+0x1d0 >> #3 0xffffffff80c5b2dc at vn_alloc_hard+0x2ac >> #4 0xffffffff80c50a12 at getnewvnode_reserve+0x92 >> #5 0xffffffff829afb12 at zfs_zget+0x22 >> #6 0xffffffff8299ca8d at zfs_dirent_lookup+0x16d >> #7 0xffffffff8299cb5f at zfs_dirlook+0x7f >> #8 0xffffffff829ac410 at zfs_lookup+0x350 >> #9 0xffffffff829a782a at zfs_freebsd_cachedlookup+0x6a >> #10 0xffffffff80c368ad at vfs_cache_lookup+0xad >> #11 0xffffffff80c3b6d8 at cache_fplookup_final_modifying+0x188 >> #12 0xffffffff80c38766 at cache_fplookup+0x356 >> #13 0xffffffff80c43fb2 at namei+0x112 >> #14 0xffffffff80c62e5b at kern_funlinkat+0x13b >> #15 0xffffffff80c62d18 at sys_unlink+0x28 >> #16 0xffffffff83b8e583 at filemon_wrapper_unlink+0x13 >> #17 0xffffffff81049a79 at amd64_syscall+0x109 >>=20 >> [00:26:28] Unmounting file systems >> Exiting with status 1 >>=20 >> # sysctl -a | grep vnode >> kern.maxvnodes: 2213808 >> kern.ipc.umtx_vnode_persistent: 0 >> kern.minvnodes: 553452 >> vm.vnode_pbufs: 2048 >> vm.stats.vm.v_vnodepgsout: 22844 >> vm.stats.vm.v_vnodepgsin: 585384 >> vm.stats.vm.v_vnodeout: 890 >> vm.stats.vm.v_vnodein: 34798 >> vfs.vnode_alloc_sleeps: 27578 >> vfs.wantfreevnodes: 553452 >> vfs.freevnodes: 61362 >> vfs.vnodes_created: 20135479 >> vfs.numvnodes: 59860 >> vfs.cache.debug.vnodes_cel_3_failures: 0 >> vfs.cache.stats.heldvnodes: 208 >> debug.vnode_domainset: >> debug.sizeof.vnode: 448 >> debug.fail_point.status_fill_kinfo_vnode__random_path: off >> debug.fail_point.fill_kinfo_vnode__random_path: off >>=20 >>=20 >> For reference (from after): >>=20 >> # kldstat >> Id Refs Address Size Name >> 1 95 0xffffffff80200000 274b308 kernel >> 2 1 0xffffffff8294c000 5d5238 zfs.ko >> 3 1 0xffffffff82f22000 7718 cryptodev.ko >> 4 1 0xffffffff83b10000 3390 acpi_wmi.ko >> 5 1 0xffffffff83b14000 3220 intpm.ko >> 6 1 0xffffffff83b18000 2178 smbus.ko >> 7 1 0xffffffff83b1b000 2240 cpuctl.ko >> 8 1 0xffffffff83b1e000 3360 uhid.ko >> 9 1 0xffffffff83b22000 4364 ums.ko >> 10 1 0xffffffff83b27000 33c0 usbhid.ko >> 11 1 0xffffffff83b2b000 3380 hidbus.ko >> 12 1 0xffffffff83b2f000 4d20 ng_ubt.ko >> 13 6 0xffffffff83b34000 abb8 netgraph.ko >> 14 2 0xffffffff83b3f000 a250 ng_hci.ko >> 15 4 0xffffffff83b4a000 2670 ng_bluetooth.ko >> 16 1 0xffffffff83b4d000 83a0 uftdi.ko >> 17 1 0xffffffff83b56000 4e58 ucom.ko >> 18 1 0xffffffff83b5b000 3360 wmt.ko >> 19 1 0xffffffff83b5f000 e268 ng_l2cap.ko >> 20 1 0xffffffff83b6e000 1bf68 ng_btsocket.ko >> 21 1 0xffffffff83b8a000 38f8 ng_socket.ko >> 22 1 0xffffffff83b8e000 3250 filemon.ko >> 23 1 0xffffffff83b92000 4758 nullfs.ko >> 24 1 0xffffffff83b97000 73c0 linprocfs.ko >> 25 3 0xffffffff83b9f000 be70 linux_common.ko >> 26 1 0xffffffff83bab000 3558 fdescfs.ko >> 27 1 0xffffffff83baf000 31b20 linux.ko >> 28 1 0xffffffff83be1000 2ed40 linux64.ko >>=20 >>=20 >> Note that before the "Cleaning up" notice, the vfs.freevnodes >> shows as being around (for example) 2210796. But after >> "Exiting with status": 61362. vfs.vnodes_created has a >> similar staging of in the ball park of up to 12222343 but >> then the change to: 20135479. Similarly, vfs.numvnodes >> 2216564 -> 59860. >>=20 >>=20 >>=20 >> Anything else I should gather and report as basic information? >>=20 >=20 > This is a known problem, but it is unclear if you should be running > into it in this setup. The changed fixed the issue: so I do run into the the issue for this setup. See below. > Can you try again but this time *revert* > 138a5dafba312ff39ce0eefdbe34de95519e600d, like so: > git revert 138a5dafba312ff39ce0eefdbe34de95519e600d >=20 > may want to switch to a different branch first, for example: git > checkout -b vfstesting # git -C /usr/main-src/ diff sys/kern/vfs_subr.c diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c index 0f3f00abfd4a..5dff556ac258 100644 --- a/sys/kern/vfs_subr.c +++ b/sys/kern/vfs_subr.c @@ -3528,25 +3528,17 @@ vdbatch_process(struct vdbatch *vd) MPASS(curthread->td_pinned > 0); MPASS(vd->index =3D=3D VDBATCH_SIZE); + mtx_lock(&vnode_list_mtx); critical_enter(); - if (mtx_trylock(&vnode_list_mtx)) { - for (i =3D 0; i < VDBATCH_SIZE; i++) { - vp =3D vd->tab[i]; - vd->tab[i] =3D NULL; - TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); - TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); - MPASS(vp->v_dbatchcpu !=3D NOCPU); - vp->v_dbatchcpu =3D NOCPU; - } - mtx_unlock(&vnode_list_mtx); - } else { - for (i =3D 0; i < VDBATCH_SIZE; i++) { - vp =3D vd->tab[i]; - vd->tab[i] =3D NULL; - MPASS(vp->v_dbatchcpu !=3D NOCPU); - vp->v_dbatchcpu =3D NOCPU; - } + for (i =3D 0; i < VDBATCH_SIZE; i++) { + vp =3D vd->tab[i]; + TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); + TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); + MPASS(vp->v_dbatchcpu !=3D NOCPU); + vp->v_dbatchcpu =3D NOCPU; } + mtx_unlock(&vnode_list_mtx); + bzero(vd->tab, sizeof(vd->tab)); vd->index =3D 0; critical_exit(); } Still with: # grep USE_TMPFS=3D /usr/local/etc/poudriere.conf # EXAMPLE: USE_TMPFS=3D"wrkdir data" #USE_TMPFS=3Dall #USE_TMPFS=3D"data" USE_TMPFS=3Dno That allowed the other builders to eventually reach "Builder started" and later activity, [00:05:50] [27] [00:02:29] Builder started being the first non-[01] to do so, no vlruwk's observed in what I saw in top: # poudriere bulk -jmain-amd64-bulk_a -a . . . [00:01:34] Building 34042 packages using up to 32 builders [00:01:34] Hit CTRL+t at any time to see build progress and stats [00:01:34] [01] [00:00:00] Builder starting [00:01:57] [01] [00:00:23] Builder started [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 [00:03:08] [01] [00:01:11] Finished ports-mgmt/pkg | pkg-1.20.4: Success [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 [00:03:21] [02] [00:00:00] Builder starting [00:03:21] [03] [00:00:00] Builder starting [00:03:21] [04] [00:00:00] Builder starting [00:03:21] [05] [00:00:00] Builder starting [00:03:21] [06] [00:00:00] Builder starting [00:03:21] [07] [00:00:00] Builder starting [00:03:21] [08] [00:00:00] Builder starting [00:03:21] [09] [00:00:00] Builder starting [00:03:21] [10] [00:00:00] Builder starting [00:03:21] [11] [00:00:00] Builder starting [00:03:21] [12] [00:00:00] Builder starting [00:03:21] [13] [00:00:00] Builder starting [00:03:21] [14] [00:00:00] Builder starting [00:03:21] [15] [00:00:00] Builder starting [00:03:21] [16] [00:00:00] Builder starting [00:03:21] [17] [00:00:00] Builder starting [00:03:21] [18] [00:00:00] Builder starting [00:03:21] [19] [00:00:00] Builder starting [00:03:21] [20] [00:00:00] Builder starting [00:03:21] [21] [00:00:00] Builder starting [00:03:21] [22] [00:00:00] Builder starting [00:03:21] [23] [00:00:00] Builder starting [00:03:21] [24] [00:00:00] Builder starting [00:03:21] [25] [00:00:00] Builder starting [00:03:21] [26] [00:00:00] Builder starting [00:03:21] [27] [00:00:00] Builder starting [00:03:21] [28] [00:00:00] Builder starting [00:03:21] [29] [00:00:00] Builder starting [00:03:21] [30] [00:00:00] Builder starting [00:03:21] [31] [00:00:00] Builder starting [00:03:21] [32] [00:00:00] Builder starting [00:03:29] [01] [00:00:08] Finished print/indexinfo | indexinfo-0.3.1: = Success [00:03:30] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 [00:04:49] [01] [00:01:19] Finished devel/gettext-runtime | = gettext-runtime-0.22: Success [00:04:52] [01] [00:00:00] Building devel/gmake | gmake-4.3_2 [00:05:20] [01] [00:00:28] Finished devel/gmake | gmake-4.3_2: Success [00:05:23] [01] [00:00:00] Building devel/libtextstyle | = libtextstyle-0.22 [00:05:50] [27] [00:02:29] Builder started [00:05:50] [27] [00:00:00] Building devel/npth | npth-1.6 [00:05:50] [08] [00:02:29] Builder started [00:05:50] [08] [00:00:00] Building devel/boost-jam | boost-jam-1.82.0 [00:05:51] [16] [00:02:30] Builder started [00:05:51] [16] [00:00:00] Building devel/libdaemon | libdaemon-0.14_1 [00:05:51] [19] [00:02:30] Builder started [00:05:51] [19] [00:00:00] Building graphics/libpotrace | = libpotrace-1.16 [00:05:51] [14] [00:02:30] Builder started [00:05:51] [14] [00:00:00] Building sysutils/gnome_subr | gnome_subr-1.0 [00:05:51] [17] [00:02:30] Builder started [00:05:51] [17] [00:00:00] Building misc/hicolor-icon-theme | = hicolor-icon-theme-0.17 [00:05:51] [26] [00:02:30] Builder started [00:05:51] [26] [00:00:00] Building devel/libinotify | = libinotify-20211018 [00:05:51] [12] [00:02:30] Builder started [00:05:51] [12] [00:00:00] Building devel/evdev-proto | evdev-proto-5.8 [00:05:51] [29] [00:02:30] Builder started [00:05:51] [29] [00:00:00] Building devel/opencl | opencl-3.0.14 [00:05:51] [05] [00:02:30] Builder started [00:05:51] [05] [00:00:00] Building devel/autoconf-switch | = autoconf-switch-20220527 [00:05:51] [11] [00:02:30] Builder started [00:05:51] [11] [00:00:00] Building textproc/html2text | = html2text-1.3.2a_1,1 [00:05:51] [25] [00:02:30] Builder started [00:05:51] [04] [00:02:30] Builder started [00:05:51] [25] [00:00:00] Building audio/speexdsp | speexdsp-1.2.1 [00:05:51] [04] [00:00:00] Building converters/libiconv | libiconv-1.17 [00:05:51] [22] [00:02:30] Builder started [00:05:51] [22] [00:00:00] Building dns/public_suffix_list | = public_suffix_list-20230630 [00:05:51] [20] [00:02:30] Builder started [00:05:51] [02] [00:02:30] Builder started [00:05:51] [24] [00:02:30] Builder started [00:05:51] [31] [00:02:30] Builder started [00:05:51] [20] [00:00:00] Building print/gsfonts | gsfonts-8.11_8 . . . I had waited a very long time with no such progress for [02]..[32] for the prior code. Now testing for the zfs deadlock issue should be possible for this setup. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 24 01:41:30 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWQmZ3zW6z4rchB for ; Thu, 24 Aug 2023 01:41:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWQmX6GhDz3dlL for ; Thu, 24 Aug 2023 01:41:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=U0Xj1Ta2; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692841306; bh=VlBqyLXuO3rW0ln2Q3DVGWpk5xZEegEDq4irev58nLI=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=U0Xj1Ta2twpjjCpHfO9jIBZMKo6qrG61WsCMl5oZ2Q7DF5bOTxT20hiVrCKxhHLoAZYJZCegEO2XtaaWMBrfH9hB9f4Jztj8YpSRFgjCkeWcqemV3bBepifqf2WmjnpZR9rd84frv/Jmg3AdFvfgZMmPIcMg9zp+Z6KaxALVXNFilDGTof1YsoMW84ltSy1PEC3jHrIsKzS14WrLt1JrSpJhf5+ZXqEK0HTeT74Nuqos1QracDOD7CVJ+fklf3OthfPsViMEZLLlkfoMTnqzIJ8attMhPA8si5JAjEqC8vaKvnEyB3hNf3bbv8Q4xI47lya4frrUuD8UPof1sZrdFQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692841306; bh=wFsQYdu4iK+ID3QsCvC3z38LHP1w3EZg0VfNuENkuzH=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=sNzYIOD/JS8gzGNFu8KIJvkdtsWv/byuHuk7Cxc3yNDrsgXWktDBYzAa6meY2BoxryanXYh0lK+ODKN9Qkjie6DpI8MDkb42v1ztEYFDqy0WMVYcGyyDzuRoh+WpFzvmoytbpRsNOEiqd7c4/QupZnzig0aXQhq5ZsYr6MdG9OLIF1/dF0jn67MRzAK/GdEFzWdcXcaPNIqK6yf+13HcKDGdJW/Yuz3R/u3yaRvm2B8MrNUXHwrmsWKTlpuN57NgjDA7dO45c+58WyMOkb6+yFEBCSwYZyMEYct7VFcxr+pV+/4bW4lw4x0d91h8rHBkma+rckFJXs7sbuwlQEPaNw== X-YMail-OSG: vBA0H48VM1ktZxHAhWkipLUkttsJAZWus_1OmI81VccVYnyOyreqLp5oCmQrHUQ sC0f6HRFRV0nTcoc3swehKOyvmeA7CyZqLNMiCGidnndH7_k3UQPoWDSlCjzc_1iZZv9wYaNMh_O XLluwDZONyc1nR8MI3wIv6eDVFRkUzZ6K5IAM.44hZj4moL4km10AipK6V2_Dkf2Lb4ncpWx6L9q C7NnUyMCN4XuWjjoNviEKE2jKHDDbtBlR25QpIXIlYdjf4tzT_17gGDk_O3HnOtny98erCpt3wbV VvHCJqwwdqheCgKtR5BUy2_90XvaBgLZiTYLdhwT5T82FihAHZ_bXrnhdrTPV43oJx7JmDZMRlkx 1FRLU_ZseMBdLn6dbr384x1LBehXv1bhFzMVmEf0U0czEZopwbpTBWqEa8CstI.WbyDsJgJYQVib Z2WRfSaOPDpPEWAYHag9FYCSjIfGUHihalm1UKTYeANWr5c0l15pp5rKbatp_VTcpUWTeVPYYb9b mItCSv487r8XN1zBoQbXLhvAaLv_D_YzhFJP.BR3tLfuStBctcXWmVTh.MbJ9xA01.ioLzUgWcYw OK3MwmbgD4p.7bL6rXN3moivfKiPSKSgEzYcI3S5mgkVPB1dOyB1xI7eYU9NhJ6QoTxtNMFwq_08 dMVZpZJuMOMnAqIbAIj5CkYlTbzaWda71OymmPvCdW5s7tqo5oC68LZpneGfl1sA7mJtlp6pnLC6 TcadkCX1DyiySBJeB_X2YUU3H.e4hi3lDMFzCAT6Y6BSmVFxuxZJ9C_XE0tmDsfttPn12chgIy.5 M.p21y0bu4uEyJOSlF7IRYpf_l.PHMPegQT7g_8691Tb7EpAnQjg3Os3I9iii.lzxh4zYkxEaOq4 08GnsgJOyUdfC8UKm.cj2jUpLgIJjdb_2g9NXVK8IqJTByE3MBIND60VYRcoD.gfEjcuNNmYaDc3 H.62jdUFSJBF2DZaHrJTHP.XSVOoSTgxmKPgtH5LBkyoESttY1tLypGj_ck3sH0iIxHsVONTfLK2 LJA4V7VG3uxrpCbS6vZXJCoGKph9xnE1e0if1N.WWbKg7DgAOEHM3PCQmPrZ2ei.5N6_kH0JtMcL xtnVUmVtBSXDxFrAY5K92aFkuKmywRpFwQnpt37jL1pXHuRJW1C3Mcu3WAaQFnmReMC.VY3oyabr UZ9HhFZWKnlMMqcrfTyfzyMTzKhGFhhYihKQpHMHCdy6l8tw8iwmn0F7ehXr4VUJrG7sre6X2vVl uOqc8Oo2jQmyfTnnBzR0wX4L7Z3LlUPh8dp7jzFwb03ynck7EievGgvgeuBUWyGYa.KN2SfUYtRi B_16b.JVAgDKufmZc4TSr0hlEw7cwIrcU0xvZGd4RspG_wLPkET4gmI2GvycB0tGGpKvlktczp_5 HVMWD3Z92OXJ_mQC4uW87fChTkpeLwNtnJWCV_3SCVTLniEkqyWCjx76gbq.GHnpDRJvIC6eJ3Jv sbvKm2.KRRIR3ElxpXN5Ri__n1oEogH6gwTnUy8s4xSLVZj9ryHaB_4sVu2LD5vntyrBIzdOiJGI taA7y5v84FmrVyEL8pCOVBIuBH4STorMqpqnNGKvhO_ZtozY.i4TTmZkBIid9n29svOD2jUyld65 33DhO5KzO2dnIT5tSbwrCy0oiDxKTEVpAs0G2SWG9YnYKGu8g355RiZlFu0MfUUiP3GSxwkLmP2U rBvurx8VZb3MS9pDNK9ibQ9jyhU8atS8OiEVWPHDXQMQmcqaUKFLuhRZ2W6x.HbgXDSel6YetcW9 mewirLK5MrX55AMvPXrB2UyH5IaY3klyEM1fiu.k9wPKMd6a173YxJ8W2Xlp858htKxX5NTF5bgu .XkhbFe03Rslum9DFqtiq.AknWVFwZTIIq4z2VUtikItqwVHMnpbefMa1YaEyeEPHG2n1w14a0gA rD2duWNEwYsUaAyVHELC0mYSf8Tw1k5CLw0gcKloW_wbTIc2rGPQepbE.UV9IqKutuB9Azb2sbYp _pWgd8enS4wOCwBPbpjy4XQBnC8Kv_mAnR7uXgcjsItb8QmXVUf3_2T4csUr3914TR9inVfjnYkl nUQi1vHMUUS021QJvn_FgguWR.IrK80N2EPQTSqOideQKHKMHYrPWDAxfQiNI0RU53koOgJPb1OC hLnTR7szz7QP7T.Mcei.ixBX8.wTwiO8ynGGUjfxmV9d9z93qne6LfHFGUDQ2w1qNtTa5AiOPBqQ A4Bj8j3jxK9z970M4JoMvxQ1vYPmQ_wC8hgc9bup3CYwJW6dNaYIxNiz7nOQey_tOVP38EHM9Ig9 P X-Sonic-MF: X-Sonic-ID: f543586c-aad4-4488-be12-a0bf6f1a16ca Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Aug 2023 01:41:46 +0000 Received: by hermes--production-gq1-6b7c87dcf5-rj4xx (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 4449366cb4b968c9513d4b8a16a35c00; Thu, 24 Aug 2023 01:41:41 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder From: Mark Millard In-Reply-To: Date: Wed, 23 Aug 2023 18:41:30 -0700 Cc: Current FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: <9519E9A0-22CE-4FCA-BE04-AB053D4DEA19@yahoo.com> References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWQmX6GhDz3dlL On Aug 23, 2023, at 18:14, Mark Millard wrote: > On Aug 23, 2023, at 15:10, Mateusz Guzik wrote: >=20 >> On 8/23/23, Mark Millard wrote: >>> [Forked off the ZFS deadlock 14 discussion, per feedback.] >>>=20 >>> On Aug 23, 2023, at 11:40, Alexander Motin wrote: >>>=20 >>>> On 22.08.2023 14:24, Mark Millard wrote: >>>>> Alexander Motin wrote on >>>>> Date: Tue, 22 Aug 2023 16:18:12 UTC : >>>>>> I am waiting for final test results from George Wilson and then = will >>>>>> request quick merge of both to zfs-2.2-release branch. = Unfortunately >>>>>> there are still not many reviewers for the PR, since the code is = not >>>>>> trivial, but at least with the test reports Brian Behlendorf and = Mark >>>>>> Maybee seem to be OK to merge the two PRs into 2.2. If somebody = else >>>>>> have tested and/or reviewed the PR, you may comment on it. >>>>> I had written to the list that when I tried to test the system >>>>> doing poudriere builds (initially with your patches) using >>>>> USE_TMPFS=3Dno so that zfs had to deal with all the file I/O, I >>>>> instead got only one builder that ended up active, the others >>>>> never reaching "Builder started": >>>>=20 >>>>> Top was showing lots of "vlruwk" for the cpdup's. For example: >>>>> . . . >>>>> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 = 4:23 >>>>> 0.00% cpdup -i0 -o ref 32 >>>>> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 = 4:20 >>>>> 0.01% cpdup -i0 -o ref 31 >>>>> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 = 4:30 >>>>> 0.01% cpdup -i0 -o ref 30 >>>>> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 = 4:18 >>>>> 0.01% cpdup -i0 -o ref 29 >>>>> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 = 4:33 >>>>> 0.01% cpdup -i0 -o ref 28 >>>>> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 = 4:28 >>>>> 0.00% cpdup -i0 -o ref 27 >>>>> . . . >>>>> But those processes did show CPU?? on occasion, as well as >>>>> *vnode less often. None of the cpdup's was stuck in >>>>> Removing your patches did not change the behavior. >>>>=20 >>>> Mark, to me "vlruwk" looks like a limit on number of vnodes. I was = not >>>> deep in that area at least recently, so somebody with more = experience >>>> there could try to diagnose it. At very least it does not look = related to >>>> the ZIL issue discussed in this thread, at least with the = information >>>> provided, so I am not surprised that the mentioned patches do not = affect >>>> it. >>>=20 >>> I did the above intending to test the deadlock in my context but >>> ended up not getting that far when I tried to make zfs handle all >>> the file I/O (USE_TMPFS=3Dno and no other use of tmpfs or the like). >>>=20 >>> The zfs context is a simple single partition on the boot media. I >>> use ZFS for bectl BE use, not for other typical reasons. The media >>> here is PCIe Optane 1.4T media. The machine is a ThreadRipper >>> 1950X, so first generation. 128 GiBytes of RAM. 491520 MiBytes of >>> swap, also on that Optane. >>>=20 >>> # uname -apKU >>> FreeBSD amd64-ZFS 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #112 >>> main-n264912-b1d3e2b77155-dirty: Sun Aug 20 10:01:48 PDT 2023 >>> = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG >>> amd64 amd64 1400096 1400096 >>>=20 >>> The GENERIC-DBG variant of the kernel did not report any issues in >>> earlier testing. >>>=20 >>> The alter referenced /usr/obj/DESTDIRs/main-amd64-poud-bulk_a was >>> installed from the same build. >>>=20 >>> # zfs list >>> NAME USED AVAIL REFER >>> MOUNTPOINT >>> zoptb 79.9G 765G 96K = /zoptb >>> zoptb/BUILDs 20.5G 765G 8.29M >>> /usr/obj/BUILDs >>> zoptb/BUILDs/alt-main-amd64-dbg-clang-alt 1.86M 765G 1.86M >>> /usr/obj/BUILDs/alt-main-amd64-dbg-clang-alt >>> zoptb/BUILDs/alt-main-amd64-nodbg-clang-alt 30.2M 765G 30.2M >>> /usr/obj/BUILDs/alt-main-amd64-nodbg-clang-alt >>> zoptb/BUILDs/main-amd64-dbg-clang 9.96G 765G 9.96G >>> /usr/obj/BUILDs/main-amd64-dbg-clang >>> zoptb/BUILDs/main-amd64-dbg-gccxtc 38.5M 765G 38.5M >>> /usr/obj/BUILDs/main-amd64-dbg-gccxtc >>> zoptb/BUILDs/main-amd64-nodbg-clang 10.3G 765G 10.3G >>> /usr/obj/BUILDs/main-amd64-nodbg-clang >>> zoptb/BUILDs/main-amd64-nodbg-clang-alt 37.2M 765G 37.2M >>> /usr/obj/BUILDs/main-amd64-nodbg-clang-alt >>> zoptb/BUILDs/main-amd64-nodbg-gccxtc 94.6M 765G 94.6M >>> /usr/obj/BUILDs/main-amd64-nodbg-gccxtc >>> zoptb/DESTDIRs 4.33G 765G 104K >>> /usr/obj/DESTDIRs >>> zoptb/DESTDIRs/main-amd64-poud 2.16G 765G 2.16G >>> /usr/obj/DESTDIRs/main-amd64-poud >>> zoptb/DESTDIRs/main-amd64-poud-bulk_a 2.16G 765G 2.16G >>> /usr/obj/DESTDIRs/main-amd64-poud-bulk_a >>> zoptb/ROOT 13.1G 765G 96K = none >>> zoptb/ROOT/build_area_for-main-amd64 5.03G 765G 3.24G = none >>> zoptb/ROOT/main-amd64 8.04G 765G 3.23G = none >>> zoptb/poudriere 6.58G 765G 112K >>> /usr/local/poudriere >>> zoptb/poudriere/data 6.58G 765G 128K >>> /usr/local/poudriere/data >>> zoptb/poudriere/data/.m 112K 765G 112K >>> /usr/local/poudriere/data/.m >>> zoptb/poudriere/data/cache 17.4M 765G 17.4M >>> /usr/local/poudriere/data/cache >>> zoptb/poudriere/data/images 96K 765G 96K >>> /usr/local/poudriere/data/images >>> zoptb/poudriere/data/logs 2.72G 765G 2.72G >>> /usr/local/poudriere/data/logs >>> zoptb/poudriere/data/packages 3.84G 765G 3.84G >>> /usr/local/poudriere/data/packages >>> zoptb/poudriere/data/wrkdirs 112K 765G 112K >>> /usr/local/poudriere/data/wrkdirs >>> zoptb/poudriere/jails 96K 765G 96K >>> /usr/local/poudriere/jails >>> zoptb/poudriere/ports 96K 765G 96K >>> /usr/local/poudriere/ports >>> zoptb/tmp 68.5M 765G 68.5M = /tmp >>> zoptb/usr 35.1G 765G 96K = /usr >>> zoptb/usr/13_0R-src 2.64G 765G 2.64G >>> /usr/13_0R-src >>> zoptb/usr/alt-main-src 96K 765G 96K >>> /usr/alt-main-src >>> zoptb/usr/home 181M 765G 181M = /usr/home >>> zoptb/usr/local 5.08G 765G 5.08G >>> /usr/local >>> zoptb/usr/main-src 833M 765G 833M >>> /usr/main-src >>> zoptb/usr/ports 26.4G 765G 26.4G >>> /usr/ports >>> zoptb/usr/src 96K 765G 96K = /usr/src >>> zoptb/var 52.6M 765G 96K = /var >>> zoptb/var/audit 356K 765G 356K >>> /var/audit >>> zoptb/var/crash 128K 765G 128K >>> /var/crash >>> zoptb/var/db 49.7M 765G 96K = /var/db >>> zoptb/var/db/pkg 49.4M 765G 49.4M >>> /var/db/pkg >>> zoptb/var/db/ports 164K 765G 164K >>> /var/db/ports >>> zoptb/var/log 1.61M 765G 1.61M = /var/log >>> zoptb/var/mail 632K 765G 632K = /var/mail >>> zoptb/var/tmp 128K 765G 128K = /var/tmp >>>=20 >>> # poudriere jail -jmain-amd64-bulk_a -i >>> Jail name: main-amd64-bulk_a >>> Jail version: 14.0-ALPHA2 >>> Jail arch: amd64 >>> Jail method: null >>> Jail mount: /usr/obj/DESTDIRs/main-amd64-poud-bulk_a >>> Jail fs: >>> Jail updated: 2021-12-04 14:55:22 >>> Jail pkgbase: disabled >>>=20 >>>=20 >>>=20 >>> So, setting up another test with some related information >>> shown before, during, and after. sysctl output is from >>> another ssh session than the bulk -a run. >>>=20 >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 0 >>> vm.stats.vm.v_vnodepgsin: 272429 >>> vm.stats.vm.v_vnodeout: 0 >>> vm.stats.vm.v_vnodein: 12461 >>> vfs.vnode_alloc_sleeps: 0 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 962766 >>> vfs.vnodes_created: 2538980 >>> vfs.numvnodes: 1056233 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 91878 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>>=20 >>> # poudriere bulk -jmain-amd64-bulk_a -a >>> . . . >>> [00:01:34] Building 34042 packages using up to 32 builders >>> [00:01:34] Hit CTRL+t at any time to see build progress and stats >>> [00:01:34] [01] [00:00:00] Builder starting >>> [00:01:57] [01] [00:00:23] Builder started >>> [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 >>> [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: = Success >>> [00:03:22] [01] [00:00:00] Building print/indexinfo | = indexinfo-0.3.1 >>> [00:03:22] [02] [00:00:00] Builder starting >>> [00:03:22] [03] [00:00:00] Builder starting >>> . . . >>> [00:03:22] [31] [00:00:00] Builder starting >>> [00:03:22] [32] [00:00:00] Builder starting >>> [00:03:31] [01] [00:00:09] Finished print/indexinfo | = indexinfo-0.3.1: >>> Success >>> [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | >>> gettext-runtime-0.22 >>> . . . >>>=20 >>> Note that only [01] makes progress: no new "Builder started" >>> notices occur. top shows 31 of the pattern: >>> cpdup -i0 -o ref ?? >>>=20 >>> Then during the 31 cpudup's showing vlruwk most of the time: >>>=20 >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 582398 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34296 >>> vfs.vnode_alloc_sleeps: 2994 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 2209662 >>> vfs.vnodes_created: 12206299 >>> vfs.numvnodes: 2214071 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 459 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>>=20 >>> Wait a while but still the mostly cpdup vlruwk status: >>>=20 >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 583527 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34396 >>> vfs.vnode_alloc_sleeps: 8053 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 2210166 >>> vfs.vnodes_created: 12212061 >>> vfs.numvnodes: 2215106 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 497 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>>=20 >>> ^C[00:14:55] Error: Signal SIGINT caught, cleaning up and exiting >>>=20 >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 584474 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34591 >>> vfs.vnode_alloc_sleeps: 17584 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 2210796 >>> vfs.vnodes_created: 12222343 >>> vfs.numvnodes: 2216564 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 539 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>>=20 >>> [main-amd64-bulk_a-default] [2023-08-23_13h58m08s] [sigint:] Queued: = 34435 >>> Built: 2 Failed: 0 Skipped: 35 Ignored: 358 Fetched: 0 >>> Tobuild: 34040 Time: 00:14:36 >>> [00:16:13] Logs: >>> = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-23_1= 3h58m08s >>> [00:16:49] Cleaning up >>> load: 5.28 cmd: sh 77057 [vlruwk] 141.63r 0.00u 30.98s 28% 6932k >>> #0 0xffffffff80b76ebb at mi_switch+0xbb >>> #1 0xffffffff80bc960f at sleepq_timedwait+0x2f >>> #2 0xffffffff80b76610 at _sleep+0x1d0 >>> #3 0xffffffff80c5b2dc at vn_alloc_hard+0x2ac >>> #4 0xffffffff80c50a12 at getnewvnode_reserve+0x92 >>> #5 0xffffffff829afb12 at zfs_zget+0x22 >>> #6 0xffffffff8299ca8d at zfs_dirent_lookup+0x16d >>> #7 0xffffffff8299cb5f at zfs_dirlook+0x7f >>> #8 0xffffffff829ac410 at zfs_lookup+0x350 >>> #9 0xffffffff829a782a at zfs_freebsd_cachedlookup+0x6a >>> #10 0xffffffff80c368ad at vfs_cache_lookup+0xad >>> #11 0xffffffff80c3b6d8 at cache_fplookup_final_modifying+0x188 >>> #12 0xffffffff80c38766 at cache_fplookup+0x356 >>> #13 0xffffffff80c43fb2 at namei+0x112 >>> #14 0xffffffff80c62e5b at kern_funlinkat+0x13b >>> #15 0xffffffff80c62d18 at sys_unlink+0x28 >>> #16 0xffffffff83b8e583 at filemon_wrapper_unlink+0x13 >>> #17 0xffffffff81049a79 at amd64_syscall+0x109 >>>=20 >>> [00:26:28] Unmounting file systems >>> Exiting with status 1 >>>=20 >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 585384 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34798 >>> vfs.vnode_alloc_sleeps: 27578 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 61362 >>> vfs.vnodes_created: 20135479 >>> vfs.numvnodes: 59860 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 208 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>>=20 >>>=20 >>> For reference (from after): >>>=20 >>> # kldstat >>> Id Refs Address Size Name >>> 1 95 0xffffffff80200000 274b308 kernel >>> 2 1 0xffffffff8294c000 5d5238 zfs.ko >>> 3 1 0xffffffff82f22000 7718 cryptodev.ko >>> 4 1 0xffffffff83b10000 3390 acpi_wmi.ko >>> 5 1 0xffffffff83b14000 3220 intpm.ko >>> 6 1 0xffffffff83b18000 2178 smbus.ko >>> 7 1 0xffffffff83b1b000 2240 cpuctl.ko >>> 8 1 0xffffffff83b1e000 3360 uhid.ko >>> 9 1 0xffffffff83b22000 4364 ums.ko >>> 10 1 0xffffffff83b27000 33c0 usbhid.ko >>> 11 1 0xffffffff83b2b000 3380 hidbus.ko >>> 12 1 0xffffffff83b2f000 4d20 ng_ubt.ko >>> 13 6 0xffffffff83b34000 abb8 netgraph.ko >>> 14 2 0xffffffff83b3f000 a250 ng_hci.ko >>> 15 4 0xffffffff83b4a000 2670 ng_bluetooth.ko >>> 16 1 0xffffffff83b4d000 83a0 uftdi.ko >>> 17 1 0xffffffff83b56000 4e58 ucom.ko >>> 18 1 0xffffffff83b5b000 3360 wmt.ko >>> 19 1 0xffffffff83b5f000 e268 ng_l2cap.ko >>> 20 1 0xffffffff83b6e000 1bf68 ng_btsocket.ko >>> 21 1 0xffffffff83b8a000 38f8 ng_socket.ko >>> 22 1 0xffffffff83b8e000 3250 filemon.ko >>> 23 1 0xffffffff83b92000 4758 nullfs.ko >>> 24 1 0xffffffff83b97000 73c0 linprocfs.ko >>> 25 3 0xffffffff83b9f000 be70 linux_common.ko >>> 26 1 0xffffffff83bab000 3558 fdescfs.ko >>> 27 1 0xffffffff83baf000 31b20 linux.ko >>> 28 1 0xffffffff83be1000 2ed40 linux64.ko >>>=20 >>>=20 >>> Note that before the "Cleaning up" notice, the vfs.freevnodes >>> shows as being around (for example) 2210796. But after >>> "Exiting with status": 61362. vfs.vnodes_created has a >>> similar staging of in the ball park of up to 12222343 but >>> then the change to: 20135479. Similarly, vfs.numvnodes >>> 2216564 -> 59860. >>>=20 >>>=20 >>>=20 >>> Anything else I should gather and report as basic information? >>>=20 >>=20 >> This is a known problem, but it is unclear if you should be running >> into it in this setup. >=20 > The changed fixed the issue: so I do run into the the issue > for this setup. See below. >=20 >> Can you try again but this time *revert* >> 138a5dafba312ff39ce0eefdbe34de95519e600d, like so: >> git revert 138a5dafba312ff39ce0eefdbe34de95519e600d >>=20 >> may want to switch to a different branch first, for example: git >> checkout -b vfstesting >=20 > # git -C /usr/main-src/ diff sys/kern/vfs_subr.c > diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c > index 0f3f00abfd4a..5dff556ac258 100644 > --- a/sys/kern/vfs_subr.c > +++ b/sys/kern/vfs_subr.c > @@ -3528,25 +3528,17 @@ vdbatch_process(struct vdbatch *vd) > MPASS(curthread->td_pinned > 0); > MPASS(vd->index =3D=3D VDBATCH_SIZE); > + mtx_lock(&vnode_list_mtx); > critical_enter(); > - if (mtx_trylock(&vnode_list_mtx)) { > - for (i =3D 0; i < VDBATCH_SIZE; i++) { > - vp =3D vd->tab[i]; > - vd->tab[i] =3D NULL; > - TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); > - TAILQ_INSERT_TAIL(&vnode_list, vp, = v_vnodelist); > - MPASS(vp->v_dbatchcpu !=3D NOCPU); > - vp->v_dbatchcpu =3D NOCPU; > - } > - mtx_unlock(&vnode_list_mtx); > - } else { > - for (i =3D 0; i < VDBATCH_SIZE; i++) { > - vp =3D vd->tab[i]; > - vd->tab[i] =3D NULL; > - MPASS(vp->v_dbatchcpu !=3D NOCPU); > - vp->v_dbatchcpu =3D NOCPU; > - } > + for (i =3D 0; i < VDBATCH_SIZE; i++) { > + vp =3D vd->tab[i]; > + TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); > + TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); > + MPASS(vp->v_dbatchcpu !=3D NOCPU); > + vp->v_dbatchcpu =3D NOCPU; > } > + mtx_unlock(&vnode_list_mtx); > + bzero(vd->tab, sizeof(vd->tab)); > vd->index =3D 0; > critical_exit(); > } >=20 > Still with: >=20 > # grep USE_TMPFS=3D /usr/local/etc/poudriere.conf > # EXAMPLE: USE_TMPFS=3D"wrkdir data" > #USE_TMPFS=3Dall > #USE_TMPFS=3D"data" > USE_TMPFS=3Dno >=20 >=20 > That allowed the other builders to eventually reach "Builder started" > and later activity, [00:05:50] [27] [00:02:29] Builder started > being the first non-[01] to do so, no vlruwk's observed in what > I saw in top: >=20 > # poudriere bulk -jmain-amd64-bulk_a -a > . . . > [00:01:34] Building 34042 packages using up to 32 builders > [00:01:34] Hit CTRL+t at any time to see build progress and stats > [00:01:34] [01] [00:00:00] Builder starting > [00:01:57] [01] [00:00:23] Builder started > [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 > [00:03:08] [01] [00:01:11] Finished ports-mgmt/pkg | pkg-1.20.4: = Success > [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 > [00:03:21] [02] [00:00:00] Builder starting > [00:03:21] [03] [00:00:00] Builder starting > [00:03:21] [04] [00:00:00] Builder starting > [00:03:21] [05] [00:00:00] Builder starting > [00:03:21] [06] [00:00:00] Builder starting > [00:03:21] [07] [00:00:00] Builder starting > [00:03:21] [08] [00:00:00] Builder starting > [00:03:21] [09] [00:00:00] Builder starting > [00:03:21] [10] [00:00:00] Builder starting > [00:03:21] [11] [00:00:00] Builder starting > [00:03:21] [12] [00:00:00] Builder starting > [00:03:21] [13] [00:00:00] Builder starting > [00:03:21] [14] [00:00:00] Builder starting > [00:03:21] [15] [00:00:00] Builder starting > [00:03:21] [16] [00:00:00] Builder starting > [00:03:21] [17] [00:00:00] Builder starting > [00:03:21] [18] [00:00:00] Builder starting > [00:03:21] [19] [00:00:00] Builder starting > [00:03:21] [20] [00:00:00] Builder starting > [00:03:21] [21] [00:00:00] Builder starting > [00:03:21] [22] [00:00:00] Builder starting > [00:03:21] [23] [00:00:00] Builder starting > [00:03:21] [24] [00:00:00] Builder starting > [00:03:21] [25] [00:00:00] Builder starting > [00:03:21] [26] [00:00:00] Builder starting > [00:03:21] [27] [00:00:00] Builder starting > [00:03:21] [28] [00:00:00] Builder starting > [00:03:21] [29] [00:00:00] Builder starting > [00:03:21] [30] [00:00:00] Builder starting > [00:03:21] [31] [00:00:00] Builder starting > [00:03:21] [32] [00:00:00] Builder starting > [00:03:29] [01] [00:00:08] Finished print/indexinfo | indexinfo-0.3.1: = Success > [00:03:30] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 > [00:04:49] [01] [00:01:19] Finished devel/gettext-runtime | = gettext-runtime-0.22: Success > [00:04:52] [01] [00:00:00] Building devel/gmake | gmake-4.3_2 > [00:05:20] [01] [00:00:28] Finished devel/gmake | gmake-4.3_2: Success > [00:05:23] [01] [00:00:00] Building devel/libtextstyle | = libtextstyle-0.22 > [00:05:50] [27] [00:02:29] Builder started > [00:05:50] [27] [00:00:00] Building devel/npth | npth-1.6 > [00:05:50] [08] [00:02:29] Builder started > [00:05:50] [08] [00:00:00] Building devel/boost-jam | boost-jam-1.82.0 > [00:05:51] [16] [00:02:30] Builder started > [00:05:51] [16] [00:00:00] Building devel/libdaemon | libdaemon-0.14_1 > [00:05:51] [19] [00:02:30] Builder started > [00:05:51] [19] [00:00:00] Building graphics/libpotrace | = libpotrace-1.16 > [00:05:51] [14] [00:02:30] Builder started > [00:05:51] [14] [00:00:00] Building sysutils/gnome_subr | = gnome_subr-1.0 > [00:05:51] [17] [00:02:30] Builder started > [00:05:51] [17] [00:00:00] Building misc/hicolor-icon-theme | = hicolor-icon-theme-0.17 > [00:05:51] [26] [00:02:30] Builder started > [00:05:51] [26] [00:00:00] Building devel/libinotify | = libinotify-20211018 > [00:05:51] [12] [00:02:30] Builder started > [00:05:51] [12] [00:00:00] Building devel/evdev-proto | = evdev-proto-5.8 > [00:05:51] [29] [00:02:30] Builder started > [00:05:51] [29] [00:00:00] Building devel/opencl | opencl-3.0.14 > [00:05:51] [05] [00:02:30] Builder started > [00:05:51] [05] [00:00:00] Building devel/autoconf-switch | = autoconf-switch-20220527 > [00:05:51] [11] [00:02:30] Builder started > [00:05:51] [11] [00:00:00] Building textproc/html2text | = html2text-1.3.2a_1,1 > [00:05:51] [25] [00:02:30] Builder started > [00:05:51] [04] [00:02:30] Builder started > [00:05:51] [25] [00:00:00] Building audio/speexdsp | speexdsp-1.2.1 > [00:05:51] [04] [00:00:00] Building converters/libiconv | = libiconv-1.17 > [00:05:51] [22] [00:02:30] Builder started > [00:05:51] [22] [00:00:00] Building dns/public_suffix_list | = public_suffix_list-20230630 > [00:05:51] [20] [00:02:30] Builder started > [00:05:51] [02] [00:02:30] Builder started > [00:05:51] [24] [00:02:30] Builder started > [00:05:51] [31] [00:02:30] Builder started > [00:05:51] [20] [00:00:00] Building print/gsfonts | gsfonts-8.11_8 > . . . >=20 > I had waited a very long time with no such progress for [02]..[32] > for the prior code. >=20 > Now testing for the zfs deadlock issue should be possible for > this setup. I'll remind that back on 2023-Apr-30 you had me make another adjustment related to poudriere bulk activity not being as expected: # sysctl vfs.zfs.per_txg_dirty_frees_percent=3D5 Someting about "tx->tx_quiesce_done_cv related blocking of processes" was involved in that exchange. So this makes 2 non-standard alternatives in use for my kinds of contexts in order to avoid otherwise having poudriere bulk performance problems. (The alternative systems are aarch64 based.) =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 24 05:54:33 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWXND3fgqz4qfsb for ; Thu, 24 Aug 2023 05:54:36 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc33.google.com (mail-oo1-xc33.google.com [IPv6:2607:f8b0:4864:20::c33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWXND1QgWz4ShW for ; Thu, 24 Aug 2023 05:54:36 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc33.google.com with SMTP id 006d021491bc7-5732a32674aso85169eaf.0 for ; Wed, 23 Aug 2023 22:54:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692856474; x=1693461274; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=AnRbJn/H4gwpP4lhYZQ9DmwF5E0TuJeillLmksw4ENs=; b=aSQdVvSXiZfIGJ7a8tZFE1uEBzMrXB3yXRhUFJDh+aBN8EkKb3GH031qX23GithGGj lm1zpDWXto6oZ4w5F9AFGc914wlIrKjl9eIX+qxWY2PIB2WEqkNCDmVUCMNXhqWm85bY lFc4J961LZIWr/hVDAkV6ma40pRhU2LG9i1LEBBtZBwYtvBHlRKNn6Z/+b1YJy1fFg4h jMsM4dZbq6LHca7T7ySUuRr0r9/2TXJeD8ujnNdtRHrncop74JMk8ee/gRgrU9WFoCGe +mXSqo7t47jbtnmniBuF+xLfsm7/UFAqk55AnKo09btEWLygJyeswVU/ZPxTxTIwHr37 +Nbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692856474; x=1693461274; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AnRbJn/H4gwpP4lhYZQ9DmwF5E0TuJeillLmksw4ENs=; b=Om2giwSp39eVqcZUSO4Tkufprkc/ntPnXWq7Dib0gLFj7JhbNptkc0d1X78YxxSCHK NC8SVQ/cxDWikjmWKtS3qoNrfIwGZjp/Qt2PM+aUMQnMS/zHLL17rX72pZZ/TjWUptVl Wu4sf5ZrLn3mrgVgrsg4ILAfts7vpl6bE47kw2UDtvk81uo7V/BtLho+N57eFWfM0skg T4WZXueT+7I3qMn7LuhxSM75kohFLFiU169Y6GqgocdXNci3E257CFFYLVMg2aqupzmX UTKQc+8wzqQ+1VLV54OszKBVIey3lUE8DFTgtCrubsbCRAda6KK9U9MyKgWuj8a0eWg9 dvtw== X-Gm-Message-State: AOJu0YxayP6t/gAKEC153sXtyTeRK3VL4MlqE/+YWPzdohB6l/Ee8LYR 8Y8AJc0H+CGEuBd7Vgg8v/IeydCATQDZkeae8C9yy+nY X-Google-Smtp-Source: AGHT+IF4E4tk1M4Rvog8aJDsKSKSmLXOkFKFLboO1+lU9dbJ7XtgNXmYbv9jmXREkAPNpH5YXnXOorA522V2+SmmesY= X-Received: by 2002:a4a:d208:0:b0:571:2b59:d697 with SMTP id c8-20020a4ad208000000b005712b59d697mr1369758oos.0.1692856474220; Wed, 23 Aug 2023 22:54:34 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:5797:0:b0:4f0:1250:dd51 with HTTP; Wed, 23 Aug 2023 22:54:33 -0700 (PDT) In-Reply-To: References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> From: Mateusz Guzik Date: Thu, 24 Aug 2023 07:54:33 +0200 Message-ID: Subject: Re: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder To: Mark Millard Cc: Current FreeBSD Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RWXND1QgWz4ShW X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On 8/24/23, Mark Millard wrote: > On Aug 23, 2023, at 15:10, Mateusz Guzik wrote: > >> On 8/23/23, Mark Millard wrote: >>> [Forked off the ZFS deadlock 14 discussion, per feedback.] >>> >>> On Aug 23, 2023, at 11:40, Alexander Motin wrote: >>> >>>> On 22.08.2023 14:24, Mark Millard wrote: >>>>> Alexander Motin wrote on >>>>> Date: Tue, 22 Aug 2023 16:18:12 UTC : >>>>>> I am waiting for final test results from George Wilson and then will >>>>>> request quick merge of both to zfs-2.2-release branch. Unfortunately >>>>>> there are still not many reviewers for the PR, since the code is not >>>>>> trivial, but at least with the test reports Brian Behlendorf and Mark >>>>>> Maybee seem to be OK to merge the two PRs into 2.2. If somebody else >>>>>> have tested and/or reviewed the PR, you may comment on it. >>>>> I had written to the list that when I tried to test the system >>>>> doing poudriere builds (initially with your patches) using >>>>> USE_TMPFS=no so that zfs had to deal with all the file I/O, I >>>>> instead got only one builder that ended up active, the others >>>>> never reaching "Builder started": >>>> >>>>> Top was showing lots of "vlruwk" for the cpdup's. For example: >>>>> . . . >>>>> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 >>>>> 0.00% cpdup -i0 -o ref 32 >>>>> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 >>>>> 0.01% cpdup -i0 -o ref 31 >>>>> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 >>>>> 0.01% cpdup -i0 -o ref 30 >>>>> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 >>>>> 0.01% cpdup -i0 -o ref 29 >>>>> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 >>>>> 0.01% cpdup -i0 -o ref 28 >>>>> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 >>>>> 0.00% cpdup -i0 -o ref 27 >>>>> . . . >>>>> But those processes did show CPU?? on occasion, as well as >>>>> *vnode less often. None of the cpdup's was stuck in >>>>> Removing your patches did not change the behavior. >>>> >>>> Mark, to me "vlruwk" looks like a limit on number of vnodes. I was not >>>> deep in that area at least recently, so somebody with more experience >>>> there could try to diagnose it. At very least it does not look related >>>> to >>>> the ZIL issue discussed in this thread, at least with the information >>>> provided, so I am not surprised that the mentioned patches do not >>>> affect >>>> it. >>> >>> I did the above intending to test the deadlock in my context but >>> ended up not getting that far when I tried to make zfs handle all >>> the file I/O (USE_TMPFS=no and no other use of tmpfs or the like). >>> >>> The zfs context is a simple single partition on the boot media. I >>> use ZFS for bectl BE use, not for other typical reasons. The media >>> here is PCIe Optane 1.4T media. The machine is a ThreadRipper >>> 1950X, so first generation. 128 GiBytes of RAM. 491520 MiBytes of >>> swap, also on that Optane. >>> >>> # uname -apKU >>> FreeBSD amd64-ZFS 14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400096 #112 >>> main-n264912-b1d3e2b77155-dirty: Sun Aug 20 10:01:48 PDT 2023 >>> root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG >>> amd64 amd64 1400096 1400096 >>> >>> The GENERIC-DBG variant of the kernel did not report any issues in >>> earlier testing. >>> >>> The alter referenced /usr/obj/DESTDIRs/main-amd64-poud-bulk_a was >>> installed from the same build. >>> >>> # zfs list >>> NAME USED AVAIL REFER >>> MOUNTPOINT >>> zoptb 79.9G 765G 96K /zoptb >>> zoptb/BUILDs 20.5G 765G 8.29M >>> /usr/obj/BUILDs >>> zoptb/BUILDs/alt-main-amd64-dbg-clang-alt 1.86M 765G 1.86M >>> /usr/obj/BUILDs/alt-main-amd64-dbg-clang-alt >>> zoptb/BUILDs/alt-main-amd64-nodbg-clang-alt 30.2M 765G 30.2M >>> /usr/obj/BUILDs/alt-main-amd64-nodbg-clang-alt >>> zoptb/BUILDs/main-amd64-dbg-clang 9.96G 765G 9.96G >>> /usr/obj/BUILDs/main-amd64-dbg-clang >>> zoptb/BUILDs/main-amd64-dbg-gccxtc 38.5M 765G 38.5M >>> /usr/obj/BUILDs/main-amd64-dbg-gccxtc >>> zoptb/BUILDs/main-amd64-nodbg-clang 10.3G 765G 10.3G >>> /usr/obj/BUILDs/main-amd64-nodbg-clang >>> zoptb/BUILDs/main-amd64-nodbg-clang-alt 37.2M 765G 37.2M >>> /usr/obj/BUILDs/main-amd64-nodbg-clang-alt >>> zoptb/BUILDs/main-amd64-nodbg-gccxtc 94.6M 765G 94.6M >>> /usr/obj/BUILDs/main-amd64-nodbg-gccxtc >>> zoptb/DESTDIRs 4.33G 765G 104K >>> /usr/obj/DESTDIRs >>> zoptb/DESTDIRs/main-amd64-poud 2.16G 765G 2.16G >>> /usr/obj/DESTDIRs/main-amd64-poud >>> zoptb/DESTDIRs/main-amd64-poud-bulk_a 2.16G 765G 2.16G >>> /usr/obj/DESTDIRs/main-amd64-poud-bulk_a >>> zoptb/ROOT 13.1G 765G 96K none >>> zoptb/ROOT/build_area_for-main-amd64 5.03G 765G 3.24G none >>> zoptb/ROOT/main-amd64 8.04G 765G 3.23G none >>> zoptb/poudriere 6.58G 765G 112K >>> /usr/local/poudriere >>> zoptb/poudriere/data 6.58G 765G 128K >>> /usr/local/poudriere/data >>> zoptb/poudriere/data/.m 112K 765G 112K >>> /usr/local/poudriere/data/.m >>> zoptb/poudriere/data/cache 17.4M 765G 17.4M >>> /usr/local/poudriere/data/cache >>> zoptb/poudriere/data/images 96K 765G 96K >>> /usr/local/poudriere/data/images >>> zoptb/poudriere/data/logs 2.72G 765G 2.72G >>> /usr/local/poudriere/data/logs >>> zoptb/poudriere/data/packages 3.84G 765G 3.84G >>> /usr/local/poudriere/data/packages >>> zoptb/poudriere/data/wrkdirs 112K 765G 112K >>> /usr/local/poudriere/data/wrkdirs >>> zoptb/poudriere/jails 96K 765G 96K >>> /usr/local/poudriere/jails >>> zoptb/poudriere/ports 96K 765G 96K >>> /usr/local/poudriere/ports >>> zoptb/tmp 68.5M 765G 68.5M /tmp >>> zoptb/usr 35.1G 765G 96K /usr >>> zoptb/usr/13_0R-src 2.64G 765G 2.64G >>> /usr/13_0R-src >>> zoptb/usr/alt-main-src 96K 765G 96K >>> /usr/alt-main-src >>> zoptb/usr/home 181M 765G 181M >>> /usr/home >>> zoptb/usr/local 5.08G 765G 5.08G >>> /usr/local >>> zoptb/usr/main-src 833M 765G 833M >>> /usr/main-src >>> zoptb/usr/ports 26.4G 765G 26.4G >>> /usr/ports >>> zoptb/usr/src 96K 765G 96K >>> /usr/src >>> zoptb/var 52.6M 765G 96K /var >>> zoptb/var/audit 356K 765G 356K >>> /var/audit >>> zoptb/var/crash 128K 765G 128K >>> /var/crash >>> zoptb/var/db 49.7M 765G 96K >>> /var/db >>> zoptb/var/db/pkg 49.4M 765G 49.4M >>> /var/db/pkg >>> zoptb/var/db/ports 164K 765G 164K >>> /var/db/ports >>> zoptb/var/log 1.61M 765G 1.61M >>> /var/log >>> zoptb/var/mail 632K 765G 632K >>> /var/mail >>> zoptb/var/tmp 128K 765G 128K >>> /var/tmp >>> >>> # poudriere jail -jmain-amd64-bulk_a -i >>> Jail name: main-amd64-bulk_a >>> Jail version: 14.0-ALPHA2 >>> Jail arch: amd64 >>> Jail method: null >>> Jail mount: /usr/obj/DESTDIRs/main-amd64-poud-bulk_a >>> Jail fs: >>> Jail updated: 2021-12-04 14:55:22 >>> Jail pkgbase: disabled >>> >>> >>> >>> So, setting up another test with some related information >>> shown before, during, and after. sysctl output is from >>> another ssh session than the bulk -a run. >>> >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 0 >>> vm.stats.vm.v_vnodepgsin: 272429 >>> vm.stats.vm.v_vnodeout: 0 >>> vm.stats.vm.v_vnodein: 12461 >>> vfs.vnode_alloc_sleeps: 0 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 962766 >>> vfs.vnodes_created: 2538980 >>> vfs.numvnodes: 1056233 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 91878 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>> >>> # poudriere bulk -jmain-amd64-bulk_a -a >>> . . . >>> [00:01:34] Building 34042 packages using up to 32 builders >>> [00:01:34] Hit CTRL+t at any time to see build progress and stats >>> [00:01:34] [01] [00:00:00] Builder starting >>> [00:01:57] [01] [00:00:23] Builder started >>> [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 >>> [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: Success >>> [00:03:22] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 >>> [00:03:22] [02] [00:00:00] Builder starting >>> [00:03:22] [03] [00:00:00] Builder starting >>> . . . >>> [00:03:22] [31] [00:00:00] Builder starting >>> [00:03:22] [32] [00:00:00] Builder starting >>> [00:03:31] [01] [00:00:09] Finished print/indexinfo | indexinfo-0.3.1: >>> Success >>> [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | >>> gettext-runtime-0.22 >>> . . . >>> >>> Note that only [01] makes progress: no new "Builder started" >>> notices occur. top shows 31 of the pattern: >>> cpdup -i0 -o ref ?? >>> >>> Then during the 31 cpudup's showing vlruwk most of the time: >>> >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 582398 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34296 >>> vfs.vnode_alloc_sleeps: 2994 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 2209662 >>> vfs.vnodes_created: 12206299 >>> vfs.numvnodes: 2214071 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 459 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>> >>> Wait a while but still the mostly cpdup vlruwk status: >>> >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 583527 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34396 >>> vfs.vnode_alloc_sleeps: 8053 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 2210166 >>> vfs.vnodes_created: 12212061 >>> vfs.numvnodes: 2215106 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 497 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>> >>> ^C[00:14:55] Error: Signal SIGINT caught, cleaning up and exiting >>> >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 584474 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34591 >>> vfs.vnode_alloc_sleeps: 17584 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 2210796 >>> vfs.vnodes_created: 12222343 >>> vfs.numvnodes: 2216564 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 539 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>> >>> [main-amd64-bulk_a-default] [2023-08-23_13h58m08s] [sigint:] Queued: >>> 34435 >>> Built: 2 Failed: 0 Skipped: 35 Ignored: 358 Fetched: 0 >>> Tobuild: 34040 Time: 00:14:36 >>> [00:16:13] Logs: >>> /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-23_13h58m08s >>> [00:16:49] Cleaning up >>> load: 5.28 cmd: sh 77057 [vlruwk] 141.63r 0.00u 30.98s 28% 6932k >>> #0 0xffffffff80b76ebb at mi_switch+0xbb >>> #1 0xffffffff80bc960f at sleepq_timedwait+0x2f >>> #2 0xffffffff80b76610 at _sleep+0x1d0 >>> #3 0xffffffff80c5b2dc at vn_alloc_hard+0x2ac >>> #4 0xffffffff80c50a12 at getnewvnode_reserve+0x92 >>> #5 0xffffffff829afb12 at zfs_zget+0x22 >>> #6 0xffffffff8299ca8d at zfs_dirent_lookup+0x16d >>> #7 0xffffffff8299cb5f at zfs_dirlook+0x7f >>> #8 0xffffffff829ac410 at zfs_lookup+0x350 >>> #9 0xffffffff829a782a at zfs_freebsd_cachedlookup+0x6a >>> #10 0xffffffff80c368ad at vfs_cache_lookup+0xad >>> #11 0xffffffff80c3b6d8 at cache_fplookup_final_modifying+0x188 >>> #12 0xffffffff80c38766 at cache_fplookup+0x356 >>> #13 0xffffffff80c43fb2 at namei+0x112 >>> #14 0xffffffff80c62e5b at kern_funlinkat+0x13b >>> #15 0xffffffff80c62d18 at sys_unlink+0x28 >>> #16 0xffffffff83b8e583 at filemon_wrapper_unlink+0x13 >>> #17 0xffffffff81049a79 at amd64_syscall+0x109 >>> >>> [00:26:28] Unmounting file systems >>> Exiting with status 1 >>> >>> # sysctl -a | grep vnode >>> kern.maxvnodes: 2213808 >>> kern.ipc.umtx_vnode_persistent: 0 >>> kern.minvnodes: 553452 >>> vm.vnode_pbufs: 2048 >>> vm.stats.vm.v_vnodepgsout: 22844 >>> vm.stats.vm.v_vnodepgsin: 585384 >>> vm.stats.vm.v_vnodeout: 890 >>> vm.stats.vm.v_vnodein: 34798 >>> vfs.vnode_alloc_sleeps: 27578 >>> vfs.wantfreevnodes: 553452 >>> vfs.freevnodes: 61362 >>> vfs.vnodes_created: 20135479 >>> vfs.numvnodes: 59860 >>> vfs.cache.debug.vnodes_cel_3_failures: 0 >>> vfs.cache.stats.heldvnodes: 208 >>> debug.vnode_domainset: >>> debug.sizeof.vnode: 448 >>> debug.fail_point.status_fill_kinfo_vnode__random_path: off >>> debug.fail_point.fill_kinfo_vnode__random_path: off >>> >>> >>> For reference (from after): >>> >>> # kldstat >>> Id Refs Address Size Name >>> 1 95 0xffffffff80200000 274b308 kernel >>> 2 1 0xffffffff8294c000 5d5238 zfs.ko >>> 3 1 0xffffffff82f22000 7718 cryptodev.ko >>> 4 1 0xffffffff83b10000 3390 acpi_wmi.ko >>> 5 1 0xffffffff83b14000 3220 intpm.ko >>> 6 1 0xffffffff83b18000 2178 smbus.ko >>> 7 1 0xffffffff83b1b000 2240 cpuctl.ko >>> 8 1 0xffffffff83b1e000 3360 uhid.ko >>> 9 1 0xffffffff83b22000 4364 ums.ko >>> 10 1 0xffffffff83b27000 33c0 usbhid.ko >>> 11 1 0xffffffff83b2b000 3380 hidbus.ko >>> 12 1 0xffffffff83b2f000 4d20 ng_ubt.ko >>> 13 6 0xffffffff83b34000 abb8 netgraph.ko >>> 14 2 0xffffffff83b3f000 a250 ng_hci.ko >>> 15 4 0xffffffff83b4a000 2670 ng_bluetooth.ko >>> 16 1 0xffffffff83b4d000 83a0 uftdi.ko >>> 17 1 0xffffffff83b56000 4e58 ucom.ko >>> 18 1 0xffffffff83b5b000 3360 wmt.ko >>> 19 1 0xffffffff83b5f000 e268 ng_l2cap.ko >>> 20 1 0xffffffff83b6e000 1bf68 ng_btsocket.ko >>> 21 1 0xffffffff83b8a000 38f8 ng_socket.ko >>> 22 1 0xffffffff83b8e000 3250 filemon.ko >>> 23 1 0xffffffff83b92000 4758 nullfs.ko >>> 24 1 0xffffffff83b97000 73c0 linprocfs.ko >>> 25 3 0xffffffff83b9f000 be70 linux_common.ko >>> 26 1 0xffffffff83bab000 3558 fdescfs.ko >>> 27 1 0xffffffff83baf000 31b20 linux.ko >>> 28 1 0xffffffff83be1000 2ed40 linux64.ko >>> >>> >>> Note that before the "Cleaning up" notice, the vfs.freevnodes >>> shows as being around (for example) 2210796. But after >>> "Exiting with status": 61362. vfs.vnodes_created has a >>> similar staging of in the ball park of up to 12222343 but >>> then the change to: 20135479. Similarly, vfs.numvnodes >>> 2216564 -> 59860. >>> >>> >>> >>> Anything else I should gather and report as basic information? >>> >> >> This is a known problem, but it is unclear if you should be running >> into it in this setup. > > The changed fixed the issue: so I do run into the the issue > for this setup. See below. > >> Can you try again but this time *revert* >> 138a5dafba312ff39ce0eefdbe34de95519e600d, like so: >> git revert 138a5dafba312ff39ce0eefdbe34de95519e600d >> >> may want to switch to a different branch first, for example: git >> checkout -b vfstesting > > # git -C /usr/main-src/ diff sys/kern/vfs_subr.c > diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c > index 0f3f00abfd4a..5dff556ac258 100644 > --- a/sys/kern/vfs_subr.c > +++ b/sys/kern/vfs_subr.c > @@ -3528,25 +3528,17 @@ vdbatch_process(struct vdbatch *vd) > MPASS(curthread->td_pinned > 0); > MPASS(vd->index == VDBATCH_SIZE); > + mtx_lock(&vnode_list_mtx); > critical_enter(); > - if (mtx_trylock(&vnode_list_mtx)) { > - for (i = 0; i < VDBATCH_SIZE; i++) { > - vp = vd->tab[i]; > - vd->tab[i] = NULL; > - TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); > - TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); > - MPASS(vp->v_dbatchcpu != NOCPU); > - vp->v_dbatchcpu = NOCPU; > - } > - mtx_unlock(&vnode_list_mtx); > - } else { > - for (i = 0; i < VDBATCH_SIZE; i++) { > - vp = vd->tab[i]; > - vd->tab[i] = NULL; > - MPASS(vp->v_dbatchcpu != NOCPU); > - vp->v_dbatchcpu = NOCPU; > - } > + for (i = 0; i < VDBATCH_SIZE; i++) { > + vp = vd->tab[i]; > + TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); > + TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); > + MPASS(vp->v_dbatchcpu != NOCPU); > + vp->v_dbatchcpu = NOCPU; > } > + mtx_unlock(&vnode_list_mtx); > + bzero(vd->tab, sizeof(vd->tab)); > vd->index = 0; > critical_exit(); > } > > Still with: > > # grep USE_TMPFS= /usr/local/etc/poudriere.conf > # EXAMPLE: USE_TMPFS="wrkdir data" > #USE_TMPFS=all > #USE_TMPFS="data" > USE_TMPFS=no > > > That allowed the other builders to eventually reach "Builder started" > and later activity, [00:05:50] [27] [00:02:29] Builder started > being the first non-[01] to do so, no vlruwk's observed in what > I saw in top: > > # poudriere bulk -jmain-amd64-bulk_a -a > . . . > [00:01:34] Building 34042 packages using up to 32 builders > [00:01:34] Hit CTRL+t at any time to see build progress and stats > [00:01:34] [01] [00:00:00] Builder starting > [00:01:57] [01] [00:00:23] Builder started > [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 > [00:03:08] [01] [00:01:11] Finished ports-mgmt/pkg | pkg-1.20.4: Success > [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 > [00:03:21] [02] [00:00:00] Builder starting > [00:03:21] [03] [00:00:00] Builder starting > [00:03:21] [04] [00:00:00] Builder starting > [00:03:21] [05] [00:00:00] Builder starting > [00:03:21] [06] [00:00:00] Builder starting > [00:03:21] [07] [00:00:00] Builder starting > [00:03:21] [08] [00:00:00] Builder starting > [00:03:21] [09] [00:00:00] Builder starting > [00:03:21] [10] [00:00:00] Builder starting > [00:03:21] [11] [00:00:00] Builder starting > [00:03:21] [12] [00:00:00] Builder starting > [00:03:21] [13] [00:00:00] Builder starting > [00:03:21] [14] [00:00:00] Builder starting > [00:03:21] [15] [00:00:00] Builder starting > [00:03:21] [16] [00:00:00] Builder starting > [00:03:21] [17] [00:00:00] Builder starting > [00:03:21] [18] [00:00:00] Builder starting > [00:03:21] [19] [00:00:00] Builder starting > [00:03:21] [20] [00:00:00] Builder starting > [00:03:21] [21] [00:00:00] Builder starting > [00:03:21] [22] [00:00:00] Builder starting > [00:03:21] [23] [00:00:00] Builder starting > [00:03:21] [24] [00:00:00] Builder starting > [00:03:21] [25] [00:00:00] Builder starting > [00:03:21] [26] [00:00:00] Builder starting > [00:03:21] [27] [00:00:00] Builder starting > [00:03:21] [28] [00:00:00] Builder starting > [00:03:21] [29] [00:00:00] Builder starting > [00:03:21] [30] [00:00:00] Builder starting > [00:03:21] [31] [00:00:00] Builder starting > [00:03:21] [32] [00:00:00] Builder starting > [00:03:29] [01] [00:00:08] Finished print/indexinfo | indexinfo-0.3.1: > Success > [00:03:30] [01] [00:00:00] Building devel/gettext-runtime | > gettext-runtime-0.22 > [00:04:49] [01] [00:01:19] Finished devel/gettext-runtime | > gettext-runtime-0.22: Success > [00:04:52] [01] [00:00:00] Building devel/gmake | gmake-4.3_2 > [00:05:20] [01] [00:00:28] Finished devel/gmake | gmake-4.3_2: Success > [00:05:23] [01] [00:00:00] Building devel/libtextstyle | libtextstyle-0.22 > [00:05:50] [27] [00:02:29] Builder started > [00:05:50] [27] [00:00:00] Building devel/npth | npth-1.6 > [00:05:50] [08] [00:02:29] Builder started > [00:05:50] [08] [00:00:00] Building devel/boost-jam | boost-jam-1.82.0 > [00:05:51] [16] [00:02:30] Builder started > [00:05:51] [16] [00:00:00] Building devel/libdaemon | libdaemon-0.14_1 > [00:05:51] [19] [00:02:30] Builder started > [00:05:51] [19] [00:00:00] Building graphics/libpotrace | libpotrace-1.16 > [00:05:51] [14] [00:02:30] Builder started > [00:05:51] [14] [00:00:00] Building sysutils/gnome_subr | gnome_subr-1.0 > [00:05:51] [17] [00:02:30] Builder started > [00:05:51] [17] [00:00:00] Building misc/hicolor-icon-theme | > hicolor-icon-theme-0.17 > [00:05:51] [26] [00:02:30] Builder started > [00:05:51] [26] [00:00:00] Building devel/libinotify | libinotify-20211018 > [00:05:51] [12] [00:02:30] Builder started > [00:05:51] [12] [00:00:00] Building devel/evdev-proto | evdev-proto-5.8 > [00:05:51] [29] [00:02:30] Builder started > [00:05:51] [29] [00:00:00] Building devel/opencl | opencl-3.0.14 > [00:05:51] [05] [00:02:30] Builder started > [00:05:51] [05] [00:00:00] Building devel/autoconf-switch | > autoconf-switch-20220527 > [00:05:51] [11] [00:02:30] Builder started > [00:05:51] [11] [00:00:00] Building textproc/html2text | > html2text-1.3.2a_1,1 > [00:05:51] [25] [00:02:30] Builder started > [00:05:51] [04] [00:02:30] Builder started > [00:05:51] [25] [00:00:00] Building audio/speexdsp | speexdsp-1.2.1 > [00:05:51] [04] [00:00:00] Building converters/libiconv | libiconv-1.17 > [00:05:51] [22] [00:02:30] Builder started > [00:05:51] [22] [00:00:00] Building dns/public_suffix_list | > public_suffix_list-20230630 > [00:05:51] [20] [00:02:30] Builder started > [00:05:51] [02] [00:02:30] Builder started > [00:05:51] [24] [00:02:30] Builder started > [00:05:51] [31] [00:02:30] Builder started > [00:05:51] [20] [00:00:00] Building print/gsfonts | gsfonts-8.11_8 > . . . > > I had waited a very long time with no such progress for [02]..[32] > for the prior code. > > Now testing for the zfs deadlock issue should be possible for > this setup. > Thanks for testing, I wrote a fix: https://people.freebsd.org/~mjg/vfs-recycle-fix.diff Applies to *stock* kernel (as in without the revert). -- Mateusz Guzik From nobody Thu Aug 24 07:22:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWZKQ5fYrz4qlLV for ; Thu, 24 Aug 2023 07:22:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.gq1.yahoo.com (sonic314-21.consmr.mail.gq1.yahoo.com [98.137.69.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWZKQ3Jzrz4btp for ; Thu, 24 Aug 2023 07:22:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692861736; bh=Ecs5TJbzYhfH7iOmtuTwm1Phwv1tkjRCyBuNaOxrBVE=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=P8T+K9myq87WD1/bqEUpgtzdyK0DfHGlsDzoRKjlBYJj5wR1f3eL1Ht+/OXMVB7foJ3s8RJsnf41hcvNbLJAHcuppwC/foE/6rxikkwMS2c4cM2Goc2LtNsZ2ma73QZk0OgUrFs/AUZXkrNbIyweryF+6nDDWVmX4tpp2O3nF4btMMu8ZaugyaiYQl7sBmXlkokGyOK0+fc9in39F0Aq8PBl8cI4R5sNPKeBcnpCGg0FS0bAK45r29wr+Of7aYHdmihY8aFGZRJVcdpEwhhZeJ94lnwXADpuEOL4f99IFlv+laBVolZsi+k5m81d/rA0QuS9OoLqJFysCBW+aY7M/w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692861736; bh=2sCIogA+PhHTe4buioRG11YKjTDOTrNw9t2GPGjlEUM=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=rrcjYJPRreJV9HekrdXH3eT3+f6hQ5920EcuFAQKdDrW+Ehliz5CtpJRAIboeQ7BW8AsQqA2DAKGPiEreEkwVWYZiEEezKW/q6ftmrHhtqxkPY7REoyXqppnGVG9YRLG+M+827EL647kGX313zj2Ed3btxx+DEWEeK1/1+M8p/p5zJRkLkrliTtNDUx5rmq12YIGLJXjLPWVDoCJk2nmfX3nih5BhdUqB/dEYxQHYPPNzBl8pO+Qj4MgRppdLONUY0HS7NsZIX1JqiqAYjbvN/JD6ak/kJzCBeMLowYC+4lDWXAYg7RBEkJXd48wdUyZyhiUTJP1wmm0xX6UjQ9pLg== X-YMail-OSG: BLppw.sVM1mzKhs0.17oslBDLwMhnFNDiyvb0ymKDMFMIVFBgUAko2H2ebtLReC c65c22y6h893SL9SgrTSJVKlICBuqGL9d2CYPEScycBih68Ei6DFR2lwtYDJ2Y5JIGLjg0c.vvLO ULRAN.cyB2jIDHU3_pjuWtQtTpeh1FotxDodV7E6_4DRfNJYMR6m8JNS6nQ7N3a6Web4WAduQ81_ lhFZ8rzTSv2DooCjLykZBvN0e_DwaAm1qs.T5sOyx5.GmFxpN0vwBvcCF8gCON.hvBgOYt7.E_r1 yZK0vCwLnsurzDH8ejve0Lh947wEsrXgj9patKi7z71JKdvzzGaEsikEiFBYfixMK5_rf7waKN2Q uBKKXh35lwop8b6ESs7C3h1fs_WhxGJ.7phA0Nngz.M454Htso_99gLb4aiuR8lf0BpydROLrdwA 23Syv6Iwpcq8H8BANs.cFm_K0mMpO7LbrkFMr0gchXtN03RywNTqtQuyAgEYmq1z.DwiAtypXyAD h06ol7Tf0nICsX_wJXGSw11iIwdVSYaRq3k.Rcs_F48dvWHPeBJJMV7dZ1TQQB2tPf72TmboyGaj QsCQWH1TK3n8FfcBdDeT3N.h31xrne92faxZiNX11XAZyNta_81LPXoOAqSMxrdRQBDIn0qmq.M6 VFGeJ5gV4_3mQe7l4cE681dpt7frDPLnFZDVl0EHZ4zAp09xoFX3B8W3Kga_VyBr.JaJQsLbkZJR nR66ZdIZC9KfSyY7M3B2XzkgwQuRdkQelQxt_lOA2omigTOcltOYveSkTZklCy.88Qn8Sau5JdIL RFr9Un6X_CW1A21tMczTuuBVybvZ2gfGklDz9wV8ng99DNW_zksHWmsvcENhFI00b0vdDWreCDKP 2dw6ay0H.6yrysNkunXdf.2zAcR7uZMX833xV7.O3uMGLJz6pHjlh1hVnjlXbjRBycwgeiuCtblv qPXkoWTSFJq3SCIZ_hXqtW.AVO35csPY3SF5spvx.pR6wgUBXX7dXFF_0fGhJgVW0C.ckk9oKgAJ 5W765si.usyD1mTc_AXeokgAKoRoeUXkL6GyDZxKNZV1oI6PJ9Jjvn3yDNqqR5eMUsmYjp_ucmqC uGa9_U68Rn6VSiMxSmQjeafYj87bNeJopUl6.fGbJPJq7QqNMftS6H4XBY1Wqqrutg0Vv3Wn1tmc DX.mTAEFXjVYo8OcV.IjBJsu94ehAzX5O7fowg4JGDnTX2UX3S7h6NdSFWmCKUUDUA42G3rVqWRE PigVCnTUMTjB8CJ1NTUFrh5sw.xowVa1FLXnwwzFySZx7oObFYWLIpCkG9sxmkqFj0BQl5l4bTgU MsI2_RL7dknV6ZG79IBR88oq6ioKojGHVUacoBsoT3bLmKZOnJJKWI7KauW_Ocx1PSqyNTPtlN.V RkVzgO8VBCbLPWstRwzqTgceT7CEWm5Zk3DopJ00Qnp.1onmyw.F12x9tudUM6sBRRmNlnUaWEt6 EpwNdLH3F0zID7NfsiuM1y6wGewDFmhoB4Yk2eKC5rBZXIbLhb6DVQDT.MfjFxvCNWKh3s8XWsub VhNBSg0rs7q9ja9ui4kN7WKYL8af5wy2.zP6yDyi8vynH4AdyXVGhNkkJW2auba6BFOg74YWuDZ_ kPwviYXgXWQEQ6SPwYcAQjA2GbSjxYj306i0WAUlKe.R2T4F.50W3234XnKlanEOklR1AKC6R7JA whWb4I1M_lzO.J9dLlPBxp_QQnHPvBGfc7PVdDTUKbGHOA5J4RpY2qeEetH3bntCW_LEYcptC.yD CRNPSrUOpCdeNJIhp8so6ycZC9Kw3J71NGAxoj_7Qqt_FDVz7FVJv._roVMJy119I.yaIT6Bva0r OMshJqey.HBOOq68kW3fJyxW5.XKsBJOY3Mwd4qCJQStSJnT.cqMFzZhrx_2n27z7Bru7twramtt sRJZJnqI_EvfmK2RwEwFF6imcXrCMZbYvyKXsJeNiInobASCTiQglIXHFj.9i8V8QCfjZYCFkgeC ivsEvIzPYTzv0c.g6maJkQHDeTUngTTOdWkG7Y33HDoFf4vHPB.udQC3U65lipA47pPvQqPjhOrE OvJsiDyMgEcMrcZxEVnd9pVLjTdb5rN8o596wTz_vseM0.On9W_cQFMdynEs0CfiY6WxQo_L_7b5 yAX227OyiU.qoiY1Se5ScBmMCqMr36vLR.Kt8N.g4griN5OMsxhwEAkbT2BSObWHwyvnC.eiS_Kz K3KP.23NnUcnlH_CSv_8QLqZmj1uKWojHBhwVce1udyRyq8GDatG_pC2C6R6OtpHH6BOCXkiXggQ D X-Sonic-MF: X-Sonic-ID: 84613e97-1997-46da-9159-6080a0c903df Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Aug 2023 07:22:16 +0000 Received: by hermes--production-gq1-6b7c87dcf5-792fm (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID f4efcc3bd0b0f3a50c7492784dcb57cf; Thu, 24 Aug 2023 07:22:11 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder From: Mark Millard In-Reply-To: Date: Thu, 24 Aug 2023 00:22:00 -0700 Cc: Current FreeBSD Content-Transfer-Encoding: 7bit Message-Id: <2CF7E1B7-3026-4485-B2F1-3D464CF0FE4F@yahoo.com> References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.700.6) X-Rspamd-Queue-Id: 4RWZKQ3Jzrz4btp X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] On Aug 23, 2023, at 22:54, Mateusz Guzik wrote: > On 8/24/23, Mark Millard wrote: >> On Aug 23, 2023, at 15:10, Mateusz Guzik wrote: >> >>> On 8/23/23, Mark Millard wrote: >>>> [Forked off the ZFS deadlock 14 discussion, per feedback.] >>>> . . . >>> >>> This is a known problem, but it is unclear if you should be running >>> into it in this setup. >> >> The changed fixed the issue: so I do run into the the issue >> for this setup. See below. >> >>> Can you try again but this time *revert* >>> 138a5dafba312ff39ce0eefdbe34de95519e600d, like so: >>> git revert 138a5dafba312ff39ce0eefdbe34de95519e600d >>> >>> may want to switch to a different branch first, for example: git >>> checkout -b vfstesting >> >> # git -C /usr/main-src/ diff sys/kern/vfs_subr.c >> diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c >> index 0f3f00abfd4a..5dff556ac258 100644 >> --- a/sys/kern/vfs_subr.c >> +++ b/sys/kern/vfs_subr.c >> @@ -3528,25 +3528,17 @@ vdbatch_process(struct vdbatch *vd) >> MPASS(curthread->td_pinned > 0); >> MPASS(vd->index == VDBATCH_SIZE); >> + mtx_lock(&vnode_list_mtx); >> critical_enter(); >> - if (mtx_trylock(&vnode_list_mtx)) { >> - for (i = 0; i < VDBATCH_SIZE; i++) { >> - vp = vd->tab[i]; >> - vd->tab[i] = NULL; >> - TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); >> - TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); >> - MPASS(vp->v_dbatchcpu != NOCPU); >> - vp->v_dbatchcpu = NOCPU; >> - } >> - mtx_unlock(&vnode_list_mtx); >> - } else { >> - for (i = 0; i < VDBATCH_SIZE; i++) { >> - vp = vd->tab[i]; >> - vd->tab[i] = NULL; >> - MPASS(vp->v_dbatchcpu != NOCPU); >> - vp->v_dbatchcpu = NOCPU; >> - } >> + for (i = 0; i < VDBATCH_SIZE; i++) { >> + vp = vd->tab[i]; >> + TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); >> + TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); >> + MPASS(vp->v_dbatchcpu != NOCPU); >> + vp->v_dbatchcpu = NOCPU; >> } >> + mtx_unlock(&vnode_list_mtx); >> + bzero(vd->tab, sizeof(vd->tab)); >> vd->index = 0; >> critical_exit(); >> } >> >> Still with: >> >> # grep USE_TMPFS= /usr/local/etc/poudriere.conf >> # EXAMPLE: USE_TMPFS="wrkdir data" >> #USE_TMPFS=all >> #USE_TMPFS="data" >> USE_TMPFS=no >> >> >> That allowed the other builders to eventually reach "Builder started" >> and later activity, [00:05:50] [27] [00:02:29] Builder started >> being the first non-[01] to do so, no vlruwk's observed in what >> I saw in top: >> >> . . . >> >> Now testing for the zfs deadlock issue should be possible for >> this setup. >> > > Thanks for testing, I wrote a fix: > > https://people.freebsd.org/~mjg/vfs-recycle-fix.diff > > Applies to *stock* kernel (as in without the revert). I'm going to leave the deadlock test running for when I sleep tonight. So it is going to be a while before I get to testing this. $ work will likely happen first as well. (No deadlock observed yet, by the way. 6+ hrs and 3000+ ports built so far.) I can easily restore the sys/kern/vfs_subr.c to then do normal 14.0-ALPHA2-ish based patching with: so not a problem. Thanks. === Mark Millard marklmi at yahoo.com From nobody Thu Aug 24 19:22:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWtJD6SF2z4rVDZ for ; Thu, 24 Aug 2023 19:22:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-54.consmr.mail.gq1.yahoo.com (sonic316-54.consmr.mail.gq1.yahoo.com [98.137.69.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWtJC5kVlz4fk3 for ; Thu, 24 Aug 2023 19:22:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=OhbABHpG; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692904937; bh=qferZE9/vX7e0w5rsqmGevvY+K8TlBkQ+mTZ0N6OLe8=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=OhbABHpGYzNsJX+TFDoKSOLVvEplInP2RZOktUrXPUyb22NMBXbCeX5wYhTimO4Qtq5lnCAxiqyaLoyy3lCjtXklPd1O+BLuQN0fFoZgzZk/tM4FSgQj3JpHbxPobw9P/H0NcSXD621g39RVrrh/Akh9VMyWeGVYCKvCgOjq1ugcEMIYHVqHb4FlazfHhmsmWpke6mmKhC2SWyQ5o7GA9Tz01GVoYckakbw1vUyEv8x9QmXDOD5N3LTPzsE/QRF+95UsAtYoV6OIWkIz5AGALV/z/Kkepnf4QUNWH+6SyrU+kijFq2G1kI2HRVDIUlICddL4GSndGj6hKPa9KbJ0Gw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692904937; bh=otEYXNqS6sHwRyeGBWw7yWmEqRrJtAsOGz60nGoFbB+=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=CdD1KG3x+uDs8D9RuVJup+hbSzNcCj/1/BzE5VfrseynxsQtRVxC8ZLf6bxSYIcRSMGoL2mDY7syeipmsYzIAk3ZZx/T90uUIC0uodwZv+khCKtapB4kpW7QMv3NGgF1aPuIE952m0OVZ3Kmh/2/qg1+hupovdhnpnUoykRYipWoHPDSA3yOKkdBbzKY5SBZgVoVqgrwXCcNHtzALTB/IcdCFt47wg7nQX0vp6Yfdvulq34+LX+Pmku5eYcyS5zKuQZQwEUgi9psBEV/bRPMJqdoHTJpR2hxubSL5AWhqHJVYZ8bqN2Cjo6klcfcCg7+HYnOGFNQ4vvl3o+qG/s/cg== X-YMail-OSG: c1wXOFEVM1lMPZYIylyU9Cxh8GbyxfZv2CkS5ZLMPEaH2Mp_vt.tM2Qhm5tOZRK UW.yy_qeHKwejmndnODvxKWPoxSi5ZbAioXAC65sZQGsNCsjXCf7.NqdENRaHr4zNDHKNMk4A_1L ck19NwGLYhhYZriPIaunI6_1AUAdAUjJ.ND7wKJsveQ1EDWnumGb6HKZaP09PIwZWwBAYFmgLyYq 1dpOfLk38Cf1X_Svztnyncn.ifogCHpsn8KlrGo.Y.Cl.AXwwD7Uj6BRg1pORJAEKaaR5rJKReU_ nM3jsqIFTtKLxao1uzDM6QtFKpJvIwmIf.KBzbd2pP2yrM2j3ePfacNWnZ0bMy3QqvxcfrsgcO0P EUlwUZPj5r0ylulToUwFHC0SZaH0MMfZyCiPpPoASVJZfwbTzrdFMMfMmTmYODHBCQZg4xJLeKL1 ygW0suOWrZJvlm9jcEtIzfBn6NFD024S6DdkfnuxWBq7IL2VoG0gLhYgca7FSez4ejon.R4fRexV weH5IPkFavIBKfXS4LWFaMcRn3Ozd9pAfbAra3QHpmU2mhQ6tjyUuSFrIbVRBa35FudIc0NDPg.6 n74L9FeaLyebzj6Ny9b7o35YZ1iLaCHZKUnCgZGDASbnpGieOLXVusKTZ74e5bjuDqFWKsCN46GU 2Jvup2XUuDIfTkZDoZyC2Xw2bNagWM8_nXAfDKESk3NR_qF1D40I3l2gfJJbW.cWvtOEu7DeEwxj h0d_SJ83Cl1vw531jBBySxOcs2zUlRK0qQrJbkgUKVr_sx.En3Ovg78VwxHLgBHi46oRIp6hTomn BjEKNUWe.v1SoMsxynwrZzytPyLLU.DWQlxLDpnOK1TlKRUe9r0zwOwmCaG2JJWrPLSk.dZxVQOa 0QZpy56_6hD6tCXudLvdaOWdKPiBRTX7m27sj.dHNwL5OVR31RbBNoMEgANos2vRfNJP7dtu.EdY AVuzdzauqS3Sr4rM3bmGzMLTaQYWsW6kNO_jKC0nxdvFwK6DrKvvffOMA1jFhIby5Yrpkdagy_rM GhDlcMw4kei9fuvQLGF18ONbZFnYrsuXi6fheUKWcyZZHDJY9p5oNH.NKz2cHDqjnAR1l0ySiOaw 8nwiEtKQxXiSUV07XFBVKOQ2n_xtAQ6ceeM1DmTcvgMxT_9TJl2z1R0AvGrIZ_fINBnXA19pQbky qQ573_ej3NTPLH3UD3op9zhj8K3STw0KYlbt3IHVubjPd_94Mn9_Uz3fmsbkrFnt8frK7WLFmICA OGd_CjDOT4plcBiz4ysrUrRpClTK37iWtyRwwdGrZVl6RVxg2_m3u.b4uV90J1ULKdNJnPxFhpq9 J_vPS8mqQt6qOdydxRSxV6zev6NhbzPYWobHrn2Ylvyvt4nh1Egm9vbyS1IaOhFPidXIjCp7VKRN W1rLgQdDizjhcvFjlqlltJI5XI1PPdFL8BtogfGq9PpPCtiHn72AcepfHSApU.EHUMcKjvF6i2u6 _80b4X3z_zjjW.Z0QEhmqiaFNMq7of6krV.bre1pHGgOE2Pgtr7OLNjeIh4itZuPS5glXvqvFlp7 6Z8uQNyCT3phbl5v3p.tA74ICWq_2wC3330xcQUqm3rnjVr2QHBL_4cx0ZhoT3Hy0v6czn_tPyZz TdWbVnaRzAtU0VEYT75t.Tutme4aoEXxKKcALw8FbELAmTs0Z2aNlezszNaavi5KAQOp5_ku6pgf X_uPkwppcma6ULLYj5Uh8CSa76oIqpk7kPTXz7sYTd7rSNgcjb2OxHdWKmls7Pihc.0JumzxA1YT eOhsvOB4RDxp15etU89YKVg9qqLPJb.PLTJ3UJusClpLxJHFRT4LGBybdajPOjxxtPEfF7fJo9Af ufe3VcF_0P.X84qaJK48O6gqN9Em.bn0Oi27XLdFMwMptWOfQAeO0laCwkYPio95l5bDllJg_3pK lUtGV4ZskpErLwMvQHKLthDwp3Z_5bbXbmm8f2BIalXpOYuytjVwRoOPP08cRUvvyYKCuxh2VppM C0zKXRYjmGG48H4gRkF1pODnCjIedDrzEOy4AbnR1KhgopPhOJpUbtzoJ1bnhRVS23xFfNSIh68. JVLloFHaEcNWdqV5Og8eXr7fhP7pk.Vgr494.K3x0RttbBIz.QFOgZcYjAHCqXqmZzjyEAhwUTA1 ioxF0xeWLVAaeTMC6a6rGF.NRn1NyiwsNNzgPtiFwFdTOT0JP3MROx7hRMyycPagmtnjhRP6uqJc g.5wR0EqzsmOG2utx.pDENwfhfnVTJFV2M..bo0gCNw8A60QQsA7FZJ7d7bPZGrveNuUYcJLGV2c - X-Sonic-MF: X-Sonic-ID: 75927ab7-ade5-4fcb-a2ce-b3a7df173b06 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Aug 2023 19:22:17 +0000 Received: by hermes--production-ne1-7b767b77cc-6vm8t (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 59bf4f7551a41c00b3a96f7c0b5dd551; Thu, 24 Aug 2023 19:22:12 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 From: Mark Millard In-Reply-To: <1AC87B79-6B65-402B-B65F-CCFFCC503861@yahoo.com> Date: Thu, 24 Aug 2023 12:22:00 -0700 Cc: Current FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: <40D0C681-C28B-47C2-B913-90A56CFD69D4@yahoo.com> References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <1AC87B79-6B65-402B-B65F-CCFFCC503861@yahoo.com> To: Alexander Motin X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.30:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.30:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWtJC5kVlz4fk3 On Aug 23, 2023, at 13:37, Mark Millard wrote: >=20 > On Aug 23, 2023, at 11:40, Alexander Motin wrote: >=20 >> On 22.08.2023 14:24, Mark Millard wrote: >>> Alexander Motin wrote on >>> Date: Tue, 22 Aug 2023 16:18:12 UTC : >>>> I am waiting for final test results from George Wilson and then = will >>>> request quick merge of both to zfs-2.2-release branch. = Unfortunately >>>> there are still not many reviewers for the PR, since the code is = not >>>> trivial, but at least with the test reports Brian Behlendorf and = Mark >>>> Maybee seem to be OK to merge the two PRs into 2.2. If somebody = else >>>> have tested and/or reviewed the PR, you may comment on it. >>> I had written to the list that when I tried to test the system >>> doing poudriere builds (initially with your patches) using >>> USE_TMPFS=3Dno so that zfs had to deal with all the file I/O, I >>> instead got only one builder that ended up active, the others >>> never reaching "Builder started": >>=20 >>> Top was showing lots of "vlruwk" for the cpdup's. For example: >>> . . . >>> 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 >>> 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 >>> 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 >>> 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 >>> 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 >>> 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 >>> . . . >>> But those processes did show CPU?? on occasion, as well as >>> *vnode less often. None of the cpdup's was stuck in >>> Removing your patches did not change the behavior. >>=20 >> Mark, to me "vlruwk" looks like a limit on number of vnodes. I was = not deep in that area at least recently, so somebody with more = experience there could try to diagnose it. At very least it does not = look related to the ZIL issue discussed in this thread, at least with = the information provided, so I am not surprised that the mentioned = patches do not affect it. >=20 > Thanks for the information. Good to know. I'll redirect this to be a = different discussion. Mateusz Guzik had me revert 138a5dafba31 ( which is for sys/kern/vfs_subr.c ), which was enough to allow me to run bulk -a with USE_TMPFS=3Dno usefully. (There is now a new sys/kern/vfs_subr.c patch for me to try instead.) So I used the reverted context to test without your patches to see if I'd get a deadlock from a bulk -a with USE_TMPFS=3Dno usage. It is past 9200 finished in about 18 hrs of building. No deadlock. (I do not plan on letting the bulk -a run to completion.) The 3 load averages are normally over 100 and the MaxObs figures for the 3 are currently: 349.68, 264.30, 243.16 (for a 32 hardware-thread system). So it looks like when I try again with Mateusz's new patch, trying with your patches would not be much of a test for preventing deadlocks for this context. More of a cross check on if other types of issues showed up vs. not. It is not clear how useful such testing might be. It might be that the high load average bulk -a style makes the deadlocks in question less likely for some reason. =3D=3D=3D Mark Millard marklmi at yahoo.com =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 24 19:44:17 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWtnq5J2wz4rWF5 for ; Thu, 24 Aug 2023 19:44:31 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWtnn5CrBz3DxJ for ; Thu, 24 Aug 2023 19:44:29 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of fbsd@www.zefox.net has no SPF policy when checking 50.1.20.27) smtp.mailfrom=fbsd@www.zefox.net; dmarc=none Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.17.1/8.15.2) with ESMTPS id 37OJiH2Q011868 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 24 Aug 2023 12:44:17 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.17.1/8.15.2/Submit) id 37OJiHrh011867 for freebsd-current@freebsd.org; Thu, 24 Aug 2023 12:44:17 -0700 (PDT) (envelope-from fbsd) Date: Thu, 24 Aug 2023 12:44:17 -0700 From: bob prohaska To: freebsd-current@freebsd.org Subject: Re: www/chromium will not build on a host w/ 8 CPU and 16G mem Message-ID: References: <2227F902-847E-4E50-B48A-B012CE51D96D.ref@yahoo.com> <2227F902-847E-4E50-B48A-B012CE51D96D@yahoo.com> <20230816163046.081be1181df386fae55065bc@dec.sakura.ne.jp> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-0.18 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_WWW(0.50)[]; WWW_DOT_DOMAIN(0.50)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-0.08)[-0.084]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[zefox.net]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Spamd-Bar: / X-Rspamd-Queue-Id: 4RWtnn5CrBz3DxJ On Fri, Aug 18, 2023 at 08:05:41AM +0200, Matthias Apitz wrote: > > sysctl vfs.read_max=128 > sysctl vfs.aio.max_buf_aio=8192 > sysctl vfs.aio.max_aio_queue_per_proc=65536 > sysctl vfs.aio.max_aio_per_proc=8192 > sysctl vfs.aio.max_aio_queue=65536 > sysctl vm.pageout_oom_seq=120 > sysctl vm.pfault_oom_attempts=-1 > Just tried these settings on a Pi4, 8GB. Seemingly no help, build of www/chromium failed again, saying only: ===> Compilation failed unexpectedly. Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer. *** Error code 1 No messages on the console at all, no indication of any swap use at all. If somebody can tell me how to invoke MAKE_JOBS_UNSAFE=yes, either locally or globally, I'll give it a try. But, if it's a system problem I'd expect at least a peep on the console.... Thanks for reading, bob prohaska From nobody Thu Aug 24 20:07:22 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWvJY1t0lz4rWwk for ; Thu, 24 Aug 2023 20:07:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-24.consmr.mail.gq1.yahoo.com (sonic311-24.consmr.mail.gq1.yahoo.com [98.137.65.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWvJX0ZPhz3HTn for ; Thu, 24 Aug 2023 20:07:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=kzr2Mr1p; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692907657; bh=CTiPCxlJlb05PibctPBVADFA3EoNMol6Ch5mBdil+lw=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=kzr2Mr1pFnyzNIPVyz1SADu+WYVy/2MTvfWP/Mb5aQVQN6DPEa/avKLYBBjWJu3Vi72kcRU/7u7Ih08U2MIL4rmyUKlgpWlqHllEDgWtvMaGnq09LcPzlFFCOdC7WLQp1a3p+/R0TrCF6Xc8/p+knFL2k7XP7Bo2wxvcBKVn/v2a9ilNpk7y5u5GGGCL9QUdAZtlvCcGQjJCJHuAPz1PVdsVACJkeVNr/s+7zY56NR3JRac1ZELGOOS3eLPJKfaKJ11pLyqtmorRlTNrOmz1GSc6jZnQHS6uSWgxr6xvUiCe4ywDugu/z3H/pUw2VatuJuw0S4UsiMd7mnCMzgv/sA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692907657; bh=ADPt5EwlwbchtujQOObRgq0SnT2ScOrSED6KgZe9XIe=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=BcUM82/jovf0TxzTwL9sd/qZSZ1F9ICX38g8WovI44ENFaCjaqAiKsMA9ZRKLTwA2j+iUe6Kl28T4IEecBNN/CHF13gf5jeD0Fqf2FxO3P9PE9BXsmxwMXXWYhftscGPzvEDxQvZv/D/j/hdbHKd+IatrCbfe3x3+vVhoAOKJH64OsrcTJWlqA+g329iUhSklu9V0zbHRSBTXUENqyZrnpn6Oo8ddSXk0jY45tY2Lt1Ag35t/qyb9/X1Btq7tnjy3DwlEjt+TrubY+bxkddYrPtS6vYhp5Epcnt9qCWARt0Ctp0Xcb2YwtcGu/XQuC306aceo0fHKqDzHjUrzrU2GQ== X-YMail-OSG: Eg4ZEQMVM1kGyRaj4VQwLLYtUWZ3b.nPvIsw0XDR5hf5x15YQG7cYdf.sUzksKt bNKqsMn9M9f1UqBw2Zzz6RJsDGWnWLb6vA0c_GOKWCjpJwfW6UjfKnNgBwL1eoIPlXmkb69KbZnF 7HEtXFFL2k24JPsjP6fEfPLmKH3J0pWtzSF1Zqhsw4kPnLtsA7YNybfZRhvv0cMyKWdNZnLtu_Rt nAqxT62BOSGScV6wtRJR7mOQLoBMg_tFam_doY_FF491159MZnjDbdvcRiFfhzElkLFjhdEdQm93 R3JZtjMMuudnbAUPbl3fLp6iWXYS5CvQGKMkYh.8Hdnj41cdiWD2VDAsGknOm4fXualeMJNxBZfR 31mBoqTw3Ho9gFzOmteonVBguc0wxZvskzYvwo9crB_EzpduU7vXciqfFoEZL0BapDisU1YBZ01Y RvCfRABn3y75rC1yo0L52tcI66dYk3nm_sZVQSMXn6xCYMbaQ0pxKXokoC03YUhgUS3q4xiY96fg FBxug2Djx4XGu1KBx5udxKXrjIMJxWFy2fYpXc4cQ8iyHkl5xl68k2aTqDDp6OyW3EE6A0qLfx3K y0u5.nSnM64PXejraOxcL0BA0TKm6SnJR1zCeVgK223wBjmtDmwa7eZIxvh3MeELWzgO9YWSEwJ4 CLIKxaRCgaqn.5sEN_BtqMiTugX4fgR3m81iCb9_wSP7Uk6IZ2L4sUxZsQ0mMlCk.UDBiDaTtotd Sd6BiATNKX.Bo5SENQhSKzAyj4q9a_QW0VXiDRu_sY75_I1mZW1gZz9C1H1GD0_wrHKpRTNR5zNk EdQcBLYeIT4OOZColbja3MB3byorwVrFiq5VULuDUIV6zs1QnICxoiSI0dOK4ejAwGuz7qfe4S1m RaAH38nqhuCU9Bfb2Bh5m.EQgp2Ewa_wN2Qs7RrMpJcOo8lCNCxrKtCW1SjMxXw.joqiCvU9zKtP A9LmPy9TNhKGMDW6QsZenaIVXT.cy8V8gWTGr1zFSoHrsjJ33V2BWo9QbEl.INcbRhoZ2TakHMgn EyZEwUAceDT1ThAe8E8m4dqWlyvhJS0IXpuH3Z1IahijRc15hQrjhWiyJnszJVGz4NiliCvRMbGe 3uyiMkHIaIdbxvQc4r9CyiMqcTqXKOtRlyszIyXG93UdhFC0FaQzTKbuP90tPJsKt1ffyVj61RuG 6bm.ePx0rYmb_ZeObPEID0OcSCY0Ygy5a_4SaTVsv_yCeHSumYW3jihNGVTM7iiq_aJJ5ZmzwhyD tbafU00mzgyDQlqRVsFBxFk6dSp.9TjL61Rva.ZLQDQenDPikZ.o3xPKftGxz6OliCDc8DPZ4p1q koqTDAOEikxe8x4A89G44aJNRCfiGgX6rkqZphmT7YXuwH76j64b4po8gKfDtBJ7n4QHGABNT9JD jN7Prsv9bgL0HS5J7WGTaQzMktrW.cMf2ZkIx8dmHXY45xN44MJF7SC5RWLKL6.40cCehBPQ2USL gKS9VzPinDvVwgbFUYfRE73Ex5MSUErQnEMFQ.VqUCPwwbnDHXhHrPSVN.S_TuU._RMvKLe.jkeo MqbOG3Qm4TE.b8ON4Kvcuy81e7nOMmaGXxMEWem.EKNZJfJ_NJdSwOeT7BwNgLItQVHs6lxfn.mP RIbiCVKj0kMNC3WY0AIqmMy76Y5w3VDEdRFYcqa.uU8m4bMBewDeuwTXi0Yj9pUj2aaCwXVDh081 BgpIkMjxEf1qyAM0I8z37pHPgGJPVbpBOM0pVfbemiXzvKuYaqkKKhIhaxVOpFsHae33c.kYgnhE RYL7lepz1IKzGXJ7CXWM2wV04qDr3iSjAbEocTSJc18HDjMfOrUUWFR.vcIHXH2dK7tEIFO9ewU4 5ILmshiXtu6_MQhzMS3M3lxSRL34.p2PXEdGntwg96PWPCH5TXT8PbWSAb3L4bEmTHxAMq.fPcwu JzwA5MJRUqMmLshka3N30axir0jjC2a1ZHeakZw5Qlnw.QC.W1gklFan989SWDpJpW1tInFKG7Li 0xvDRLoHheFklZ1H.uzFtnHST7jhqfm5_qyjCiLmu1.BTObkuPBetqPWaH5.e6XOALGNWqVdHPaS 8fGG8cR3O83tph9daKz2qxwdhpSqeZPbLMjliaPlm_tR86gdjkupZxEFguNgcT4tTe0y9SPEIVXp xDOU1XkwzuLkl6TPllbXFHRzqXPX3m2w4LdCDhmXg_iwoRuedInzxfYVW31uDmX2UwK0P2DtFvLF 9gqLjHafDHwSeUkrJI2_hgu6fJa7XC6vwsFYmfH6Zu7veC.qxRBmjYPl8zFaABMc1nWsQyFGlZeU yVEM- X-Sonic-MF: X-Sonic-ID: b588e2a7-4e7d-474d-9e3f-9ddfc3cb6cd1 Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Aug 2023 20:07:37 +0000 Received: by hermes--production-ne1-7b767b77cc-7tm2h (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID cc93bd5d4a5e33ed98b91a6cbd942207; Thu, 24 Aug 2023 20:07:34 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder From: Mark Millard In-Reply-To: <2CF7E1B7-3026-4485-B2F1-3D464CF0FE4F@yahoo.com> Date: Thu, 24 Aug 2023 13:07:22 -0700 Cc: Current FreeBSD Content-Transfer-Encoding: quoted-printable Message-Id: References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> <2CF7E1B7-3026-4485-B2F1-3D464CF0FE4F@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.205:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.205:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWvJX0ZPhz3HTn On Aug 24, 2023, at 00:22, Mark Millard wrote: > On Aug 23, 2023, at 22:54, Mateusz Guzik wrote: >=20 >> On 8/24/23, Mark Millard wrote: >>> On Aug 23, 2023, at 15:10, Mateusz Guzik wrote: >>>=20 >>>> On 8/23/23, Mark Millard wrote: >>>>> [Forked off the ZFS deadlock 14 discussion, per feedback.] >>>>> . . . >>>>=20 >>>> This is a known problem, but it is unclear if you should be running >>>> into it in this setup. >>>=20 >>> The changed fixed the issue: so I do run into the the issue >>> for this setup. See below. >>>=20 >>>> Can you try again but this time *revert* >>>> 138a5dafba312ff39ce0eefdbe34de95519e600d, like so: >>>> git revert 138a5dafba312ff39ce0eefdbe34de95519e600d >>>>=20 >>>> may want to switch to a different branch first, for example: git >>>> checkout -b vfstesting >>>=20 >>> # git -C /usr/main-src/ diff sys/kern/vfs_subr.c >>> diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c >>> index 0f3f00abfd4a..5dff556ac258 100644 >>> --- a/sys/kern/vfs_subr.c >>> +++ b/sys/kern/vfs_subr.c >>> @@ -3528,25 +3528,17 @@ vdbatch_process(struct vdbatch *vd) >>> MPASS(curthread->td_pinned > 0); >>> MPASS(vd->index =3D=3D VDBATCH_SIZE); >>> + mtx_lock(&vnode_list_mtx); >>> critical_enter(); >>> - if (mtx_trylock(&vnode_list_mtx)) { >>> - for (i =3D 0; i < VDBATCH_SIZE; i++) { >>> - vp =3D vd->tab[i]; >>> - vd->tab[i] =3D NULL; >>> - TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); >>> - TAILQ_INSERT_TAIL(&vnode_list, vp, = v_vnodelist); >>> - MPASS(vp->v_dbatchcpu !=3D NOCPU); >>> - vp->v_dbatchcpu =3D NOCPU; >>> - } >>> - mtx_unlock(&vnode_list_mtx); >>> - } else { >>> - for (i =3D 0; i < VDBATCH_SIZE; i++) { >>> - vp =3D vd->tab[i]; >>> - vd->tab[i] =3D NULL; >>> - MPASS(vp->v_dbatchcpu !=3D NOCPU); >>> - vp->v_dbatchcpu =3D NOCPU; >>> - } >>> + for (i =3D 0; i < VDBATCH_SIZE; i++) { >>> + vp =3D vd->tab[i]; >>> + TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); >>> + TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); >>> + MPASS(vp->v_dbatchcpu !=3D NOCPU); >>> + vp->v_dbatchcpu =3D NOCPU; >>> } >>> + mtx_unlock(&vnode_list_mtx); >>> + bzero(vd->tab, sizeof(vd->tab)); >>> vd->index =3D 0; >>> critical_exit(); >>> } >>>=20 >>> Still with: >>>=20 >>> # grep USE_TMPFS=3D /usr/local/etc/poudriere.conf >>> # EXAMPLE: USE_TMPFS=3D"wrkdir data" >>> #USE_TMPFS=3Dall >>> #USE_TMPFS=3D"data" >>> USE_TMPFS=3Dno >>>=20 >>>=20 >>> That allowed the other builders to eventually reach "Builder = started" >>> and later activity, [00:05:50] [27] [00:02:29] Builder started >>> being the first non-[01] to do so, no vlruwk's observed in what >>> I saw in top: >>>=20 >>> . . . >>>=20 >>> Now testing for the zfs deadlock issue should be possible for >>> this setup. >>>=20 >>=20 >> Thanks for testing, I wrote a fix: >>=20 >> https://people.freebsd.org/~mjg/vfs-recycle-fix.diff >>=20 >> Applies to *stock* kernel (as in without the revert). >=20 > I'm going to leave the deadlock test running for when > I sleep tonight. So it is going to be a while before > I get to testing this. $ work will likely happen first > as well. (No deadlock observed yet, by the way. 6+ hrs > and 3000+ ports built so far.) >=20 > I can easily restore the sys/kern/vfs_subr.c to then > do normal 14.0-ALPHA2-ish based patching with: so not > a problem. Thanks. >=20 I stopped the deadlock experiment, cleaned out the partial bulk -a, put back the modern sys/kern/vfs_subr.c , applied your patch, built, installed, rebooted, and started another bulk -a run. It made progress on all the builders to and past "Builder started": . . . [00:01:34] Building 34042 packages using up to 32 builders [00:01:34] Hit CTRL+t at any time to see build progress and stats [00:01:34] [01] [00:00:00] Builder starting [00:01:57] [01] [00:00:23] Builder started [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: Success [00:03:22] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 [00:03:22] [02] [00:00:00] Builder starting [00:03:22] [03] [00:00:00] Builder starting [00:03:22] [04] [00:00:00] Builder starting [00:03:22] [05] [00:00:00] Builder starting [00:03:22] [06] [00:00:00] Builder starting [00:03:22] [07] [00:00:00] Builder starting [00:03:22] [08] [00:00:00] Builder starting [00:03:22] [09] [00:00:00] Builder starting [00:03:22] [10] [00:00:00] Builder starting [00:03:22] [11] [00:00:00] Builder starting [00:03:22] [12] [00:00:00] Builder starting [00:03:22] [13] [00:00:00] Builder starting [00:03:22] [14] [00:00:00] Builder starting [00:03:22] [15] [00:00:00] Builder starting [00:03:22] [16] [00:00:00] Builder starting [00:03:22] [17] [00:00:00] Builder starting [00:03:22] [18] [00:00:00] Builder starting [00:03:22] [19] [00:00:00] Builder starting [00:03:22] [20] [00:00:00] Builder starting [00:03:22] [21] [00:00:00] Builder starting [00:03:22] [22] [00:00:00] Builder starting [00:03:22] [23] [00:00:00] Builder starting [00:03:22] [24] [00:00:00] Builder starting [00:03:22] [25] [00:00:00] Builder starting [00:03:22] [26] [00:00:00] Builder starting [00:03:22] [27] [00:00:00] Builder starting [00:03:22] [28] [00:00:00] Builder starting [00:03:22] [29] [00:00:00] Builder starting [00:03:22] [30] [00:00:00] Builder starting [00:03:22] [31] [00:00:00] Builder starting [00:03:22] [32] [00:00:00] Builder starting [00:03:30] [01] [00:00:08] Finished print/indexinfo | indexinfo-0.3.1: = Success [00:03:30] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 [00:04:42] [01] [00:01:12] Finished devel/gettext-runtime | = gettext-runtime-0.22: Success [00:04:48] [01] [00:00:00] Building devel/libtextstyle | = libtextstyle-0.22 [00:05:46] [19] [00:02:24] Builder started [00:05:46] [15] [00:02:24] Builder started [00:05:46] [19] [00:00:00] Building graphics/libpotrace | = libpotrace-1.16 [00:05:46] [15] [00:00:00] Building devel/libdaemon | libdaemon-0.14_1 [00:05:46] [25] [00:02:24] Builder started [00:05:46] [25] [00:00:00] Building audio/speexdsp | speexdsp-1.2.1 [00:05:46] [29] [00:02:24] Builder started [00:05:46] [29] [00:00:00] Building devel/opencl | opencl-3.0.14 . . . Thanks. I'll let it run as another deadlock test. The prior run built over 9400 in about 18.5 hr before I stopped it (no deadlocks observed). =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 24 22:20:50 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWyGV3m8Kz4rfSR for ; Thu, 24 Aug 2023 22:21:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-25.consmr.mail.gq1.yahoo.com (sonic304-25.consmr.mail.gq1.yahoo.com [98.137.68.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWyGT0Jj0z3bCC for ; Thu, 24 Aug 2023 22:21:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=CmVcVUUb; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692915662; bh=G7vFLD3bCO1RCS6K+iggZt3uLMk/ErRMMGmaQiGAHxk=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=CmVcVUUbtpFeF8QNwdQOpMlX1XFJ7u/A0oEmA05Ekn/34qf1bN0wCwxKm5+IIWvX3mqaSuMi3zZ1Dc40GfepRh366Mu+BtgiF4Msh2GprMxzyAF+IhEWYN+LYWXZOHx8+B4s/0m6rvkVfNCFZpeKBLxeabq6IyjEI0w0ehh1gIwG54ZXE8uqeBDuTZk5XdHyoDdyj8zPqSdkIMctb3I+q8zzL7T68x8FY6nXsVR/wdyJUjmNbOHsry07VRsz1U3gm4FnkHbMIocsLs8Nu7c2sq4eYxF0JlB31sXfJffd9YBtDu052Fi+mFvJkIy1+/8+9Wt7xJ8p7l3iordroPrmxg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692915662; bh=ahlb4l5vIiGgtVPA77CyMmdhtMUB/VA3nRundIEI7K0=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=OeZdoCTrDCq63OSxJWb/r3nz2f5xZdDwS+AMwZCfUyb8p6VmNbLD723JRmTrAbjl9AaKNs0hIFP3ZllKPsy8P+Mm6oFEnJcMOW6tSa6itHYEbQP+BSB2MNUctjVMX2ay3FNOJ1ok1qJWZfTtMNPO4CRrX7yOjZwMfNZLfwtJlJI4fCrVAxHGLyGML04OxG802nuWkjYybkerUi+vOXbnA3sA1ASzB31kGbTCXWI4jAwt8AVeTjQsEVgH1XC72rRkwJiLzSVH79EkQvxCu4HLrv9mzxB8dwwvS9hdy6/M69uKlba3FniXtRa6TYKbhM5CIl+jzV1NJP7w80P42eKI7A== X-YMail-OSG: ZDvzPeQVM1kiBYZrDGIOYrZfM0YLVFG4tFEGPzFhJXrekNKLADrzCiFus7yu.E3 q34X.txgtb4AU98EpSD44p1ifCSEHgJkjlX2xzInUMCiEqK_l.SLpbVOQf41Vq8CE9bFE77OKXtx EmiYPFA7kbZpcYRNrDyldiBYfR_Wz6Dhjh09XWn9Xa_HXT40owbZp4_Qn7TsMmYuAjzFnaV1poDU 4ADE_Itz4Ni8XGNaQGm4iC_kDiwW6J7u920JOU5V0c150pD65VnIix60pESLiXSOy7qnurmobeX0 9bh.9IYvi304VpObZrWUeJ3J5iHXE7NZZ1AcS91qbV8g2FgWOf85Gx5gNTAIG2W6NDVmBx._rme0 FnlNaSWXSTXelnRVIYgDjm4psOCGGXCr9kwE4snW5jYfwZRKT5itv47xCppvhsLbDyd6xtIaAVos nNbyVvd0dz4.zdmy7BT4DjXojaMU5PD9GmkmYio2qdkQgKPvbpReDfHbLamF7XNieN_j.vb6ecH9 NybSPqDrON.ROMqeaMVgoKmwB_V2qWHqtbB3zW2j.HXckP.ytv8.oRm7VSMX_T0WkUU.Pwc40KWv SoCYBd.CeDD5s66aDsT2cmTgFbNJD7yiuxTapZX1jAGu3qWH0A47QqsPaXFf_EfahBHy4UnVmx1d 10W62.sR4huynDdisZGJ53ctv9upxz6Rg3Ih1brMEJpNzPGjh7aAyPyK3OzaZEaEe0JL_fbDyUKH i1smG7udjmzUJLYwHYE8Owi4UnwXCa0Uuq9NCA0iwAX2y_WDK2dPN5QhKaP_6jmTr6FpC5EgkRTY THyanPycLBLvs3a8f8FdnGPBkxocm3_bbMRTNSBzKDhLgL8AZMlyX9Ok6ddrLlfR3ZwIYtBqt1CV RJXVfFUgUk0R1Tquac1SgDUB_phqoStJ1sa854.DxRM3NRWyoiY1_awMhzZusiXrQBcOHGL2AaM7 GLxosuelyk6huF7KB_qhMAIfaOXHWRS2ncBQ6grTpoWBsqFoGdpgxliCKZDKvG7bpm1PwHfxsX4u rg.86YeMqo8vqpar_K0MJ3rJsSVNoFvMDFbnCn4PlLRXkcEO2lT3tr.682cfH9tlSXfbKnXLf1WM Vvo5niuTqHnGmHb.YylmpAFFnDQqeFQrvN_eWFFHUvZYgK_2mrWNAAEQqRyV6lLZ_rgelQ6U9rn. GkPGNam2bp8UnMdGewTOydTgDrtoufCPfzOdiradGQP.9UsN5lmo.cpMaXxchFNnNbtzYffQUBNK 4u8EMFWFsAuSe7wKY3wMencLIQdMgqabCfLcuF_KDvrfSrFDeGX8wyyJWIIFOUvHTNBiGMBBlc7C KXV4qCRV3yZYJkyhwxMnKbsV.k5Mb_Vu42iMKh86Xe.zF7n.H6a26s1HlRX4dXd74kTZBnK.W8AQ b7U.nNOQv1IC6Yhp1GCFDwUAyEkSHZeOXyCHdUhn4qoQw7bvO3R3e2ftjBrwgiE7C9T3I2mlw4o_ OEO2YrDIampA7tqiaSEwzAR.sbLlF8xvLJIMFgwCnFRb.izKh1N0PxLLWo212w8EKxRgeKia2scJ dgEMncZw2W8QuB6.k4z4iljDOLSQ12LH6YfcwWYu7UVWhB7CyOvlGLqAUbz71vyFJMaTsFo7GCFW khMLq6dVPyadEOCPUQt6Ict4EuhiZsdEmKwko0Tu8zr9.ln.1qgCUnrzKZpqvo0sGltUBSS1C5GW Ev9Vp_Qr3h._dqXTTbBvY8hyC.DVygyS3wHrPbVGRtilW4x7Zs3mAzle8JMNWzr2as1QA_meBaHJ Y5BVY6bAdIs2Ml6dOyIgWazs5GXH43A35EiOzRh.8Sj4AyYvJgEftldZMgR9JUS_S42eXKbSJg.V FSCTZbMvUOAtj0lgnDA0jJ.eVfHuONNO0qOzdN64Zr_5PnNeufxtRcOQrzCbej2WC94PrzpIvXod ZH.adkLRQfa8HGu.tPjRuXJHOZsWV_Swc2x4DuS.EcP2JLoZRuqLrUUTZ.9yQFdYmX7UZegam6ri mlO6p1.23uAny3J3vdJm7Tr868ZL.EPSsstXqcjQLZlXVCRcVBhUaRVG1sYRJ3FUsp34yJPn.wbr vuGb0.I2s1g3TBRR_2it1UE2.THeL.ZWHHbAecFLSsjhtgQVspfD7qxCrNzQKbMffdRuv1dnwEcp shwBBVwIMchzdS7z.JTWUK_pG6mAF5JK10PsJ8LrMwWAxzaC3DWR3tdQvyMf_ODmaMWla6AWTKRl ensC.r90w_FnFIg_r6EAyyDiiFQXqKBGqoAz0E2vsBNbQSz7Ts0UoGQxTp1jxMIjpHZcAU6aog0E r X-Sonic-MF: X-Sonic-ID: f3b482b4-16d9-4b54-add9-f951ae6369e3 Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Aug 2023 22:21:02 +0000 Received: by hermes--production-ne1-7b767b77cc-q899j (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a2020579cda67b73abb996e7f78ec2e5; Thu, 24 Aug 2023 22:21:01 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: www/chromium will not build on a host w/ 8 CPU and 16G mem [RPi4B 8 GiByte example] Message-Id: <804E6287-71B7-4D2C-A72C-6FA681311139@yahoo.com> Date: Thu, 24 Aug 2023 15:20:50 -0700 To: bob prohaska , Current FreeBSD X-Mailer: Apple Mail (2.3731.700.6) References: <804E6287-71B7-4D2C-A72C-6FA681311139.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.206:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.206:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWyGT0Jj0z3bCC bob prohaska wrote on Date: Thu, 24 Aug 2023 19:44:17 UTC : > On Fri, Aug 18, 2023 at 08:05:41AM +0200, Matthias Apitz wrote: > >=20 > > sysctl vfs.read_max=3D128 > > sysctl vfs.aio.max_buf_aio=3D8192 > > sysctl vfs.aio.max_aio_queue_per_proc=3D65536 > > sysctl vfs.aio.max_aio_per_proc=3D8192 > > sysctl vfs.aio.max_aio_queue=3D65536 > > sysctl vm.pageout_oom_seq=3D120 > > sysctl vm.pfault_oom_attempts=3D-1=20 > >=20 >=20 > Just tried these settings on a Pi4, 8GB. Seemingly no help, > build of www/chromium failed again, saying only: >=20 > =3D=3D=3D> Compilation failed unexpectedly. > Try to set MAKE_JOBS_UNSAFE=3Dyes and rebuild before reporting the = failure to > the maintainer. > *** Error code 1 >=20 > No messages on the console at all, no indication of any swap use at = all. > If somebody can tell me how to invoke MAKE_JOBS_UNSAFE=3Dyes, either > locally or globally, I'll give it a try. But, if it's a system problem > I'd expect at least a peep on the console.... Are you going to post the log file someplace? You may have missed an earlier message. How long had it run before stopping? How does that match up with the MAX_EXECUTION_TIME and NOHANG_TIME and the like that you have poudriere set up to use ( /usr/local/etc/poudriere.conf ). Something relevant for the question is what you have for: # Grep build logs to determine a possible build failure reason. This is # only shown on the web interface. # Default: yes DETERMINE_BUILD_FAILURE_REASON=3Dno Using DETERMINE_BUILD_FAILURE_REASON leads to large builds running for a long time after it starts the process of stopping from a timeout> the grep activity takes a long time and the build activity is not stopped during the grep. vm.pageout_oom_seq=3D120 and vm.pfault_oom_attempts=3D-1 make sense to me for certain kinds of issues involved in large builds, presuming sufficient RAM+SWAP for how it is set up to operate. vm.pageout_oom_seq is associated with console/log messages. if one runs out of RAM+SWAP, vm.pfault_oom_attempts=3D-1 tends to lead to deadlock. But it allows slow I/O to have the time to complete and so can be useful. I'm not sure that any vfs.aio.* is actually involved: special system calls are involved, splitting requests vs. retrieving the status of completed requests later. Use of aio has to be explicit in the running software from what I can tell. I've no information about which software builds might be using aio during the build activity. # sysctl -d vfs.aio vfs.aio: Async IO management vfs.aio.max_buf_aio: Maximum buf aio requests per process vfs.aio.max_aio_queue_per_proc: Maximum queued aio requests per process vfs.aio.max_aio_per_proc: Maximum active aio requests per process vfs.aio.aiod_lifetime: Maximum lifetime for idle aiod vfs.aio.num_unmapped_aio: Number of aio requests presently handled by = unmapped I/O buffers vfs.aio.num_buf_aio: Number of aio requests presently handled by the buf = subsystem vfs.aio.num_queue_count: Number of queued aio requests vfs.aio.max_aio_queue: Maximum number of aio requests to queue, globally vfs.aio.target_aio_procs: Preferred number of ready kernel processes for = async IO vfs.aio.num_aio_procs: Number of presently active kernel processes for = async IO vfs.aio.max_aio_procs: Maximum number of kernel processes to use for = handling async IO=20 vfs.aio.unsafe_warningcnt: Warnings that will be triggered upon failed = IO requests on unsafe files vfs.aio.enable_unsafe: Permit asynchronous IO on all file types, not = just known-safe types vfs.read_max may well change the disk access sequences: # sysctl -d vfs.read_max vfs.read_max: Cluster read-ahead max block count That might well help some spinning rust or other types of I/O. MAKE_JOBS_UNSAFE=3Dyes is, for example, put in makefiles of ports that have problems with parallel build activity. It basically disables having parallel activity in the build context involved. I've no clue if you use the likes of, say, /usr/local/etc/poudriere.d/make.conf with conditional logic inside such as use of notation like: .if ${.CURDIR:M*/www/chromium} STUFF HERE .endif but you could. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 24 23:24:11 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWzgL1kPKz4rjLF; Thu, 24 Aug 2023 23:24:14 +0000 (UTC) (envelope-from gjb@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWzgL1D6Vz4FWh; Thu, 24 Aug 2023 23:24:14 +0000 (UTC) (envelope-from gjb@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692919454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=sOz3mI0W1i/Hk7ukOmrbqEYCmnMk9BkH6r62wZTMvgg=; b=KC136/GQnIYvvYVQPIfODT0FPEGrHH7Xe+3BOabwVN7iM0gdVWtD+w1B772H9lBDmgjBNP yzmGWl9aMyQzr0IufdBmuwy2lhw5GBo5mqthhSoI17ZTVrBLnDMa50RAXzdBYCvn+jPUjx v/xPckSwegtx29oGgfeKo9CICEQyzWjwKDfSb/f1q/bodrt3/EZ3crZh4C+rec+O7IwKMV PWt4Z1RMV7r/VasJfn65qLWD/e6URK12AdRmGllx9i/WF0I3S6nFvvZA5FXUTIFBE5VUlK wd2MqhMxRcFMGZNuIkHMUy0FEjw33bJWGc3n+/YyahFJrN1EdkpA5/effwsfKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692919454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=sOz3mI0W1i/Hk7ukOmrbqEYCmnMk9BkH6r62wZTMvgg=; b=C1I5odombH95Zexk2D0FrLS1CWaihkindCMnyprZfYBUy/DGG0lhiI65VpPcKT1rmlnHv8 oYECJVpGbeLXKipnqTuHQb/TqKbo/QaTsLPwALD/D8F8oSH8jS5vg7dFe0N+/R5W3vW78k WfXInCfvRTAksNIEcCu8TryWKJ0L9T8vbBDjcd85nuEo+1vVoqeIbdL9TvfO8AQgGfXoz7 rDqQZwtcyqd0hIR7SQcwEhktAuk0T1Z55ukoCJSo8Gl8lc7tUVCpSJ1eot8v9fVujAG2uK srah5ACmFr5SNiP/WOkGWrkVfeHKSOuAGTgxtB+9G8y2svTuPrdjzQ9YwXgsnQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692919454; a=rsa-sha256; cv=none; b=xfw7kDZY/YxrzROMtZWqwLhbjZYQ2l11uYd2un8NnOEhK+lu+quahq0Qf9GZaTB6v32Gt6 RtfYzh9SjCAXTSuWcUIx4pp7896CbiKbI2/JZK/BXhmk7QdXNChCHoKX60lQBbAIRsryXD n9Z8s0IziZrzt6q1aaL7oRKYZU4KI5w4g19U0AScUL5tQg9Wwyeu9BPVQB2uNDDuTz2Vps uTeDB47qIQJ5vADqrQKlwzWRnI3i4BIQYmfeM602UQLNAWP0XVAQreECI003bpknv500sh TYlreULyJQWLWbJcdJj+5nCyJ3mCS64D4tpz6W3HrbF8/12zLqB5oZ/vkHL/HQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id C95166237; Thu, 24 Aug 2023 23:24:13 +0000 (UTC) (envelope-from gjb@freebsd.org) Date: Thu, 24 Aug 2023 23:24:11 +0000 From: Glen Barber To: freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org Subject: HEADS-UP: stable/14 branched Message-ID: <20230824232411.GI4090@FreeBSD.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; x-action=pgp-signed -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 No email was sent for the branch commit, presumably because I pushed the branch and did subsequent changes (i.e., version numbers, etc.) afterward. The stable/14 branch has been created, from where the remainder of the 14.0 series will live. Glen On behalf of: re@ Please consider donating to help support my FreeBSD work: https://www.gofundme.com/f/gjbbsd https://paypal.me/gjbbsd -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmTn5psACgkQAxRYpUeP 4pM7oQ//USj527XH8Gfy2r6VL1N6l/NI8z0Rc6cIZbYTANQ1DJol0IEFlHPP5Xl3 FZ3X0ukWIAa+pDtsPq1d6bYCBfEfZ87Jxhrw12fZWrdam0p84hC1fVuRWDXEqHe4 ecBePE8MLz4cX+mG4H4cqgyypIS6Wcg+ICqI3/TpitTK2oIzctHSh7Md0HCRzV6i Rj5rGidyWz/tIgJSA6IHHkkTjXlvSBHRFkbVbiprkqcsSGCOa42j2kJxoHED3/S3 QPsKmjmpr2VhLjRBt+4R05phdkTXGcR1QZ4OdTd6Qo+JqGfo2j8dOXdS4awwh03v oTqSfK7BN/Vwo/nNGoMerXhg7IL40T4WtNx9Z99b5YVOCszFbPI3YMbojWHE+jpZ 4GD5CFPh7BfroJ+/Soyoeujro+4y7FssM1FKBfwHQiZAmJVLDqVnDUmff6F5wc2X ZfI1tawQl1e0IlFbc4fRku8HdwJn0esfGX7YiUFGnZEiX1EharHvFyhI8aUah0oO CYnBFFMyqV5sbr4Rl1UwI9EfoZ/XfLjh6G1Rd2st33OpgpzFHV1ksjkfJjmFDOc6 JmYK9We9Ei+BopVYkyGCk8tbxRo4zbsmpL/phk9S3EYKgERDoWJ2scP4w0dKWEsf oY2xvwR5yp2dZGCXMqqGPTJi/rd45KSQ3cgo86z8eGJcCUqXJlc= =+xEe -----END PGP SIGNATURE----- From nobody Fri Aug 25 00:41:31 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RX1Nb71jyz4qYqw for ; Fri, 25 Aug 2023 00:41:35 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yb1-xb31.google.com (mail-yb1-xb31.google.com [IPv6:2607:f8b0:4864:20::b31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RX1NZ3Zmhz4PqN; Fri, 25 Aug 2023 00:41:34 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=Mis8L1xa; spf=pass (mx1.freebsd.org: domain of mavbsd@gmail.com designates 2607:f8b0:4864:20::b31 as permitted sender) smtp.mailfrom=mavbsd@gmail.com; dmarc=none Received: by mail-yb1-xb31.google.com with SMTP id 3f1490d57ef6-d7260fae148so422907276.1; Thu, 24 Aug 2023 17:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692924093; x=1693528893; h=content-transfer-encoding:in-reply-to:references:cc:to:from :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=3SfsElmT6CGI5pVpN3LlqUHYOXhtTgBGtd5fu5gUJM8=; b=Mis8L1xanOJWEQb8KFMPiQiBzksWpJktVw28J8soTTENCCuY41y1446qVrgFORzoKX U4QDN/CZK1NIQdL/5/C0D0GrTHIAcCD3WbB/e/A2eSG0r87hS8ztbO2eEQban8iR5ZBh T3IxWld1NH5pv89f+632kr6nBDN6glkWDwt07g0wl966e00CXskvxz2kjCls4r77NTmp ZZtZxbXZlfVb1/gh3X9BCpX866kJi4dv8aE1hVu6K9cQIwcJa0qIghs1WpV+QKMXmJOH TtN4ooZkBKstxVHyG5i9er5on2a6BTEhFv0uHPWRz94H7BziYP/S+BVnbHY9/5Ewp7mO nWPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692924093; x=1693528893; h=content-transfer-encoding:in-reply-to:references:cc:to:from :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3SfsElmT6CGI5pVpN3LlqUHYOXhtTgBGtd5fu5gUJM8=; b=Sb5a3r6u6PEuCi4VJWtXIb4NgqGeKt264tglXB5fBTYppdwGJ/PNeCS8jomL76g8Ii 22Kj9z164JXpXzweLIrl9OHsJMTGM9784IRpmzgPDQGm5lhOKYwqhFZOXtkHRG3QXh7a L5r7RHqJgXcwRVb7gYNP0xdRrOZkz2jdc7m0knDdD4nX67qgwpEJBPTAIipHrLOUul5Y E/8i1mBnCwg3+XLkiVKQGSwuFEET0a5rXgIZ6ltcuHYWpUQ1y15Q4oSyIJbVn1a2PMTu r/wLyrsFXiMfgrL3MYaSgYRUggYbwvjNoGkDn5mSEOgpHRgQsHtztD/VY9qkGp9eB3W/ tFdg== X-Gm-Message-State: AOJu0Yxgt+RNFiCzwIExIMJBNeGLAMKJGm27RNCoQA6zxvk30sRVsoFV 9qIf+6tCpRiZWEUBFRZwAInliYiSYoo= X-Google-Smtp-Source: AGHT+IGJUbCJH2S0lIdWUnxXbKEcsDde7Zzgb7jrNNXqsYfBu2Blvj6FOZBByKDaTq9DRnAVKOCyrw== X-Received: by 2002:a25:ad0a:0:b0:d4d:d260:44bf with SMTP id y10-20020a25ad0a000000b00d4dd26044bfmr17023989ybi.59.1692924093339; Thu, 24 Aug 2023 17:41:33 -0700 (PDT) Received: from [192.168.1.66] (104-55-12-234.lightspeed.knvltn.sbcglobal.net. [104.55.12.234]) by smtp.gmail.com with ESMTPSA id q12-20020a25bfcc000000b00d13b72fae3esm142017ybm.2.2023.08.24.17.41.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 24 Aug 2023 17:41:32 -0700 (PDT) Message-ID: <2b628cf5-77cf-f148-bf47-92b2e43eee34@FreeBSD.org> Date: Thu, 24 Aug 2023 20:41:31 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: ZFS deadlock in 14 Content-Language: en-US From: Alexander Motin To: Martin Matuska Cc: current@freebsd.org References: <86leeltqcb.fsf@ltc.des.no> <86h6p4s64h.fsf@ltc.des.no> <86a5utrafp.fsf@ltc.des.no> <86350kqokl.fsf@ltc.des.no> <86y1icp95t.fsf@ltc.des.no> <86ttt0p8wv.fsf@ltc.des.no> <197ead1e-210a-6be6-7e24-5c56b14bb777@FreeBSD.org> <86350hpiby.fsf@ltc.des.no> <0fd64567-59cd-9276-cc0c-17a09611f11a@FreeBSD.org> <20230822102622.Horde.tUEwNRTMDn0AjOBiO1UB7OX@webmail.your-server.de> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.18 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.984]; FORGED_SENDER(0.30)[mav@FreeBSD.org,mavbsd@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; FROM_NEQ_ENVFROM(0.00)[mav@FreeBSD.org,mavbsd@gmail.com]; MLMMJ_DEST(0.00)[current@freebsd.org]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b31:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RX1NZ3Zmhz4PqN Martin, The PR was just merged to upstream master. Merge to zfs-2.2-release should follow shortly: https://github.com/openzfs/zfs/pull/15204 , same as some other 2.2 fixes: https://github.com/openzfs/zfs/pull/15205 . Can't wait to get back in sync with ZFS master in FreeBSD main. ;) On 22.08.2023 12:18, Alexander Motin wrote: > Hi Martin, > > I am waiting for final test results from George Wilson and then will > request quick merge of both to zfs-2.2-release branch.  Unfortunately > there are still not many reviewers for the PR, since the code is not > trivial, but at least with the test reports Brian Behlendorf and Mark > Maybee seem to be OK to merge the two PRs into 2.2.  If somebody else > have tested and/or reviewed the PR, you may comment on it. > > On 22.08.2023 04:26, Martin Matuska wrote: >> as 15107 is a prerequisite for 15122, >> would it be possible to have https://github.com/openzfs/zfs/pull/15107 >> merged into the OpenZFS zfs-2.2-release branch (and of course later >> 15122)? >> >> If the patches help I can cherry-pick them into main. >> >> Alexander Motin wrote: >> >>> On 17.08.2023 15:41, Dag-Erling Smørgrav wrote: >>>> Alexander Motin writes: >>>>> Trying to run your test (so far without reproduction) I see it >>>>> producing a substantial amount of ZIL writes.  The range of commits >>>>> you reduced the scope to so far includes my ZIL locking refactoring, >>>>> where I know for sure are some deadlocks.  I am already waiting for 3 >>>>> weeks now for reviews and tests for PR that should fix it: >>>>> https://github.com/openzfs/zfs/pull/15122 .  It would be good if you >>>>> could test it, though it seems to depend on few more earlier patches >>>>> not merged to FreeBSD yet. >>>> >>>> Do you have a FreeBSD branch with your patch applied? >>> >>> I don't have a FreeBSD branch, but these two patches apply clean and >>> build on top of today's FreeBSD main branch: >>> >>> https://github.com/openzfs/zfs/pull/15107 >>> https://github.com/openzfs/zfs/pull/15122 >>> >>> And if you still experience the issue, please show all stacks, or at >>> least include ZFS sync threads. > -- Alexander Motin From nobody Fri Aug 25 04:57:42 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RX74J1QhFz4qpgx; Fri, 25 Aug 2023 04:57:52 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RX74H433Nz3Hr7; Fri, 25 Aug 2023 04:57:51 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Authentication-Results: mx1.freebsd.org; none Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.17.1/8.15.2) with ESMTPS id 37P4vh64013209 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 24 Aug 2023 21:57:43 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.17.1/8.15.2/Submit) id 37P4vgvI013208; Thu, 24 Aug 2023 21:57:42 -0700 (PDT) (envelope-from fbsd) Date: Thu, 24 Aug 2023 21:57:42 -0700 From: bob prohaska To: Mark Millard Cc: Current FreeBSD , freebsd-arm@freebsd.org Subject: Re: www/chromium will not build on a host w/ 8 CPU and 16G mem [RPi4B 8 GiByte example] Message-ID: References: <804E6287-71B7-4D2C-A72C-6FA681311139.ref@yahoo.com> <804E6287-71B7-4D2C-A72C-6FA681311139@yahoo.com> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <804E6287-71B7-4D2C-A72C-6FA681311139@yahoo.com> X-Rspamd-Queue-Id: 4RX74H433Nz3Hr7 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US] On Thu, Aug 24, 2023 at 03:20:50PM -0700, Mark Millard wrote: > bob prohaska wrote on > Date: Thu, 24 Aug 2023 19:44:17 UTC : >=20 > > On Fri, Aug 18, 2023 at 08:05:41AM +0200, Matthias Apitz wrote: > > >=20 > > > sysctl vfs.read_max=3D128 > > > sysctl vfs.aio.max_buf_aio=3D8192 > > > sysctl vfs.aio.max_aio_queue_per_proc=3D65536 > > > sysctl vfs.aio.max_aio_per_proc=3D8192 > > > sysctl vfs.aio.max_aio_queue=3D65536 > > > sysctl vm.pageout_oom_seq=3D120 > > > sysctl vm.pfault_oom_attempts=3D-1=20 > > >=20 > >=20 > > Just tried these settings on a Pi4, 8GB. Seemingly no help, > > build of www/chromium failed again, saying only: > >=20 > > =3D=3D=3D> Compilation failed unexpectedly. > > Try to set MAKE_JOBS_UNSAFE=3Dyes and rebuild before reporting the fail= ure to > > the maintainer. > > *** Error code 1 > >=20 > > No messages on the console at all, no indication of any swap use at all. > > If somebody can tell me how to invoke MAKE_JOBS_UNSAFE=3Dyes, either > > locally or globally, I'll give it a try. But, if it's a system problem > > I'd expect at least a peep on the console.... >=20 > Are you going to post the log file someplace?=20 http://nemesis.zefox.com/~bob/data/logs/bulk/main-default/2023-08-20_16h11m= 59s/logs/errors/chromium-115.0.5790.170_1.log > You may have missed an earlier message.=20 Yes, I did. Some (very long) lines above there is: [ 96% 53691/55361] "python3" "../../build/toolchain/gcc_link_wrapper.py" --= output=3D"./v8_context_snapshot_generator" -- c++ -fuse-ld=3Dlld -Wl,--buil= d-id=3Dsha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--icf=3Dal= l -Wl,--color-diagnostics -Wl,--undefined-version -Wl,-mllvm,-enable-machin= e-outliner=3Dnever -no-canonical-prefixes -Wl,-O2 -Wl,--gc-sections -rdynam= ic -pie -Wl,--disable-new-dtags -Wl,--icf=3Dnone -L/usr/local/lib -fstack-= protector-strong -L/usr/local/lib -o "./v8_context_snapshot_generator" -Wl= ,--start-group @"./v8_context_snapshot_generator.rsp" -Wl,--end-group -lp= thread -lgmodule-2.0 -lglib-2.0 -lgobject-2.0 -lgthread-2.0 -lintl -licui18= n -licuuc -licudata -lnss3 -lsmime3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -ldl = -lkvm -lexecinfo -lutil -levent -lgio-2.0 -ljpeg -lpng16 -lxml2 -lxslt -lex= pat -lwebp -lwebpdemux -lwebpmux -lharfbuzz-subset -lharfbuzz -lfontconfig = -lopus -lopenh264 -lm -lz -ldav1d -lX11 -lXcomposite -lXdamage -lXext -lXfi= xes -lXrender -lXrandr -lXtst -lepoll-shim -ldrm -lxcb -lxkbcommon -lgbm -l= Xi -lGL -lpci -lffi -ldbus-1 -lpangocairo-1.0 -lpango-1.0 -lcairo -latk-1.0= -latk-bridge-2.0 -lsndio -lFLAC -lsnappy -latspi=20 FAILED: v8_context_snapshot_generator=20 Then, a bit further down in the file a series of=20 d.lld: error: relocation R_AARCH64_ABS64 cannot be used against local symbo= l; recompile with -fPIC complaints. Unclear if the two kinds of complaints are related, nor whether they're the= first.. > How long had it run before stopping?=20 95 hours, give or take. Nothing about timeout was reported > How does that match up with the MAX_EXECUTION_TIME > and NOHANG_TIME and the like that you have poudriere set > up to use ( /usr/local/etc/poudriere.conf ).=20 NOHANG_TIME=3D44400 MAX_EXECUTION_TIME=3D1728000 MAX_EXECUTION_TIME_EXTRACT=3D144000 MAX_EXECUTION_TIME_INSTALL=3D144000 MAX_EXECUTION_TIME_PACKAGE=3D11728000 Admittedly some are plain silly, I just started tacking on zeros after getting timeouts and being unable to match the error message and variable name.. I checked for duplicates this time, however. > Something relevant for the question is what you have for: >=20 > # Grep build logs to determine a possible build failure reason. This is > # only shown on the web interface. > # Default: yes > DETERMINE_BUILD_FAILURE_REASON=3Dno >=20 > Using DETERMINE_BUILD_FAILURE_REASON leads to large builds > running for a long time after it starts the process of > stopping from a timeout the grep activity takes a long > time and the build activity is not stopped during the > grep. >=20 >=20 > vm.pageout_oom_seq=3D120 and vm.pfault_oom_attempts=3D-1 make > sense to me for certain kinds of issues involved in large > builds, presuming sufficient RAM+SWAP for how it is set > up to operate. vm.pageout_oom_seq is associated with > console/log messages. if one runs out of RAM+SWAP, > vm.pfault_oom_attempts=3D-1 tends to lead to deadlock. But > it allows slow I/O to have the time to complete and so > can be useful. >=20 > I'm not sure that any vfs.aio.* is actually involved: special > system calls are involved, splitting requests vs. retrieving > the status of completed requests later. Use of aio has to be > explicit in the running software from what I can tell. I've > no information about which software builds might be using aio > during the build activity. >=20 > # sysctl -d vfs.aio > vfs.aio: Async IO management > vfs.aio.max_buf_aio: Maximum buf aio requests per process > vfs.aio.max_aio_queue_per_proc: Maximum queued aio requests per process > vfs.aio.max_aio_per_proc: Maximum active aio requests per process > vfs.aio.aiod_lifetime: Maximum lifetime for idle aiod > vfs.aio.num_unmapped_aio: Number of aio requests presently handled by unm= apped I/O buffers > vfs.aio.num_buf_aio: Number of aio requests presently handled by the buf = subsystem > vfs.aio.num_queue_count: Number of queued aio requests > vfs.aio.max_aio_queue: Maximum number of aio requests to queue, globally > vfs.aio.target_aio_procs: Preferred number of ready kernel processes for = async IO > vfs.aio.num_aio_procs: Number of presently active kernel processes for as= ync IO > vfs.aio.max_aio_procs: Maximum number of kernel processes to use for hand= ling async IO=20 > vfs.aio.unsafe_warningcnt: Warnings that will be triggered upon failed IO= requests on unsafe files > vfs.aio.enable_unsafe: Permit asynchronous IO on all file types, not just= known-safe types >=20 >=20 > vfs.read_max may well change the disk access sequences: >=20 > # sysctl -d vfs.read_max > vfs.read_max: Cluster read-ahead max block count >=20 > That might well help some spinning rust or other types of > I/O. There don't seem to be any indications of disk speed being a problem, despite using "spinning rust" 8-) >=20 >=20 > MAKE_JOBS_UNSAFE=3Dyes is, for example, put in makefiles of > ports that have problems with parallel build activity. It > basically disables having parallel activity in the build > context involved. I've no clue if you use the likes of, > say, > =20 > /usr/local/etc/poudriere.d/make.conf >=20 > with conditional logic inside such as use of notation > like: >=20 > .if ${.CURDIR:M*/www/chromium} > STUFF HERE > .endif >=20 > but you could. That wasn't needed when the Pi4 last compiled www/chromium. A Pi3 did benefit from tuning of that sort.=20 It sounds like the sysctl settings were unlikely to be=20 a source of the trouble seen, if not actively helpful. For the moment the machine is updating world and kernel. That should finish by tomorrow, at which point I'll try to add something like =20 .if ${.CURDIR:M*/www/chromium} MAKE_JOBS_UNSAFE=3Dyes .endif to /usr/local/etc/poudriere.d/make.conf Thanks very much for writing. bob prohaska From nobody Fri Aug 25 09:21:33 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXDwt1MVWz4r6Kh for ; Fri, 25 Aug 2023 09:21:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-19.consmr.mail.gq1.yahoo.com (sonic305-19.consmr.mail.gq1.yahoo.com [98.137.64.82]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXDws5zBsz3fDK for ; Fri, 25 Aug 2023 09:21:49 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692955306; bh=GmahZgEvkzuH/FQXJOuFXYDQVGgJJLrxPjfr2s9lglo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=i5cjHnbWWUIWTIw5faKRhq3vCCVmQzgUumfIqQ2Zr9xfcW3s+dgokAzOQLIRbS4FRsywE5WnUe/9qHeJ2eQAIfjsx+1q85en07zqiS6/KpMCxqTGE7eiqH1NYYb9z5Zruf3Hg6RyfYfCsZHMqw4G2OZO+smE4cxDeiazgDMGSQdtC2qzduHw1LpRwHCBLehYS9yuPLcvQtN9kqTog3Fez3+lnnpsJFcsL1SQyNBZ3Jvfo3sUYjTCbeW7eZnIPFVl/8pf9pu6Zh1EH/cNNJMuue2NjEgfJRjtLDsMlPTaTsmreLMpairpSR1vZJAzrW5duaI+t7JJKJc/wTkQCcUaiw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692955306; bh=TK9UmyAg7UXiCWLz+DzrYzrGF/1JhBTW7fDi88gAlSh=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=Bw0uU+2pK8RB7F5O2b/0314b7ytprxgjbi+UtjeRdzJCp1I5L5MKuBpeNsAGRLcn89ZU4s1jD0ijxEKLC6cFXaeH4SaTCGv262ExQ2XpIKTVzCJHtQhXuBBgTCyIStO7YtcUixH8syLggJwM7MY70hTj9pZ6jOZxx2brGb84BtFc77upXs88L2Axx2n9WZzUr37qRKaT09YzmJ8yh+sFwBWUqzNPddZW0lht6eRC7RgkMAWx7r2PxJYfuw9EjyXLB15tYZ1lBI91S8LyeSotv5W1SFRuHjYGcxGrIA5D2ZSC2Fbyn39GZL1lKdiQXwlwguMsIJ2mzVesrgtIyjEKCw== X-YMail-OSG: I6SaPP4VM1kDFTU7tOdeKJMo6WeWSluFD5izrsd2lEGaaJfpuqC7Ppo2oluj0Ed 673iOvQyInQG.O3Dyi1El5LV3mSBwmYEi1qe6SurXu5THrlg57w6YXpytnTNneW_KWm3nvywoemj JUXB7IyyeoIxpgFJUryO.vzCzCFSt9iYedAH0etokDOOm7nCECK5DrKSRCAkSYf0P_n_EfJML9aC EuW.EKYL1_K28Q9Ebqp4IlQ9cwrhDflSguvZorYU7.7kgY1O32Dt4W.qaug2Akwd5HndvKN8Lm1r CJwHemvCp_xbWOf_O1TCIW6Y3UEJEyGiGWT.iSoj5uwGxgG0kqFAsUyT2WK0Y4AyHLg2CLCz0V5P 7K1sBFKo6hoKfUca8vwJMrQPTxOM_uhGSKG4shC5tcy_h2X_BUFqK5Kf.BIdgSK6KP5FfCn4iN8k _ebPSbzIw62ES7HBs1Pek2LquZAwfQM5i0vSa3L7Sd7BQQWiJoI8jiNQPVigJXlBg6YwBiIvtZ4o 5KJlnKOzJYh3sIBpBr6qP5ineV2lnIENCPht3WQ.OrYYtE7L0RWGWvoiEBPRBgxG9pf8XLeGYHZx N11OzoXCSAGEFBcbCJ2prZceJn7w7slmDEBwgZaTZ_NyV375xeka2.fPvRRNYzp2Fa1KRMdIs4MI FSmbqtG546qWaUsShYKYCP3SOZvyzZ4c94mgcIAsRWPL_NVWpOj1Mv88Szchw4RKUNFJellroQa1 QibiG3pMIXabjFGJ44eyz6M8BxN1z3NBnGr4ieFLi8TMVBZDoQ5CXQ2VwkbfbVLlt.1aAuJeH5S_ qcxPIYkH72lDJu_z_tsWN9Mqbp7bCiVXkLu14txKswm5FGIXBMlj_KLQU36dKf2GC3kxJJw.OQR2 o4elS8Jt_pzD0UpQ0o6cTXFpXL6gSl.spBKtZclH63NIDdsXi2.4F7PAls7nyWPf0CLdEMFhhLiV VMrO_pG0CwPb6_QDZvXPXlnFGkdVCQNeTnnqS159uZvgiGz3n.YuGoDp_yLhncmCmkzO_9M4pH_R pRGE1GZRzgyUuS2Lau.yDdLmK7kEW56iDy62qSDVEUy.OXmQ59mAbmssJ.i5O43N6LjXEPPSAv81 8EX9.Qvsb5udYtIwTFFKBg7ilrTquddj3vo8Pbd2gGWUi_cSTmPoFrjJxVsKaMOsNGDEqEaFXxGT fUMfK7uvJYXiA7CDy4lqqA278ZxrmTBWd.Bk37xcEQwS7hYT4WyYybkpSwbMYkpWyH_0aM_MURMd SpBlyGWpTw0PK4gBQtB61aah6N9w0Sq73ZBPWHHsDY6mDTJFmerFW7w7Oz6SJOlDLESriPXydBTR X3a1mE6M7gxu7SbRdFpceY2313K_69xNh4BO2hUfcu8H0DzZmdCdrsYxdpfw5ogwVrbaxNks0rJo EdGfpMe3rWQuImHrCyr2RHcQVvhauT5qLC06FMKI9Aq0buapVmkOnfNgpMmxH2TXFHkJCWAVYBIZ 0369ycKuMkVNRT526N03SxZdh8xUIcGOTO3pa6XbvaKuD0gzoMFFTJfMdIdPdaIIYlbihzjUbI3n bJ1FKBwASXHId_vF0WNTj1yS.aLjf6G4qz2m8gpM74K9djmVxVgs6esCZEbbf22nRFjLULE3sDiC 1boG8OvbblIbPjxH3vzZ4DTmHGWz.s0BDDLYjPfP_rbQMU_3ls2I9kb.gSGnl3bJ3eDTa4qZ_KAz ieQG8viHw3L0VonZg_jbwuBinKBIfWC7Im67mglXLLKt8lBkhmoaWKhmPpYxDOMlNPQ4Z1cG2DYI gH6ika3hA4yRCaWDwq8n85VoPt_iifOT4AWJdsRG3tMj4opQrFwe_Xdl9sKJHMQ1S8PxRXntOakT k32lLY_ocezhOjefgP4l3SLYdGCgO8X3DhTevcXiuwKiVqc.nncDClg4ydJXa6Fk.ec7dxuE88nP ldOHtXvESpz9xPFQxEVr0bjLedmFzgpDZBGpFe0M_hru8aV4kodg31usxy_fHzDPuyuRWfzPZU__ lmilUw5EwjwNvo8pJfRFuwburRC2oCo2bBzuUZSO1HePHUkNUqmj3c550nc34iRf6iZenuTmK0C3 vGoSOxU3_.FjQBwk0a30PgfI0TylJYBrgW9yzZLb6ADeS08l_zMB47.Is40s3DiJty7HoQ_aJABF IYoUejDIvULK6JtHXlQvuj2dBWOSqvPO6g26CP8habLFkPqW0c_sg_Kkt1A5OsTqTY9ZQQdO2zlK .lTJ__eil6c3wYe0EV3bh67vvFAaAiOQ9vW7P9vF2I0AiR1Wvnw_FjULSWxGDg9nEHlJTF4_Vt_P Bfp4- X-Sonic-MF: X-Sonic-ID: f917baca-d30e-4667-9bad-2eb2c1c22601 Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Fri, 25 Aug 2023 09:21:46 +0000 Received: by hermes--production-gq1-6b7c87dcf5-9dzwt (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID ac5e9bae4323d5f3ea8d390ba87d248b; Fri, 25 Aug 2023 09:21:44 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: www/chromium will not build on a host w/ 8 CPU and 16G mem [RPi4B 8 GiByte example] From: Mark Millard In-Reply-To: Date: Fri, 25 Aug 2023 02:21:33 -0700 Cc: Current FreeBSD , freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <73355E51-CF0A-4397-8B63-92A4497F1090@yahoo.com> References: <804E6287-71B7-4D2C-A72C-6FA681311139.ref@yahoo.com> <804E6287-71B7-4D2C-A72C-6FA681311139@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3731.700.6) X-Rspamd-Queue-Id: 4RXDws5zBsz3fDK X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] On Aug 24, 2023, at 21:57, bob prohaska wrote: > On Thu, Aug 24, 2023 at 03:20:50PM -0700, Mark Millard wrote: >> bob prohaska wrote on >> Date: Thu, 24 Aug 2023 19:44:17 UTC : >>=20 >>> On Fri, Aug 18, 2023 at 08:05:41AM +0200, Matthias Apitz wrote: >>>>=20 >>>> sysctl vfs.read_max=3D128 >>>> sysctl vfs.aio.max_buf_aio=3D8192 >>>> sysctl vfs.aio.max_aio_queue_per_proc=3D65536 >>>> sysctl vfs.aio.max_aio_per_proc=3D8192 >>>> sysctl vfs.aio.max_aio_queue=3D65536 >>>> sysctl vm.pageout_oom_seq=3D120 >>>> sysctl vm.pfault_oom_attempts=3D-1=20 >>>>=20 >>>=20 >>> Just tried these settings on a Pi4, 8GB. Seemingly no help, >>> build of www/chromium failed again, saying only: >>>=20 >>> =3D=3D=3D> Compilation failed unexpectedly. >>> Try to set MAKE_JOBS_UNSAFE=3Dyes and rebuild before reporting the = failure to >>> the maintainer. >>> *** Error code 1 >>>=20 >>> No messages on the console at all, no indication of any swap use at = all. >>> If somebody can tell me how to invoke MAKE_JOBS_UNSAFE=3Dyes, either >>> locally or globally, I'll give it a try. But, if it's a system = problem >>> I'd expect at least a peep on the console.... >>=20 >> Are you going to post the log file someplace?=20 >=20 >=20 > = http://nemesis.zefox.com/~bob/data/logs/bulk/main-default/2023-08-20_16h11= m59s/logs/errors/chromium-115.0.5790.170_1.log >=20 >> You may have missed an earlier message.=20 >=20 > Yes, I did. Some (very long) lines above there is: >=20 > [ 96% 53691/55361] "python3" = "../../build/toolchain/gcc_link_wrapper.py" = --output=3D"./v8_context_snapshot_generator" -- c++ -fuse-ld=3Dlld = -Wl,--build-id=3Dsha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now = -Wl,--icf=3Dall -Wl,--color-diagnostics -Wl,--undefined-version = -Wl,-mllvm,-enable-machine-outliner=3Dnever -no-canonical-prefixes = -Wl,-O2 -Wl,--gc-sections -rdynamic -pie -Wl,--disable-new-dtags = -Wl,--icf=3Dnone -L/usr/local/lib -fstack-protector-strong = -L/usr/local/lib -o "./v8_context_snapshot_generator" -Wl,--start-group = @"./v8_context_snapshot_generator.rsp" -Wl,--end-group -lpthread = -lgmodule-2.0 -lglib-2.0 -lgobject-2.0 -lgthread-2.0 -lintl -licui18n = -licuuc -licudata -lnss3 -lsmime3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -ldl = -lkvm -lexecinfo -lutil -levent -lgio-2.0 -ljpeg -lpng16 -lxml2 -lxslt = -lexpat -lwebp -lwebpdemux -lwebpmux -lharfbuzz-subset -lharfbuzz = -lfontconfig -lopus -lopenh264 -lm -lz -ldav1d -lX11 -lXcomposite = -lXdamage -lXext -lXfixes -lXrender -lXrandr -lXtst -lepoll-shim -ldrm = -lxcb -lxkbcommon -lgbm -lXi -lGL -lpci -lffi -ldbus-1 -lpangocairo-1.0 = -lpango-1.0 -lcairo -latk-1.0 -latk-bridge-2.0 -lsndio -lFLAC -lsnappy = -latspi=20 > FAILED: v8_context_snapshot_generator=20 That FAILED line is 64637. > Then, a bit further down in the file a series of=20 > d.lld: error: relocation R_AARCH64_ABS64 cannot be used against local = symbol; recompile with -fPIC > complaints. The first R_AARCH64_ABS64 lines is 64339. After that are the next 2 = lines, with: defined in = obj/third_party/ffmpeg/libffmpeg_internal.a(ffmpeg_internal/autorename_lib= avcodec_aarch64_fft_neon.o) and: referenced by = ffmpeg_internal/autorename_libavcodec_aarch64_fft_neon.o:(fft_tab_neon) = in archive obj/third_party/ffmpeg/libffmpeg_internal.a > Unclear if the two kinds of complaints are related, nor whether = they're the first.. >=20 >> How long had it run before stopping?=20 >=20 > 95 hours, give or take. Nothing about timeout was reported >=20 >> How does that match up with the MAX_EXECUTION_TIME >> and NOHANG_TIME and the like that you have poudriere set >> up to use ( /usr/local/etc/poudriere.conf ).=20 >=20 > NOHANG_TIME=3D44400 > MAX_EXECUTION_TIME=3D1728000 > MAX_EXECUTION_TIME_EXTRACT=3D144000 > MAX_EXECUTION_TIME_INSTALL=3D144000 > MAX_EXECUTION_TIME_PACKAGE=3D11728000 > Admittedly some are plain silly, I just started > tacking on zeros after getting timeouts and being > unable to match the error message and variable name.. >=20 > I checked for duplicates this time, however. Not stopped for time. >> Something relevant for the question is what you have for: >>=20 >> # Grep build logs to determine a possible build failure reason. This = is >> # only shown on the web interface. >> # Default: yes >> DETERMINE_BUILD_FAILURE_REASON=3Dno >>=20 >> Using DETERMINE_BUILD_FAILURE_REASON leads to large builds >> running for a long time after it starts the process of >> stopping from a timeout the grep activity takes a long >> time and the build activity is not stopped during the >> grep. >>=20 >>=20 >> vm.pageout_oom_seq=3D120 and vm.pfault_oom_attempts=3D-1 make >> sense to me for certain kinds of issues involved in large >> builds, presuming sufficient RAM+SWAP for how it is set >> up to operate. vm.pageout_oom_seq is associated with >> console/log messages. if one runs out of RAM+SWAP, >> vm.pfault_oom_attempts=3D-1 tends to lead to deadlock. But >> it allows slow I/O to have the time to complete and so >> can be useful. >>=20 >> I'm not sure that any vfs.aio.* is actually involved: special >> system calls are involved, splitting requests vs. retrieving >> the status of completed requests later. Use of aio has to be >> explicit in the running software from what I can tell. I've >> no information about which software builds might be using aio >> during the build activity. >>=20 >> # sysctl -d vfs.aio >> vfs.aio: Async IO management >> vfs.aio.max_buf_aio: Maximum buf aio requests per process >> vfs.aio.max_aio_queue_per_proc: Maximum queued aio requests per = process >> vfs.aio.max_aio_per_proc: Maximum active aio requests per process >> vfs.aio.aiod_lifetime: Maximum lifetime for idle aiod >> vfs.aio.num_unmapped_aio: Number of aio requests presently handled by = unmapped I/O buffers >> vfs.aio.num_buf_aio: Number of aio requests presently handled by the = buf subsystem >> vfs.aio.num_queue_count: Number of queued aio requests >> vfs.aio.max_aio_queue: Maximum number of aio requests to queue, = globally >> vfs.aio.target_aio_procs: Preferred number of ready kernel processes = for async IO >> vfs.aio.num_aio_procs: Number of presently active kernel processes = for async IO >> vfs.aio.max_aio_procs: Maximum number of kernel processes to use for = handling async IO=20 >> vfs.aio.unsafe_warningcnt: Warnings that will be triggered upon = failed IO requests on unsafe files >> vfs.aio.enable_unsafe: Permit asynchronous IO on all file types, not = just known-safe types >>=20 >>=20 >> vfs.read_max may well change the disk access sequences: >>=20 >> # sysctl -d vfs.read_max >> vfs.read_max: Cluster read-ahead max block count >>=20 >> That might well help some spinning rust or other types of >> I/O. > There don't seem to be any indications of disk speed being > a problem, despite using "spinning rust" 8-) Nope: R_AARCH64_ABS64 misuse is not a disk speed issue. >>=20 >>=20 >> MAKE_JOBS_UNSAFE=3Dyes is, for example, put in makefiles of >> ports that have problems with parallel build activity. It >> basically disables having parallel activity in the build >> context involved. I've no clue if you use the likes of, >> say, >>=20 >=20 >> /usr/local/etc/poudriere.d/make.conf >>=20 >> with conditional logic inside such as use of notation >> like: >>=20 >> .if ${.CURDIR:M*/www/chromium} >> STUFF HERE >> .endif >>=20 >> but you could. The actual R_AARCH64_ABS64 use is in: = obj/third_party/ffmpeg/libffmpeg_internal.a(ffmpeg_internal/autorename_lib= avcodec_aarch64_fft_neon.o) not directly in chromium. The solution is not clear to me. > That wasn't needed when the Pi4 last compiled www/chromium. > A Pi3 did benefit from tuning of that sort.=20 >=20 > It sounds like the sysctl settings were unlikely to be=20 > a source of the trouble seen, if not actively helpful. Yep, the sysctl's were not relevant. > For the moment the machine is updating world and kernel. > That should finish by tomorrow, at which point I'll try > to add something like =20 >=20 > .if ${.CURDIR:M*/www/chromium} > MAKE_JOBS_UNSAFE=3Dyes > .endif >=20 > to /usr/local/etc/poudriere.d/make.conf That will not help avoid the R_AARCH64_ABS64 abuse, unfortunately. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Fri Aug 25 12:50:58 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXKZl2hMTz4rHpB for ; Fri, 25 Aug 2023 12:51:27 +0000 (UTC) (envelope-from yasu@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXKZl2D7jz4Qb2; Fri, 25 Aug 2023 12:51:27 +0000 (UTC) (envelope-from yasu@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692967887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zjwQSyJE8DSLxW5DJMrhFKQ6KzosKr7BPjaVYExE89g=; b=b/uvJF3UhYfb0yxwGJNLlg+HY5FtiZe0eCi5ob0Yave5qjHMcpocjQZgFuNIRKOvBvuySU H4nCnbSw6tCPm8lwD+tbf5/eOlHq51/xZB5TW7aiY+nmZDkSrfbFBkkXsKOVmfeMWQuo2B SeSGg4tSZwMCWA1q4In430osh/a7qmn4C+A5R7nm5ipUTDtUWlEEXMwWYy5tdbeabCMaiW ba6kuSZyh0QpojEIU3WqkeHuayVue1/z/z0Kupqrpin5dFMBtoHU4n2wZzmLVTnS0/QUid M2RvKR/VQ1TwitGwWqZgCkyRRWgrE4MhSFJ0p7o1ufCF6OTb7OL8QMdX/cQBSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692967887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=zjwQSyJE8DSLxW5DJMrhFKQ6KzosKr7BPjaVYExE89g=; b=YFL/98UaVj6PJsbcnsCwsXgjHP1856FV2QOR60m++BcBwrF5QE7iI6u2xMAFTtf1JZeMo9 NUuWw4sH/36enlxAWiNSAw8WceFWUUUT5yK2gXgTpR2HxCBkaARt6RyxTJH40M1E1E2wyf 6JzUIrh5+tn6OTpBpmDotdSlMm1O/j3EbArhNU00Z7Jsqo88hasUdUjwWH1SYsfNPjHWGP nPSvYhcpQ+5tGPtidhemaEU6inXhZSzvxtpA76KlrEH5U0R9MkqduzX++hWSPJbK1lF8qR Fsrpf3zbZaQkFDCaOekB7sFSqrfrLcJelPKxs1CdeD0RWr/QAeSrRJjeoa0sdw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692967887; a=rsa-sha256; cv=none; b=b6ssuFjD4yFLxIGxOe8Fc/TZdSCPRXOvbqkEPmZGidu5mStm6xAgrzO+b3PabJwljACZJ/ U9sx21FrFJ2Ri5tu45d1qVHkjVEflTrdNeak0EUmlj1J4FYMhFN6Iivr9bajyre1G3rZq/ BJhIWWeisEr+cKLuALeE4DRIKy9FeJt/dJXgp4UXV4cwQOEViVR0O2p7ZrL02QZad9/rWP 0RKzU5+yO4lMtCzISarpT/DAhHnoJIft98sOcCqaUarM/5lL4FNg2u26GsyYpRsbtFZTk8 SGYDKi9UQGiP7MCu9PmuZ4vZnV4yyELZ6nSEN/+U0/jU0ixF9+Chw+tcfmZpOA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from localhost (unknown [IPv6:240b:11:220:fe00::174:11]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) (Authenticated sender: yasu/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RXKZk4Mzlz1R05; Fri, 25 Aug 2023 12:51:26 +0000 (UTC) (envelope-from yasu@FreeBSD.org) Date: Fri, 25 Aug 2023 21:50:58 +0900 (JST) Message-Id: <20230825.215058.859665587566428364.yasu@FreeBSD.org> To: freebsd-current@freebsd.org Subject: Panic with 15.0-CURRENT From: Yasuhiro Kimura X-Mailer: Mew version 6.9 on Emacs 30.0.50 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hello, I made regular update of my amd64 system from main-n264870-e5e6a865358 to main-n265022-1554ba03b65 and system crashed with panic while building packages with poudriere. Screenshot of console: https://people.freebsd.org/~yasu/FreeBSD-15-CURRENT-amd64-main-n265022-1554ba03b65.20230825.panic.png --- Yasuhiro Kimura From nobody Fri Aug 25 14:29:02 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXMlP4vDZz4rNpL for ; Fri, 25 Aug 2023 14:29:05 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXMlP1tRnz4bdj; Fri, 25 Aug 2023 14:29:05 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ot1-x329.google.com with SMTP id 46e09a7af769-6bd066b0fd4so705041a34.2; Fri, 25 Aug 2023 07:29:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692973743; x=1693578543; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=H0EJFSZ3894cgayFu7/wCAQbJrVejhKwQo5tq5i+zE4=; b=ieb7Bpkfp8MZ8L0auZ3BFCV6MROHpOfIHtM6gg72eqYhdKbSmZy1YQfXOkk1ttuKXa VF+Cp0d3sDQsHFb9/g1clpGv9odZGMJiFGe4hs45Ui5a5Nj/zExXHJpmjPYg3KWu2tBU xzEUXMoGCOgblFmnK7z1di5cZJ8ND7UU4m/Rx9ahiuoVQNjxIIITfQndEfhnwOS2fjdd YOEgdCYyx0FeptFKOWwOudDSUWhW0BMcEnCetJdwJcaIG2+e7nDLgKP14f2mdAHuhiXU 6fVA+FTaGn+ClaMR+m3jc1sxzf/oGee63Ia/wEwoIfjIryq5/1TTLlve7ByZUL4rCW+1 sbZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692973743; x=1693578543; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=H0EJFSZ3894cgayFu7/wCAQbJrVejhKwQo5tq5i+zE4=; b=B675hstUJMv0ue2UCMV0vqNFEWQ8poKtRMyGhuMyJZLe/XY/Ch285WvjEVgsqsqLxb 6cvCbjYeZot6tuRl01AhhNOyT2qFQbjjL7+p3ccQFi3vGve3BA14v97Qe2ujp+tubkHq +pc+b2AgFpPHIgrPFeRT+JqtEw1C4S2563kH9vOlLsGjBAEzuymK0XSccrCV2Ji1456i PRj/HBvbJuWCMyWV7JKnYhcpsQHQ7wozufiw1qOJQyUr9XA9igoczDPMvIGu2SjEFjaq AxLwSHFWf9NKQb/iCtwovRBnauf7QlfrNrF0djetQUycsynG/ICJyo26mlQGZy3AtI2O Wk1g== X-Gm-Message-State: AOJu0YwejlC2UrJQGDo6qOuGQ+uDr+9gRFGwIIIgWvpkmjc9F9PeqPQO 2YL5WnAPC5RSYEOAcsPHC4wilXF6zrzTrq+NMkzszUQq X-Google-Smtp-Source: AGHT+IFrtQUok8o4QJBAPQDZrYoyvuhqOFTGvwrfjgb7mXHo5vUiOc2lINdvp6dyiwgttH2l3kq4+vJD8Wfqv450sXU= X-Received: by 2002:a05:6870:808a:b0:1bb:85c3:929e with SMTP id q10-20020a056870808a00b001bb85c3929emr2942645oab.48.1692973743429; Fri, 25 Aug 2023 07:29:03 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1141:0:b0:4f0:1250:dd51 with HTTP; Fri, 25 Aug 2023 07:29:02 -0700 (PDT) In-Reply-To: <20230825.215058.859665587566428364.yasu@FreeBSD.org> References: <20230825.215058.859665587566428364.yasu@FreeBSD.org> From: Mateusz Guzik Date: Fri, 25 Aug 2023 16:29:02 +0200 Message-ID: Subject: Re: Panic with 15.0-CURRENT To: Yasuhiro Kimura Cc: freebsd-current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RXMlP1tRnz4bdj X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On 8/25/23, Yasuhiro Kimura wrote: > Hello, > > I made regular update of my amd64 system from main-n264870-e5e6a865358 > to main-n265022-1554ba03b65 and system crashed with panic while > building packages with poudriere. > > Screenshot of console: > https://people.freebsd.org/~yasu/FreeBSD-15-CURRENT-amd64-main-n265022-1554ba03b65.20230825.panic.png > this is a fallout from the recent timerfd commit. I'll fix it in few. -- Mateusz Guzik From nobody Fri Aug 25 14:47:53 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXN981fy3z4rPpn for ; Fri, 25 Aug 2023 14:47:56 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc2d.google.com (mail-oo1-xc2d.google.com [IPv6:2607:f8b0:4864:20::c2d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXN972jDlz4f4q; Fri, 25 Aug 2023 14:47:55 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=bZhpJUSJ; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::c2d as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-xc2d.google.com with SMTP id 006d021491bc7-57129417cecso578485eaf.1; Fri, 25 Aug 2023 07:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692974874; x=1693579674; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Zt4rdELMxAfC+tqbsGLlMequicv8aLTclKh/II2TX/Q=; b=bZhpJUSJKDR7l0p6aqOMLpwvG2HZpFCD70Tlg7SDj5FsB6mqn7q1cvucakIZBvdjKM gAPO60p+h0Ly2/KeV22+NPVPK/fg9Q2FuHxZoAA9T8XGYrRSIAH0o6ciY74Whzp0uEM/ aK2Y7LKsBDvBM4g9qM3wdJFeRliZbAgy5BgiZIVnWVQUZBaEjlNeR3b+JRV4P36HdkwY dk0f3m7XvMe+kZD1g+lDc6UFzg1x7DP7fp5p4Mzr3d8lPmmpho0BmvyrvOY63TafD6tu m+JN4B/MIjKUVcSLGtAi3bcMVJwQi+jUEytDDwL+5ARx9/H20YgHz90Z3zM/TszcMBP/ TI9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692974874; x=1693579674; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Zt4rdELMxAfC+tqbsGLlMequicv8aLTclKh/II2TX/Q=; b=OOXtmqT4OGdibekxCqHPyKNnNPQIA4E+vKvXvYPDQN1Qmf5+gBMaOXNsg4+FdsmSLM q+w5f9zNUBj3YDZKSAY5fXLCEIFoCSUu65X2jGsxHgzkzsXtooHXYvRjz/VHFptNGRhP Y9Bj64gGaMSAasHstuhFKgmN+sweY4v98ebS7riHW+lqYB2O40H6SjoMRNMZqK8jpUtk oSlczuehBTKWJDoF7E+z3oKgs8B+E7CdO6ammljA7UDeH8bg6vPlCjBQxsggQE/C4ygg ym4OYA1p8LPyiJQSm2c57hxRHbwXPQTE9kcQlf4bKayB6brLTcmW8yjgWKIMPeahgOeu 6NSg== X-Gm-Message-State: AOJu0Yzq8qfR6AP47dqnN4KRfQvhBqySbQvU8miwyKU0DHrn2lRnTzs5 KkDD2qK5dTN7/K6kRQP0Dx5Bv+nRfLg5lCsolv02Qt0l X-Google-Smtp-Source: AGHT+IHXRSStHXD0zMt+zNMhtXNQBSQ8kSf+I4/9N6nGwAuBoYlpUCswyAIHNoPuEE2q1Tkz+zrUG4fhhfw0qfVmTYI= X-Received: by 2002:a4a:de43:0:b0:56e:9c66:e8a3 with SMTP id z3-20020a4ade43000000b0056e9c66e8a3mr3340802oot.4.1692974873824; Fri, 25 Aug 2023 07:47:53 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1141:0:b0:4f0:1250:dd51 with HTTP; Fri, 25 Aug 2023 07:47:53 -0700 (PDT) In-Reply-To: References: <20230825.215058.859665587566428364.yasu@FreeBSD.org> From: Mateusz Guzik Date: Fri, 25 Aug 2023 16:47:53 +0200 Message-ID: Subject: Re: Panic with 15.0-CURRENT To: Yasuhiro Kimura Cc: freebsd-current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c2d:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RXN972jDlz4f4q On 8/25/23, Mateusz Guzik wrote: > On 8/25/23, Yasuhiro Kimura wrote: >> Hello, >> >> I made regular update of my amd64 system from main-n264870-e5e6a865358 >> to main-n265022-1554ba03b65 and system crashed with panic while >> building packages with poudriere. >> >> Screenshot of console: >> https://people.freebsd.org/~yasu/FreeBSD-15-CURRENT-amd64-main-n265022-1554ba03b65.20230825.panic.png >> > > this is a fallout from the recent timerfd commit. I'll fix it in few. > fixed in https://cgit.freebsd.org/src/commit/?id=02f534b57f84d6f4f97c337b05b383c8b3aaf18c -- Mateusz Guzik From nobody Fri Aug 25 15:45:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXPRd0KVsz4rRwB; Fri, 25 Aug 2023 15:45:33 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXPRc3pXPz4kWj; Fri, 25 Aug 2023 15:45:32 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Authentication-Results: mx1.freebsd.org; none Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.17.1/8.15.2) with ESMTPS id 37PFjR90015225 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 25 Aug 2023 08:45:27 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.17.1/8.15.2/Submit) id 37PFjQkR015224; Fri, 25 Aug 2023 08:45:26 -0700 (PDT) (envelope-from fbsd) Date: Fri, 25 Aug 2023 08:45:26 -0700 From: bob prohaska To: Mark Millard Cc: Current FreeBSD , freebsd-arm@freebsd.org Subject: Re: www/chromium will not build on a host w/ 8 CPU and 16G mem [RPi4B 8 GiByte example] Message-ID: References: <804E6287-71B7-4D2C-A72C-6FA681311139.ref@yahoo.com> <804E6287-71B7-4D2C-A72C-6FA681311139@yahoo.com> <73355E51-CF0A-4397-8B63-92A4497F1090@yahoo.com> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <73355E51-CF0A-4397-8B63-92A4497F1090@yahoo.com> X-Rspamd-Queue-Id: 4RXPRc3pXPz4kWj X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US] On Fri, Aug 25, 2023 at 02:21:33AM -0700, Mark Millard wrote: > > That will not help avoid the R_AARCH64_ABS64 abuse, > unfortunately. > > Thank you for the analysis. I've posted a bug,id=273349. Sounds like I shouldn't hold my breath 8-( bob prohaska > From nobody Fri Aug 25 17:54:42 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXSJn3QTjz4qZw7 for ; Fri, 25 Aug 2023 17:54:49 +0000 (UTC) (envelope-from felix@palmen-it.de) Received: from stef.palmen-it.de (stef.palmen-it.de [IPv6:2001:470:1f0b:bbb:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXSJm0FMvz3FlB; Fri, 25 Aug 2023 17:54:48 +0000 (UTC) (envelope-from felix@palmen-it.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=palmen-it.de header.s=20200414 header.b=zI0ppiMU; spf=pass (mx1.freebsd.org: domain of felix@palmen-it.de designates 2001:470:1f0b:bbb:1::1 as permitted sender) smtp.mailfrom=felix@palmen-it.de; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=palmen-it.de; s=20200414; h=Content-Type:MIME-Version:Message-ID:Subject:Cc :To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=oVPqDcHYSxagmMM4V1NJTAEXjnazfBWcEJPJDYIVFUE=; b=zI0ppiMUFJ3pbe2sa7o5tA0fHa EVbZFHiUbsdsYgscSr8z0dMLUMXGYMTpHDvQxwpP7FDnwNh4cfz+nZPJ5hNlItfDkFSnhWq419/+1 Jbobcf0OS3sypj/h2OouNmAN7VQSLzyVHoke8AHSgLspIdclXzRV01p66cB4wiNqShbFiPJA98bvr v4D55OTvMtJV4GUX2deQ54pqlDDegc2NQQRgW5PhTwh78l23sSZ1/np0WGnVz+4Ce9hY95PT2SI/Q OSg4s1K4xn9NBgzeN8KAAO0AIjTXAGwcvQB6g7MvJdG1enGtdwF9V3abKGkdsc36MYkhJkIDg7GJO 8n9bANpA==; Received: from [192.168.71.101] (helo=mail.home.palmen-it.de) by stef.palmen-it.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qZb19-00Ar16-UR; Fri, 25 Aug 2023 19:54:43 +0200 Received: from nexus.home.palmen-it.de ([192.168.99.2]) by mail.home.palmen-it.de with esmtpsa (TLS1.3) tls TLS_CHACHA20_POLY1305_SHA256 (Exim 4.96 (FreeBSD)) (envelope-from ) id 1qZb19-000051-1w; Fri, 25 Aug 2023 17:54:43 +0000 Date: Fri, 25 Aug 2023 19:54:42 +0200 From: Felix Palmen To: current@freebsd.org Cc: Dmitry Chagin Subject: Possible issue with linux xattr support? Message-ID: Mail-Followup-To: current@freebsd.org, Dmitry Chagin X-Face: /1K@t"h.}e~pR@]c7HorQ!T`F^RJCa'BCr#e>IKA{>C/9OTGB4|xh"y2{?1Z5M i2w"AH^pN_LlHR^{+f',_Np~;.B;!M/bL}*qk]p5*r7F5vW};{:@4u5S?T&f0$7BJ-71Q5SV]:v$`5 A0[DZ:=?S52x8HJ~5@^P_\T@MsjG{R( Organization: FreeBSD.org List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="2ln4onhn5xy4ls2n" Content-Disposition: inline User-Agent: NeoMutt/20230517 X-Spamd-Result: default: False [-6.50 / 15.00]; SIGNED_PGP(-2.00)[]; DWL_DNSWL_MED(-2.00)[palmen-it.de:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[zirias@freebsd.org,felix@palmen-it.de]; RCVD_IN_DNSWL_MED(-0.20)[2001:470:1f0b:bbb:1::1:from]; R_DKIM_ALLOW(-0.20)[palmen-it.de:s=20200414]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f0b:bbb:1::1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; FROM_HAS_DN(0.00)[]; FROM_NEQ_ENVFROM(0.00)[zirias@freebsd.org,felix@palmen-it.de]; RCVD_COUNT_TWO(0.00)[2]; HAS_ORG_HEADER(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DKIM_TRACE(0.00)[palmen-it.de:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_TLS_ALL(0.00)[] X-Spamd-Bar: ------ X-Rspamd-Queue-Id: 4RXSJm0FMvz3FlB --2ln4onhn5xy4ls2n Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi all and Dmitry, (CC to Dmitry because he added this linux xattr support in 22dca7acf7756c07d3ccbfdc8dfd10fd9ad3f9cf) I'm currently working on an idea to build a userland for Linuxulator =66rom source here: https://github.com/Zirias/zfbsd-ports/commits/linux Now I ran into an issue with the first port using the full "base" userland for building (I tried expat2 for this): The "install" tool from GNU coreutils was unable to set file permissions, and truss showed me this: | linux_fsetxattr(0x4,0x401860e8,0x134dd0,0x1c,0x0) ERR#-1 'Operation | not permitted' To verify, I removed xattr support completely from coreutils (and also sed) in commit "linuxsrc: Disable usage of xattr" and indeed, with this change, GNU's install works as it should. So, hoping for someone with some knowledge how to debug this ;) Thanks, Felix --=20 Felix Palmen {private} felix@palmen-it.de -- ports committer -- {web} http://palmen-it.de {pgp public key} http://palmen-it.de/pub.txt {pgp fingerprint} 6936 13D5 5BBF 4837 B212 3ACC 54AD E006 9879 F231 --2ln4onhn5xy4ls2n Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEABYKAH0WIQRpNhPVW79IN7ISOsxUreAGmHnyMQUCZOjq118UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0Njkz NjEzRDU1QkJGNDgzN0IyMTIzQUNDNTRBREUwMDY5ODc5RjIzMQAKCRBUreAGmHny MWGnAQDcDejICs8th3DSl9+udk1G+nCOgzBpB4sD9WtRErR1PQEAvrjUTihNalkA c5gnxMO/dqQ7RE/6UZ4vitxePAlL8gU= =yvya -----END PGP SIGNATURE----- --2ln4onhn5xy4ls2n-- From nobody Fri Aug 25 19:12:47 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXV2r1wdwz4qgV4 for ; Fri, 25 Aug 2023 19:12:52 +0000 (UTC) (envelope-from felix@palmen-it.de) Received: from stef.palmen-it.de (stef.palmen-it.de [IPv6:2001:470:1f0b:bbb:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXV2q55W0z3Qct; Fri, 25 Aug 2023 19:12:51 +0000 (UTC) (envelope-from felix@palmen-it.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=palmen-it.de header.s=20200414 header.b="t2Ch/Kpw"; spf=pass (mx1.freebsd.org: domain of felix@palmen-it.de designates 2001:470:1f0b:bbb:1::1 as permitted sender) smtp.mailfrom=felix@palmen-it.de; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=palmen-it.de; s=20200414; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=nf4MLyOT70nBtiZ0cREZHyDTIiY+yYs+MWBJcYS1Aok=; b=t2Ch/KpwuG91I8xqynG9CAKxor etoU1JHlRLsh45+pAHgTfOjyuJFkI0kyagokBNkTVq72f7SEkL7euk/n/F7YAjwPcZsAZq9ojbBxs t1I9Eec6fKDGvWX9OAUWOSoN/6FntJfD3CcWAk3z03Fr7uOw7bC+2GOjYA03fIlnn7vuaQ5KvvZx2 4epGIvtpkviYM6BV3g3nKCGIIhMva5PTyIG5IgM5S9UjP5M09vUVuNZPU9rHO96wRwusLAatJfr+n QcKBKpv/aNk1OS89Uv92/ZCdE7hcXVY768klnB2UmmtpZHkieb/5ATuB6R/uCZeNjE7W7H9eArdoR Eqr1RbPg==; Received: from [192.168.71.101] (helo=mail.home.palmen-it.de) by stef.palmen-it.de with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qZcEj-00ArId-EF; Fri, 25 Aug 2023 21:12:49 +0200 Received: from nexus.home.palmen-it.de ([192.168.99.2]) by mail.home.palmen-it.de with esmtpsa (TLS1.3) tls TLS_CHACHA20_POLY1305_SHA256 (Exim 4.96 (FreeBSD)) (envelope-from ) id 1qZcEi-0000R5-2s; Fri, 25 Aug 2023 19:12:49 +0000 Date: Fri, 25 Aug 2023 21:12:47 +0200 From: Felix Palmen To: current@freebsd.org Cc: Dmitry Chagin Subject: Re: Possible issue with linux xattr support? Message-ID: <72n5izanidp74gqb67xuzqcowaokayk3v5dtsxl3pcsdw35gy3@qnrwhd7sjkpi> Mail-Followup-To: current@freebsd.org, Dmitry Chagin X-Face: /1K@t"h.}e~pR@]c7HorQ!T`F^RJCa'BCr#e>IKA{>C/9OTGB4|xh"y2{?1Z5M i2w"AH^pN_LlHR^{+f',_Np~;.B;!M/bL}*qk]p5*r7F5vW};{:@4u5S?T&f0$7BJ-71Q5SV]:v$`5 A0[DZ:=?S52x8HJ~5@^P_\T@MsjG{R( Organization: FreeBSD.org References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="xotyndfv3zwpnqdt" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20230517 X-Spamd-Result: default: False [-6.47 / 15.00]; SIGNED_PGP(-2.00)[]; DWL_DNSWL_MED(-2.00)[palmen-it.de:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.966]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[zirias@freebsd.org,felix@palmen-it.de]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:1f0b:bbb:1::1:c]; RCVD_IN_DNSWL_MED(-0.20)[2001:470:1f0b:bbb:1::1:from]; R_DKIM_ALLOW(-0.20)[palmen-it.de:s=20200414]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; FROM_HAS_DN(0.00)[]; FROM_NEQ_ENVFROM(0.00)[zirias@freebsd.org,felix@palmen-it.de]; RCVD_COUNT_TWO(0.00)[2]; HAS_ORG_HEADER(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[palmen-it.de:+]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_TLS_ALL(0.00)[] X-Spamd-Bar: ------ X-Rspamd-Queue-Id: 4RXV2q55W0z3Qct --xotyndfv3zwpnqdt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Felix Palmen [20230825 19:54]: > So, hoping for someone with some knowledge how to debug this ;) Information I forgot to add, sorry: I did my tests on an aarch64 machine with a kernel built from a98a0090b2ba64ff0bc3cf71a00fb5f9e31fc1d3. Cheers, Felix --=20 Felix Palmen {private} felix@palmen-it.de -- ports committer -- {web} http://palmen-it.de {pgp public key} http://palmen-it.de/pub.txt {pgp fingerprint} 6936 13D5 5BBF 4837 B212 3ACC 54AD E006 9879 F231 --xotyndfv3zwpnqdt Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iNUEABYKAH0WIQRpNhPVW79IN7ISOsxUreAGmHnyMQUCZOj9L18UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0Njkz NjEzRDU1QkJGNDgzN0IyMTIzQUNDNTRBREUwMDY5ODc5RjIzMQAKCRBUreAGmHny MX2kAQCCLgT3bdzXzYKkPMgsp7NEuy5Ddtxn0zIgieaM9ZFWXwEA7dlsnB0OXdZl gNnGqt+L9t1G4943F7GHPzxwkqf1XQE= =tPId -----END PGP SIGNATURE----- --xotyndfv3zwpnqdt-- From nobody Fri Aug 25 20:58:24 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXXNg21Gvz4qnBb; Fri, 25 Aug 2023 20:58:27 +0000 (UTC) (envelope-from gjb@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXXNg1Yx1z3ZgL; Fri, 25 Aug 2023 20:58:27 +0000 (UTC) (envelope-from gjb@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692997107; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=WOMEmdQorh4WlUkC3acBFJ3700Xxh9YmUSRDJb/jDEI=; b=uShpprZtbIvYhHziLvUQDxWNkOgF/DlYQ2qTfdnt+jqbc5L5EAbuGe/kt4fMhpnpaGq/ii IaSfVx3Va2qVw9x7mj9mwkMimpFMD8WwXjDOOzvUQuGkMENF/6tZnqEz6ur6dP9Rqpy7Cb DM5n3VCvyuKt4J+lcvGyu1Bj4XjTcwXbb1Qmkj59h1DhW/e91c0xVyQVILDnr1BURbf0NE oxqz6mWVtRsZYNJJfQXDNKRzG9pAEi+OXKAFBVDT0FXAnIBbnn9ei3bp+dtyD2Isc468O6 8FIQA2xHDzjynUYrHNliLFynYgndWmmDJ0n7CxspiQmil7+qiQuEtaq02o2BpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692997107; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=WOMEmdQorh4WlUkC3acBFJ3700Xxh9YmUSRDJb/jDEI=; b=SmFU9qTUO85OXIaHAjEYprJWJG2oJ/BVzzu45zzp9zXmixdyahbZjOaop0N6QN0XB/ktrA +Bnxcpzr1bexBnW9g66dkyHvjorVFuiK2WQhxsH23QHkkMU9LuCaTObjt6/c78qWi1aNb+ ZuabXawa+DQddJi2rQJmf1ibJuyjDDrO6tQcPfDnjXd/wMZX9X3B2j+D9No4PE/0ojxxLP FgzegnDDE+YaK72GhGXRxJGlu+GTNvIQdNFudSGIcG6Ev2FZ4Kg9y4y7QvVBa7Pn+4SEu0 U0e+TS3SUdgZ4mgve4fKUFcQrrMEriaJ9sih8L0KLx2t+j9Pn3ijPwN5Ol2T3A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692997107; a=rsa-sha256; cv=none; b=cpt78QI8Oxo+0nlqbTa11n8c5SULk6xcuE9t7KK8qD49tYKY/zKHqR9tZ1IXMhauc7/fB2 xXgur2vgCE5sJVeEucrlHrVmEN6pQxoP0M+Z9cPQmadGMN9KOTErXri8/9Hvs044doEZHP aiSV1tAfW8h+zKMIJbMW1MfEPBzeNyq1DWKeOGxVBkLdCSbXQ4qDuBuSR3oVYCQEuCDmzc rmYt1KC+2ud1lD7pxGgPqdxDE58bJ3e1MU93K51Lotty3S8Qvq14VTF3az8xmwJm7wMPnh qNzW8fTCjQlUuR6o+hmIUv+Tz1ezF+lQiFlt691jmuGRsygkvZ7NULN7hwdepw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id BC946862C; Fri, 25 Aug 2023 20:58:26 +0000 (UTC) (envelope-from gjb@freebsd.org) Date: Fri, 25 Aug 2023 20:58:24 +0000 From: Glen Barber To: freebsd-snapshots@freebsd.org, freebsd-stable@freebsd.org, freebsd-current@freebsd.org Cc: FreeBSD Release Engineering Team Subject: New FreeBSD snapshots available: stable/14 (ALPHA3) (20230825 2af9390e54ed) Message-ID: <20230825205824.GL4090@FreeBSD.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; x-action=pgp-signed -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 New FreeBSD development branch installation ISOs and virtual machine disk images have been uploaded to the FreeBSD Project mirrors. As with any development branch, the installation snapshots are not intended for use on production systems. We do, however, encourage testing on non-production systems as much as possible. Please also consider installing the sysutils/panicmail port, which can help in providing FreeBSD developers the necessary information regarding system crashes. Checksums for the installation ISOs and the VM disk images follow at the end of this email. === Installation ISOs === Installation images are available for: o 14.0-ALPHA3 amd64 GENERIC o 14.0-ALPHA3 i386 GENERIC o 14.0-ALPHA3 powerpc GENERIC o 14.0-ALPHA3 powerpc64 GENERIC64 o 14.0-ALPHA3 powerpc64le GENERIC64LE o 14.0-ALPHA3 powerpcspe MPC85XXSPE o 14.0-ALPHA3 armv7 GENERICSD o 14.0-ALPHA3 aarch64 GENERIC o 14.0-ALPHA3 aarch64 RPI o 14.0-ALPHA3 aarch64 PINE64 o 14.0-ALPHA3 aarch64 PINE64-LTS o 14.0-ALPHA3 aarch64 PINEBOOK o 14.0-ALPHA3 aarch64 ROCK64 o 14.0-ALPHA3 aarch64 ROCKPRO64 o 14.0-ALPHA3 riscv64 GENERIC o 14.0-ALPHA3 riscv64 GENERICSD Note, armv6 RPI-B images have been removed from the list of builds as it is intended to be removed during the 14.0-RELEASE cycle. Note regarding arm/armv{6,7} images: For convenience for those without console access to the system, a freebsd user with a password of freebsd is available by default for ssh(1) access. Additionally, the root user password is set to root, which it is strongly recommended to change the password for both users after gaining access to the system. Snapshots may be downloaded from the corresponding architecture directory from: https://download.freebsd.org/snapshots/ISO-IMAGES/ Please be patient if your local mirror has not yet caught up with the changes. Problems, bug reports, or regression reports should be reported through the Bugzilla PR system or the appropriate mailing list such as -current@ or -stable@ . === Virtual Machine Disk Images === VM disk images are available for the following architectures: o 14.0-ALPHA3 amd64 o 14.0-ALPHA3 i386 o 14.0-ALPHA3 aarch64 o 14.0-ALPHA3 riscv64 o 14.0-ALPHA3 amd64 BASIC-CI Disk images may be downloaded from the following URL (or any of the FreeBSD Project mirrors): https://download.freebsd.org/snapshots/VM-IMAGES/ Images are available in the following disk image formats: ~ RAW ~ QCOW2 (qemu) ~ VMDK (qemu, VirtualBox, VMWare) ~ VHD (qemu, xen) The partition layout for UFS is: ~ 512k - freebsd-boot GPT partition type (bootfs GPT label) ~ 1GB - freebsd-swap GPT partition type (swapfs GPT label) ~ 24GB - freebsd-ufs GPT partition type (rootfs GPT label) Note regarding arm64/aarch64 virtual machine images: a modified QEMU EFI loader file is needed for qemu-system-aarch64 to be able to boot the virtual machine images. This is provided by the emulators/qemu port. To boot the VM image, run: % qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt -bios /usr/local/share/qemu/edk2-aarch64-code.fd -serial telnet::4444,server -nographic -drive if=none,file=VMDISK,id=hd0 -device virtio-blk-device,drive=hd0 -device virtio-net-device,netdev=net0 -netdev user,id=net0 Be sure to replace "VMDISK" with the path to the virtual machine image. BASIC-CI images can be found at: https://download.freebsd.org/snapshots/VM-IMAGES/ === Amazon EC2 AMI Images === FreeBSD amd64 and aarch64 EC2 AMIs are available for both UFS and ZFS. These AMI IDs can be retrieved from the Systems Manager Parameter Store in each region using the keys: /aws/service/freebsd/amd64/base/ufs/14.0/ALPHA3 /aws/service/freebsd/amd64/base/zfs/14.0/ALPHA3 /aws/service/freebsd/arm64/base/ufs/14.0/ALPHA3 /aws/service/freebsd/arm64/base/zfs/14.0/ALPHA3 === Vagrant Images === FreeBSD/amd64 images are available on the Hashicorp Atlas site for the VMWare Desktop and VirtualBox providers, and can be installed by running: % vagrant init freebsd/FreeBSD-14.0-ALPHA3 % vagrant up == ISO CHECKSUMS == o 14.0-ALPHA3 amd64 GENERIC: SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-bootonly.iso) = 7c7dd389aba9de05bc5a44c3268541fc336aa2bd4ede294a98d913ff1a79776a1bf974a828c81fcc7e44351685b89c5d6154563baec6e8c11859e546b3ee8a79 SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 6a48dbddd901da8312a891a9f26309a443f9100dcbb88d9c78c20305917ba5da750de2d381e53118a704f88e83a29aec3d5365a87a37696ef01b98c6c08ab79b SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-disc1.iso) = f6903f80d8d6bf4498bc9df32d7d605e16061743fb7cd3935ed9aba337c18ac048c751443718bb395b8d20c178f31778f1057104ee225f0a4f7cbfa36563ddaa SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-disc1.iso.xz) = bb7b872330f89d441ff3f1263f9f882c423b46bdf2f9ac5180a79ae116f7531b6b4985f5d040d376842389c540bb25f7c16cd9b6aa5b9c50c23bb80c462d648f SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-memstick.img) = d6f4e4e76478cc2eff61ecd0e3756e1d04db6645e8ec276f5ebaf8d9e5c1cd4811af8a773e882f1451a6f214441d860ec2d06693647955d83fd28a30abb0cc6f SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-memstick.img.xz) = 124284ed029fffaa95c69f89f2bc9e3dfca9cddfcccabe3f338b64b0d49e628f11641624b3773b44756bea33a028a98aa9365ac346fd7096734cbd6b12561bf0 SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-mini-memstick.img) = 3a07928784ff7b32b6bc550a5e963163294e69a9b4ef11d5476e9aa631f609e8f45112723338af7094c6ffe8eb3125fa32009dc02f5061ca5f3cbd7128817b07 SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = 41145f8794f7adeff3dda6587871e82f20245a002e8c201af108adff07bdf022981b597ec2ded6d88c5487efcdbecf89a30d696bf8c6f5bcd6a678a3d9f076d8 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-bootonly.iso) = 9bce20bfc6ed3804ca1dd80ad9f3dea35fda9a5f0fbdb051240c8cfa876c6679 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = ffd00be8e92656b12f81e531b0871fdd508a079037453d7ac1c30418ad840048 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-disc1.iso) = 20a6db92240bfb1d25fce548ce8dde2447164da671b302c8ca7c02b177d547b7 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-disc1.iso.xz) = 6d23bc5dec10051bfa532ef4089087523737ebc958bb3a2d6b8dfddc7b1c20a6 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-memstick.img) = e6b7540e816794158847a1672e21cf0d9d6a63c224a180547f64667d5cb48cda SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-memstick.img.xz) = 85f1a069a286e4cc5237e164a1fa32565b4c32c70525a1c1538b511d5798a06f SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-mini-memstick.img) = d936ed300b9b8153fb320f3ecc8b531dcac6de048464c26871b8343ee46ccf56 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = a0c0afb8eb1e167f5a9f4b1191d07cd50448197c96c7b193c0de4197ceb1d61f o 14.0-ALPHA3 i386 GENERIC: SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-bootonly.iso) = 3b32d5eb746079b1385463dae96f4dd259f44ab698a6f3fab8141fc50be23536fb62f8ef3f13b05dfb972eefc37a0fd5e5876b4bb75b5f6ea29837ebcf56ed0c SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-bootonly.iso.xz) = dc14e82071af058108966ea7fb16af2f8ef395b6a0ec63f1db5f6c041ef1d20de0e4638d30e8b9ce85428dbbbefb67a29189e67495836f478ea673757a486c6c SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-disc1.iso) = 97c339559962326e9c56f1d5703b5a7b2da2be2ea574a65bd865061b51036805d8918a2714715ddb99e392bc78234bd264b44cb031e444d756d6ec78f6354b6a SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-disc1.iso.xz) = 13187f770aa15633980fc12ce4f10b818f29eca442ba1720f05fabd5987f0e93c9072e13169524af7bfb973ff284715b9a2933e5828cd5003f0657527be9ef0a SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-memstick.img) = 24c677b48a82a9098c35e0275d01156b5b0e7e51803f2a45f72b6a2992ee08398178472f011e92e2a5dbc49cef78e36e62c08024d58f4307bc4ebf2f1303c86d SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-memstick.img.xz) = ffc92f4a3f29e577e868a5ee166ef2e97f9f88ab52116b33d2c4d2d05d7ce895ef1b8856b1cecf46b566eec720a650943c3549039972755b5c27a603f8fd05b6 SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-mini-memstick.img) = 3677a9b6902b34d5bd113438559bb0830a24df7037b13bcc41761e5be799fdff70dec2c7291596c5d0d560ff7880fb52f3adb4c453a443ed7bcb22c1450a9898 SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = 3c40728b9aa761e93cb3b4271467e2d96590bfea6677c0ef0454eef21ccffef296a51f5c59ccde1e60f6442fccd675edb8cc7fb0be88d2453ac7ca87da8b4ad6 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-bootonly.iso) = a1ed4a570c3815d7fffa1f70f777fb13b964e7cc37e8c08575a502bec1fee357 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 5f72153a75718d4d85ac4af09d008fb172c242562bec7ebf51ccac102a1636f3 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-disc1.iso) = 39e966918d9f3d8a2d592f925ccf54950ff6e12ad14ed5420b9371ba996674ae SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-disc1.iso.xz) = 9466035cca190433699d3479900c201ec1aa31795a5881874074e0d54a47313e SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-memstick.img) = 2adbfd56b4d14db7d701c117ac780fd7b76dc69a12aeb594e882d113f9050231 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-memstick.img.xz) = 494176bc961d597662cae59e7d5880ba2d1a8a0b8957d74a198c98e6dc782b32 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-mini-memstick.img) = 6fdec86da214754fb69a68eef83fc4d6a1df85ba79263edc1b7f41df019680e0 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = ed315b1c042b16f0c145927c472318cffb960030839d305e67a1594a796d1d35 o 14.0-ALPHA3 powerpc GENERIC: SHA512 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-bootonly.iso) = 6f4a58b2bde393cb9c0c56f2cddeed407585f9e220e0a3b34d62bf0625dd3f1c7697518017b015838930ab5b702a249a8c33615da7f2ad3bb1432efbaacfd88a SHA512 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 4cedae95a2070731d1899b91ed3c72cead963e95e635a55e8fb18ed76127a15033cfc103ddb9c1f74746b17f39db88dfb33fb747b524fd76301b536dd9db1ad4 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-disc1.iso) = dcb6a2a82f702f43c82a22aa21ad406192239fde12859b7d93da8bb039651e60baf0bf1300a3ef29e30cdccb09a56f82cd90987c522864ebf20c282265dfd55a SHA512 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-disc1.iso.xz) = def99c0d77d652ae4fee2931b323e2d5d222a095f6a7f63e1bed589824f86d7fe5b14e71e11010faef5c3b4cb735f590eb9d2c7be8677951aa4777c81cf8f391 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-bootonly.iso) = 4a2e1939a0c63b3de970c19624591715e901eeb7e5c3352679362da9cad8a856 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-bootonly.iso.xz) = ba3619ec004c51b5223e3f1c2adf42198a1d2d6679f9203ff944f8684725e2a7 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-disc1.iso) = b7096ad28eb91991ae925fa516928a32168847b5580a697aa7cb63e765c30b84 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-20230825-2af9390e54ed-265022-disc1.iso.xz) = 6400085be2a9a936c79f72daf511b36d7d516413bc9b0f3018cb0607bffb7494 o 14.0-ALPHA3 powerpc64 GENERIC64: SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-bootonly.iso) = b83b26617a685648b59d45d31f835aa5c5298d268bef28a4e4b5c93d2cc0598d1b625fca7c18d755e3c3c483896ba640cfb96bf7cd68bf7bc469fefbb6c491b6 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 55e75a911ff89a99373e50aecb417e0332244a08eb24b57da0244ef83b9210d620cc9cbccaa59e18cbfde921a626893a20699e080ff527f6f63698a8c461adec SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-disc1.iso) = 125894cfa5a0e4f18bd27c7ab1292a0bd90cd1017dd15830ebfc904d28d6695dd3799b3c7115501b8e95dc2e1c4eace5ff0fa3fcbb443218d069b2925e4986a6 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-disc1.iso.xz) = 7551d58284c30c86cca44aec69d748957e2ec5b924d115ce76ba1de4a1e34ced1e81ff0918a21e590dc2020e4a50f07d8f618a8ce2a5f66616fe605c0e006bde SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-bootonly.iso) = 2a1a30bbf20220d68e52a0e0b95f8a2ab29500de88dad4ad4a84f6c4d5897210 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 8e93e3e7ed1858f97663572802c0ae1f02ad5df202c2b345ef2a45c70601cf8b SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-disc1.iso) = 18e72bf650d4fda8e847715deaf2b0eb67ff499e21d1da5873aa5bb1d49b6642 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64-20230825-2af9390e54ed-265022-disc1.iso.xz) = f7a0947089e895e22704a5de0c5c18ce0c8a2d01fbdca6c562ca5af055629813 o 14.0-ALPHA3 powerpc64le GENERIC64LE: SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-bootonly.iso) = 8f13bed203c54523592f953961a4bd679ecdc9cde92bcc933d8e1bb0cde513dc998207c16159413eb6d675266e7662725fff92e7ce3d2beebe0410f367faea9d SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-bootonly.iso.xz) = be9636f44eb5df11bdb7265569c3fc1a6872a79ab933fa1272c58bcecc318e42b9514bbfa1a714318d94e4ed14c811e9e0ff809e8f57bbba9adaf68e6edaa618 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-disc1.iso) = 8ec594bcfbfaa586c8af57ac97e8ea0b5d5afeb521ba015fb056c5c2d93d21472eeedc5592bd7e4349b4dbbd1a81ab4a56849bd52191fdf9ae2c413737d02041 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-disc1.iso.xz) = 3244d03b9ebfefd3792201dc5d0075cba5e084ea1e241b4ca282eb0ab6247ca598b791b2b6b4ccf622cfd121be737e67d33f12fa5e83328f89142f60f85ba2b8 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-bootonly.iso) = f954f7be39fe61e22a9ebcf420c8ce1148d975f1691aa6404b9ac4a6cd2bd639 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 08d458259baf3593e862cfa8f8595984b7ba8e0dd9398517d36d292be14a7b56 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-disc1.iso) = 2699db05c2495071acb2a45a0b424c47ff75d057b65be34d0d4ca2c2f8ccf37b SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpc64le-20230825-2af9390e54ed-265022-disc1.iso.xz) = fb5d705305d70ee18102f42ad609d40551acfbe44bac99a2a2a577a3e9738d32 o 14.0-ALPHA3 powerpcspe MPC85XXSPE: SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-bootonly.iso) = 730a23a0963e03bc9a591b095f50fc54f764969b99a009ec5e433fc8965c415c4ce16780e2233254e0be969044c8c82a44d6b8a14c99f5780479f351bb770539 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-bootonly.iso.xz) = cfffc69722d53166a16698a25f5127e2ab463828430ae9bb1edeaf5c7e84314e38e26634a88e5f79eae9e6fa94e6199ae161288d46b9ede83905e7c7a17ff633 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-disc1.iso) = f4b6be0058f2e5838eea788f7c0a6b2d60f7b223f18626496329fa21337270bd1192ad13355d85dc1f9706c17d75af4f8d6a13d736877a57025c324196f14f05 SHA512 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-disc1.iso.xz) = 92e840d241d5666dfd611161d66bb25d787323a88003aaf25e074ec7979fe9c210e6346bbcd54994cfc4e57ac8f7de558b70ea398e2c4d4a9b27f6cffa6b8625 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-bootonly.iso) = caafe240be306b9f953c0252e1eb98e13946e3a1dcdfd2585447819de49a8a63 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-bootonly.iso.xz) = e8badf9286451eb30a95957e6ee38e8237d3f47d41a7179c1b6034393ecbf1d3 SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-disc1.iso) = 3f363a0c63f42d9d4a17c13b49706e3bd06357bff7326aabf6710f6cf74054bb SHA256 (FreeBSD-14.0-ALPHA3-powerpc-powerpcspe-20230825-2af9390e54ed-265022-disc1.iso.xz) = abe1f3657ea386c3b4bed728f687ea773b1b191311ec5a55176fc53185c290c6 o 14.0-ALPHA3 armv7 GENERICSD: SHA512 (FreeBSD-14.0-ALPHA3-arm-armv7-GENERICSD-20230825-2af9390e54ed-265022.img.xz) = c8ab58204d43d61cdf3fa06f45f53d75e44cc81d2676cd421fe514eb7d0ad1d23de1ae5da5e041b14447a80c90630a890cfb8b159f68c82e6fd63b4d07072dc4 SHA256 (FreeBSD-14.0-ALPHA3-arm-armv7-GENERICSD-20230825-2af9390e54ed-265022.img.xz) = 404a4386573c9cec2e768b5dfd882307ef393f77866b77b64f8df1fb16d6b5eb o 14.0-ALPHA3 aarch64 GENERIC: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-bootonly.iso) = 3651ed7e68f50ba5be3fdf7cc04c722dbffc2ab1369e21574319e44d2a02f60223a74bd149ff543d6296ad89c082b6171d4d6f8c9bdae1d046df7c8a7feb13d5 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 9b0273323b78c04f5b3e68664e206533abefdb95024325858c5605cb1b75500486898022e3c06c6e026726f8f8996974236dcfc1f59e3336ec1ee512d8e67632 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-disc1.iso) = 71acd3386653998f87ecede3f114ab3cedef1c683c19585c03ad18a0d6c1f5bc1f8dc3441e5506fd8696f9ffe6b53da0063e159c5a58f79c352d29f81be477a4 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-disc1.iso.xz) = 66cc7a1592de00edfe710e29a90201fb8227a51befe2c8f7d829557787229d726b52689178c7d913c70425b7d4632be556fac049a8cc47f0122dc80c9f04ab0e SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-memstick.img) = d42075d886aed685a47c437988e38a1f2694bb12e9282dda77577d6274850fffb8421b3d1e47d70053a8eb652a9963f7154f25783720201a119af181907a0243 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-memstick.img.xz) = a9615f5643b5f03f223f3f6748926fd0b019ededd329e9233bcacf7025c32830cd5166321c98d65bae3a65ce51c65effaf9aea75af1e0a8681cf9e0e96764864 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-mini-memstick.img) = 38c973872cf4c49a6470c04d4116580783cd2131b558a91470f3f8a8557babac46a73831f8cb5749d51e41037e04e44c2e6c35ab7da7d1804a9df16396f4986e SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = f6a22b8974e2c3c9f1713533c3a932471f190e0f1bdc852f0ef483ebb21a3f7fb3b7970c1213581f8754abc4a819b55c37133e604955f34ba001f124c446b046 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-bootonly.iso) = 76f1da4bc9f8b7586296bac0ffd105032194dc5dc99acc482bf3f4fb62086228 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 8e377536fa256dbc883a301e7b3626f127c9a14c713c7e09cdb80aa303c3a03f SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-disc1.iso) = 8f295527fe69e12d0f23b0373bfd0ed059957a9c623aa25b4eb634fb3ae95ce2 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-disc1.iso.xz) = cb3c1cce18b9f75830606027d185ee65be5db8e42b46a2143301ed376768b985 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-memstick.img) = 2ac7540d409c0f6611c202bbfb24c14ad3b33001c566fbd5c89fd5f99da94e80 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-memstick.img.xz) = e7f5ced2a302f00b6deae994ad7eecd4614b5fc4270f174dfc2278bf3b0e10c7 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-mini-memstick.img) = ce623a7d3783b21e740b1b48ae599669fb3449eb2a042419eb96a4f1709a7779 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = 13743559dd6f16912d40b4f4a6d24b755bc9b359fb49786c5e35f8c950f63fc3 o 14.0-ALPHA3 aarch64 RPI: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-RPI-20230825-2af9390e54ed-265022.img.xz) = 4f260597783313966eea31df2cd29244265312a7d9713f8a824353c039769e2e62a91d60aa4eb943a1b5d757af068622f7b592750137ee738c31a8bdbdbf1eb3 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-RPI-20230825-2af9390e54ed-265022.img.xz) = 625bf0064c23807049bbfba0273f9ab70a433be2c3b08ef2a948a651a0a02866 o 14.0-ALPHA3 aarch64 PINE64: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-PINE64-20230825-2af9390e54ed-265022.img.xz) = 8263ba2c9121f7f5835797a9b261fe736c55a3edff42cf02bfcfd273d79815892b7f7ea9705ab2ee16e84f36e58ce8c13bac0adc48ae587785295f26b1ddfa7b SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-PINE64-20230825-2af9390e54ed-265022.img.xz) = 10f4a83c78115c4909c2cab66e7fed532588c2dd2853e24abaed0b3829da72f1 o 14.0-ALPHA3 aarch64 PINE64-LTS: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-PINE64-LTS-20230825-2af9390e54ed-265022.img.xz) = acc83ee86f19df948cdb1126c19a3a0e15728bb21ef76e19beee78e96f46fa50643cb28832f933e2f1d7e5f8a621817b61385e47d479a98dd60f6c3eac100e7f SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-PINE64-LTS-20230825-2af9390e54ed-265022.img.xz) = 03c2e3ccd17189c70d6fcda9fc7593474dbfcda9dca2c98d02bc31667234bf7b o 14.0-ALPHA3 aarch64 PINEBOOK: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-PINEBOOK-20230825-2af9390e54ed-265022.img.xz) = 721b94bad066a4a8d34ac4a121a3c5e295593ccdadf4cb455cfbcbc4069c958670ac012f7040b1660e1010363edddf9676cbf7bbb4ac2422bb7192bfdf1b7d17 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-PINEBOOK-20230825-2af9390e54ed-265022.img.xz) = 63e46b0fb066f1091bdb59959148111353eda46cf900a19a896080369c6a2f7b o 14.0-ALPHA3 aarch64 ROCK64: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ROCK64-20230825-2af9390e54ed-265022.img.xz) = 93612dfe9f970d8f6966085eb0d5a2f5321fd02f34183b0f0c3dac97c5eff3681a80985a334c963ae9af860fe28ab21921b04265d66b14d7a698c3b811f6e1a2 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ROCK64-20230825-2af9390e54ed-265022.img.xz) = 5712216d81ce2a19fd3146897919ac1e29a3092ae26a451627f91b6f05e42496 o 14.0-ALPHA3 aarch64 ROCKPRO64: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ROCKPRO64-20230825-2af9390e54ed-265022.img.xz) = 95f036833bc983eaa56a1eb2bcd23eb8b30c95bd0ee0431b5450762b9b001cf725f669231b4633b6d0c5b34bdce932d9bd54bba1c60475a8bd2262d81289b310 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ROCKPRO64-20230825-2af9390e54ed-265022.img.xz) = 3fb0e78fa2edb39d669ef09880f6b31b43585e33a823cca13a220e0d0623e22e o 14.0-ALPHA3 riscv64 GENERIC: SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-bootonly.iso) = 73fb7740e3a8ccb6a3fc4441425d0a96bcd147c1b55ea633c5c4ff46c5e630c8ca8211f59bab11f9522623afe793452e4763ee2228db2d0e45886b2b466c21d2 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = fa55daa7f26c6f5333ed749408a158af91a3de4f17979f125fa3fdb317e647b0687e51e807e6e60efd08da0f0c0f5a6128e30d5b15c466c56e1a8fdcf5d4bbfe SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-disc1.iso) = 6a01cb1a067ef745a273f90e0dcee1e41d85ab67962e70b996853edfdc26bef2cf7eb51cb76d46515581b4552bc10749b7b0c201b6c223f087cfd91ed944ca51 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-disc1.iso.xz) = 2cf8664276d5b4b7bf091d0f6720a7c080cbcb85c07dd9ac6c48bc7290b6c3e95f096c4192d063136059804eb1721c981f7711dbbe171666d51f42ba43f7dff3 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-memstick.img) = 7fcc1fe852890ba1cf8d3a23c31a9e81e89bb71af30ba37c5d5670ed65cf431e9048b598363dbf8f72388878a3bb572323c686f9f1f825f63e188010f71dcd66 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-memstick.img.xz) = 13ba404bd4591c30bc19bacc25cc0d7d3f3e63be7757ac70d6dfb4c0ad421368ad2235d2337c10bb1377c65fdbcf83ea73109e938c5253703bea284440423b31 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-mini-memstick.img) = 649921c83596f10e43a91e2617aea073ebea493cd1719e368587bfc095a5e6f98802c08d26350aed6733897e3b8ad1175a3af2f49c1f4bef4b3eacb5a5c5c8dc SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = d9de0ea3fbfe663020cc40b9ef75322abec07400f7df0c7c5af39a835d48fb0c089098ada19a8ba8acbafc2ba04341e1de10fa86eaca1beb7e3882e094d66aff SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-bootonly.iso) = 8ca420ec4b4478c11d2dfef19579d8e4322c6f5020b4f48fa9ddd2412d4753c8 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-bootonly.iso.xz) = 65d0450634b352e7285314df949cdf97696bcf605bb18bc4e94f871fcc89ad62 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-disc1.iso) = ade128c1fb51f0b84408939df5fa1e976c502d7b511a7301feea138be7abb3c6 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-disc1.iso.xz) = f7b3873b1f85d13dfc71382ac05666e6aee03d94fc4eea3ce4a094cc9ca0e543 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-memstick.img) = a5d7e4bd81600bec95d16a6d5ce64b896175eaddeffebba25a0bb3c9adcdebb9 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-memstick.img.xz) = e7377067c8a3188deadde4504a48157f55a1063e279806112769b6aa4fd8046a SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-mini-memstick.img) = 3a42ad6bf82626498895b8cc01cd87cadde24e8143f0191f9b980dd632306c35 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022-mini-memstick.img.xz) = ab810174e273c627541cdd0bc41bf17d78935bedab865e5ec9c221baa0f9b0f7 o 14.0-ALPHA3 riscv64 GENERICSD: SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-GENERICSD-20230825-2af9390e54ed-265022.img.xz) = 85642fa97b4dbd44eeba2af9e95ee836a7b92dc93e4664356bccd3b4e7394ca39e8eda31739bd42a9e47f195cc9af659dd589d1f1e13e2e31d4da7069f30bbf4 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-GENERICSD-20230825-2af9390e54ed-265022.img.xz) = f454ee2b106347021d8afec64e927f2bba6a0bdccdf09ffd5ef33c08bf3ab83d == VM IMAGE CHECKSUMS == o 14.0-ALPHA3 amd64: SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.qcow2.xz) = 9c6c96603de2d89bd9eb297a538fe8a111f6ab1356a6d0c7da31cfba8682494d436b6254c351ba07629eb718e5a9feb4a15e5f34102290fbf9d82a12be0df386 SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.raw.xz) = 5bef4fbb68ccc4777fe2e1fd197c0315c0ab072ee66d3ac8b4af8629761bdbec1c1128aab9676378a124c609f98652d6b4fe51364e0106c044242767b46df1a0 SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.vhd.xz) = c06367ff56e536040aedf414f829de61627ac493b3e81541591bd264687e43e9f0ad2b96f0da8b130c2d5b9768850c1e8fb73a808584b13ec107eda194983467 SHA512 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.vmdk.xz) = 3f2036fa0466abf76c885158015000e2f91e01eb2d13b133791434ea4b8856634375ea20e07d92d89ca89d27284010a55e25f72fcea823e45918709839754861 SHA512 (FreeBSD-14.0-ALPHA3-amd64-ufs.qcow2.xz) = 9c6c96603de2d89bd9eb297a538fe8a111f6ab1356a6d0c7da31cfba8682494d436b6254c351ba07629eb718e5a9feb4a15e5f34102290fbf9d82a12be0df386 SHA512 (FreeBSD-14.0-ALPHA3-amd64-ufs.raw.xz) = 5bef4fbb68ccc4777fe2e1fd197c0315c0ab072ee66d3ac8b4af8629761bdbec1c1128aab9676378a124c609f98652d6b4fe51364e0106c044242767b46df1a0 SHA512 (FreeBSD-14.0-ALPHA3-amd64-ufs.vhd.xz) = c06367ff56e536040aedf414f829de61627ac493b3e81541591bd264687e43e9f0ad2b96f0da8b130c2d5b9768850c1e8fb73a808584b13ec107eda194983467 SHA512 (FreeBSD-14.0-ALPHA3-amd64-ufs.vmdk.xz) = 3f2036fa0466abf76c885158015000e2f91e01eb2d13b133791434ea4b8856634375ea20e07d92d89ca89d27284010a55e25f72fcea823e45918709839754861 SHA512 (FreeBSD-14.0-ALPHA3-amd64-zfs.qcow2.xz) = f69a512ef7c987f54c08704e3f90c869cb406ee039f8e1fc2f93b3014a51e250453ed0d8fa979b642511fd9a3cc0d652f28ce9c057ee9fce58ead86544c35bd2 SHA512 (FreeBSD-14.0-ALPHA3-amd64-zfs.raw.xz) = 6da9d6bebba08059a0b628b7627dfb65b63e03804b76f67b789c13e1904724347c81f097ba038c9796a29e1c38a9951992bb8c7c3d16517330bbb2ad033e482e SHA512 (FreeBSD-14.0-ALPHA3-amd64-zfs.vhd.xz) = 942b46101677e88906d2d844d12df0ff9055ffc13a7385d96da605cbd1beeb64d0c97c055b15abe243b68a9905466b5b4ce8e0c5907ac0183582dc1d6eea9bb0 SHA512 (FreeBSD-14.0-ALPHA3-amd64-zfs.vmdk.xz) = d0e7d3e09ae73f204468a479a66b84da5003e577cc2d69f92d622d41f6357558da5432feafbae8001d3ecd27a99ebadda40fad0e9dd950d03756b59fe463515e SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.qcow2.xz) = 04eb05142969619c7183fa3dcf3b10e19972a095c55ec5cbbc914562d21f7a9d SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.raw.xz) = 046e9bdfae2754937ae07af41d459d3fcd40f08761d570613b29d76ce5443162 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.vhd.xz) = f38cdfe7222696d17169ff10eb467168322c178774001abb63d24bdec0165287 SHA256 (FreeBSD-14.0-ALPHA3-amd64-20230825-2af9390e54ed-265022.vmdk.xz) = 486c50b104b4a33a09856ae0c4aff82178b7e1a27b6c2dc71e6c4ac7c511d2b4 SHA256 (FreeBSD-14.0-ALPHA3-amd64-ufs.qcow2.xz) = 04eb05142969619c7183fa3dcf3b10e19972a095c55ec5cbbc914562d21f7a9d SHA256 (FreeBSD-14.0-ALPHA3-amd64-ufs.raw.xz) = 046e9bdfae2754937ae07af41d459d3fcd40f08761d570613b29d76ce5443162 SHA256 (FreeBSD-14.0-ALPHA3-amd64-ufs.vhd.xz) = f38cdfe7222696d17169ff10eb467168322c178774001abb63d24bdec0165287 SHA256 (FreeBSD-14.0-ALPHA3-amd64-ufs.vmdk.xz) = 486c50b104b4a33a09856ae0c4aff82178b7e1a27b6c2dc71e6c4ac7c511d2b4 SHA256 (FreeBSD-14.0-ALPHA3-amd64-zfs.qcow2.xz) = 8b6047f397fccdeab654c439057de746c8a50baed5e7a6b5aef79478ca4b56eb SHA256 (FreeBSD-14.0-ALPHA3-amd64-zfs.raw.xz) = 6a50f601910c5ce6b43f6ac56394937d2394d4f37a6f3a0219db433c6ca63ba7 SHA256 (FreeBSD-14.0-ALPHA3-amd64-zfs.vhd.xz) = be7bfeb96463530ad292140fddb1a34f962051c9cbd779901071f5ded81c9d95 SHA256 (FreeBSD-14.0-ALPHA3-amd64-zfs.vmdk.xz) = 1021b82cf62fbbda70da6724353fa9868d053a35dddd96feace16a4be801ca09 o 14.0-ALPHA3 i386: SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.qcow2.xz) = 796c5d02268d149ec221365af2e03e092bef52daec01f1953d6a0b6ddc98ed1aa21acb78cb5fbe8e8054e4bb6e6d10dc0783c514313720013ea4ab58c1645e70 SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.raw.xz) = 3d30474e10bf3f5e1e0dd0012af4074faf938c002bdc1b8d1f9c661ed1e9816fbf8064bcf5b285e47a83e48c20875a40b96681d58b7a6efa1b9ae273708f113d SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.vhd.xz) = 5062df28ba9b3848fd287243f4f2e3120c684a2ae7b3e85b8ed6f9217b84f12edd38ee90a64f13e12383ebb3f4703fe73c1fb8050ced5dcaf51ca32f36088a38 SHA512 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.vmdk.xz) = 608ff19389838e0a8132fcba939f982e11ae027a86475e4a5123631593957adcb1ea363e2436d8b06a5bb7205b999f12b43485e335e672660ad4269b1194c1e4 SHA512 (FreeBSD-14.0-ALPHA3-i386-ufs.qcow2.xz) = 796c5d02268d149ec221365af2e03e092bef52daec01f1953d6a0b6ddc98ed1aa21acb78cb5fbe8e8054e4bb6e6d10dc0783c514313720013ea4ab58c1645e70 SHA512 (FreeBSD-14.0-ALPHA3-i386-ufs.raw.xz) = 3d30474e10bf3f5e1e0dd0012af4074faf938c002bdc1b8d1f9c661ed1e9816fbf8064bcf5b285e47a83e48c20875a40b96681d58b7a6efa1b9ae273708f113d SHA512 (FreeBSD-14.0-ALPHA3-i386-ufs.vhd.xz) = 5062df28ba9b3848fd287243f4f2e3120c684a2ae7b3e85b8ed6f9217b84f12edd38ee90a64f13e12383ebb3f4703fe73c1fb8050ced5dcaf51ca32f36088a38 SHA512 (FreeBSD-14.0-ALPHA3-i386-ufs.vmdk.xz) = 608ff19389838e0a8132fcba939f982e11ae027a86475e4a5123631593957adcb1ea363e2436d8b06a5bb7205b999f12b43485e335e672660ad4269b1194c1e4 SHA512 (FreeBSD-14.0-ALPHA3-i386-zfs.qcow2.xz) = e4dd5383e1c7c8c079013ad5abc7953a902423fc4a3f7ed5eb74b8e1b21f5eeb51acbc6cf7def400c8d295fdf4ed0fd4574164a719168a4af6c538e65338c4e6 SHA512 (FreeBSD-14.0-ALPHA3-i386-zfs.raw.xz) = acf03fbccfdb95eb7ed7e42192546acf0f5f23e20785e3d47e395a69617f45fd7a5462775a31478c571ed62789a5daaf1ae7b486819ba15fbe87ab6e6c5f2dcc SHA512 (FreeBSD-14.0-ALPHA3-i386-zfs.vhd.xz) = 6e17d0463851320a743760157786220482a8ca971a54e7bab5839bfc90726b0088c2094cd5fd1f4341723048e12f5c01fa9911f7b65fe32ea3ad36ba57ec8515 SHA512 (FreeBSD-14.0-ALPHA3-i386-zfs.vmdk.xz) = e36d53dbb136f7b28fe53c3bd77c1a4db50333886044e4c5c660d8f1d0aec6958679007cd288c5429b0e31d2c63bcd6650effa670fcc62afd696a966f7badf9d SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.qcow2.xz) = b9f05ef211d068dc814e109df8a19f25c37be522fc5324b8adef3067bbf46217 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.raw.xz) = b82fad6a90a631fa214948c8d4633b16088f20a36e104fea02c3e2b4a2546c18 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.vhd.xz) = 10d5a9828dd06fcdc2a14141c24ba5cf43e94da37659bea089438839f07ff2c7 SHA256 (FreeBSD-14.0-ALPHA3-i386-20230825-2af9390e54ed-265022.vmdk.xz) = 691f32699e66d1a912866024d737a26202dfb66beae41674ff9509411e9991a4 SHA256 (FreeBSD-14.0-ALPHA3-i386-ufs.qcow2.xz) = b9f05ef211d068dc814e109df8a19f25c37be522fc5324b8adef3067bbf46217 SHA256 (FreeBSD-14.0-ALPHA3-i386-ufs.raw.xz) = b82fad6a90a631fa214948c8d4633b16088f20a36e104fea02c3e2b4a2546c18 SHA256 (FreeBSD-14.0-ALPHA3-i386-ufs.vhd.xz) = 10d5a9828dd06fcdc2a14141c24ba5cf43e94da37659bea089438839f07ff2c7 SHA256 (FreeBSD-14.0-ALPHA3-i386-ufs.vmdk.xz) = 691f32699e66d1a912866024d737a26202dfb66beae41674ff9509411e9991a4 SHA256 (FreeBSD-14.0-ALPHA3-i386-zfs.qcow2.xz) = ef5cc3f29cbaaa638fe6766233ca70cc2efc79c4ee0d32aad94125cde52e03ac SHA256 (FreeBSD-14.0-ALPHA3-i386-zfs.raw.xz) = d5eb8c70d4ccfa568a5d3880e32e3ec9146d9cfcecc3a98b24ecf11d721ea491 SHA256 (FreeBSD-14.0-ALPHA3-i386-zfs.vhd.xz) = 9f91153ccee727e233af384d01d231725a9303788766df932a3cb3a7bf977c9d SHA256 (FreeBSD-14.0-ALPHA3-i386-zfs.vmdk.xz) = 0c9606d6f85aa9568b1cb0cd0cf045b6b67157de994f7d6a1314514a86ebbaa5 o 14.0-ALPHA3 aarch64: SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.qcow2.xz) = 025038f3fcf083e094052cb762d83e6546c068798d56ffcf3ae15c3856d0c2345d5b5d4d8106d8673e7e0fc14cba6274242e6e11ae8825d42a0013d6cb93de0e SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.raw.xz) = a2b79e04871dbc1ca2dd8ef6a4238f08fb61ac372793850d5d15882aeae4a36088ec867ee8c58b5c7cd74feed1959f078a321a4fefe8cea38cf7b112e9f38bad SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.vhd.xz) = 215cbf1da739985fd327656151c97099a1a8d8946fd1c408cff27af350c6d1b61fbe3d0613ed8635b5069a42e37d4100ab8087914d4157c5885804a67283497a SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.vmdk.xz) = 58ba492c37b74488e656fdf0174d2a0bf07ca5f6174e5a948b5c7d5825f62234bc1ad22cd2922182f1c95701a5ffcf838872ccebad39b1bec1554d6d23eccb7c SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.qcow2.xz) = 025038f3fcf083e094052cb762d83e6546c068798d56ffcf3ae15c3856d0c2345d5b5d4d8106d8673e7e0fc14cba6274242e6e11ae8825d42a0013d6cb93de0e SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.raw.xz) = a2b79e04871dbc1ca2dd8ef6a4238f08fb61ac372793850d5d15882aeae4a36088ec867ee8c58b5c7cd74feed1959f078a321a4fefe8cea38cf7b112e9f38bad SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.vhd.xz) = 215cbf1da739985fd327656151c97099a1a8d8946fd1c408cff27af350c6d1b61fbe3d0613ed8635b5069a42e37d4100ab8087914d4157c5885804a67283497a SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.vmdk.xz) = 58ba492c37b74488e656fdf0174d2a0bf07ca5f6174e5a948b5c7d5825f62234bc1ad22cd2922182f1c95701a5ffcf838872ccebad39b1bec1554d6d23eccb7c SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.qcow2.xz) = f41f16dbd7243fe7df083ed1ac509dabd4e6a6c5c56b2cc667442c422f881702c634aef4472310674de280f09d93c58450194a149952871e51460c6de6a4d7d4 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.raw.xz) = 57798ce6f6177ad611e8289bb083e36c8d1f320cb79b02cdaaf05c34e2b2a9fbb182cfd9dc881bd9c132c3996268f85a61aa0184093c8de81024624c32a98365 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.vhd.xz) = 3f1da0f9d1a6a0ab6e9002082336296689a031be5bcce73b7db7a051d8311d1e8db7eaa3ca8dc58fcd58d3e2ab8af51f51586ac8f509b29d866e482b56747d57 SHA512 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.vmdk.xz) = c9549d4976c9276ab40a23bb7352034d59a84f8d0735828a6a9db55b3948a1ef28a70b3c690509b509536e7389e9146109ed91f0e43b9e8e314683d6499e535f SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.qcow2.xz) = 8c4e8a98c13ff681422a4777a4680d90d0bd68198a22d0882845f69d1abb9905 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.raw.xz) = d6c156a0de0f75fe180c58ff0d6d17d3f33f935dfe68242ae001091d9e61e3b5 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.vhd.xz) = e3f9f9bb13c39ebd12c76d3075005c99d462a1732e3261cf8204ad5f4f9a0805 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-20230825-2af9390e54ed-265022.vmdk.xz) = a13c5527addc29e5cb336231944bb8bae65c4ee72bd889184e93ce383380ad27 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.qcow2.xz) = 8c4e8a98c13ff681422a4777a4680d90d0bd68198a22d0882845f69d1abb9905 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.raw.xz) = d6c156a0de0f75fe180c58ff0d6d17d3f33f935dfe68242ae001091d9e61e3b5 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.vhd.xz) = e3f9f9bb13c39ebd12c76d3075005c99d462a1732e3261cf8204ad5f4f9a0805 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-ufs.vmdk.xz) = a13c5527addc29e5cb336231944bb8bae65c4ee72bd889184e93ce383380ad27 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.qcow2.xz) = dd4e1c0916fb1de5ae7b3642a5d5e9e4418b4292ea628ecaef2c357cfb061b8a SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.raw.xz) = 9692ac3aeaad0494cadec886d76ff501abad78d73e668319f1e5a80ea6224730 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.vhd.xz) = daec08824b431ee2e861d1207ec35ff43cc95571fc969ecb2066cc6605a9f5c0 SHA256 (FreeBSD-14.0-ALPHA3-arm64-aarch64-zfs.vmdk.xz) = b003ba0c9f70db1b56b80063e62968b716a832796a69f710979e3a2b5683d197 o 14.0-ALPHA3 riscv64: SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.qcow2.xz) = 500511b656aabaab7e0221878c24a7de53f51abe61ab82d7fdfee87b501e51be5ccaf828175696e372a000fade9a6a12360c1d46366f1a6ae610172f075b3513 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.raw.xz) = a4809e4b43019e6c6d58b2b886a5db1228f0a94e880f0317ef97ed84d0863bf9f09f1f0d63b3bcc219845d101e426c05165686b93e2b4daca5be949c49b6e97c SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.vhd.xz) = ac846673a55fe31432c70f1c62696c61ea1ffe42737e48ca725de98ff79dbb5fc0dbbb0465e62809881ebe8728d6c7b9fa10b35991ce9c1699f6e022fd6ceb0d SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.vmdk.xz) = be5357b7970e6992d08ff734e55b058659df600cdf9dba8083789c8a96e9dbc3496c4f2f649b86875b5065bba793a8349236d590a017366e1b6d7e750774d142 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.qcow2.xz) = 500511b656aabaab7e0221878c24a7de53f51abe61ab82d7fdfee87b501e51be5ccaf828175696e372a000fade9a6a12360c1d46366f1a6ae610172f075b3513 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.raw.xz) = a4809e4b43019e6c6d58b2b886a5db1228f0a94e880f0317ef97ed84d0863bf9f09f1f0d63b3bcc219845d101e426c05165686b93e2b4daca5be949c49b6e97c SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.vhd.xz) = ac846673a55fe31432c70f1c62696c61ea1ffe42737e48ca725de98ff79dbb5fc0dbbb0465e62809881ebe8728d6c7b9fa10b35991ce9c1699f6e022fd6ceb0d SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.vmdk.xz) = be5357b7970e6992d08ff734e55b058659df600cdf9dba8083789c8a96e9dbc3496c4f2f649b86875b5065bba793a8349236d590a017366e1b6d7e750774d142 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.qcow2.xz) = 993a2d189a3a22a6972b50eb9a257c02bd5d92475e92c062e310b17700c73a962978b940bed33b552247b5cdfb49befd3e38bdfd4abc4cdfdf7137acda297178 SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.raw.xz) = 5ea420612e8d0909be6693d0d4981c35bb230b1f4c393f9de3040e9cfa300b0623db5a99cc090bc7ba27b7b93eea5f2f615b2fc82fbd856b2c18ed532d99911d SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.vhd.xz) = af43d11afd74c78150b63c7424fac746a2c31ec56d809333b8e66a7643738d5377398b6fadbd3f71b623b7bf6d4bffc409de42d686549c6a452d182157ed917d SHA512 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.vmdk.xz) = 3a19a23045cac5bad3e6bcf436989ff101f4c6718801c0b7f2018f8e0330a977a3d3bbb285f5551dbbfb39df06b5517c6e2166387f0ef7a8e5a4dc0e0ffe6a21 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.qcow2.xz) = c1df3962fe42ddc128420553cf3daa1aaa262c8ff70d5e4130e573d2e218a9e9 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.raw.xz) = 4691d87500f92f2867c414539a9659c9ad63a3c843a31930ada449cad75ee0ea SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.vhd.xz) = a65d09d9e4738017a0acf91dd4cd312d4a8b01ba9cc24f803688fa9d47d620c4 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-20230825-2af9390e54ed-265022.vmdk.xz) = 1711ad2cf3b9bf0d7b30bdea031a8f22259323f482f3b05e8996906fbae4250a SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.qcow2.xz) = c1df3962fe42ddc128420553cf3daa1aaa262c8ff70d5e4130e573d2e218a9e9 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.raw.xz) = 4691d87500f92f2867c414539a9659c9ad63a3c843a31930ada449cad75ee0ea SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.vhd.xz) = a65d09d9e4738017a0acf91dd4cd312d4a8b01ba9cc24f803688fa9d47d620c4 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-ufs.vmdk.xz) = 1711ad2cf3b9bf0d7b30bdea031a8f22259323f482f3b05e8996906fbae4250a SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.qcow2.xz) = 0b37dfd65fe49a53a96fe828838a885a11717810bcf3a1c31f1c04e823a8c0a6 SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.raw.xz) = 6f706833b22bcc9cf2033eafc23b4b7aab74d10ea38f39e5a7afcf5664f0d23a SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.vhd.xz) = 67b9b30fec48977b6c1d4d23ef6d01b3f99bfb52037328a171ddacaee547110d SHA256 (FreeBSD-14.0-ALPHA3-riscv-riscv64-zfs.vmdk.xz) = fb1817fb74056087d8c5e0b023040c15cc9e9b3c1a7b44eb450f799f7e79d2fa o 14.0-ALPHA3 amd64 BASIC-CI: SHA512 (FreeBSD-14.0-ALPHA3-amd64-BASIC-CI-20230825-2af9390e54ed-265022.raw.xz) = 441de4f13a792a72651c6e61f63b590e8ad103672198419147c432d60cfaad13f9993374b43a3fe53bf582f8632a5ea9a74c932382c2d5adacca96d904aadcc2 SHA256 (FreeBSD-14.0-ALPHA3-amd64-BASIC-CI-20230825-2af9390e54ed-265022.raw.xz) = 4ea2dd37d500b11234d730fdc595331354f32a15cb5337e59902d9b2e4020a78 Regards, Glen Please consider donating to help support my FreeBSD work: https://www.gofundme.com/f/gjbbsd https://paypal.me/gjbbsd Love FreeBSD? Support this and future releases with a donation to the FreeBSD Foundation! https://www.freebsdfoundation.org/donate/ -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmTpFfAACgkQAxRYpUeP 4pNdYhAAi0jqIh2J6urLpXRRxafTcLrMa9vaWUkDbEr7aTD7c5DuuzsPGEhfLC2H avPDP2oKT67mJp44Tp0A3akljs6DRwmYL8rz0cjeVbV2dET72m8WNJ1swEsoS8rw PZHq1M0O+nEPbE72bGKSL01T4f1OoWhtjrBLMylW7psgynCtXj5pJoE/MwP9orDu 9HtRcu3dwy6WB3JbJBbWQTZUrxHSgoRH6XCDJBcZjcBtwxmHC6j9fvESlkG02/xz FIpy6OWMYGh+sP83Azfqo/4+nMbxKIKn5DHU0I2Ri4+/j34X9c2p/5LVapO/Gj2Q 20sf1KHf7dazI9aazB8Y2W9bKUn1kgRyhfllpWWxa/oJq+an4r4+IYoum8R3/vMT yzAnF35G1epmbbm0Ue9q63AaiSPHH8KeOE8MK+191HWt8qjDHOQ13d2hdRW9bu4q e5mt+u+lgturgN2MD1pTeF3SsF7XRtJdvRBH3op23ueAKksccUq9LpYatHehnFnI SoGnLLwnKxUZzQRuCXmSg6EYgZPo57YSVsoNeuzSG+tRIVq4n1K+it2iJKNTUQp5 Fs5pxx5HyYHro3hhx/d5h51eL5caOPPurHIMNx8oZMcKnG0osDqczNpuAUSvk0uK f6jwZfTxzrIGEFc4m8RdPKkqzuZENrZ/gGYGXVNf2lFtt/IfX9A= =EP+q -----END PGP SIGNATURE-----