From nobody Sat Dec 14 00:42:13 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y96qV6SfGz5gNW8 for ; Sat, 14 Dec 2024 00:42:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-20.consmr.mail.gq1.yahoo.com (sonic301-20.consmr.mail.gq1.yahoo.com [98.137.64.146]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y96qT6vk3z43fS for ; Sat, 14 Dec 2024 00:42:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=L62ywvGe; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1734136947; bh=aQfB+R4hK6sg21ca9Qq/oraUFAVHkd8QNaGky3toy/s=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=L62ywvGeSYELZSpR6QY8Wotis23QeQpk19Xt+DlXP4coEOPPm8Dxv+yf8prpQTmC19q3U0WBdy6lJvSP0Rx/36tO5DOW32lAe1lSfZvtlPAzm4xIx49Er1zkD/xx7lwm3Mtds3VaBlNNmBBZKmEeiCO7uKeFT7Q38SUV/uYMzxTomIGPsd0yCyZd9OhrSICVO5L3uOMg2Rb4LRVoI1hlqLYruNMZJDxtGICZY6EXxB1Uvp5Oz7gAJlCQUHQnedNoI+ynMDrsNzxIsUD94d2aWP1lwCY7GxEKKxviyPs4ORgP6nahiAdYLlZeUnQyAmnoB9yAWoISjMaDqdmBPXrGOg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1734136947; bh=dh3IVh3kIZa7D2FOUvAua4Y8MMSDRrXZnXiIuR252sg=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=p6HY1J/d9m673YPb/MaubxGnoTXppbNFCs4ADG2rD75A7fhfAmxn/JJ9w8cDOPFgSOtBEzI5w1CwmGF9XNYXptYtAIMybjST/6kDuwZYY/y+GmWtLg/8fsw+hZnJeBnG9ZTf8uqsXE3I9MJg9Vm5Pfd28Da4pluAN2cCVwZbA4PtPUdBDcoRAiw36WkxBJGQXs5lkWzrnzNuP+h37VYMK0m+PN1VPqIUMfiswrWLXnqvvY90ty0F7u58WVaZ3ZJ6dGmymTpQMKLngqf2bPVNz/xY4GLwmp71xKw1KwuxqDaz2gUDSS/8Q8l270UJLcVw6wIrWprjFT61WFUACXiNzA== X-YMail-OSG: j3utuvcVM1lZaTvi42n1JQ4I2XquhCYw92iVr1LI8zJmYq5tjFrjjlDqkmLN0YR ebtcYu8YWF73tkciVeVi719WKOOm60agSQa2jEN_nED4fkPYXTlosELyHNi7ypPBgy2iFQ73llr2 FgD08lciSf0ENNL5z1bMuA_zMlTrOBwl1f8LNOoCDzysv3N3rWKoZkF0Wrp_DDf4pGxWrr1N26y1 OznP5Ksg.8b2yIaG0IYWPcedkShSUvGdhJ2VT5IRkuc1co9wjiLNZKx4vSOsHQXaO5w7MObGXhYy UiM4MU7J.js1uFGwDAoPoQFSWUQf4O2jrneOKzd3U6WL.PP.xD4Aizk4b.Mv33kwcxPbdgcXtmK3 ptDPVzUoZN2pID02cndl.L_NpmEJT7mceIatEN.kCBAJci5q_O36qnTDmUBmUTJraQKhapTHWxYt 0IbpKs2X5wamtDMLtZ_fIBnBv6b2n9s3v4_nYdNxWB_PYSNq7uUrpUgRH5dPUyKNwwf11_vlDU.o Ya3fLBAXvvzm5x3vs5oooUG56YfMUs5DLPyGVJzJCbdCAhwIPFZWZRdBEu8Q.5E1vDzsLchDgwO7 wLEDKbVwgBM58I25xMuGp11lGU7TjLBsB1sfJpm0Ivla8ff9JY4OjI3qfHvtqpw8Ap4w7AB9nYRx JT0TSLUJYvym8CEDJqagrzrqgTQCT1B6cmpsgJNyhBKOzKuu.mgR2EXS1vZJezdakntoA7jBf3tN XkUuylrorw7hUpGHMy0E64xJDiC0NctXMcfLQH.lmi2cru93TFWT3TO90KwlVLzS.M6.P7tRtTty r4WuzAJFNyyXtSfzGbWD3xeoxYUrCRT5KLyTl.xDu9JF_w5D9fZhjHPRubm5SCPywzfe_AFq0_nr _4q4iDxiqj9sU.yX3YaL.K6tTAdG241gcFQs_zY8Rh4eh8latcQIqRcLlPe342HZcCu6sg3bJo8x EKMvT8yxHrHh04Y67FQy2HhwFEiXwiU9a_iod6c8M5x8fq._S9P..PxGzoXxbvgC.SxUI5xDERM6 qJaZFaA029D6bMb2al4dBgBTcGsQEz2pHhANcCFD2Fxrv.XsZKI3IcZVJLcgeah_lKpLAQ1qH5m7 XNDokBNNzlWcl_F3DnkIM.tzHQXErly2PC7a9SBbjzA03.kgP7wdHgIR0nThVE24_HyoEn4aQfdY SawmynvuUBV3t0aPRytBUgxl3MEEq48NMJrRT4w3BozZMqnHCGXh7iWmBGoWLOUN0J6t8Mv55CQ_ fcHjlHaJJsJ9DpwSbNDQ0bEoLnBKZXmuyOy4IsGnhhhjVBF8dVZcDGd9SGxC9HonIlANDelsPCOT X198AP5NX.nnw.aeUvT26hwODYvKEm8VmmBrmNPG74dt7zUvkrGIOLn4NII6rl_D2EtF59o.Pf6j 0vY2NTGBE.SgT5KMVwcON5jefaQSThdrS.lZ3vdwk9qr2VY0VurWc6VvNBsxQOwmume4oaU7Km57 9RKG8OcXK_PmQmFbRtXwcDJxL8w_Xu4seudMGBMHn6IxrNGVCM6KxDZOw7p5vtPcr6oZik1fNJ85 4tu3wEC_6MAc4XkTymkEYVg.1jjnJc3Ef7IJY1Fa.V300zb90FjR71kRxqqtDHdr9HHKNNXAiR98 oQA4Bu.7WBGB4luW0f6in3gwV8v8UNRizdJMJwXZm7KYrGt0o.dcSQvncPapT_.iCsd9zJwrWQ.2 mdk6XSVJeCqc54GhoOzOmU5eZY0WEi0rFoY6dgRQp8Q_1Qt0YqIH_orjVCw20lRSelIiTcHVGQZR EnJqgnnrYCAg8qUCHA1UCS0h4Fv3DFFOEbBR1KIJAacOq0ii0aFmJDHlwIRtqF9pVp6JFkmkBSRK ezAsTFIKflr0rSRbE7dojTtM_Jvf0iYzd3NH7ZXrYsHC_YbrZCDj8er7T6_UwW4e4n.HrdK251IJ AiKrF2Z80Q_SbpZW3BrtUEm0235mjoIb6A1R9chZAKkvaFlYbLEun.Wr.wfvPEi.vBInNpqWjkK5 dsXH6cMlnTHSYFJgwBgbEf6eAlkGb05a9rDEeWQwJiLmXhDc3886d59QO.C37GU7sZgqEqENK1UF A73vLBSmLaEW0y56rvvK3_cEX1tHEGXvXUtsxNph6hQcaP4tlC6drfCFeJvvBTshj3hHu1gKDSgx O99tnGMF2Y0zdmC4osbbiuA3trlVRNj0U8NJEQ2qKfg5iyL0_ZtnH77AY362Kj.rLoTNoVqYuHj2 R_vyD.X2_RfpL9Kepnam0FcZ51UjrC03K2srH3p62Y5ySvX5G0GjxVTOVagcdhxd7.0R6J1iDaG5 sEKY- X-Sonic-MF: X-Sonic-ID: 2fd65a6f-3011-43fe-abe5-486d794d8f2a Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sat, 14 Dec 2024 00:42:27 +0000 Received: by hermes--production-gq1-5dd4b47f46-k4d2j (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1237c3cdea7bd40e34820087282e386d; Sat, 14 Dec 2024 00:42:24 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.300.87.4.3\)) Subject: RE: Switching release media dist sets to .tzst (tar + zstd)? Message-Id: Date: Fri, 13 Dec 2024 16:42:13 -0800 To: Ed Maste , FreeBSD Current X-Mailer: Apple Mail (2.3826.300.87.4.3) References: X-Spamd-Result: default: False [-2.24 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-0.74)[-0.740]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.146:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.146:from] X-Rspamd-Queue-Id: 4Y96qT6vk3z43fS X-Spamd-Bar: -- Ed Maste wrote on Date: Fri, 13 Dec 2024 21:15:45 UTC " > I have been reviewing parts of the release artifact build process, > including ISO and memstick images, and came across the distribution > sets (e.g., base.txz, src.txz) used by the installer to populate new > file systems. I=E2=80=99d like to discuss switching these to .tzst = (tar + > zstd) compression. My notes are mostly not about the specific compression format --but are about the dist-sets potentially going away. I tend to use https://artifact.ci.freebsd.org/snapshot/*/*/*/*/*.txz for crude bisecting without needing to do builds. For example, grabbing something like: = https://artifact.ci.freebsd.org/snapshot/main/926905796749750da6464b97ec4f= 8eec0882cc0e/arm64/aarch64/kernel.txz = https://artifact.ci.freebsd.org/snapshot/main/926905796749750da6464b97ec4f= 8eec0882cc0e/arm64/aarch64/kernel-dbg.txz to see compare/contrast behavior with other builds. This also avoids questions about if my builds create any problems vs. official builds. > While I haven=E2=80=99t yet conducted detailed benchmarks comparing = zstd and > xz specifically for this use case, here are some initial > considerations: >=20 > Pros of zstd: > - Faster compression and decompression speeds. > - Aligns with the compression method used for FreeBSD packages. >=20 > Cons of zstd: > - Somewhat larger compressed file sizes. > - Requires updates to tools that interact with distribution sets. > - May have limited availability on some other operating systems (?). >=20 > I have a review open to demonstrate the extent of the change in the > build system & installer: https://reviews.freebsd.org/D48042 >=20 > It might be that this is not worth pursuing, as dist sets will most > likely go away with the migration to pkgbase, I tend to use https://artifact.ci.freebsd.org/snapshot/*/*/*/*/*.txz for crude bisecting without needing to do builds. Are you saying that such will no longer be a possibility? (This is not a which-compression-style question.) I've also used the likes of the below with kgdb to look at reported backtraces from problems that have been reported --for versions of FreeBSD that I do not have a boot environment for. (Not wanting to do a normal install on other media and to boot/configure it --just to look around.) = http://ftp3.freebsd.org/pub/FreeBSD/releases/amd64/14.2-RELEASE/kernel.txz= = http://ftp3.freebsd.org/pub/FreeBSD/releases/amd64/14.2-RELEASE/kernel-dbg= .txz http://ftp3.freebsd.org/pub/FreeBSD/releases/amd64/14.2-RELEASE/src.txz (But there is no equivalent for patch revisions of *.*-RELEASE 's .) = http://ftp3.freebsd.org/pub/FreeBSD/snapshots/arm64/13.4-STABLE/kernel.txz= = http://ftp3.freebsd.org/pub/FreeBSD/snapshots/arm64/13.4-STABLE/kernel-dbg= .txz http://ftp3.freebsd.org/pub/FreeBSD/snapshots/arm64/13.4-STABLE/src.txz Similar question for those: no longer to be possible? > but I would like to > discuss and make an explicit decision. We can separately consider > compression on the release media images themselves. >=20 > Feedback Requested: >=20 > Is there support for this idea? Are there objections to pursuing this? > Are there other factors I should consider, especially compatibility = concerns? =3D=3D=3D Mark Millard marklmi at yahoo.com