From nobody Tue Dec 2 15:02:39 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dLPCQ6mZ5z6KYj1 for ; Tue, 02 Dec 2025 15:02:58 +0000 (UTC) (envelope-from wschnr@googlemail.com) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dLPCP64scz45Qm for ; Tue, 02 Dec 2025 15:02:52 +0000 (UTC) (envelope-from wschnr@googlemail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of wschnr@googlemail.com designates 209.85.221.42 as permitted sender) smtp.mailfrom=wschnr@googlemail.com Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-42e2e2eccd2so1934986f8f.1 for ; Tue, 02 Dec 2025 07:02:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764687771; x=1765292571; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=X8IuXSpfGR2RSmE1CcTDKX551BkuPzPYeER/TLh53k8=; b=ohVTyf/UfbQhWh3Gum6J/kQceJrJsWIgvbJCsVF2VQpFRrqnB8ooC5/c6ZJ5Ms7j0B WB3YNy2jFZd1QrOclZPa/ioDf1aeXl66rqRSmENhVHh+bjO8VrwEnk5Npg7+fwcSmJ+F bXyCuuJdWD5EHkN+R2eKWF3k0U5jLy+bSMLjsQmALRmAbF4O4cZzaKaVOIJsWX+9lj6S QLDMi45WCl796mUy2DzJ5quE4KNLE9RPl1TONShZT0898l5iykr/tqoA3wr36zti5YVX VhrwaiKNv+nNrbdLfs4L9TXUAP1HgxSibAJgBi8XcEIjE0kmTQhG6HbxBjkF+T70m0zu 8UBQ== X-Forwarded-Encrypted: i=1; AJvYcCXr2EE0Y97sVzLDL7jKGfOfH/muCmHxaG/kHjThT5bq/qB21u8oBzQzfCIyMCML/NqGoaIS8swD11zaQbwIWhMmbsxz@freebsd.org X-Gm-Message-State: AOJu0YyEWsaIKvgEStUsX4S45+R3KIQE0fwnej6DRT7Vl70r/Aa8TGMh kt0FR0TTJkOJN54mTE3+Gr8Il9bctQtgzquV3w+DN0R0SpnafFhNfMIozNquJLsyf3Q+NvpY2HL aHGF8P7QTOr3WIczzfZj1HGO0s0n/IR4= X-Gm-Gg: ASbGncv6AZQvIQVk9aCvOvDFvxSpBf1VSV7bzQJZwvZiACbOHAeWjNZmkyFtvK1X21U tSyse5jJ8qlj36n5nTPLo0nUAwJKw0itpBQV7Tdmp3WOzrpTQIuxOZtgFErZ975H1grClYFuGpo vo7ZC5BiYHt7H2kDNO7LeAD9zFYkvi+SOQS3mNahVbDQiRmtInP82QvORFGLQ2JvC92arzCtRDf 16wbVvyxChM5SvpMAwxrJOwIu+XKTeXIkAEgDeq6HslMnqCRl/ZgNqXq9wD7Sj3jGH+gmS/BSUO AhlonlNy/l3F6PBua/6OL4R0Fcs8yq2u1EtJOG6g9yRL3xJw/w== X-Google-Smtp-Source: AGHT+IGS3IbH/aKalU7p1Nwk1E+wbPWfkzehqLfjfhzmrCzx381Zj9uBbS0y10KBHZrErGawuKOisssW6Ro/wE1x9Xo= X-Received: by 2002:a05:6000:240b:b0:42b:43b4:2870 with SMTP id ffacd0b85a97d-42cc1d0897cmr48051659f8f.26.1764687770968; Tue, 02 Dec 2025 07:02:50 -0800 (PST) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 References: <202510140252.59E2qpmW046357@gitrepo.freebsd.org> In-Reply-To: <202510140252.59E2qpmW046357@gitrepo.freebsd.org> From: Wolfram Schneider Date: Tue, 2 Dec 2025 16:02:39 +0100 X-Gm-Features: AWmQ_bnKKPKmXqsE-QuDGXMh8wvJXwR-sBOFqt5zPc7TpJFD7N0xoiDZRDGvQC4 Message-ID: Subject: Re: git: 16155cccc4bf - main - packages: Compress in parallel To: Colin Percival Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: multipart/mixed; boundary="000000000000ba7dde0644f96368" X-Spamd-Bar: / X-Spamd-Result: default: False [-0.40 / 15.00]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.956]; NEURAL_SPAM_MEDIUM(0.36)[0.361]; FORGED_SENDER(0.30)[wosch@freebsd.org,wschnr@googlemail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_BASE64_TEXT(0.10)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; HAS_ATTACHMENT(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; PREVIOUSLY_DELIVERED(0.00)[dev-commits-src-all@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[wosch@freebsd.org,wschnr@googlemail.com]; RCVD_IN_DNSWL_NONE(0.00)[209.85.221.42:from]; MLMMJ_DEST(0.00)[dev-commits-src-all@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[209.85.221.42:server fail]; R_DKIM_NA(0.00)[]; MISSING_XM_UA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.221.42:from]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4dLPCP64scz45Qm --000000000000ba7dde0644f96368 Content-Type: text/plain; charset="UTF-8" On Tue, 14 Oct 2025 at 04:53, Colin Percival wrote: > > The branch main has been updated by cperciva: > > URL: https://cgit.FreeBSD.org/src/commit/?id=16155cccc4bf42a5a76806128e7ef637f9ae61fe > > commit 16155cccc4bf42a5a76806128e7ef637f9ae61fe > Author: Colin Percival > AuthorDate: 2025-10-12 21:44:14 +0000 > Commit: Colin Percival > CommitDate: 2025-10-14 02:50:52 +0000 > > packages: Compress in parallel > > The FreeBSD-base pkg repository is currrently dominated by a handful of > large packages: Out of a compressed repository size of ~960 MB, about > 160 MB is taken up by the FreeBSD-src package, 128 MB (on amd64) is > used by the FreeBSD-kernel-generic-dbg package, and 91 MB is used by > the FreeBSD-src-sys package. Consequently, running 'make packages -jN' > provides less benefit than one might hope, as most of the packages > finish building quickly, ultimately leaving the FreeBSD-src package > building by itself for a couple minutes while all the other CPUs are > idle. > > Pass -T${PKG_CTHREADS} to the 'pkg create' commands (with a default > of -T0) in order to instruct pkg's zstd compression to use multiple > threads. > > Testing on an EC2 r7i.48xlarge instance with -j192, this reduces the > time taken by 'make packages' from 6m17s to 1m39s; package creation > time itself (excluding the initial process of installing into world > and kernel staging directories) dropped from 5m37s to 59s. Great work! `make packages' is now twice as fast on my old macbook pro m1 (10 CPUs, 8P+2E, cross compiling), from 8.5min down to 4.5min. 100 seconds for installation, and 180 seconds for pkg. Most of the time is spent with the usual suspects, the toolchain and debug libs (WITHOUT_TOOLCHAIN and WITHOUT_DEBUG_FILES), see the CSV attachment. -Wolfram > > Reviewed by: ivy > MFC after: 3 days > Sponsored by: https://www.patreon.com/cperciva > Differential Revision: https://reviews.freebsd.org/D53053 > --- > Makefile.inc1 | 15 ++++++++------- > 1 file changed, 8 insertions(+), 7 deletions(-) > > diff --git a/Makefile.inc1 b/Makefile.inc1 > index e079a23552f1..a86dead09aa1 100644 > --- a/Makefile.inc1 > +++ b/Makefile.inc1 > @@ -1964,6 +1964,7 @@ REPODIR?= ${OBJROOT}repo > PKG_FORMAT?= tzst > PKG_LEVEL?= -1 > PKG_CLEVEL?= ${"${PKG_FORMAT:Mtar}" != "":?:-l ${PKG_LEVEL}} > +PKG_CTHREADS?= 0 > PKG_REPO_SIGNING_KEY?= # empty > PKG_OUTPUT_DIR?= ${PKG_VERSION} > PKG_ABI_FILE?= ${WSTAGEDIR}/usr/bin/uname > @@ -2144,7 +2145,7 @@ create-source-src-package: _pkgbootstrap .PHONY > ${SSTAGEDIR}/src.ucl > ${PKG_CMD} -o ABI=${PKG_ABI} \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${SSTAGEDIR}/src.ucl \ > -p ${SSTAGEDIR}/src.plist \ > -r ${SRCDIR} \ > @@ -2170,7 +2171,7 @@ create-source-src-sys-package: _pkgbootstrap .PHONY > ${SSTAGEDIR}/src-sys.ucl > ${PKG_CMD} -o ABI=${PKG_ABI} \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${SSTAGEDIR}/src-sys.ucl \ > -p ${SSTAGEDIR}/src-sys.plist \ > -r ${SRCDIR} \ > @@ -2210,7 +2211,7 @@ create-world-package-${pkgname}: .PHONY > ' ${WSTAGEDIR}/${pkgname}.ucl > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${WSTAGEDIR}/${pkgname}.ucl \ > -p ${WSTAGEDIR}/${pkgname}.plist \ > -r ${WSTAGEDIR} \ > @@ -2229,7 +2230,7 @@ create-sets-packages: .PHONY > @for manifest in ${WSTAGEDIR}/set-*.ucl; do \ > echo "--> Processing manifest: $$manifest"; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M $$manifest \ > -o "${REPODIR}/${PKG_ABI}/${PKG_OUTPUT_DIR}" \ > || exit 1; \ > @@ -2259,7 +2260,7 @@ create-dtb-package: .PHONY > ${KSTAGEDIR}/${DISTDIR}/dtb.ucl ; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${KSTAGEDIR}/${DISTDIR}/dtb.ucl \ > -p ${KSTAGEDIR}/${DISTDIR}/dtb.plist \ > -r ${KSTAGEDIR}/${DISTDIR} \ > @@ -2296,7 +2297,7 @@ create-kernel-packages-flavor${flavor:C,^""$,${_default_flavor},}: _pkgbootstrap > ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.ucl ; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.ucl \ > -p ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.plist \ > -r ${KSTAGEDIR}/${DISTDIR} \ > @@ -2339,7 +2340,7 @@ create-kernel-packages-extra-flavor${flavor:C,^""$,${_default_flavor},}-${_kerne > ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.ucl ; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.ucl \ > -p ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.plist \ > -r ${KSTAGEDIR}/kernel.${_kernel} \ -- Wolfram Schneider https://wolfram.schneider.org --000000000000ba7dde0644f96368 Content-Type: text/csv; charset="US-ASCII"; name="pkg-time.csv" Content-Disposition: attachment; filename="pkg-time.csv" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_miopc5dg0 dGltZTogcmVhbCAxMTQuNDIgdXNlciAyMTkuNDYgc3lzIDEuMTQgc3RhcnQ6IDE3NjQ1ODY5ODUu Mzc5MTgxMDAwIGVuZDogMTc2NDU4NzA5OS44MTk5NzAwMDAgZXhpdDogMCBtZXNzYWdlOiBwa2cg Y2xhbmctZGJnCnRpbWU6IHJlYWwgNzguOTQgdXNlciAzOC4xNyBzeXMgMS4yNSBzdGFydDogMTc2 NDU4NzA5MS44ODE1ODkwMDAgZW5kOiAxNzY0NTg3MTcwLjg0ODkwMjAwMCBleGl0OiAwIG1lc3Nh Z2U6IHBrZyByZXNjdWUKdGltZTogcmVhbCA2OS4wMiB1c2VyIDg3LjU1IHN5cyAwLjQ2IHN0YXJ0 OiAxNzY0NTg2OTg1LjM5MDA2NjAwMCBlbmQ6IDE3NjQ1ODcwNTQuNDMwODkzMDAwIGV4aXQ6IDAg bWVzc2FnZTogcGtnIGNsYW5nCnRpbWU6IHJlYWwgNTMuNDAgdXNlciAzMC4zMiBzeXMgMS4wNCBz dGFydDogMTc2NDU4Njk5Ni4wNjcwNDYwMDAgZW5kOiAxNzY0NTg3MDQ5LjQ5Njk5NTAwMCBleGl0 OiAwIG1lc3NhZ2U6IHBrZyBjbGlicy1kZXYKdGltZTogcmVhbCAzOS42NSB1c2VyIDE2LjQwIHN5 cyAwLjI0IHN0YXJ0OiAxNzY0NTg3MDYxLjU2OTUyMjAwMCBlbmQ6IDE3NjQ1ODcxMDEuMjUzMTYy MDAwIGV4aXQ6IDAgbWVzc2FnZTogcGtnIGxvY2FsZXMKdGltZTogcmVhbCAzNS45MCB1c2VyIDE3 LjAwIHN5cyAxLjMxIHN0YXJ0OiAxNzY0NTg3MDcxLjM3NjMxMjAwMCBlbmQ6IDE3NjQ1ODcxMDcu MzA0Njg2MDAwIGV4aXQ6IDAgbWVzc2FnZTogcGtnIG9wZW5zc2wtZGV2CnRpbWU6IHJlYWwgMzMu MDEgdXNlciAyNC42MiBzeXMgMC4xOSBzdGFydDogMTc2NDU4Njk4NS4zNzY4NTIwMDAgZW5kOiAx NzY0NTg3MDE4LjQxNTQyMTAwMCBleGl0OiAwIG1lc3NhZ2U6IHBrZyBjbGFuZy1kZXYKdGltZTog cmVhbCAzMS4yNSB1c2VyIDIwLjExIHN5cyAwLjExIHN0YXJ0OiAxNzY0NTg3MDU4LjA2MjE5MjAw MCBlbmQ6IDE3NjQ1ODcwODkuMzQxMTk3MDAwIGV4aXQ6IDAgbWVzc2FnZTogcGtnIGxsZGItZGJn CnRpbWU6IHJlYWwgMzEuMTUgdXNlciA0MC41OSBzeXMgMS40OSBzdGFydDogMTc2NDU4NzEwMy4z Mjg0ODkwMDAgZW5kOiAxNzY0NTg3MTM0LjUxNjk4NTAwMCBleGl0OiAwIG1lc3NhZ2U6IHBrZyB0 ZXN0cwp0aW1lOiByZWFsIDI4LjA2IHVzZXIgMTEuMzggc3lzIDAuMDkgc3RhcnQ6IDE3NjQ1ODY5 ODUuODEwMjM4MDAwIGVuZDogMTc2NDU4NzAxMy44OTUwNTYwMDAgZXhpdDogMCBtZXNzYWdlOiBw a2cgY2xpYnMtZGV2LWxpYjMyCg== --000000000000ba7dde0644f96368--