From nobody Tue Dec 2 15:02:39 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dLPCM06Wtz6KYt0 for ; Tue, 02 Dec 2025 15:02:55 +0000 (UTC) (envelope-from wschnr@googlemail.com) Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dLPCJ69Lsz4520 for ; Tue, 02 Dec 2025 15:02:52 +0000 (UTC) (envelope-from wschnr@googlemail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of wschnr@googlemail.com designates 209.85.221.50 as permitted sender) smtp.mailfrom=wschnr@googlemail.com Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-42e2e3c0dccso1546858f8f.2 for ; Tue, 02 Dec 2025 07:02:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764687771; x=1765292571; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=X8IuXSpfGR2RSmE1CcTDKX551BkuPzPYeER/TLh53k8=; b=P1lr8qw9zXXoFinU5MC7oQFGn45Jb+OWxzU4PTsMYi8Ked9LBe0jE0+Wr4yymcOtaV CyNW3X48I2nt2hqIFJbzv1Pit+GFmjKwOZkUxGaJU/X/cXktTAb8M6cDszhT6RdJdai0 xwNePSN5OljMRWyR0URt6i8QU4AjPyI4QqYMEAEiV9HOj9K/vXEr23BVcuOgzno7DLJ1 A1GcjZEnTf2HFet3HymTgYHxO3MuyJwvx+6Fsj5Y2Q5RFM5lf3HsO8hedTGfOofUVmvO 6tIv3yGDR+UmCIQZewvbGEehc/K9/T2LIDEQ/qkxTEFP6bj+TfDkBrrm5vhXIJWBhN8v u32Q== X-Forwarded-Encrypted: i=1; AJvYcCWmpadzRAClfkeaHwUmCkoQULxVdi6e6YFDZdpz6Al4q9e1GI1+FBxR4Bgi1zPdO0zfm6VBuenO605g3Wz1NifGnqfhVA==@freebsd.org X-Gm-Message-State: AOJu0YyMw2341faCK+eGQotrmmjI6muk22A/b9MDgMZEnJ8sP0DIegEK 7BMt4Mn4ByaWOSpCDe1RE/jatA8KMtjNUdL+ku2ZoN62vrXggXbW6sbveWkV5cVbrNkSlYGwIVZ nGqEnTdUchV046sufLunJvxS8GPtjFnw= X-Gm-Gg: ASbGncsmk0hzcuLtAIP0lDFNofCyPhQ82l8kF8Fw0/a+4B0VaWuZj7a6FtZA59Ow6Ri Ky1TTRBCAkWWJjp6rgZmACzNoS0TduCv+BfB0B+F8AHV88H/s9iKImgH8PiOygTqafe8h/fj6ws RyFkKew+caZR2IlGfPeTCWCqMt87TdW7UI4hDy7EwPNY5yB2HUGVcnq5DaYoNc1CHrg+fhIncY4 SJro5SxyQDz8XIO2LDmNV3hps6dWm4clCHTFbyTfedOTOCzPdKrmjk0dNvApR6misI47O/cEySb SlLLmtPYFQD0gPj/kVThqi/af4Tx0mB/tROmqWh+gPZcpGkBPA== X-Google-Smtp-Source: AGHT+IGS3IbH/aKalU7p1Nwk1E+wbPWfkzehqLfjfhzmrCzx381Zj9uBbS0y10KBHZrErGawuKOisssW6Ro/wE1x9Xo= X-Received: by 2002:a05:6000:240b:b0:42b:43b4:2870 with SMTP id ffacd0b85a97d-42cc1d0897cmr48051659f8f.26.1764687770968; Tue, 02 Dec 2025 07:02:50 -0800 (PST) List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 References: <202510140252.59E2qpmW046357@gitrepo.freebsd.org> In-Reply-To: <202510140252.59E2qpmW046357@gitrepo.freebsd.org> From: Wolfram Schneider Date: Tue, 2 Dec 2025 16:02:39 +0100 X-Gm-Features: AWmQ_bnKKPKmXqsE-QuDGXMh8wvJXwR-sBOFqt5zPc7TpJFD7N0xoiDZRDGvQC4 Message-ID: Subject: Re: git: 16155cccc4bf - main - packages: Compress in parallel To: Colin Percival Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: multipart/mixed; boundary="000000000000ba7dde0644f96368" X-Spamd-Bar: / X-Spamd-Result: default: False [-0.40 / 15.00]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.956]; NEURAL_SPAM_MEDIUM(0.36)[0.361]; FORGED_SENDER(0.30)[wosch@freebsd.org,wschnr@googlemail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_BASE64_TEXT(0.10)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; FREEMAIL_ENVFROM(0.00)[googlemail.com]; HAS_ATTACHMENT(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; PREVIOUSLY_DELIVERED(0.00)[dev-commits-src-main@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[wosch@freebsd.org,wschnr@googlemail.com]; RCVD_IN_DNSWL_NONE(0.00)[209.85.221.50:from]; MLMMJ_DEST(0.00)[dev-commits-src-main@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[209.85.221.50:server fail]; R_DKIM_NA(0.00)[]; MISSING_XM_UA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.221.50:from]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4dLPCJ69Lsz4520 --000000000000ba7dde0644f96368 Content-Type: text/plain; charset="UTF-8" On Tue, 14 Oct 2025 at 04:53, Colin Percival wrote: > > The branch main has been updated by cperciva: > > URL: https://cgit.FreeBSD.org/src/commit/?id=16155cccc4bf42a5a76806128e7ef637f9ae61fe > > commit 16155cccc4bf42a5a76806128e7ef637f9ae61fe > Author: Colin Percival > AuthorDate: 2025-10-12 21:44:14 +0000 > Commit: Colin Percival > CommitDate: 2025-10-14 02:50:52 +0000 > > packages: Compress in parallel > > The FreeBSD-base pkg repository is currrently dominated by a handful of > large packages: Out of a compressed repository size of ~960 MB, about > 160 MB is taken up by the FreeBSD-src package, 128 MB (on amd64) is > used by the FreeBSD-kernel-generic-dbg package, and 91 MB is used by > the FreeBSD-src-sys package. Consequently, running 'make packages -jN' > provides less benefit than one might hope, as most of the packages > finish building quickly, ultimately leaving the FreeBSD-src package > building by itself for a couple minutes while all the other CPUs are > idle. > > Pass -T${PKG_CTHREADS} to the 'pkg create' commands (with a default > of -T0) in order to instruct pkg's zstd compression to use multiple > threads. > > Testing on an EC2 r7i.48xlarge instance with -j192, this reduces the > time taken by 'make packages' from 6m17s to 1m39s; package creation > time itself (excluding the initial process of installing into world > and kernel staging directories) dropped from 5m37s to 59s. Great work! `make packages' is now twice as fast on my old macbook pro m1 (10 CPUs, 8P+2E, cross compiling), from 8.5min down to 4.5min. 100 seconds for installation, and 180 seconds for pkg. Most of the time is spent with the usual suspects, the toolchain and debug libs (WITHOUT_TOOLCHAIN and WITHOUT_DEBUG_FILES), see the CSV attachment. -Wolfram > > Reviewed by: ivy > MFC after: 3 days > Sponsored by: https://www.patreon.com/cperciva > Differential Revision: https://reviews.freebsd.org/D53053 > --- > Makefile.inc1 | 15 ++++++++------- > 1 file changed, 8 insertions(+), 7 deletions(-) > > diff --git a/Makefile.inc1 b/Makefile.inc1 > index e079a23552f1..a86dead09aa1 100644 > --- a/Makefile.inc1 > +++ b/Makefile.inc1 > @@ -1964,6 +1964,7 @@ REPODIR?= ${OBJROOT}repo > PKG_FORMAT?= tzst > PKG_LEVEL?= -1 > PKG_CLEVEL?= ${"${PKG_FORMAT:Mtar}" != "":?:-l ${PKG_LEVEL}} > +PKG_CTHREADS?= 0 > PKG_REPO_SIGNING_KEY?= # empty > PKG_OUTPUT_DIR?= ${PKG_VERSION} > PKG_ABI_FILE?= ${WSTAGEDIR}/usr/bin/uname > @@ -2144,7 +2145,7 @@ create-source-src-package: _pkgbootstrap .PHONY > ${SSTAGEDIR}/src.ucl > ${PKG_CMD} -o ABI=${PKG_ABI} \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${SSTAGEDIR}/src.ucl \ > -p ${SSTAGEDIR}/src.plist \ > -r ${SRCDIR} \ > @@ -2170,7 +2171,7 @@ create-source-src-sys-package: _pkgbootstrap .PHONY > ${SSTAGEDIR}/src-sys.ucl > ${PKG_CMD} -o ABI=${PKG_ABI} \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${SSTAGEDIR}/src-sys.ucl \ > -p ${SSTAGEDIR}/src-sys.plist \ > -r ${SRCDIR} \ > @@ -2210,7 +2211,7 @@ create-world-package-${pkgname}: .PHONY > ' ${WSTAGEDIR}/${pkgname}.ucl > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${WSTAGEDIR}/${pkgname}.ucl \ > -p ${WSTAGEDIR}/${pkgname}.plist \ > -r ${WSTAGEDIR} \ > @@ -2229,7 +2230,7 @@ create-sets-packages: .PHONY > @for manifest in ${WSTAGEDIR}/set-*.ucl; do \ > echo "--> Processing manifest: $$manifest"; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M $$manifest \ > -o "${REPODIR}/${PKG_ABI}/${PKG_OUTPUT_DIR}" \ > || exit 1; \ > @@ -2259,7 +2260,7 @@ create-dtb-package: .PHONY > ${KSTAGEDIR}/${DISTDIR}/dtb.ucl ; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${KSTAGEDIR}/${DISTDIR}/dtb.ucl \ > -p ${KSTAGEDIR}/${DISTDIR}/dtb.plist \ > -r ${KSTAGEDIR}/${DISTDIR} \ > @@ -2296,7 +2297,7 @@ create-kernel-packages-flavor${flavor:C,^""$,${_default_flavor},}: _pkgbootstrap > ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.ucl ; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.ucl \ > -p ${KSTAGEDIR}/${DISTDIR}/kernel.${INSTALLKERNEL}${flavor}.plist \ > -r ${KSTAGEDIR}/${DISTDIR} \ > @@ -2339,7 +2340,7 @@ create-kernel-packages-extra-flavor${flavor:C,^""$,${_default_flavor},}-${_kerne > ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.ucl ; \ > ${PKG_CMD} -o ABI=${PKG_ABI} -o ALLOW_BASE_SHLIBS=yes \ > -o OSVERSION="${SRCRELDATE}" \ > - create -f ${PKG_FORMAT} ${PKG_CLEVEL} \ > + create -f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS} \ > -M ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.ucl \ > -p ${KSTAGEDIR}/kernel.${_kernel}/kernel.${_kernel}${flavor}.plist \ > -r ${KSTAGEDIR}/kernel.${_kernel} \ -- Wolfram Schneider https://wolfram.schneider.org --000000000000ba7dde0644f96368 Content-Type: text/csv; charset="US-ASCII"; name="pkg-time.csv" Content-Disposition: attachment; filename="pkg-time.csv" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_miopc5dg0 dGltZTogcmVhbCAxMTQuNDIgdXNlciAyMTkuNDYgc3lzIDEuMTQgc3RhcnQ6IDE3NjQ1ODY5ODUu Mzc5MTgxMDAwIGVuZDogMTc2NDU4NzA5OS44MTk5NzAwMDAgZXhpdDogMCBtZXNzYWdlOiBwa2cg Y2xhbmctZGJnCnRpbWU6IHJlYWwgNzguOTQgdXNlciAzOC4xNyBzeXMgMS4yNSBzdGFydDogMTc2 NDU4NzA5MS44ODE1ODkwMDAgZW5kOiAxNzY0NTg3MTcwLjg0ODkwMjAwMCBleGl0OiAwIG1lc3Nh Z2U6IHBrZyByZXNjdWUKdGltZTogcmVhbCA2OS4wMiB1c2VyIDg3LjU1IHN5cyAwLjQ2IHN0YXJ0 OiAxNzY0NTg2OTg1LjM5MDA2NjAwMCBlbmQ6IDE3NjQ1ODcwNTQuNDMwODkzMDAwIGV4aXQ6IDAg bWVzc2FnZTogcGtnIGNsYW5nCnRpbWU6IHJlYWwgNTMuNDAgdXNlciAzMC4zMiBzeXMgMS4wNCBz dGFydDogMTc2NDU4Njk5Ni4wNjcwNDYwMDAgZW5kOiAxNzY0NTg3MDQ5LjQ5Njk5NTAwMCBleGl0 OiAwIG1lc3NhZ2U6IHBrZyBjbGlicy1kZXYKdGltZTogcmVhbCAzOS42NSB1c2VyIDE2LjQwIHN5 cyAwLjI0IHN0YXJ0OiAxNzY0NTg3MDYxLjU2OTUyMjAwMCBlbmQ6IDE3NjQ1ODcxMDEuMjUzMTYy MDAwIGV4aXQ6IDAgbWVzc2FnZTogcGtnIGxvY2FsZXMKdGltZTogcmVhbCAzNS45MCB1c2VyIDE3 LjAwIHN5cyAxLjMxIHN0YXJ0OiAxNzY0NTg3MDcxLjM3NjMxMjAwMCBlbmQ6IDE3NjQ1ODcxMDcu MzA0Njg2MDAwIGV4aXQ6IDAgbWVzc2FnZTogcGtnIG9wZW5zc2wtZGV2CnRpbWU6IHJlYWwgMzMu MDEgdXNlciAyNC42MiBzeXMgMC4xOSBzdGFydDogMTc2NDU4Njk4NS4zNzY4NTIwMDAgZW5kOiAx NzY0NTg3MDE4LjQxNTQyMTAwMCBleGl0OiAwIG1lc3NhZ2U6IHBrZyBjbGFuZy1kZXYKdGltZTog cmVhbCAzMS4yNSB1c2VyIDIwLjExIHN5cyAwLjExIHN0YXJ0OiAxNzY0NTg3MDU4LjA2MjE5MjAw MCBlbmQ6IDE3NjQ1ODcwODkuMzQxMTk3MDAwIGV4aXQ6IDAgbWVzc2FnZTogcGtnIGxsZGItZGJn CnRpbWU6IHJlYWwgMzEuMTUgdXNlciA0MC41OSBzeXMgMS40OSBzdGFydDogMTc2NDU4NzEwMy4z Mjg0ODkwMDAgZW5kOiAxNzY0NTg3MTM0LjUxNjk4NTAwMCBleGl0OiAwIG1lc3NhZ2U6IHBrZyB0 ZXN0cwp0aW1lOiByZWFsIDI4LjA2IHVzZXIgMTEuMzggc3lzIDAuMDkgc3RhcnQ6IDE3NjQ1ODY5 ODUuODEwMjM4MDAwIGVuZDogMTc2NDU4NzAxMy44OTUwNTYwMDAgZXhpdDogMCBtZXNzYWdlOiBw a2cgY2xpYnMtZGV2LWxpYjMyCg== --000000000000ba7dde0644f96368--