From nobody Tue May 20 20:34:05 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4b25qx5Hshz5vwlc; Tue, 20 May 2025 20:34:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4b25qx4WPYz3MFt; Tue, 20 May 2025 20:34:05 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1747773245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/7na5+JeIbqW/0YlreqYRhzFVQYiMst0Cbg2tRwUM8o=; b=Tv3XLx6OERZblUcOmjo2TFblHONVLKfskJrAP99MIqf3rDPf5TeuiCzkTb1T/5IAU8/kkJ +le/VvjbiGPoWDD4CD9wjYAf3pSO1iHmGN8QYzp2J+v3ieIm+y3lLz+ak6eY4B+PpyEfRd +OscBjmV/RkaJcrZJHHervOX9xnChoVgO8X0PNbcVk/r3HW3a8HPn+cdxdVv7VAnGLlVlm olY2IMoj9Hnf6Wy8EMtDyLqtU1RVWOnHUEb0LXqP9VozBz3uoYLD3ztiC3E4SoKwSY58XN N5nvdUPJjDVZom79ig8S8iWyB2x8LDzSSoAJaMppgftlXub7C3DIpaWEByGuCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1747773245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=/7na5+JeIbqW/0YlreqYRhzFVQYiMst0Cbg2tRwUM8o=; b=UtA2w0ImcYTJ6SriD3Dk5Ef+oqkFaQssPyknFCoiobamhkMC5LpUb/bbyYJqclJGrOaEKy ZNk+6yMSNAwIDxuHnmXsLA5D9plffbixjqAFf5y7FrNky7IgjgqpZ2LRwdALFcSs5J4wX8 8gIl4cvleo4bMR7R0DShZCRKmDqtB08eBpNz0LsVNNjHmbtt8CJckus3EeWEUIsIOkUW9t WXlfUiwgwKzbfvLEp6Rvo4+ptF8ZphrayqR7hy9E5fcJk86C702x6oF4MJpqBwYPGpskwh hpxiBk8ghvLjEbtUbThkjV5JgPIU4AnxlxD7jmLOsFuwjU8JyUaMLnfWS4nEgA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1747773245; a=rsa-sha256; cv=none; b=coEt/FT3VnlHW/J4j7jCRVtIp1F05ozQ6RcLx/QFIha3fY3AdcFsEOhO8RqyvAYgt6f67D szCRdtahFxIkPX6t9q6pcrGZ4+2TBmzFrwmN3IS/qy2EdzS4lmsc7C1O/4C09vucehrNTz 01RWjodkqhoenA1WcsIXJKN/OtSaqXTbIRM5YEBic7zpmXMik6f0B2FMXfjl88J7pVxH0x Pveg8ZBucEVKzpO2hUI5Vv2vsKHuH4at4I8jScsIWgYreHgjJ7A1k/sMQi2K4OTYA3Jat8 HKmuxKRuDDAYBeyo7EHZqqKzN7M/0omGvXiCIoEWX/jvQB0O/N8cF4duiyQZKA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4b25qx46b2z14YD; Tue, 20 May 2025 20:34:05 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 54KKY51G091173; Tue, 20 May 2025 20:34:05 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 54KKY5Sd091170; Tue, 20 May 2025 20:34:05 GMT (envelope-from git) Date: Tue, 20 May 2025 20:34:05 GMT Message-Id: <202505202034.54KKY5Sd091170@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Ed Maste Subject: git: 62d18f8c4c10 - main - release: Add -DPKGBASE option to include pkgbase packages List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: emaste X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 62d18f8c4c10a5e680b0b795040abffc7964977e Auto-Submitted: auto-generated The branch main has been updated by emaste: URL: https://cgit.FreeBSD.org/src/commit/?id=62d18f8c4c10a5e680b0b795040abffc7964977e commit 62d18f8c4c10a5e680b0b795040abffc7964977e Author: Isaac Freund AuthorDate: 2025-05-10 21:36:53 +0000 Commit: Ed Maste CommitDate: 2025-05-20 20:30:12 +0000 release: Add -DPKGBASE option to include pkgbase packages If this option is set, an offline repo of pkgbase packages corresponding to base.txz and kernel.txz will be included in the disc1 release media rather than the base.txz and kernel.txz tarballs. Reviewed by: bapt Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D50346 --- release/Makefile | 53 +++++++++++- release/scripts/FreeBSD-base-offline.conf | 4 + release/scripts/pkgbase-stage.lua | 138 ++++++++++++++++++++++++++++++ 3 files changed, 193 insertions(+), 2 deletions(-) diff --git a/release/Makefile b/release/Makefile index 1a64c51d3400..578df226ff49 100644 --- a/release/Makefile +++ b/release/Makefile @@ -21,6 +21,8 @@ # Variables affecting the build process: # WORLDDIR: location of src tree -- must have built world and default kernel # (by default, the directory above this one) +# PKGBASE: if set, include pkgbase packages rather than dist tarballs in +# disc1 and dvd1 installation media # PORTSDIR: location of ports tree to distribute (default: /usr/ports) # XTRADIR: xtra-bits-dir argument for /mkisoimages.sh # NOPKG: if set, do not distribute third-party packages @@ -124,7 +126,7 @@ CLEANFILES+= ${I}.xz .if defined(WITH_DVD) && !empty(WITH_DVD) CLEANFILES+= pkg-stage .endif -CLEANDIRS= dist ftp disc1 disc1-disc1 disc1-memstick bootonly bootonly-bootonly bootonly-memstick dvd +CLEANDIRS= dist pkgbase-repo pkgbase-repo-conf ftp disc1 disc1-disc1 disc1-memstick bootonly bootonly-bootonly bootonly-memstick dvd beforeclean: chflags -R noschg . .include @@ -170,7 +172,16 @@ ports.txz: --exclude 'usr/ports/INDEX*' --exclude work usr/ports | \ ${XZ_CMD} > ${.OBJDIR}/ports.txz ) -disc1: packagesystem +.if defined(PKGBASE) +PKGBASE_REPO= pkgbase-repo +PKG_ABI= $$(${PKG_CMD} -o ABI_FILE=${.TARGET}/usr/bin/uname config ABI) +.endif + +pkgbase-repo: + mkdir -p pkgbase-repo + ( ${IMAKE} -C ${WORLDDIR} packages REPODIR=${.OBJDIR}/pkgbase-repo ) + +disc1: packagesystem ${PKGBASE_REPO} # Install system mkdir -p ${.TARGET} ( cd ${WORLDDIR} && ${IMAKE} installworld installkernel distribution \ @@ -181,6 +192,24 @@ disc1: packagesystem MK_RESCUE=no MK_DICT=no \ MK_KERNEL_SYMBOLS=no MK_TESTS=no MK_DEBUG_FILES=no \ -DDB_FROM_SRC -DNO_ROOT) +.if defined(PKGBASE) +# Create offline pkgbase repo on release media + mkdir -p ${.TARGET}/usr/freebsd-packages/repos/ + ${.CURDIR}/scripts/pkgbase-stage.lua disc \ + ${.OBJDIR}/pkgbase-repo/${PKG_ABI}/latest \ + ${.TARGET}/usr/freebsd-packages/offline \ + "${_ALL_libcompats}" + cp ${.CURDIR}/scripts/FreeBSD-base-offline.conf \ + ${.TARGET}/usr/freebsd-packages/repos/ + mtree -c -p ${.TARGET}/usr/freebsd-packages | \ + mtree -C -k type,mode,link,size | \ + sed 's|^\.|./usr/freebsd-packages|g' >> ${.TARGET}/METALOG +# Copy manifest only (no distfiles) to get checksums + mkdir -p ${.TARGET}/usr/freebsd-dist + cp MANIFEST ${.TARGET}/usr/freebsd-dist + echo "./usr/freebsd-dist type=dir uname=root gname=wheel mode=0755" >> ${.TARGET}/METALOG + echo "./usr/freebsd-dist/MANIFEST type=file uname=root gname=wheel mode=0644" >> ${.TARGET}/METALOG +.else # Copy distfiles mkdir -p ${.TARGET}/usr/freebsd-dist for dist in MANIFEST $$(ls *.txz | grep -v container | grep -vE -- '(${base ${_ALL_libcompats}:L:ts|})-dbg'); \ @@ -190,6 +219,7 @@ disc1: packagesystem for dist in MANIFEST $$(ls *.txz | grep -v container | grep -vE -- '(${base ${_ALL_libcompats}:L:ts|})-dbg'); \ do echo "./usr/freebsd-dist/$${dist} type=file uname=root gname=wheel mode=0644" >> ${.TARGET}/METALOG; \ done +.endif .if ${.MAKE.OS} == "FreeBSD" && (!defined(NOPKG) || empty(NOPKG)) # Install packages onto release media. ${PKG_INSTALL} pkg || true @@ -256,6 +286,24 @@ dvd: packagesystem DESTDIR=${.OBJDIR}/${.TARGET} MK_RESCUE=no MK_KERNEL_SYMBOLS=no \ MK_TESTS=no MK_DEBUG_FILES=no \ -DDB_FROM_SRC -DNO_ROOT) +.if defined(PKGBASE) +# Create offline pkgbase repo on release media + mkdir -p ${.TARGET}/usr/freebsd-packages/repos/ + ${.CURDIR}/scripts/pkgbase-stage.lua dvd \ + ${.OBJDIR}/pkgbase-repo/${PKG_ABI}/latest \ + ${.TARGET}/usr/freebsd-packages/offline \ + "${_ALL_libcompats}" + cp ${.CURDIR}/scripts/FreeBSD-base-offline.conf \ + ${.TARGET}/usr/freebsd-packages/repos/ + mtree -c -p ${.TARGET}/usr/freebsd-packages | \ + mtree -C -k type,mode,link,size | \ + sed 's|^\.|./usr/freebsd-packages|g' >> ${.TARGET}/METALOG +# Copy manifest only (no distfiles) to get checksums + mkdir -p ${.TARGET}/usr/freebsd-dist + cp MANIFEST ${.TARGET}/usr/freebsd-dist + echo "./usr/freebsd-dist type=dir uname=root gname=wheel mode=0755" >> ${.TARGET}/METALOG + echo "./usr/freebsd-dist/MANIFEST type=file uname=root gname=wheel mode=0644" >> ${.TARGET}/METALOG +.else # Copy distfiles mkdir -p ${.TARGET}/usr/freebsd-dist for dist in MANIFEST $$(ls *.txz | grep -v container); \ @@ -265,6 +313,7 @@ dvd: packagesystem for dist in MANIFEST $$(ls *.txz | grep -v container); \ do echo "./usr/freebsd-dist/$${dist} type=file uname=root gname=wheel mode=0644" >> ${.TARGET}/METALOG; \ done +.endif .if ${.MAKE.OS} == "FreeBSD" && (!defined(NOPKG) || empty(NOPKG)) # Install packages onto release media. ${PKG_INSTALL} pkg || true diff --git a/release/scripts/FreeBSD-base-offline.conf b/release/scripts/FreeBSD-base-offline.conf new file mode 100644 index 000000000000..b77334e7a4aa --- /dev/null +++ b/release/scripts/FreeBSD-base-offline.conf @@ -0,0 +1,4 @@ +FreeBSD-base: { + url: "file:///usr/freebsd-packages/offline", + enabled: yes +} diff --git a/release/scripts/pkgbase-stage.lua b/release/scripts/pkgbase-stage.lua new file mode 100755 index 000000000000..01eec8c44e49 --- /dev/null +++ b/release/scripts/pkgbase-stage.lua @@ -0,0 +1,138 @@ +#!/usr/libexec/flua + +-- SPDX-License-Identifier: BSD-2-Clause +-- +-- Copyright(c) 2025 The FreeBSD Foundation. +-- +-- This software was developed by Isaac Freund +-- under sponsorship from the FreeBSD Foundation. + +-- Run a command using the OS shell and capture the stdout +-- Strips exactly one trailing newline if present, does not strip any other whitespace. +-- Asserts that the command exits cleanly +local function capture(command) + local p = io.popen(command) + local output = p:read("*a") + assert(p:close()) + -- Strip exactly one trailing newline from the output, if there is one + return output:match("(.-)\n$") or output +end + +local function append_list(list, other) + for _, item in ipairs(other) do + table.insert(list, item) + end +end + +-- Returns a list of packages to be included in the given media +local function select_packages(pkg, media, all_libcompats) + local components = { + kernel = {}, + kernel_dbg = {}, + base = {}, + base_dbg = {}, + src = {}, + tests = {}, + } + + for compat in all_libcompats:gmatch("%S+") do + components["lib" .. compat] = {} + components["lib" .. compat .. "_dbg"] = {} + end + + local rquery = capture(pkg .. "rquery -U -r FreeBSD-base %n") + for package in rquery:gmatch("[^\n]+") do + if package == "FreeBSD-src" or package:match("^FreeBSD%-src%-.*") then + table.insert(components["src"], package) + elseif package == "FreeBSD-tests" or package:match("^FreeBSD%-tests%-.*") then + table.insert(components["tests"], package) + elseif package:match("^FreeBSD%-kernel%-.*") then + -- Kernels other than FreeBSD-kernel-generic are ignored + if package == "FreeBSD-kernel-generic" then + table.insert(components["kernel"], package) + elseif package == "FreeBSD-kernel-generic-dbg" then + table.insert(components["kernel_dbg"], package) + end + elseif package:match(".*%-dbg$") then + table.insert(components["base_dbg"], package) + else + local found = false + for compat in all_libcompats:gmatch("%S+") do + if package:match(".*%-dbg%-lib" .. compat .. "$") then + table.insert(components["lib" .. compat .. "_dbg"], package) + found = true + break + elseif package:match(".*%-lib" .. compat .. "$") then + table.insert(components["lib" .. compat], package) + found = true + break + end + end + if not found then + table.insert(components["base"], package) + end + end + end + assert(#components["kernel"] == 1) + assert(#components["base"] > 0) + + local selected = {} + if media == "disc" then + append_list(selected, components["base"]) + append_list(selected, components["kernel"]) + append_list(selected, components["kernel_dbg"]) + append_list(selected, components["src"]) + append_list(selected, components["tests"]) + for compat in all_libcompats:gmatch("%S+") do + append_list(selected, components["lib" .. compat]) + end + else + assert(media == "dvd") + append_list(selected, components["base"]) + append_list(selected, components["base_dbg"]) + append_list(selected, components["kernel"]) + append_list(selected, components["kernel_dbg"]) + append_list(selected, components["src"]) + append_list(selected, components["tests"]) + for compat in all_libcompats:gmatch("%S+") do + append_list(selected, components["lib" .. compat]) + append_list(selected, components["lib" .. compat .. "_dbg"]) + end + end + + return selected +end + +local function main() + -- Determines package subset selected + local media = assert(arg[1]) + assert(media == "disc" or media == "dvd") + -- Local repository to fetch from + local source = assert(arg[2]) + -- Directory to create new repository + local target = assert(arg[3]) + -- =hitespace separated list of all libcompat names (e.g. "32") + local all_libcompats = assert(arg[4]) + + assert(os.execute("mkdir -p pkgbase-repo-conf")) + local f = assert(io.open("pkgbase-repo-conf/FreeBSD-base.conf", "w")) + assert(f:write(string.format([[ + FreeBSD-base: { + url: "file://%s", + enabled: yes + } + ]], source))) + assert(f:close()) + + local pkg = "pkg -o ASSUME_ALWAYS_YES=yes -o IGNORE_OSVERSION=yes " .. + "-o INSTALL_AS_USER=1 -o PKG_DBDIR=./pkgdb -R ./pkgbase-repo-conf " + + assert(os.execute(pkg .. "update")) + + local packages = select_packages(pkg, media, all_libcompats) + + assert(os.execute(pkg .. "fetch -o " .. target .. " " .. table.concat(packages, " "))) + assert(os.execute(pkg .. "repo " .. target)) +end + +main()