From owner-freebsd-pkgbase@freebsd.org Mon Jan 18 18:25:26 2021 Return-Path: Delivered-To: freebsd-pkgbase@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 40E0B4F2C30 for ; Mon, 18 Jan 2021 18:25:26 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKKw61Mq8z4nsQ; Mon, 18 Jan 2021 18:25:26 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 1CE5622C03; Mon, 18 Jan 2021 18:25:26 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qt1-f170.google.com with SMTP id z6so4857096qtn.0; Mon, 18 Jan 2021 10:25:26 -0800 (PST) X-Gm-Message-State: AOAM532hf/+FTJkTvtGIz3XWT2o4wsIyeHzwqmBSdm/pTc7fSHjf/A3F MXyE/Rv7DyKrJV4FJXeKVTnNnl5w0TLJ/i+7AMs= X-Google-Smtp-Source: ABdhPJxICIXn/HdRfCOQKnPGpqodv6cg5JAvp3UH2b6g9yPtqmbJk2tXKyhjUbrWkBzGhy7N1HTa6i03m68/SV/7qrY= X-Received: by 2002:ac8:7141:: with SMTP id h1mr835654qtp.211.1610994325554; Mon, 18 Jan 2021 10:25:25 -0800 (PST) MIME-Version: 1.0 References: <202011021823.0A2INo93086367@repo.freebsd.org> In-Reply-To: <202011021823.0A2INo93086367@repo.freebsd.org> From: Kyle Evans Date: Mon, 18 Jan 2021 12:25:12 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: svn commit: r367277 - head To: Emmanuel Vadot Cc: freebsd-pkgbase@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-pkgbase@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Packaging the FreeBSD base system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 18:25:26 -0000 On Mon, Nov 2, 2020 at 12:24 PM Emmanuel Vadot wrote: > > Author: manu > Date: Mon Nov 2 18:23:50 2020 > New Revision: 367277 > URL: https://svnweb.freebsd.org/changeset/base/367277 > > Log: > pkgbase: Add incremental packages > > This adds a new target update-packages which will create the new packages > compared to the last run. > > This is how to use it: > At this point we cut a release > $ make buildworld ... > $ make buildkernel > $ make packages > > There is now a PKG_VERSION directory with latest link pointing to it > Distribute the packages to server > > $ something something that update the source tree > $ make buildworld ... > $ make buildkernel > $ make update-packages > You know have a PKG_VERSION directory in the REPODIR and latest link pointing to it. > In PKG_VERSION dir only the packages which differs from the latest run are > named PKG_VERSION, otherwise the old packages are there. > > The process is : > Build the new packages in the PKG_VERSION directory > Compare the internal data with the PKG_VERSION_FROM version. The comparison is done > by checking the internal hash of the packages. > By default PKG_VERSION_FROM is set to what the latest link points to. > If the old and new version matches, we rm the new package and cp the old one. > > Differential Revision: https://reviews.freebsd.org/D25984 > > Modified: > head/Makefile > head/Makefile.inc1 > > [.. snip ..] > Modified: head/Makefile.inc1 > ============================================================================== > --- head/Makefile.inc1 Mon Nov 2 17:39:59 2020 (r367276) > +++ head/Makefile.inc1 Mon Nov 2 18:23:50 2020 (r367277) > [... snip ...] > @@ -1867,6 +1880,28 @@ package-pkg: .PHONY > > real-packages: stage-packages create-packages sign-packages .PHONY > > +real-update-packages: stage-packages .PHONY > + ${_+_}${MAKE} -C ${.CURDIR} PKG_VERSION=${PKG_VERSION} create-packages > + @echo "==> Checking for new packages (comparing ${PKG_VERSION} to ${PKG_VERSION_FROM})" > + @for pkg in ${REPODIR}/${PKG_ABI}/${PKG_VERSION_FROM}/${PKG_NAME_PREFIX}-*; do \ > + pkgname=$$(pkg query -F $${pkg} '%n' | sed 's/${PKG_NAME_PREFIX}-\(.*\)/\1/') ; \ > + newpkgname=${PKG_NAME_PREFIX}-$${pkgname}-${PKG_VERSION}.${PKG_FORMAT} ; \ > + oldsum=$$(pkg query -F $${pkg} '%X') ; \ > + if [ ! -f ${REPODIR}/${PKG_ABI}/${PKG_VERSION}/$${newpkgname} ]; then \ > + continue; \ > + fi ; \ > + newsum=$$(pkg query -F ${REPODIR}/${PKG_ABI}/${PKG_VERSION}/$${newpkgname} '%X') ; \ > + if [ "$${oldsum}" == "$${newsum}" ]; then \ > + echo "==> Keeping old ${PKG_NAME_PREFIX}-$${pkgname}-${PKG_VERSION_FROM}.${PKG_FORMAT}" ; \ > + rm ${REPODIR}/${PKG_ABI}/${PKG_VERSION}/$${newpkgname} ; \ > + cp $${pkg} ${REPODIR}/${PKG_ABI}/${PKG_VERSION} ; \ > + else \ > + echo "==> New package $${newpkgname}" ; \ > + fi ; \ > + done > + ${_+_}@cd ${.CURDIR}; \ > + ${MAKE} -f Makefile.inc1 PKG_VERSION=${PKG_VERSION} sign-packages > + > stage-packages-world: .PHONY > @mkdir -p ${WSTAGEDIR} > ${_+_}@cd ${.CURDIR}; \ Any objection to making update-packages harmless to run the first time, too? It'd be nicer for some external scripting to not worry about the distinction between packages and update-packages if the real-update-packages target skipped the loop entirely if ${REPODIR}/${PKG_ABI} didn't exist at all. Thanks, Kyle Evans From owner-freebsd-pkgbase@freebsd.org Mon Jan 18 18:32:45 2021 Return-Path: Delivered-To: freebsd-pkgbase@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E582E4F3085 for ; Mon, 18 Jan 2021 18:32:45 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mx.blih.net (mx.blih.net [212.83.155.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx.blih.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DKL4Y3Vh6z4pMv; Mon, 18 Jan 2021 18:32:44 +0000 (UTC) (envelope-from manu@bidouilliste.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bidouilliste.com; s=mx; t=1610994761; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m2Mzy/IlXQqclgBHoPPnbFlBcrWdwok+sivYZ8NTSHY=; b=IcW3oyhZoE1m1VdXrDCN6NL0SeGGEbX7h/cW5kEj5ngI/Qo7HgwNIcOwvv24FQKlYxPu2d KHgXs9jM/zB2reX8ZJn+VrySecwpFCUmphBETa5VH2w1eDBrItBXOkgtFsDeu4a990MFQ3 g0IxFYiXXoOCkH6bizU6jGTOOw6RbHc= Received: from amy.home (lfbn-idf2-1-745-114.w86-247.abo.wanadoo.fr [86.247.192.114]) by mx.blih.net (OpenSMTPD) with ESMTPSA id e8a1961b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Jan 2021 18:32:41 +0000 (UTC) Date: Mon, 18 Jan 2021 19:32:41 +0100 From: Emmanuel Vadot To: Kyle Evans Cc: Emmanuel Vadot , freebsd-pkgbase@freebsd.org Subject: Re: svn commit: r367277 - head Message-Id: <20210118193241.3465309db73fd85e456d179f@bidouilliste.com> In-Reply-To: References: <202011021823.0A2INo93086367@repo.freebsd.org> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DKL4Y3Vh6z4pMv X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-pkgbase@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Packaging the FreeBSD base system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jan 2021 18:32:46 -0000 On Mon, 18 Jan 2021 12:25:12 -0600 Kyle Evans wrote: > On Mon, Nov 2, 2020 at 12:24 PM Emmanuel Vadot wrote: > > > > Author: manu > > Date: Mon Nov 2 18:23:50 2020 > > New Revision: 367277 > > URL: https://svnweb.freebsd.org/changeset/base/367277 > > > > Log: > > pkgbase: Add incremental packages > > > > This adds a new target update-packages which will create the new packages > > compared to the last run. > > > > This is how to use it: > > At this point we cut a release > > $ make buildworld ... > > $ make buildkernel > > $ make packages > > > > There is now a PKG_VERSION directory with latest link pointing to it > > Distribute the packages to server > > > > $ something something that update the source tree > > $ make buildworld ... > > $ make buildkernel > > $ make update-packages > > You know have a PKG_VERSION directory in the REPODIR and latest link pointing to it. > > In PKG_VERSION dir only the packages which differs from the latest run are > > named PKG_VERSION, otherwise the old packages are there. > > > > The process is : > > Build the new packages in the PKG_VERSION directory > > Compare the internal data with the PKG_VERSION_FROM version. The comparison is done > > by checking the internal hash of the packages. > > By default PKG_VERSION_FROM is set to what the latest link points to. > > If the old and new version matches, we rm the new package and cp the old one. > > > > Differential Revision: https://reviews.freebsd.org/D25984 > > > > Modified: > > head/Makefile > > head/Makefile.inc1 > > > > [.. snip ..] > > Modified: head/Makefile.inc1 > > ============================================================================== > > --- head/Makefile.inc1 Mon Nov 2 17:39:59 2020 (r367276) > > +++ head/Makefile.inc1 Mon Nov 2 18:23:50 2020 (r367277) > > [... snip ...] > > @@ -1867,6 +1880,28 @@ package-pkg: .PHONY > > > > real-packages: stage-packages create-packages sign-packages .PHONY > > > > +real-update-packages: stage-packages .PHONY > > + ${_+_}${MAKE} -C ${.CURDIR} PKG_VERSION=${PKG_VERSION} create-packages > > + @echo "==> Checking for new packages (comparing ${PKG_VERSION} to ${PKG_VERSION_FROM})" > > + @for pkg in ${REPODIR}/${PKG_ABI}/${PKG_VERSION_FROM}/${PKG_NAME_PREFIX}-*; do \ > > + pkgname=$$(pkg query -F $${pkg} '%n' | sed 's/${PKG_NAME_PREFIX}-\(.*\)/\1/') ; \ > > + newpkgname=${PKG_NAME_PREFIX}-$${pkgname}-${PKG_VERSION}.${PKG_FORMAT} ; \ > > + oldsum=$$(pkg query -F $${pkg} '%X') ; \ > > + if [ ! -f ${REPODIR}/${PKG_ABI}/${PKG_VERSION}/$${newpkgname} ]; then \ > > + continue; \ > > + fi ; \ > > + newsum=$$(pkg query -F ${REPODIR}/${PKG_ABI}/${PKG_VERSION}/$${newpkgname} '%X') ; \ > > + if [ "$${oldsum}" == "$${newsum}" ]; then \ > > + echo "==> Keeping old ${PKG_NAME_PREFIX}-$${pkgname}-${PKG_VERSION_FROM}.${PKG_FORMAT}" ; \ > > + rm ${REPODIR}/${PKG_ABI}/${PKG_VERSION}/$${newpkgname} ; \ > > + cp $${pkg} ${REPODIR}/${PKG_ABI}/${PKG_VERSION} ; \ > > + else \ > > + echo "==> New package $${newpkgname}" ; \ > > + fi ; \ > > + done > > + ${_+_}@cd ${.CURDIR}; \ > > + ${MAKE} -f Makefile.inc1 PKG_VERSION=${PKG_VERSION} sign-packages > > + > > stage-packages-world: .PHONY > > @mkdir -p ${WSTAGEDIR} > > ${_+_}@cd ${.CURDIR}; \ > > Any objection to making update-packages harmless to run the first > time, too? It'd be nicer for some external scripting to not worry > about the distinction between packages and update-packages if the > real-update-packages target skipped the loop entirely if > ${REPODIR}/${PKG_ABI} didn't exist at all. > > Thanks, > > Kyle Evans Nope, feel free :) -- Emmanuel Vadot