From owner-dev-commits-ports-main@freebsd.org Sun Jul 25 17:06:33 2021 Return-Path: Delivered-To: dev-commits-ports-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4C4BA65D0B1; Sun, 25 Jul 2021 17:06:33 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GXqGK0cYqz3ky2; Sun, 25 Jul 2021 17:06:33 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1627232793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A5mc5xjFy9Y8ulsb5B3/BdIBwZCy2/k7sLmsv4fe+RA=; b=gj6Xi8k1csIPEdju+9ljZLQHmnAXRx8hDRgzKfOw+8ftCkIpNsHnwVtON+zvzJEXYIUlhk +NJWFhuzpI6mRG926AY9fZtP4UIls9ripLYQwMTePDDmpw2YUmziO/tsPz7Ca/C+R2lwMu XrK2BK+Nt4WxNfEu921UvXP2Oni/FIOcdyw26TrI4oN2RxBwYiSVOlOP4raB9RBjoTYMhw xJTyDsIIZ9uuXKAdpAMQZkJs5VJ9uAi9a6uhydpMysfa6u96QoQWwNtaoSmCncklZb+s/T zaAVuQcATELJ8bTYF/eLTxx3e1LCoSO4aSJxPdLKKgzG3H14AeDy6mLEb+hE3w== Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 9DA90B965; Sun, 25 Jul 2021 17:06:32 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id C3746160DB; Sun, 25 Jul 2021 17:06:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id yBg14ihPwukC; Sun, 25 Jul 2021 17:06:24 +0000 (UTC) Subject: Re: git: e497a16a2869 - main - ports-mgmt/pkg*: Release 1.17.0 DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com AA684160CF To: Kevin Bowling Cc: Baptiste Daroussin , ports-committers , dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org References: <202107220806.16M862GC052436@gitrepo.freebsd.org> From: Bryan Drewery Organization: FreeBSD Message-ID: <2a176f0b-3a52-346f-dd25-7cedf6ca3f38@FreeBSD.org> Date: Sun, 25 Jul 2021 10:06:23 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1627232793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A5mc5xjFy9Y8ulsb5B3/BdIBwZCy2/k7sLmsv4fe+RA=; b=f8EAMyeFIq/ez6WGX40n0KwTJnc1VSXAe7QEv9RTQwRiuvKl4U0eLbcHqhnWOWGig4qWBi j+X/N3sfgWOWn35jTz0rWub6ONB2OknEOeWX2zP6tjbe20qwW7E8o0igymmnlidV8HQ9LF EMen9X6w0OeXa/41bdpKClOhPx+KNXZjHatRLzohdiUNqugReJ1imEzNG24eb3twD5lpru jXtif0cRzNt3P8wLKgyDKl9Z+iGbr7BPnc/4A9eYNV0ko/zDhEI56tXABCgSnScOEZYuW6 ripjQ01tkegZt95QqTLZmo476duhht4zR7agBUqUisgzMkvrJX8T06LuNlbV9Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1627232793; a=rsa-sha256; cv=none; b=r3q3WLpQiG6/VmVbBXqwFgyi+kJzy/gld7PlCK4Oy4kzzOw1KxHJX3C5Ura+/K5NAyX2s8 6xO7Xs2jmjmGZOueT1rt/vIg5aQPqiSLyfoVr5GGUpYsZTUj3MGeEthP6zNOet2ZSLPBOk KZSA1+nqOSb9wQmtEY/ajkyJR7JsMbYb75pEM+J9xIRbudeFOFxKlz4oTShxuqBqm/WFRG ho/5LssrTzesAR/FGJO/D2VXp0yBjFOe2P2mh6/SF7xGSROLJlQwdv/JsR6Nb8bbtTdC3F e4mQPxsA7j/sYescsaV0dPNtJU4230u55YgIeiFcpF2C+mUi4rV2M4t7lVHUGw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-BeenThere: dev-commits-ports-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commits to the main branch of the FreeBSD ports repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jul 2021 17:06:33 -0000 On 7/24/21 9:48 PM, Kevin Bowling wrote: > On Sat, Jul 24, 2021 at 1:36 PM Bryan Drewery wrote: >> >> On 7/22/2021 1:06 AM, Baptiste Daroussin wrote: >>> The branch main has been updated by bapt: >>> >>> URL: https://cgit.FreeBSD.org/ports/commit/?id=e497a16a286972bfcab908209b11ee6a13d99dc9 >>> >>> commit e497a16a286972bfcab908209b11ee6a13d99dc9 >>> Author: Baptiste Daroussin >>> AuthorDate: 2021-07-22 07:54:38 +0000 >>> Commit: Baptiste Daroussin >>> CommitDate: 2021-07-22 08:05:59 +0000 >>> >>> ports-mgmt/pkg*: Release 1.17.0 >>> >>> TL;DR: new uniq package file extension: '.pkg' >>> >>> Full changelog for pkg: >>> - add a new "snap" prefix, to deal with pkg base snapshots >>> - pkg repo now accepts packages with different compression formats >>> - pkg now have a single extension: ".pkg" and a backward compatible >>> symlink is created when the package is created to help transitioning >>> - Default compression level for zstd is now set to 19 >>> - the default compression level is now a configuration option >>> - plenty of portability fixes >>> - fix plenty of typos >>> - expose the name of the package to lua and shell scripts >>> - plist: >>> remove internal support for @*exec (this is now in the ports tree) >>> remove support for @ignore >>> remove support for @dirrm/@dirrmtry >>> remove support for @pkgdep >>> remove stub support for @stopdaemon >>> remove stub support for @display >>> remove stub support for @mtree >>> remove stub support for @conflict >>> - VUXML: fetch .xz compressed version of the file by default >>> - triggers are deferred to later command (firstboot is planned) if run >>> with pkg -r >>> - pkg triggers commands has been added and can execute the deferred >>> triggers for example at firstboot >>> >>> Changes in the framework: >>> - Add a backward compatibility layer, so people are not forced to move >>> to pkg 1.17 yet (new quarter we will enforce the switch for triggers >>> anyway, but it gives time to people to switch) >>> - Introduce a new PKG_COMPRESSION_FORMAT for users to specify the >>> compression format they are willing to use when creating packages: >>> txz, tar, tgz, tbz, tzst are the valid ones (note that tzst is only >>> usable on FreeBSD 13 and 14) >>> - Add a backward compatibility for people who specified the compression >>> format already via PKG_SUFX and issue a warning to tell them about the >>> deprecation of PKG_SUFX >>> --- >>> Mk/bsd.port.mk | 58 +++++++++++++++++++++++-------------------- >>> ports-mgmt/pkg-devel/Makefile | 4 +-- >>> ports-mgmt/pkg-devel/distinfo | 6 ++--- >>> ports-mgmt/pkg/Makefile | 2 +- >>> ports-mgmt/pkg/distinfo | 6 ++--- >>> 5 files changed, 40 insertions(+), 36 deletions(-) >>> >>> diff --git a/Mk/bsd.port.mk b/Mk/bsd.port.mk >>> index 228644a1845f..5eba5d89da95 100644 >>> --- a/Mk/bsd.port.mk >>> +++ b/Mk/bsd.port.mk >>> @@ -1211,6 +1211,14 @@ _OSVERSION_MAJOR= ${OSVERSION:C/([0-9]?[0-9])([0-9][0-9])[0-9]{3}/\1/} >>> .if !defined(_PKG_VERSION) >>> _PKG_VERSION!= ${PKG_BIN} -v >>> .endif >>> +# XXX hack for smooth transition towards pkg 1.17 >>> +_PKG_BEFORE_PKGEXT!= ${PKG_BIN} version -t ${_PKG_VERSION:C/-.*//g} 1.17.0 >>> +.if ${_PKG_BEFORE_PKGEXT} == "<" >>> +_PKG_TRANSITIONING_TO_NEW_EXT= yes >>> +_EXPORTED_VARS+= _PKG_TRANSITIONING_TO_NEW_EXT >>> +.warning "It is strongly recommanded to upgrade to a newer vertion of pkg first" >>> +.endif >>> +# XXX End of hack >>> _PKG_STATUS!= ${PKG_BIN} version -t ${_PKG_VERSION:C/-.*//g} ${MINIMAL_PKG_VERSION} >>> .if ${_PKG_STATUS} == "<" >>> IGNORE= pkg(8) must be version ${MINIMAL_PKG_VERSION} or greater, but you have ${_PKG_VERSION}. You must upgrade the ${PKG_ORIGIN} port first >>> @@ -2227,23 +2235,29 @@ _PKGMESSAGES+= ${PKGMESSAGE} >>> >>> TMPPLIST?= ${WRKDIR}/.PLIST.mktmp >>> >>> -.if ${WITH_PKG} == devel >>> -PKG_SUFX?= .pkg >>> +# backward compatibility for users >>> +.if defined(_PKG_TRANSITIONING_TO_NEW_EXT) >>> .if defined(PKG_NOCOMPRESS) >>> -PKG_OLDSUFX?= .tar >>> +PKG_SUFX?= .tar >>> .else >>> -#.if ${OSVERSION} > 1400000 >>> -#PKG_OLDSUFX?= .tzst >>> -#.else >>> -PKG_OLDSUFX?= .txz >>> -#.endif >>> +PKG_SUFX?= .txz >>> .endif >>> +PKG_COMPRESSION_FORMAT?= ${PKG_SUFX:S/.//} >>> .else >>> +.if defined(PKG_SUFX) >>> +PKG_COMPRESSION_FORMAT?= ${PKG_SUFX:S/.//} >>> +.warning PKG_SUFX is defined, if should be replace with PKG_COMPRESSION_FORMAT >>> +.endif >>> +PKG_SUFX= .pkg >>> +.endif >> >> I love the idea here but can this be reverted for now please? Poudriere >> is not prepared for a default PKG_EXT change. >> >> 1. Now that it finds a new PKG_EXT of .pkg, it cannot find the existing >> bootstrap pkg.txz because it looks for Latest/pkg.pkg -> >> ../All/pkg-*.pkg even though the *existing* one is .txz. I'm not sure >> how this symlink is added before pkg is even built but it's not done in >> a compatible way. > > I haven't seen an issue on my end with the symlinks for pkg which > seems to bootstrap ok and > I don't fully understand the problem statement. I'd like to see what > bapt has to say for this one. > >> 2. The new PKG_EXT makes Poudriere now find no existing packages so >> everything is rebuilt. Not a deal breaker in itself but a problem for >> future builds... > > See below, maybe this is an acceptable transition hammer whenever > someone changes it. > >> 3. The rebuild won't remove all the .txz files. I assume once `pkg repo` >> runs it will find both .txz and .pkg files and cause duplicate version >> problems in `pkg repo` > > This one is poudriere specific, do you think an acceptable solution > would be to detect changes to the variable from the config versus the > repo by storing a canary in the repo and run the pkgclean module if it > changes? I can try and do that tomorrow with your design approval or > guidance. > Sorry for lashing out yesterday. All of these issues are poudriere-specific. None are actually hard. Considering the right fixes and testing is why I'm asking for a revert. So we can do it right and give other tools time to adapt. Even today I got a bug about portupgrade being broken. I don't care to fix it myself but a notice in advance to users that a breaking change is coming would be nice. I do think the official package builds (<13) will be broken for the next week or two. Looking at beefy6 right now (12.2 amd64) I do see both .txz and .pkg files residing in the repository. Soon it will actually run `pkg repo`, sync, and then be serving multiple versions of packages. It's the first run since this commit. pkgclean does indeed fix the problem. I've thought about, in the past, having pkgclean basically always run. That is, when deleting packages for new versions and new dependencies and such also delete files that are unknown. I think in poudriere-devel all that is missing in delete_old_pkg() that pkgclean does extra is delete files not matching *.PKG_EXT. Sounds trivial to add that in. But the official builders have an older version of Poudriere that would need to be updated or we do testing of the new versions. Either way some special care is needed to make sure it works right on the builders. Ideally we shouldn't deleting them here. We should rename them and save the user and world resources from needless rebuilds. A fix for pre-commit and/or post-commit of PKG_EXT isn't as obvious to me and now that it is committed I feel there is no time to consider it so we waste a ton of global resources when we could have done some renames with more planning. Related to rebuilds, but not this thread, I have been testing not auto deleting packages just because a dependency was updated (and addressing the dependency version problem) [1]. I was going to release a prototype this week but now will wait for PKG_EXT to settle down. It could lead to very weird runtime dependency problems if not done right. We need to be able to figure out what fallout is from what cause. [1] https://github.com/freebsd/poudriere/issues/822 > >> I see something creates a .txz -> .pkg symlink. What deles that symlink >> if Poudriere only deletes/rebuilds the .pkg? > > Unsure. > >> *** >> "all those .txz entries are actually _duplicates_ in packagesite.yaml" >> seen on IRC just now by someone. >> I can confirm that looking in a packagesite.yaml I just generated. Both >> the older .txz version of a port and a new revision-bumped one appear in >> the yaml. >> *** >> >> 4. I've seen reports of packagesite.pkg which is concerning because >> Poudriere manages packagesite.txz with its own symlinks for >> hardlink-copied repositories. It does not use PKG_EXT there as they >> weren't packages, but I suspect we need special handling even with >> PKG_EXT added there. >> >> -- >> Regards, >> Bryan Drewery >> -- Regards, Bryan Drewery