From owner-dev-commits-ports-all@freebsd.org Sun Jul 25 02:07:28 2021 Return-Path: Delivered-To: dev-commits-ports-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9651166DB8D for ; Sun, 25 Jul 2021 02:07:28 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-yb1-xb2b.google.com (mail-yb1-xb2b.google.com [IPv6:2607:f8b0:4864:20::b2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GXRJv5Zr4z3jPb for ; Sun, 25 Jul 2021 02:07:27 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-yb1-xb2b.google.com with SMTP id a201so8789981ybg.12 for ; Sat, 24 Jul 2021 19:07:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=JcFp3wYWIqejVvBxuASs7SIEmS7ENR5wL10IWej8f0k=; b=jMmhsibaBbxbZT1n09LjLTv9nRggFk+uCQ3ix6HCm9pf437aWiQmc4z0+SFaTABmmP OaUpFyWZCOGLeaTipOgNkkIyZROYZqXYUhhb8xwiR4Vbt/7+uvGgUYGfk4oTF4II7g/z AV1k+GITBDCtvul6Ecy37YuxqdFJdwXEVtYt8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=JcFp3wYWIqejVvBxuASs7SIEmS7ENR5wL10IWej8f0k=; b=Y9x3v6N2pdK+DkPLaFo4uZnOf8jAjIs9Ljz3CB04NApZAozA3CKI6WS4ZIstjvBlXZ IYfd8ODJF37kEwY0COGXTugkWCphuPhfS+CEbL5tgKaq8glfqXv8bWOOdQ39FjJZfinH 1HJkyYWt2+26+O8tbwigsi/e5r+gfpq0E1yeKyAouauoteD6ygkZbx0Izo9u1TvRypGj 38LxNYmB3cbkEuWGCpm23PAPyAFZ4mTLLuxY+iVAjj29EF9rCaJN+dHRBxpufh1RuR5r 9buyo6hySAdhr6EhGPZ2Rdo3bXsXej2C9hMo66BFTVk+ndsMT0fN+5dala4HjUR6nGkG e8AQ== X-Gm-Message-State: AOAM530FnHQ1c9BDHgbs7lk6qu14+Qo7DEn/Io6FX6PLGudossoQCM/q DG6beH8yb7MgtPe0VMvAlT/gZtmfgj5TkMN0IlqVmg== X-Google-Smtp-Source: ABdhPJyHsZR+0xikmB4hvwHXP9nojd0wNzLG+JK9Ty5oO7tMDhIkS19TrMcYGggBdrn0pgKF0z1e0CvZyVA9TJ4bvi8= X-Received: by 2002:a25:ad51:: with SMTP id l17mr1174725ybe.123.1627178846757; Sat, 24 Jul 2021 19:07:26 -0700 (PDT) MIME-Version: 1.0 References: <202107220806.16M862GC052436@gitrepo.freebsd.org> In-Reply-To: From: Kevin Bowling Date: Sat, 24 Jul 2021 19:07:15 -0700 Message-ID: Subject: Re: git: e497a16a2869 - main - ports-mgmt/pkg*: Release 1.17.0 To: Bryan Drewery Cc: Baptiste Daroussin , dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org, ports-committers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4GXRJv5Zr4z3jPb X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none ("invalid DKIM record") header.d=kev009.com header.s=google header.b=jMmhsiba; dmarc=none; spf=pass (mx1.freebsd.org: domain of kevin.bowling@kev009.com designates 2607:f8b0:4864:20::b2b as permitted sender) smtp.mailfrom=kevin.bowling@kev009.com X-Spamd-Result: default: False [-0.30 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[dev-commits-ports-all@freebsd.org]; DMARC_NA(0.00)[kev009.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[kev009.com:~]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b2b:from]; R_DKIM_PERMFAIL(0.00)[kev009.com:s=google]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[dev-commits-ports-all]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: dev-commits-ports-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the ports repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jul 2021 02:07:28 -0000 On Sat, Jul 24, 2021 at 3:24 PM Bryan Drewery wrote: > > We have a storm brewing on the package builders right now. They will > soon be serving repositories with duplicated/multiple versions of package= s. > > Frankly your response only angered me as it was dismissive and focus > stealing top-post. The details I gave are backed up by many reports > coming in on IRC and github. If you look into your packagesite.yaml and > inspect for both .txz and .pkg you may find duplicates. I'm sorry for upsetting you. I asked a technical question to you and the entire CC line and triggered you, I don't see a causal link so you'll have to forgive me for top posting. > I have a life and work priorities of my own. I can't drop everything to > fix this right now. I am skeptical of any fix that does not meticulously > handle the transition and simply tosses in an unlink. Meaning I will > want to see or do my own testing. If I had known the default PKG_EXT was > changing I would have thrown up my arms and blocked this and asked for a > migration period. Poudriere isn't even all there is. You have my solidarity. Times get better and worse for everyone each day and committing to FreeBSD as a volunteer (which I assume we both are) is not always easy nor possible. I'm not seeing where anyone asked you to do any work. If you give it a couple days you probably wont have to, thanks for raising the issues to bapt and the entire ports community. > I think users would appreciate having a tool that does the right thing > when the change comes in rather than being forced to go through the > wrong broken thing to get the fixed tool. The longer this is in the > bigger the damage is. Ok. There are always unknown unknowns. I don't know all the details here but I saw bapt lay out very good reasoning for these changes, they are working for some including me, and I support him in resolving the matter. I'll take a look tonight and see if I can help. I support pkg and ports, and I like anyone that wants to make things better. Regards, Kevin > > > > On 7/24/2021 2:50 PM, Kevin Bowling wrote: > > It=E2=80=99s all working fine for me on main and 13.0. Which of these = issues > > cannot be solved in the next few days? > > > > On Sat, Jul 24, 2021 at 1:36 PM Bryan Drewery > > wrote: > > > > On 7/22/2021 1:06 AM, Baptiste Daroussin wrote: > > > The branch main has been updated by bapt: > > > > > > URL: > > https://cgit.FreeBSD.org/ports/commit/?id=3De497a16a286972bfcab9082= 09b11ee6a13d99dc9 > > > > > > > > commit e497a16a286972bfcab908209b11ee6a13d99dc9 > > > Author: Baptiste Daroussin > > > AuthorDate: 2021-07-22 07:54:38 +0000 > > > Commit: Baptiste Daroussin > > > CommitDate: 2021-07-22 08:05:59 +0000 > > > > > > ports-mgmt/pkg*: Release 1.17.0 > > > > > > TL;DR: new uniq package file extension: '.pkg' > > > > > > Full changelog for pkg: > > > - add a new "snap" prefix, to deal with pkg base snapshots > > > - pkg repo now accepts packages with different compression fo= rmats > > > - pkg now have a single extension: ".pkg" and a backward > > compatible > > > symlink is created when the package is created to help > > transitioning > > > - Default compression level for zstd is now set to 19 > > > - the default compression level is now a configuration option > > > - plenty of portability fixes > > > - fix plenty of typos > > > - expose the name of the package to lua and shell scripts > > > - plist: > > > remove internal support for @*exec (this is now in the port= s > > tree) > > > remove support for @ignore > > > remove support for @dirrm/@dirrmtry > > > remove support for @pkgdep > > > remove stub support for @stopdaemon > > > remove stub support for @display > > > remove stub support for @mtree > > > remove stub support for @conflict > > > - VUXML: fetch .xz compressed version of the file by default > > > - triggers are deferred to later command (firstboot is > > planned) if run > > > with pkg -r > > > - pkg triggers commands has been added and can execute the > > deferred > > > triggers for example at firstboot > > > > > > Changes in the framework: > > > - Add a backward compatibility layer, so people are not force= d > > to move > > > to pkg 1.17 yet (new quarter we will enforce the switch for > > triggers > > > anyway, but it gives time to people to switch) > > > - Introduce a new PKG_COMPRESSION_FORMAT for users to specify= the > > > compression format they are willing to use when creating > > packages: > > > txz, tar, tgz, tbz, tzst are the valid ones (note that tzst > > is only > > > usable on FreeBSD 13 and 14) > > > - Add a backward compatibility for people who specified the > > compression > > > format already via PKG_SUFX and issue a warning to tell the= m > > about the > > > deprecation of PKG_SUFX > > > --- > > > Mk/bsd.port.mk | 58 > > +++++++++++++++++++++++-------------------- > > > ports-mgmt/pkg-devel/Makefile | 4 +-- > > > ports-mgmt/pkg-devel/distinfo | 6 ++--- > > > ports-mgmt/pkg/Makefile | 2 +- > > > ports-mgmt/pkg/distinfo | 6 ++--- > > > 5 files changed, 40 insertions(+), 36 deletions(-) > > > > > > diff --git a/Mk/bsd.port.mk b/Mk/bsd.port.mk > > > > > index 228644a1845f..5eba5d89da95 100644 > > > --- a/Mk/bsd.port.mk > > > +++ b/Mk/bsd.port.mk > > > @@ -1211,6 +1211,14 @@ _OSVERSION_MAJOR=3D > > ${OSVERSION:C/([0-9]?[0-9])([0-9][0-9])[0-9]{3}/\1/} > > > .if !defined(_PKG_VERSION) > > > _PKG_VERSION!=3D ${PKG_BIN} -v > > > .endif > > > +# XXX hack for smooth transition towards pkg 1.17 > > > +_PKG_BEFORE_PKGEXT!=3D ${PKG_BIN} version -t > > ${_PKG_VERSION:C/-.*//g} 1.17.0 > > > +.if ${_PKG_BEFORE_PKGEXT} =3D=3D "<" > > > +_PKG_TRANSITIONING_TO_NEW_EXT=3D yes > > > +_EXPORTED_VARS+=3D _PKG_TRANSITIONING_TO_NEW_EXT > > > +.warning "It is strongly recommanded to upgrade to a newer > > vertion of pkg first" > > > +.endif > > > +# XXX End of hack > > > _PKG_STATUS!=3D ${PKG_BIN} version -t > > ${_PKG_VERSION:C/-.*//g} ${MINIMAL_PKG_VERSION} > > > .if ${_PKG_STATUS} =3D=3D "<" > > > IGNORE=3D pkg(8) must be version > > ${MINIMAL_PKG_VERSION} or greater, but you have ${_PKG_VERSION}. Yo= u > > must upgrade the ${PKG_ORIGIN} port first > > > @@ -2227,23 +2235,29 @@ _PKGMESSAGES+=3D ${PKGMESSAGE} > > > > > > TMPPLIST?=3D ${WRKDIR}/.PLIST.mktmp > > > > > > -.if ${WITH_PKG} =3D=3D devel > > > -PKG_SUFX?=3D .pkg > > > +# backward compatibility for users > > > +.if defined(_PKG_TRANSITIONING_TO_NEW_EXT) > > > .if defined(PKG_NOCOMPRESS) > > > -PKG_OLDSUFX?=3D .tar > > > +PKG_SUFX?=3D .tar > > > .else > > > -#.if ${OSVERSION} > 1400000 > > > -#PKG_OLDSUFX?=3D .tzst > > > -#.else > > > -PKG_OLDSUFX?=3D .txz > > > -#.endif > > > +PKG_SUFX?=3D .txz > > > .endif > > > +PKG_COMPRESSION_FORMAT?=3D ${PKG_SUFX:S/.//} > > > .else > > > +.if defined(PKG_SUFX) > > > +PKG_COMPRESSION_FORMAT?=3D ${PKG_SUFX:S/.//} > > > +.warning PKG_SUFX is defined, if should be replace with > > PKG_COMPRESSION_FORMAT > > > +.endif > > > +PKG_SUFX=3D .pkg > > > +.endif > > > > I love the idea here but can this be reverted for now please? Poudr= iere > > is not prepared for a default PKG_EXT change. > > > > 1. Now that it finds a new PKG_EXT of .pkg, it cannot find the exis= ting > > bootstrap pkg.txz because it looks for Latest/pkg.pkg -> > > ../All/pkg-*.pkg even though the *existing* one is .txz. I'm not su= re > > how this symlink is added before pkg is even built but it's not don= e in > > a compatible way. > > > > 2. The new PKG_EXT makes Poudriere now find no existing packages so > > everything is rebuilt. Not a deal breaker in itself but a problem f= or > > future builds... > > > > 3. The rebuild won't remove all the .txz files. I assume once `pkg = repo` > > runs it will find both .txz and .pkg files and cause duplicate vers= ion > > problems in `pkg repo` > > > > I see something creates a .txz -> .pkg symlink. What deles that sym= link > > if Poudriere only deletes/rebuilds the .pkg? > > > > *** > > "all those .txz entries are actually _duplicates_ in packagesite.ya= ml" > > seen on IRC just now by someone. > > I can confirm that looking in a packagesite.yaml I just generated. = Both > > the older .txz version of a port and a new revision-bumped one appe= ar in > > the yaml. > > *** > > > > 4. I've seen reports of packagesite.pkg which is concerning because > > Poudriere manages packagesite.txz with its own symlinks for > > hardlink-copied repositories. It does not use PKG_EXT there as they > > weren't packages, but I suspect we need special handling even with > > PKG_EXT added there. > > > > -- > > Regards, > > Bryan Drewery > > > > > -- > Regards, > Bryan Drewery >