Date: Sat, 24 Jul 2021 15:24:19 -0700 From: Bryan Drewery <bdrewery@FreeBSD.org> To: Kevin Bowling <kevin.bowling@kev009.com> Cc: Baptiste Daroussin <bapt@freebsd.org>, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org, ports-committers@freebsd.org Subject: Re: git: e497a16a2869 - main - ports-mgmt/pkg*: Release 1.17.0 Message-ID: <e86c9567-afd0-3ec0-8118-0d3143f8c36e@FreeBSD.org> In-Reply-To: <CAK7dMtC%2BGgQmWaGOBK434h_zHhrt=5=G1CwV7dBpE0wxBJcaNw@mail.gmail.com> References: <202107220806.16M862GC052436@gitrepo.freebsd.org> <f5d0be1a-7f7c-7dd5-61d0-f88328847722@FreeBSD.org> <CAK7dMtC%2BGgQmWaGOBK434h_zHhrt=5=G1CwV7dBpE0wxBJcaNw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --OJyh7sYpT2911NYcZpaVuK3vqIU7I6rpg Content-Type: multipart/mixed; boundary="qS70xC1GGjYYmZiNyhLR6RvnUzWMg51Jk"; protected-headers="v1" From: Bryan Drewery <bdrewery@FreeBSD.org> To: Kevin Bowling <kevin.bowling@kev009.com> Cc: Baptiste Daroussin <bapt@freebsd.org>, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org, ports-committers@freebsd.org Message-ID: <e86c9567-afd0-3ec0-8118-0d3143f8c36e@FreeBSD.org> Subject: Re: git: e497a16a2869 - main - ports-mgmt/pkg*: Release 1.17.0 References: <202107220806.16M862GC052436@gitrepo.freebsd.org> <f5d0be1a-7f7c-7dd5-61d0-f88328847722@FreeBSD.org> <CAK7dMtC+GgQmWaGOBK434h_zHhrt=5=G1CwV7dBpE0wxBJcaNw@mail.gmail.com> In-Reply-To: <CAK7dMtC+GgQmWaGOBK434h_zHhrt=5=G1CwV7dBpE0wxBJcaNw@mail.gmail.com> --qS70xC1GGjYYmZiNyhLR6RvnUzWMg51Jk Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable We have a storm brewing on the package builders right now. They will soon be serving repositories with duplicated/multiple versions of package= s. Frankly your response only angered me as it was dismissive and focus stealing top-post. The details I gave are backed up by many reports coming in on IRC and github. If you look into your packagesite.yaml and inspect for both .txz and .pkg you may find duplicates. I have a life and work priorities of my own. I can't drop everything to fix this right now. I am skeptical of any fix that does not meticulously handle the transition and simply tosses in an unlink. Meaning I will want to see or do my own testing. If I had known the default PKG_EXT was changing I would have thrown up my arms and blocked this and asked for a migration period. Poudriere isn't even all there is. I think users would appreciate having a tool that does the right thing when the change comes in rather than being forced to go through the wrong broken thing to get the fixed tool. The longer this is in the bigger the damage is. On 7/24/2021 2:50 PM, Kevin Bowling wrote: > It=E2=80=99s all working fine for me on main and 13.0.=C2=A0 Which of t= hese issues > cannot be solved in the next few days? >=20 > On Sat, Jul 24, 2021 at 1:36 PM Bryan Drewery <bdrewery@freebsd.org > <mailto:bdrewery@freebsd.org>> wrote: >=20 > On 7/22/2021 1:06 AM, Baptiste Daroussin wrote: > > The branch main has been updated by bapt: > > > > URL: > https://cgit.FreeBSD.org/ports/commit/?id=3De497a16a286972bfcab9082= 09b11ee6a13d99dc9 > <https://cgit.FreeBSD.org/ports/commit/?id=3De497a16a286972bfcab908= 209b11ee6a13d99dc9> > > > > commit e497a16a286972bfcab908209b11ee6a13d99dc9 > > Author:=C2=A0 =C2=A0 =C2=A0Baptiste Daroussin <bapt@FreeBSD.org> > > AuthorDate: 2021-07-22 07:54:38 +0000 > > Commit:=C2=A0 =C2=A0 =C2=A0Baptiste Daroussin <bapt@FreeBSD.org> > > CommitDate: 2021-07-22 08:05:59 +0000 > > > >=C2=A0 =C2=A0 =C2=A0ports-mgmt/pkg*: Release 1.17.0 > >=C2=A0 =C2=A0 =C2=A0 > >=C2=A0 =C2=A0 =C2=A0TL;DR: new uniq package file extension: '.pkg'= > >=C2=A0 =C2=A0 =C2=A0 > >=C2=A0 =C2=A0 =C2=A0Full changelog for pkg: > >=C2=A0 =C2=A0 =C2=A0- add a new "snap" prefix, to deal with pkg ba= se snapshots > >=C2=A0 =C2=A0 =C2=A0- pkg repo now accepts packages with different= compression formats > >=C2=A0 =C2=A0 =C2=A0- pkg now have a single extension: ".pkg" and = a backward > compatible > >=C2=A0 =C2=A0 =C2=A0 =C2=A0symlink is created when the package is = created to help > transitioning > >=C2=A0 =C2=A0 =C2=A0- Default compression level for zstd is now se= t to 19 > >=C2=A0 =C2=A0 =C2=A0- the default compression level is now a confi= guration option > >=C2=A0 =C2=A0 =C2=A0- plenty of portability fixes > >=C2=A0 =C2=A0 =C2=A0- fix plenty of typos > >=C2=A0 =C2=A0 =C2=A0- expose the name of the package to lua and sh= ell scripts > >=C2=A0 =C2=A0 =C2=A0- plist: > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove internal support for @*exec (thi= s is now in the ports > tree) > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove support for @ignore > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove support for @dirrm/@dirrmtry > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove support for @pkgdep > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove stub support for @stopdaemon > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove stub support for @display > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove stub support for @mtree > >=C2=A0 =C2=A0 =C2=A0 =C2=A0remove stub support for @conflict > >=C2=A0 =C2=A0 =C2=A0- VUXML: fetch .xz compressed version of the f= ile by default > >=C2=A0 =C2=A0 =C2=A0- triggers are deferred to later command (firs= tboot is > planned) if run > >=C2=A0 =C2=A0 =C2=A0 =C2=A0with pkg -r > >=C2=A0 =C2=A0 =C2=A0- pkg triggers commands has been added and can= execute the > deferred > >=C2=A0 =C2=A0 =C2=A0 =C2=A0triggers for example at firstboot > >=C2=A0 =C2=A0 =C2=A0 > >=C2=A0 =C2=A0 =C2=A0Changes in the framework: > >=C2=A0 =C2=A0 =C2=A0- Add a backward compatibility layer, so peopl= e are not forced > to move > >=C2=A0 =C2=A0 =C2=A0 =C2=A0to pkg 1.17 yet (new quarter we will en= force the switch for > triggers > >=C2=A0 =C2=A0 =C2=A0 =C2=A0anyway, but it gives time to people to = switch) > >=C2=A0 =C2=A0 =C2=A0- Introduce a new PKG_COMPRESSION_FORMAT for u= sers to specify the > >=C2=A0 =C2=A0 =C2=A0 =C2=A0compression format they are willing to = use when creating > packages: > >=C2=A0 =C2=A0 =C2=A0 =C2=A0txz, tar, tgz, tbz, tzst are the valid = ones (note that tzst > is only > >=C2=A0 =C2=A0 =C2=A0 =C2=A0usable on FreeBSD 13 and 14) > >=C2=A0 =C2=A0 =C2=A0- Add a backward compatibility for people who = specified the > compression > >=C2=A0 =C2=A0 =C2=A0 =C2=A0format already via PKG_SUFX and issue a= warning to tell them > about the > >=C2=A0 =C2=A0 =C2=A0 =C2=A0deprecation of PKG_SUFX > > --- > >=C2=A0 Mk/bsd.port.mk <http://bsd.port.mk>=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | 58 > +++++++++++++++++++++++-------------------- > >=C2=A0 ports-mgmt/pkg-devel/Makefile |=C2=A0 4 +-- > >=C2=A0 ports-mgmt/pkg-devel/distinfo |=C2=A0 6 ++--- > >=C2=A0 ports-mgmt/pkg/Makefile=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 2= +- > >=C2=A0 ports-mgmt/pkg/distinfo=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 6= ++--- > >=C2=A0 5 files changed, 40 insertions(+), 36 deletions(-) > > > > diff --git a/Mk/bsd.port.mk <http://bsd.port.mk>; b/Mk/bsd.port.mk= > <http://bsd.port.mk>; > > index 228644a1845f..5eba5d89da95 100644 > > --- a/Mk/bsd.port.mk <http://bsd.port.mk>; > > +++ b/Mk/bsd.port.mk <http://bsd.port.mk>; > > @@ -1211,6 +1211,14 @@ _OSVERSION_MAJOR=3D=C2=A0 =C2=A0 =C2=A0 > ${OSVERSION:C/([0-9]?[0-9])([0-9][0-9])[0-9]{3}/\1/} > >=C2=A0 .if !defined(_PKG_VERSION) > >=C2=A0 _PKG_VERSION!=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0${PKG_BIN} -v > >=C2=A0 .endif > > +# XXX hack for smooth transition towards pkg 1.17 > > +_PKG_BEFORE_PKGEXT!=3D ${PKG_BIN} version -t > ${_PKG_VERSION:C/-.*//g} 1.17.0 > > +.if ${_PKG_BEFORE_PKGEXT} =3D=3D "<" > > +_PKG_TRANSITIONING_TO_NEW_EXT=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0yes > > +_EXPORTED_VARS+=3D=C2=A0 =C2=A0 =C2=A0_PKG_TRANSITIONING_TO_NEW_= EXT > > +.warning "It is strongly recommanded to upgrade to a newer > vertion of pkg first" > > +.endif > > +# XXX End of hack > >=C2=A0 _PKG_STATUS!=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0 ${PKG_BIN} versi= on -t > ${_PKG_VERSION:C/-.*//g} ${MINIMAL_PKG_VERSION} > >=C2=A0 .if ${_PKG_STATUS} =3D=3D "<" > >=C2=A0 IGNORE=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 p= kg(8) must be version > ${MINIMAL_PKG_VERSION} or greater, but you have ${_PKG_VERSION}. Yo= u > must upgrade the ${PKG_ORIGIN} port first > > @@ -2227,23 +2235,29 @@ _PKGMESSAGES+=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0= ${PKGMESSAGE} > >=C2=A0 > >=C2=A0 TMPPLIST?=3D=C2=A0 =C2=A0${WRKDIR}/.PLIST.mktmp > >=C2=A0 > > -.if ${WITH_PKG} =3D=3D devel > > -PKG_SUFX?=3D=C2=A0 =C2=A0.pkg > > +# backward compatibility for users > > +.if defined(_PKG_TRANSITIONING_TO_NEW_EXT) > >=C2=A0 .if defined(PKG_NOCOMPRESS) > > -PKG_OLDSUFX?=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0 .tar > > +PKG_SUFX?=3D=C2=A0 =C2=A0.tar > >=C2=A0 .else > > -#.if ${OSVERSION} > 1400000 > > -#PKG_OLDSUFX?=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0.tzst > > -#.else > > -PKG_OLDSUFX?=3D=C2=A0 =C2=A0 =C2=A0 =C2=A0 .txz > > -#.endif > > +PKG_SUFX?=3D=C2=A0 =C2=A0.txz > >=C2=A0 .endif > > +PKG_COMPRESSION_FORMAT?=3D=C2=A0 =C2=A0 =C2=A0${PKG_SUFX:S/.//} > >=C2=A0 .else > > +.if defined(PKG_SUFX) > > +PKG_COMPRESSION_FORMAT?=3D=C2=A0 =C2=A0 =C2=A0${PKG_SUFX:S/.//} > > +.warning PKG_SUFX is defined, if should be replace with > PKG_COMPRESSION_FORMAT > > +.endif > > +PKG_SUFX=3D=C2=A0 =C2=A0 .pkg > > +.endif >=20 > I love the idea here but can this be reverted for now please? Poudr= iere > is not prepared for a default PKG_EXT change. >=20 > 1. Now that it finds a new PKG_EXT of .pkg, it cannot find the exis= ting > bootstrap pkg.txz because it looks for Latest/pkg.pkg -> > ../All/pkg-*.pkg even though the *existing* one is .txz. I'm not su= re > how this symlink is added before pkg is even built but it's not don= e in > a compatible way. >=20 > 2. The new PKG_EXT makes Poudriere now find no existing packages so= > everything is rebuilt. Not a deal breaker in itself but a problem f= or > future builds... >=20 > 3. The rebuild won't remove all the .txz files. I assume once `pkg = repo` > runs it will find both .txz and .pkg files and cause duplicate vers= ion > problems in `pkg repo` >=20 > I see something creates a .txz -> .pkg symlink. What deles that sym= link > if Poudriere only deletes/rebuilds the .pkg? >=20 > *** > "all those .txz entries are actually _duplicates_ in packagesite.ya= ml" > seen on IRC just now by someone. > I can confirm that looking in a packagesite.yaml I just generated. = Both > the older .txz version of a port and a new revision-bumped one appe= ar in > the yaml. > *** >=20 > 4. I've seen reports of packagesite.pkg which is concerning because= > Poudriere manages packagesite.txz with its own symlinks for > hardlink-copied repositories. It does not use PKG_EXT there as they= > weren't packages, but I suspect we need special handling even with > PKG_EXT added there. >=20 > --=20 > Regards, > Bryan Drewery >=20 --=20 Regards, Bryan Drewery --qS70xC1GGjYYmZiNyhLR6RvnUzWMg51Jk-- --OJyh7sYpT2911NYcZpaVuK3vqIU7I6rpg Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEE+Rc8ssOq6npcih8JNddxu25Gl88FAmD8kxMFAwAAAAAACgkQNddxu25Gl8+Y cwf8C5b9d/OARB1EUlwZILRlbWg5Euc9qO21/azd2adtvBSBkzSzqC1klQEyugp3wtgcrqp/wGX/ b8Y+v86Mxu+O9LpxFbHRzlbF+SoseknZE5EGIBBDxlp1xPFQdWLqONx91TXVlr5GvgciC2dSB/3s vwveHglWtsIM0CtboHuULGWm0BzaOkNnz1qo0gCCDi3A3lrLqtan1dl1D+t1T8HsB6KF/rpcK9rm nVvtFmFDD4tZtehtazgvbGG4QFL+OrPal9Z+xw+YXOQib5gbz2+VbkYZ0TsM6ydd+S+Jmg7qNh07 Bm9KyoQsPR4UnkcBNMd1sSvzqOGToO4K4LYvItwNfA== =qWIj -----END PGP SIGNATURE----- --OJyh7sYpT2911NYcZpaVuK3vqIU7I6rpg--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e86c9567-afd0-3ec0-8118-0d3143f8c36e>