Date: Fri, 26 Jan 2024 09:01:17 +0100 From: Alexander Leidinger <Alexander@Leidinger.net> To: Luca Pizzamiglio <pizzamig@freebsd.org> Cc: Stefan Esser <se@freebsd.org>, freebsd-ports <ports@freebsd.org>, portmgr <portmgr@freebsd.org>, FreeBSD Core Team <core@freebsd.org> Subject: Re: This is going to break port building without poudriere! Message-ID: <d6a7c9725edd734aca842d6ce85b0be2@Leidinger.net> In-Reply-To: <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com> References: <CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg@mail.gmail.com> <cd0c0cb0-6035-45b4-b3e8-d99115e6c013@FreeBSD.org> <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_afcc090eeb35f4cae43fa1bcd152af2b Content-Type: multipart/alternative; boundary="=_9bd8e07784132f49b91be5cdcb1f51e3" --=_9bd8e07784132f49b91be5cdcb1f51e3 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Am 2024-01-25 20:57, schrieb Luca Pizzamiglio: > Hi Stefan, > > I did reply to your first email, but not to your second one. > I preferred (in agreement with portmgr@) to open the discussion with > everyone, instead of keeping it just between you and me. > > As you can read in the email you have linked, I didn't ignore your > comments. > >> This implementation will break port dependencies, since there is no >> way >> a port can depend on a specific sub-package - there even is no way a >> non-default sub-package can be built without manual selection of the >> options that activate its creation. > > A port can depend on a specific sub-package. The category/origin~subpkg > is the chosen format. > Non-default subpackages shouldn't exist. In the aforementioned email, I > explicitly say that IF a subpackage is enabled by an option, the option > must be enabled by default. > Your previous comments contributed to making this point clearer. > >> Dependencies stated in the port Makefile are converted into package >> dependencies that can be resolved by the "pkg" command, but that >> cannot >> be directly used to build and install the requested sub-package from >> a port. > > I don't know where you get this idea, but it's not how it works. > If a port has a subpackage dependency, the category/origin~subpkg is > the chosen format to express that dependency. > The related port (category/origin) has to be built and installed. > By running `make install`, the whole port is installed, subpackages as > well. > The only "issue" I see is that via 'make install' you cannot install > only the subpackage, but the entire port only. > However this is not different than before. I think Stefans expectations about such a feature are different from the understand of the implementor of the feature in our tree. Somewhat a clash of an idealistic view and reality. To me (and I assume to Stefan too) it looks like slave ports will go away and subpackages will be used instead. A slave port may only compile a subset, and a subpackages aware port will compile everything (simplified view, not true where slave ports exclude a feature instead of excluding a file). From this point of view, a port can not depend on a subpackage in the sense of a port can not depend only on a subset of the php extensions included in the main php build (if/when it is converted to subpackages). As such a build from ports will have all php extensions included in the php port installed, whereas a pkp based install can limit the amount of installed php extensions to what is required. At least this is what I understand based upon what I have read about subpackages. As the documentation is not ready and I haven't looked at the code, this understanding may off course be wrong. >> Ports that do not create sub-packages can still be depended on by >> other >> ports, but as critical dependencies have been depended to sub-packages >> a ever large fraction of ports will only build in poudriere. > > Again, no. > If you run make install on any port, it will be built. The package and > the subpackages are all installed. > You cannot install a portion of a port, and it has never been possible. Is my understanding correct that subpackages aware ports are supposed to replace master/slave ports? In the sense of slave ports will be deleted once a port is converted to a subpackages aware port (except where slave ports exclude features from binaries like git-lite... yes, git-lite is implemented as a flavour and as such we cover this in a different way and such slave ports if they still exist can maybe be converted to use flavours)? I assume yes to both questions. With this assumption: With master/slave ports this is possible. And replacing master/slave ports with a subpackages aware port will remove this possibility. > Nothing is broken here and I fail to understand where you get the idea > of this behavior. > I wrote tests and examples to implement the feature and get them > working before adding support to poudriere. > >> This change does also obviously break port management tools like >> portmaster, >> which took me significant effort to adapt to FLAVOR support (which >> also had >> been implemented without consideration for other tools than >> poudriere), and >> which I have been maintaining since then. > > Yes, portmaster need to be able to parse the new dependencies format, > by removing the ~subpkg. By removing the suffix, you get the > category/port and everything is fine. > As announced, we are keeping subpackages adoption blocked with a > git-hook to give time to maintainers to add support to subpkg and to > introduce the feature slowly. > portmgr@ doesn't officially support neither portmaster nor portupgrade. > We simply lack the manpower to do it. > >> The reason given for sub-packages support is reduced build time for >> some >> packages that share a common distfile (e.g. qt5) when building >> official >> packages. > > As explained, in the short term there are this benefit and getting rid > of many master/slave ports (hard to maintain) > But there are several use cases in the future. > For instance, work is in place to provide debug symbols as subpackage, > in a similar way as pkg-base. > >> But this comes at a high cost for all builds outside the package build >> cluster, since now lots of unnecessary sub-trees will be compiled and >> installed, if only one program (i.e. sub-package) is desired. > > This is the reason why OPTIONS with SUBPACAKGES have been introduced, > to reduce build time for port builders. The complaint/fear of Stefan is that there does not seem to be a way to e.g.: 1) have php converted to a subpackages aware port which will remove some of the php extensions ports as they will be handled as subpackages of the lang/php port 2) depend upon one of the extensions which are now integrated as a subpackage of the php port but do not depend upon all extensions which are available as subpackages 3) as a result have all php extensions build and installed (and as such loaded at run time automatically), even those which are not needed, when installing via ports I think what Stefan is asking for, is to have a way that only build a subpackage if desired (default behavior = build everything, but the possibility to only build the subpackage specific part in case the upstream source allows this, like in the php example). >> You are breaking use-cases of a large number of users that still build >> ports without poudriere. This is especially causing a barrier to entry >> of new port developers, since this will require them to setup >> poudriere >> before they can begin port development. (This will only become an >> issue >> over time, as more and more dependencies will have been converted to >> sub-packages, but then there will be no way back.) > > I still fail to see what is going to break. Non-poudriere users can > experience longer build time for dependencies, but dependencies can be > installed via packages, at least most of the time. What is broken is that you _have_ to install such dependencies via packages in this case if you want to have the minimal install. With master/slave ports this is not required, the user is free to install everything as a port. "cd /usr/ports/port/which/depends/upon/a/subset/of/the/php/extensions; make install" will be different than "pkg install port/which/depends/upon/a/subset/of/the/php/extensions". And in this example this can also have a security implication (php stuff may be exposed to the internet and more extensions which can have a security issue may as such be exposed when installed as a port but not when installed as a package). Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_9bd8e07784132f49b91be5cdcb1f51e3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; charset= =3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family: Verdana,Gen= eva,sans-serif'> <p id=3D"reply-intro">Am 2024-01-25 20:57, schrieb Luca Pizzamiglio:</p> <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0"> <div id=3D"replybody1"> <div dir=3D"ltr"> <div dir=3D"ltr"> <div>Hi Stefan,</div> <div> </div> <div>I did reply to your first email, but not to your second one.<br />I pr= eferred (in agreement with portmgr@) to open the discussion with everyone, = instead of keeping it just between you and me.</div> <div> </div> <div>As you can read in the email you have linked, I didn't ignore your com= ments.</div> </div> <div> </div> <div class=3D"v1gmail_quote"> <blockquote class=3D"v1gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; bor= der-left: 1px solid #cccccc; padding-left: 1ex;">This implementation will b= reak port dependencies, since there is no way<br />a port can depend on a s= pecific sub-package - there even is no way a<br />non-default sub-package c= an be built without manual selection of the<br />options that activate its = creation.</blockquote> <div>A port can depend on a specific sub-package. The category/origin~subpk= g is the chosen format.</div> <div>Non-default subpackages shouldn't exist. In the aforementioned email, = I explicitly say that IF a subpackage is enabled by an option, the option m= ust be enabled by default.<br />Your previous comments contributed to makin= g this point clearer.</div> <div> </div> <blockquote class=3D"v1gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; bor= der-left: 1px solid #cccccc; padding-left: 1ex;">Dependencies stated in the= port Makefile are converted into package<br />dependencies that can be res= olved by the "pkg" command, but that cannot<br />be directly used to build = and install the requested sub-package from<br />a port.</blockquote> <div>I don't know where you get this idea, but it's not how it works.</div> </div> <div>If a port has a subpackage dependency, the category/origin~subpkg is t= he chosen format to express that dependency.<br />The related port (categor= y/origin) has to be built and installed.</div> <div>By running `make install`, the whole port is installed, subpackages as= well.</div> <div>The only "issue" I see is that via 'make install' you cannot install o= nly the subpackage, but the entire port only.</div> <div>However this is not different than before.</div> </div> </div> </blockquote> <div id=3D"replybody1"> <div dir=3D"ltr"> <div> </div> <div>I think Stefans expectations about such a feature are different from t= he understand of the implementor of the feature in our tree. Somewhat a cla= sh of an idealistic view and reality.</div> <div> </div> <div>To me (and I assume to Stefan too) it looks like slave ports will go a= way and subpackages will be used instead. A slave port may only compile a s= ubset, and a subpackages aware port will compile everything (simplified vie= w, not true where slave ports exclude a feature instead of excluding a file= ). From this point of view, a port can not depend on a subpackage in the se= nse of a port can not depend only on a subset of the php extensions include= d in the main php build (if/when it is converted to subpackages). As such a= build from ports will have all php extensions included in the php port ins= talled, whereas a pkp based install can limit the amount of installed php e= xtensions to what is required. At least this is what I understand based upo= n what I have read about subpackages. As the documentation is not ready and= I haven't looked at the code, this understanding may off course be wrong.<= /div> <div> </div> </div> </div> <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0"> <div id=3D"replybody1"> <div dir=3D"ltr"> <div class=3D"v1gmail_quote"> <blockquote class=3D"v1gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; bor= der-left: 1px solid #cccccc; padding-left: 1ex;">Ports that do not create s= ub-packages can still be depended on by other<br />ports, but as critical d= ependencies have been depended to sub-packages<br />a ever large fraction o= f ports will only build in poudriere. </blockquote> <div>Again, no.</div> <div>If you run make install on any port, it will be built. The package and= the subpackages are all installed.</div> <div>You cannot install a portion of a port, and it has never been possible= =2E</div> </div> </div> </div> </blockquote> <div id=3D"replybody1"> <div dir=3D"ltr"> <div class=3D"v1gmail_quote"> <div> </div> <div>Is my understanding correct that subpackages aware ports are supposed = to replace master/slave ports? In the sense of slave ports will be deleted = once a port is converted to a subpackages aware port (except where slave po= rts exclude features from binaries like git-lite... yes, git-lite is implem= ented as a flavour and as such we cover this in a different way and such sl= ave ports if they still exist can maybe be converted to use flavours)? I as= sume yes to both questions. With this assumption:</div> <div> </div> <div>With master/slave ports this is possible. And replacing master/slave p= orts with a subpackages aware port will remove this possibility.</div> <div> </div> </div> </div> </div> <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0"> <div id=3D"replybody1"> <div dir=3D"ltr"> <div class=3D"v1gmail_quote"> <div>Nothing is broken here and I fail to understand where you get the idea= of this behavior.</div> <div>I wrote tests and examples to implement the feature and get them worki= ng before adding support to poudriere.</div> <div> </div> <blockquote class=3D"v1gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; bor= der-left: 1px solid #cccccc; padding-left: 1ex;">This change does also obvi= ously break port management tools like portmaster,<br />which took me signi= ficant effort to adapt to FLAVOR support (which also had<br />been implemen= ted without consideration for other tools than poudriere), and<br />which I= have been maintaining since then.</blockquote> <div>Yes, portmaster need to be able to parse the new dependencies format, = by removing the ~subpkg. By removing the suffix, you get the category/port = and everything is fine.</div> <div>As announced, we are keeping subpackages adoption blocked with a git-h= ook to give time to maintainers to add support to subpkg and to introduce t= he feature slowly.</div> <div>portmgr@ doesn't officially support neither portmaster nor portupgrade= =2E We simply lack the manpower to do it.</div> <div> </div> <blockquote class=3D"v1gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; bor= der-left: 1px solid #cccccc; padding-left: 1ex;">The reason given for sub-p= ackages support is reduced build time for some<br />packages that share a c= ommon distfile (e.g. qt5) when building official<br />packages.</blockquote> <div>As explained, in the short term there are this benefit and getting rid= of many master/slave ports (hard to maintain)</div> <div>But there are several use cases in the future.</div> <div>For instance, work is in place to provide debug symbols as subpackage,= in a similar way as pkg-base.</div> <div> </div> <blockquote class=3D"v1gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; bor= der-left: 1px solid #cccccc; padding-left: 1ex;">But this comes at a high c= ost for all builds outside the package build<br />cluster, since now lots o= f unnecessary sub-trees will be compiled and<br />installed, if only one pr= ogram (i.e. sub-package) is desired.</blockquote> <div>This is the reason why OPTIONS with SUBPACAKGES have been introduced, = to reduce build time for port builders.</div> </div> </div> </div> </blockquote> <div id=3D"replybody1"> <div dir=3D"ltr"> <div class=3D"v1gmail_quote"> <div> </div> <div>The complaint/fear of Stefan is that there does not seem to be a way t= o e.g.:</div> <div> 1) have php converted to a subpackages aware port which will rem= ove some of the php extensions ports as they will be handled as subpackages= of the lang/php port</div> <div> 2) depend upon one of the extensions which are now integrated as= a subpackage of the php port but do not depend upon all extensions which a= re available as subpackages</div> <div> 3) as a result have all php extensions build and installed (and = as such loaded at run time automatically), even those which are not needed,= when installing via ports</div> <div> </div> <div>I think what Stefan is asking for, is to have a way that only build a = subpackage if desired (default behavior =3D build everything, but the possi= bility to only build the subpackage specific part in case the upstream sour= ce allows this, like in the php example).</div> <div> </div> </div> </div> </div> <blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2= px solid; margin: 0"> <div id=3D"replybody1"> <div dir=3D"ltr"> <div class=3D"v1gmail_quote"> <blockquote class=3D"v1gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; bor= der-left: 1px solid #cccccc; padding-left: 1ex;">You are breaking use-cases= of a large number of users that still build<br />ports without poudriere. = This is especially causing a barrier to entry<br />of new port developers, = since this will require them to setup poudriere<br />before they can begin = port development. (This will only become an issue<br />over time, as more a= nd more dependencies will have been converted to<br />sub-packages, but the= n there will be no way back.)</blockquote> <div>I still fail to see what is going to break.</div> Non-poudriere users can experience longer build time for dependencies, but = dependencies can be installed via packages, at least most of the time.</div> </div> </div> </blockquote> <div id=3D"replybody1"> <div dir=3D"ltr"> <div class=3D"v1gmail_quote"> </div> <div class=3D"v1gmail_quote">What is broken is that you _have_ to install s= uch dependencies via packages in this case if you want to have the minimal = install. With master/slave ports this is not required, the user is free to = install everything as a port. "cd /usr/ports/port/which/depends/upon/a/subs= et/of/the/php/extensions; make install" will be different than "pkg install= port/which/depends/upon/a/subset/of/the/php/extensions". And in this examp= le this can also have a security implication (php stuff may be exposed to t= he internet and more extensions which can have a security issue may as such= be exposed when installed as a port but not when installed as a package).<= /div> <div class=3D"v1gmail_quote"> </div> <div class=3D"v1gmail_quote">Bye,<br />Alexander.</div> <div class=3D"v1gmail_quote"> </div> </div> </div> <div id=3D"signature">-- <br /> <div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">= <a href=3D"http://www.Leidinger.net" target=3D"_blank" rel=3D"noopener nore= ferrer">http://www.Leidinger.net</a>; <a href=3D"mailto:Alexander@Leidinger.= net:">Alexander@Leidinger.net:</a> PGP 0x8F31830F9F2772BF<br /><a href=3D"h= ttp://www.FreeBSD.org" target=3D"_blank" rel=3D"noopener noreferrer">http:/= /www.FreeBSD.org</a> <a href=3D"mailto:netchild@FreeBSD.org">n= etchild@FreeBSD.org</a> : PGP 0x8F31830F9F2772BF</div> </div> </body></html> --=_9bd8e07784132f49b91be5cdcb1f51e3-- --=_afcc090eeb35f4cae43fa1bcd152af2b Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmWzZt4ACgkQEg2wmwP4 2IZqIQ//S2Wka4A3HjAcX5aEPJ++eST8tyvs6Eq+4wXp1oumgL7378cilqNG5NMB QZmi9CoX3OxTJhbmL/HM6iKCc79Nu2cIuBMk6AbaGWG3VzeZIYTpOVQJUtJWgHD1 8TelAR+iiwb7fe2hUjEqaYekYl4bJu+oQxuX0GpPFFaw0UQJp7Fccw2KxISHe/L9 MOMsdqjvh0qj05Lmr94a14xdzAKWW6g5TUDdeSnqq4WEWs1tm/RzQy7EuiC+Wohp Mbw+dt3uO2TTQrJ0wdv/txxsmESkg5lUicx2zkl8bqZosEsOmvbYjR0vnPnclQj/ v6514e4VL38nobJqniC92TvjEioWztzx9vckp72d90BEiqIJQlSdixOmP41IXlyX f/2vhWmoBnJH/Rt2/j2Big9ONnvHF69Q4jkAU3PIMu3GSqBCoH8UNZVwYUqXnBM3 R8N8xTKkjjoh/1Dn0him7JjGWbJoqK/3xhav/cKogUVL3kA5Kwizz3GVBfalreKc UGC/2ctQ9G17+WPe3hRD+MR6JsCaW+HwoVL2vmDa1aQyBrWQK8seyBiHrUdFtguB yKYAWrN9ybj72wh6LppTiF0WpvoPcDWHCo53RhRcyYo/IKw0hT69FFQt+HXdkqWs Lh/MT2blx6mfJwt6mRlJZIBEAq03LssKfIWyqMw+oETlq0QIiLY= =tIcU -----END PGP SIGNATURE----- --=_afcc090eeb35f4cae43fa1bcd152af2b--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d6a7c9725edd734aca842d6ce85b0be2>