Date: Tue, 30 Jan 2024 14:21:03 +0100 From: Alexander Leidinger <Alexander@Leidinger.net> To: Luca Pizzamiglio <pizzamig@freebsd.org> Cc: FreeBSD ports <freebsd-ports@freebsd.org>, portmgr <portmgr@freebsd.org>, FreeBSD Core Team <core@freebsd.org> Subject: Re: This is going to break port building without poudriere! Message-ID: <e3571856669357edaeed847f9a59859d@Leidinger.net> In-Reply-To: <CAB88xy_2W2%2BZGraFKSsNxBOKTy0vThCd=6nuTDnpVkBeDpQ7ow@mail.gmail.com> References: <CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg@mail.gmail.com> <cd0c0cb0-6035-45b4-b3e8-d99115e6c013@FreeBSD.org> <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com> <609374cf-7f1f-4643-8379-f368b23ccb09@freebsd.org> <CAB88xy-AkwJWhUtdfjernOumzshahJ_KRH1wOqRROceFbq3=GA@mail.gmail.com> <1e27bce44f2ba85f0097fbc6aba1dac1@Leidinger.net> <CAB88xy-DesXEH9natthizUYOayw=nmGXXm-WzYgYVOkaE=81kA@mail.gmail.com> <282c88d38f7d9125a3cfad911370aa40@Leidinger.net> <CAB88xy_2W2%2BZGraFKSsNxBOKTy0vThCd=6nuTDnpVkBeDpQ7ow@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_deba9ff91a50bd5416fe042e22d25a98 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed Am 2024-01-30 13:24, schrieb Luca Pizzamiglio: > Hi Alexander, > > On Tue, Jan 30, 2024 at 11:31 AM Alexander Leidinger > <Alexander@leidinger.net> wrote: > >> Am 2024-01-30 10:42, schrieb Luca Pizzamiglio: >> >>> Hi Alexander. >>> >>> Subpackage modularization of existing ports (i.e. qt6-tools) provides >>> benefits to package users/builders: smaller dependency footprint, >>> smaller usage on disk (useful for embedded systems), smaller jails. >>> There are no benefits nor regressions for port builders for this >>> specific use case. >> >> Nicely worded. I agree to that. At the same time it is a regression >> for >> port builders. Installing from ports is not on par with installing >> from >> pkg anymore in this case. What you describe means we can use >> subpackages >> only for leaf ports. Ports which are a dependency can not converted to >> subpackages (in the sense of "no slave port available for the >> subpackages"). > > I disagree that this is regression, but I agree that it's an unexpected > change. And at the same time you talk about a breaking change in the next part. > Installing from ports can install more than installing from pkg. > I don't call it a regression, as it doesn't introduce bugs, ports can > be successfully built as before. > However, it introduces a divergence that requires more attention. A divergence which requires more attention is a nice way of telling it is a regression. >>> On the other hand, moving master/slave ports to subpackages comes >>> with >>> the cost of losing the ability to build the slave ports >>> independently. >>> For package users there is no change and some benefit for package >>> builders. >>> The issue can be mitigated by introducing options to select which >>> subpackage to build (available for make install as well), but, still, >>> a >>> single subpackage cannot be built independently. >> >> This is not a proper mitigation. I used mail/roundcube and lang/php as >> a >> specific example for this particular case where it is not a proper >> mitigation. > > It's a mitigation, as you can reduce what you need to build, but it's > not a solution and it doesn't fix the breaking change. > It only offers a way to install less, but no way to install a > subpackage only. > >>> This limitation _can_ be unacceptable in some cases. For example, >>> bofh@, the php maintainer, considers subpackages not the way to go, >>> so >>> the master/slave approach will stay. >> >> Do you see that you just told that one of the best role models for >> subpackages according to you previous mail will not use subpackages >> because of the limitations I highlighted? > > Yes, your arguments contributed to change my opinion on this point. Is > this a problem? No. >>> We introduced subpackages in the framework to explore all use cases, >>> by >>> trying and testing its adoption. >>> By doing so we have already identified some issues (i.e. USES is not >>> subpackage-aware yet) and interesting new use cases (subpackage with >>> debug symbols). >> >> You are surely aware that you haven't mentioned one of the points I >> talked about as an issue, don't you? I have not seen any technical >> answer which shows that my technical description of issues is wrong. I >> want to make you aware that I understand the responses from you and >> the >> others in this thread as: "we do not care about those regressions and >> do >> not want to fix them" (I understand it like that because I see no >> acknowledgement of those issues, just answers which look like a >> smokescreen and pointing into directions of good faith). If you would >> acknowledge the technical issues highlighted in this thread (or show >> evidence that those regressions can not happen) and tell that the >> adoption of subpackages is monitored to not introduce those >> regressions >> I talk about, my understanding of the situation would change. > > So you are saying that my answers are "smokescreen and pointing into > directions of good faith". > Am I trying to deceive you? Are you assuming that I have an ill intent > that needs to be masked? > Those seem serious accusations and they trouble me. I do not think you try to deceive me or anyone else. I do not think you have ill intents. When we met some years ago in Essen I got a favorable impression about you (I remember you as smart and nice) and this has not changed (and you can be assured that I'm not angry or similar, and that you can imagine my "voice" in this thread as calm as in my interaction with everyone back then in Essen). What I "accuse" (to pick up one of your words, I rather call it "giving the impression in your answers") you is that you postulate that subpackages as implemented does not create issues or play down the issues when converting master/slave ports to subpackages, or implicitly tell that "make install" is not worth it / something you support / ... and people shall use packages instead (I use packages myself in most cases, but sometimes "make install" is the most pragmatic solution, and then users shall not fall into a trap... "make install" is an asset, not a burden). > As per: "we don't care about those regressions and do not want to fix > them". > I've already acknowledged that subpackages are going to introduce a > breaking change. > It's possible to have different outcomes if you use packages and ports. We always had different outcomes regarding build dependencies. Having those differences is not the point. The point is that this change will cripple the ports collection. Converting a port to make use of subpackages removes the possibility to have fine grained control over what is installed which is exposed to a sensible security context (aka the internet). Yes, sorry, I'm riding on the lang/php example to death... it's a good example. > We are going to have a portmgr meeting soon and we will see how to > proceed. I'm looking forward to the outcome. > The solutions you propose (subpackge+slave port) don't make any sense > to me, I would prefer to focus on a long term solution. It doesn't make sense to me either, but is the only way the current implementation of subpackages will not cripple the functionality of "make install" in the mid-term. I also would prefer a long term solution. Namely making subpackages first class items for pkg (= no different naming -> s/~/-/, and having the origin recorded inside the metadata), and having the possibility to target subpackages for a make install (as a quick idea: introduce "make install TARGET_SUBPACKAGES=a,b,c", have the port build as needed (full or only the subpackage), build the subpackage out of the plist, and only install the subpackage with pkg). > Technically, it should be possible to install a portion of a port. > It's more difficult to build only a part of a port, but this aspect is > less relevant. Only building the subpackage is an optimization problem (for lang/php this is possible, for other ports maybe not), not a problem of a crippled feature. Yes, it would be nice if it is possible (if upstream allows it) to build subpackages, but this is not something I would ask portmgr to policy. But I ask portmgr to policy the security relevant features, as an example "make install" of mail/roundcube (or take any other port which doesn't require lang/php but falls into the same category of security implications). >>> As per master/slave merge use case, we will let the maintainers >>> decide >>> if they want to move forward with subpackage adoptions, knowing the >>> regression for port builders, but we won't push them in this >>> direction. >> >> I consider it unfortunate that you describe it like that. I was hoping >> you would tell that portmgr will prevent the removal of slave packages >> and enforce the rule of having slave ports for subpackages aware ports >> to prevent regressions for users which install from ports (until the >> technical valid issues I have pointed out are resolved -- and they can >> be fixed, a first step would be to make the names of subpackages match >> normal packages names = replacing the '~' in the name with a '-' ASAP >> to >> prevent churn later). >> >> Note, in src some big discussion like this of having several >> committers >> identify regressions and no immediate fix would lead to a backout >> until >> it is fixed. I do not ask for a backout of this, but I ask for a >> strong >> lock/policy by portmgr on the subpackages feature like I already >> described in my previous mails (until the regressions the use of >> subpackages would create are fixed). This would allow for more >> experimentation by a lot of people without the need to patch the ports >> tree and without introducing regressions for a simple "make install" >> of >> ports with a lot of dependencies. > > Are you asking to completely block the introduction of subpackages? A > complete block seems excessive to me, but some additional restriction > can be considered. I ask portmgr to - not accept conversions of existing ports to subpackages which remove slave ports (and as such touches the dependencies of other ports). - not accept the removal of slave ports for existing subpackages. - not accept new ports with subpackages without slave ports to be able to target the subpackages for "make install". All this until the implementation of subpackages is fixed in a way that we can make lang/php a subpackages aware port and do not install more dependencies in "make install" in mail/roundcube. Again, lang/php serves here as an example of a perfect-fit port for subpackages, and at the same time a perfect example of the issues we get on "make install". So no, I do not ask to block it completely. I simply ask to not remove features. I'm aware that this introduces a burden on package building. If those which operate a package building infrastructure want to have some more restrictions because of this, then this is off course at the discretion of them to speak up. Personally I'm willing to take the hit for my poudriere runs, if this allows to improve the subpackages feature in the tree. I additionally suggest to rethink the naming scheme of subpackages (replacing the '~' with a '-' to have the same filename on disk for the slave ports and the corresponding subpackage) before new subpackages aware ports are introduced. In my opinion this facilitates future updates / improvements to the current implementation of subpackages. I'm open to discuss specifics, but I suggest to open a new thread for this... in case there is interest to discuss it. > In any case, an update on the subpackage topic needs to be shared in > the next few days. I'm looking forward to it. I consider subpackages a very nice feature in the general case. I have simply seen some bugs/issues/regressions in the current implementation which make me believe it is not yet ready to let us use it without the above mentioned governance until those issues are fixed. Bye, Alexander. > Best regards, > Luca > >> Bye, >> Alexander. >> >>> Best regards, >>> Luca >>> >>> On Mon, Jan 29, 2024 at 10:04 AM Alexander Leidinger >>> <Alexander@leidinger.net> wrote: >>> >>>> Am 2024-01-27 16:59, schrieb Luca Pizzamiglio: >>>> >>>>> Hi Stefan. >>>>> >>>>> Let's start from the beginning, as it seems that things are not >>>>> clear. >>>>> >>>>> Subpackages is a feature to create multiple packages from one build >>>>> Those subpackages can depend on the main package. >>>>> The main package cannot depend on any subpackages. >>>>> This limits the cases where subpackages can be applied. The main >>>>> package MUST be independent from its subpackages. Subpackages can >>>>> only >>>>> add features (like a plugin). >>>>> To recall your NLS example (ref >>>>> https://reviews.freebsd.org/D16457#715443): this is not a use-case >>>>> for >>>>> subpackages. If the main program/library needs to be compiled >>>>> differently with or without NLS, this is not viable for >>>>> subpackages. >>>>> If a port needs to be built multiple times to create different >>>>> subpackages, this is not a viable case for subpackages. >>>>> A good candidate is qt6-tools: this port contains multiple tools >>>>> (designer, linguist, help, and so on). Those tools could be put in >>>>> different subpackages. >>>>> >>>>> I hope this explanation helps to clarify and address some of your >>>>> concerns. >>>> >>>> As I read this, lang/php is the best example of a port where >>>> subpackages >>>> has a benefit (in the sense it matches your description above to the >>>> letter, the main port independent from the subpacakges, and what can >>>> be >>>> build as subpackages is a plugin/extension). >>>> >>>>> OPTIONS and SUBPACKAGES >>>>> Do we have to convert OPTIONS to SUBPACKAGES? No. >>>>> Can a SUBPACKAGE be built only if an OPTION is enabled? Yes. >>>>> The only viable use cases for SUBPACKAGES being enabled/disabled by >>>>> OPTIONS is limited to those portions of the port that do not affect >>>>> the >>>>> main package. >>>>> Examples are: additional libraries, additional data files, and so >>>>> on. >>>>> >>>>> Consolidate master/slave ports in one bigger ports >>>>> About this topic, I guess your concerns are mainly related to >>>>> potential >>>>> explosion of build time of the consolidated ports. >>>>> This is a justified concern. >>>>> In those cases, we are suggesting to convert slave ports in >>>>> SUBPACKAGES >>>>> enabled via OPTIONS. >>>>> This will allow port builders to configure the bigger ports to not >>>>> build all SUBPACKAGES, but only the needed ones. This should >>>>> restore >>>>> the previous build time. >>>>> >>>>> However, as for the php case, the maintainer is going to evaluate >>>>> if >>>>> the consolidation makes sense or not. >>>>> If a consolidation is going to result more problematic than >>>>> beneficial, >>>>> it can be reverted and subpackages not adopted for the use case. >>>> >>>> If I understand the sum of all the above correct, you suggest to >>>> remove >>>> slave ports and to use subpackages instead (where this makes sense >>>> in >>>> terms of the current implementation of subpackages). Or worded >>>> differently to the same effect (as I only care about the effect and >>>> not >>>> the intention), when someone converts a port to subpackages, the >>>> corresponding slave packages shall be removed (or for new ports: >>>> slave >>>> ports shall not be introduced in this case). >>>> >>>> Removing slave ports means we can not depend upon specific parts >>>> anymore >>>> when installing from ports, as the subpackages can not be targeted >>>> for >>>> install directly and my example of a subpackages aware php results >>>> in >>>> security implications of to much being installed and active if >>>> installed >>>> via make install in /usr/ports/something/with_webinterface. -> best >>>> example of lang/php to use subpacakges is the best example of why >>>> not >>>> to >>>> use subpackages / shows what is a regression in the features we >>>> provide >>>> in our ports collection. >>>> >>>> While qt6-tools may be a good example where subpackages makes sense, >>>> we >>>> can not depend on subpacakges for "make install", and as if the port >>>> would be converted to have subpackages but no slave ports are >>>> introduced, pkg install and make install would differ in the amount >>>> of >>>> installed files. >>>> >>>>> For port builders >>>>> >>>>> Port builders can experience longer build times in the future, as >>>>> master/slave ports could be consolidated in one single larger >>>>> ports. >>>>> If this is the case, the larger ports should provide OPTIONS to not >>>>> build unneeded subpackages. >>>>> If no OPTION is available, please work with the maintainer to >>>>> introduce >>>>> one. >>>> >>>> I fail to see the benefit: >>>> We either lose the possibility to target parts of a package (when >>>> slave >>>> ports are removed / not introduced) on make install (with a similar >>>> amount of build time for the ports tree as it is right now), or get >>>> higher build times for the package builders. In both cases we do not >>>> gain something significant. >>>> >>>> If we want to keep the (useful in some cases) feature to install >>>> from >>>> ports (there is the case of py39-rpds-py failing to build in my >>>> poudriere which I tried to debug with the author of py-maturin due >>>> to >>>> a >>>> runtime issue in maturin, which shows up in my the cross build on >>>> amd64-intel for amd64-athlon64... which in the end leads me to build >>>> py-rpds-py on the target machine from ports), the current >>>> implementation >>>> of subpackages has to come with the requirement to create slave >>>> ports >>>> for each subpackage. >>>> >>>> That's my concern, and that's the reason why I have the opinion, >>>> that >>>> portmgr has to keep the lock on subpackages and reject any >>>> subpackage >>>> which don't come with a slave port. >>>> >>>> I would be OK to lift this restriction, when the current >>>> implementation >>>> is changed to be able to only install the files of a subpacakge on >>>> make >>>> install (an implementation could be to require "make install >>>> TARGET_SUBPACAKGES=sub1,sub2,sub3" or "make install-subpackage1 >>>> install-subpacakge2"... as long as recursive dependencies are >>>> handled >>>> according to this requirement, those people designing, implementing >>>> and >>>> reviewing this can argue about such details). >>>> >>>> Keeping the current implementation (with the restriction of always >>>> introducing slave packages for subpackages) would need a way to >>>> denote >>>> that a slave port covers a specific subpackage which would allow >>>> package >>>> builders to skip those slave ports (and the subpacakges would need >>>> to >>>> have the same package name as the slave port, no idea if this has a >>>> technical disadvantage in terms of having 2 different origins for >>>> the >>>> same package name, but it surely would be confusing for people at >>>> first >>>> look). >>>> >>>> TLDR: for the use cases you specified in the beginning, I do not see >>>> a >>>> benefit, only drawbacks. Can you please provide an example of a >>>> benefit >>>> I fail to see (yes, more modularity for qt6-tools may be beneficial >>>> for >>>> some people, but the cost/benefit between qt6-tools (something which >>>> we >>>> don't provide right now) and "make install" (what we provide right >>>> now) >>>> or "build time reduction for package builders" (which would have a >>>> benefit for a lot of use cases) is in my books much more in the >>>> direction of "make install" and "build time reduction" than towards >>>> the >>>> modularization of qt6-tools)? >>>> >>>> Bye, >>>> Alexander. >>>> >>>> -- >>>> http://www.Leidinger.net Alexander@Leidinger.net: PGP >>>> 0x8F31830F9F2772BF >>>> http://www.FreeBSD.org netchild@FreeBSD.org : PGP >>>> 0x8F31830F9F2772BF >> >> -- >> http://www.Leidinger.net Alexander@Leidinger.net: PGP >> 0x8F31830F9F2772BF >> http://www.FreeBSD.org netchild@FreeBSD.org : PGP >> 0x8F31830F9F2772BF -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_deba9ff91a50bd5416fe042e22d25a98 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmW4988ACgkQEg2wmwP4 2IY4ZRAAmFeMb/O0SK1Nr7ngjhqOiKvzeLLJ/oAPzActoRDQ7PlsAupFmUkNYpKl a5Nyuggt5384/6jmWDBsli1aaNkOYmiNzIHhgb6Q7jWFKNI5Wsbt+4fS9+mwvu99 4AZsC7ohIw1Kw+x12TMfgzxIBZ9t+mfXNnyYEy0zj4UfgHrgxyHSyyPEwAzvwe0w DVevcEgUWyFcX6rmNb+FH3GxOtI/QpsLxbLHJmVnyNnLs1dOdOk/85c1B5I3dQJ0 SxTOoqYqjzQqeKyqwhLvgKKu3So2HXgUh++xOjCzCxq9FMvg30f+zDNwFPv6ChPA js+Ax5h4K6lpr1i2DeTEyI2PxJ8r+gbMe5qn/js7nzGxdL0cd0Nn+h5Cf5RIb998 3EtdZSpwWgdQbVxvvjPIDakbSARAcUY+wyxdcEnSkWKos3Ey1a3GFnpLMJn8dVKW XjvB14nNlVnUOfRFVIOBqIqN/FTVEj+pTdgad5FXacnHZhB2VYdaxo7blX19/hnp Vb/IWCW5BRu3mLkxD8+nRQLkRX/Xq9bZosqi90iK5mUwzrazUuuPOoYCPC0tXSKK oqx+p+beQvomDMgFUeqWIOAzhJJDGjs0i5XIT9ma0M+Xid9/ee94ulsRAJ6tiJKe 7jHsKthj6yVenS9Q9cS9lFW1eBgh/e+Tl0RvFRPgW7k51zuDHfk= =luvr -----END PGP SIGNATURE----- --=_deba9ff91a50bd5416fe042e22d25a98--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e3571856669357edaeed847f9a59859d>