Date: Tue, 30 Jan 2024 10:42:20 +0100 From: Luca Pizzamiglio <pizzamig@freebsd.org> To: Alexander Leidinger <Alexander@leidinger.net> Cc: FreeBSD ports <freebsd-ports@freebsd.org>, portmgr <portmgr@freebsd.org>, FreeBSD Core Team <core@freebsd.org> Subject: Re: This is going to break port building without poudriere! Message-ID: <CAB88xy-DesXEH9natthizUYOayw=nmGXXm-WzYgYVOkaE=81kA@mail.gmail.com> In-Reply-To: <1e27bce44f2ba85f0097fbc6aba1dac1@Leidinger.net> References: <CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg@mail.gmail.com> <cd0c0cb0-6035-45b4-b3e8-d99115e6c013@FreeBSD.org> <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com> <609374cf-7f1f-4643-8379-f368b23ccb09@freebsd.org> <CAB88xy-AkwJWhUtdfjernOumzshahJ_KRH1wOqRROceFbq3=GA@mail.gmail.com> <1e27bce44f2ba85f0097fbc6aba1dac1@Leidinger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000019db290610269536 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Alexander. Subpackage modularization of existing ports (i.e. qt6-tools) provides benefits to package users/builders: smaller dependency footprint, smaller usage on disk (useful for embedded systems), smaller jails. There are no benefits nor regressions for port builders for this specific use case. On the other hand, moving master/slave ports to subpackages comes with the cost of losing the ability to build the slave ports independently. For package users there is no change and some benefit for package builders. The issue can be mitigated by introducing options to select which subpackage to build (available for make install as well), but, still, a single subpackage cannot be built independently. This limitation *can* be unacceptable in some cases. For example, bofh@, the php maintainer, considers subpackages not the way to go, so the master/slave approach will stay. We introduced subpackages in the framework to explore all use cases, by trying and testing its adoption. By doing so we have already identified some issues (i.e. USES is not subpackage-aware yet) and interesting new use cases (subpackage with debug symbols). As per master/slave merge use case, we will let the maintainers decide if they want to move forward with subpackage adoptions, knowing the regression for port builders, but we won't push them in this direction. Best regards, Luca On Mon, Jan 29, 2024 at 10:04=E2=80=AFAM Alexander Leidinger < Alexander@leidinger.net> wrote: > Am 2024-01-27 16:59, schrieb Luca Pizzamiglio: > > > Hi Stefan. > > > > Let's start from the beginning, as it seems that things are not clear. > > > > Subpackages is a feature to create multiple packages from one build > > Those subpackages can depend on the main package. > > The main package cannot depend on any subpackages. > > This limits the cases where subpackages can be applied. The main > > package MUST be independent from its subpackages. Subpackages can only > > add features (like a plugin). > > To recall your NLS example (ref > > https://reviews.freebsd.org/D16457#715443): this is not a use-case for > > subpackages. If the main program/library needs to be compiled > > differently with or without NLS, this is not viable for subpackages. > > If a port needs to be built multiple times to create different > > subpackages, this is not a viable case for subpackages. > > A good candidate is qt6-tools: this port contains multiple tools > > (designer, linguist, help, and so on). Those tools could be put in > > different subpackages. > > > > I hope this explanation helps to clarify and address some of your > > concerns. > > As I read this, lang/php is the best example of a port where subpackages > has a benefit (in the sense it matches your description above to the > letter, the main port independent from the subpacakges, and what can be > build as subpackages is a plugin/extension). > > > OPTIONS and SUBPACKAGES > > Do we have to convert OPTIONS to SUBPACKAGES? No. > > Can a SUBPACKAGE be built only if an OPTION is enabled? Yes. > > The only viable use cases for SUBPACKAGES being enabled/disabled by > > OPTIONS is limited to those portions of the port that do not affect the > > main package. > > Examples are: additional libraries, additional data files, and so on. > > > > Consolidate master/slave ports in one bigger ports > > About this topic, I guess your concerns are mainly related to potential > > explosion of build time of the consolidated ports. > > This is a justified concern. > > In those cases, we are suggesting to convert slave ports in SUBPACKAGES > > enabled via OPTIONS. > > This will allow port builders to configure the bigger ports to not > > build all SUBPACKAGES, but only the needed ones. This should restore > > the previous build time. > > > > However, as for the php case, the maintainer is going to evaluate if > > the consolidation makes sense or not. > > If a consolidation is going to result more problematic than beneficial, > > it can be reverted and subpackages not adopted for the use case. > > If I understand the sum of all the above correct, you suggest to remove > slave ports and to use subpackages instead (where this makes sense in > terms of the current implementation of subpackages). Or worded > differently to the same effect (as I only care about the effect and not > the intention), when someone converts a port to subpackages, the > corresponding slave packages shall be removed (or for new ports: slave > ports shall not be introduced in this case). > > Removing slave ports means we can not depend upon specific parts anymore > when installing from ports, as the subpackages can not be targeted for > install directly and my example of a subpackages aware php results in > security implications of to much being installed and active if installed > via make install in /usr/ports/something/with_webinterface. -> best > example of lang/php to use subpacakges is the best example of why not to > use subpackages / shows what is a regression in the features we provide > in our ports collection. > > While qt6-tools may be a good example where subpackages makes sense, we > can not depend on subpacakges for "make install", and as if the port > would be converted to have subpackages but no slave ports are > introduced, pkg install and make install would differ in the amount of > installed files. > > > For port builders > > > > Port builders can experience longer build times in the future, as > > master/slave ports could be consolidated in one single larger ports. > > If this is the case, the larger ports should provide OPTIONS to not > > build unneeded subpackages. > > If no OPTION is available, please work with the maintainer to introduce > > one. > > I fail to see the benefit: > We either lose the possibility to target parts of a package (when slave > ports are removed / not introduced) on make install (with a similar > amount of build time for the ports tree as it is right now), or get > higher build times for the package builders. In both cases we do not > gain something significant. > > If we want to keep the (useful in some cases) feature to install from > ports (there is the case of py39-rpds-py failing to build in my > poudriere which I tried to debug with the author of py-maturin due to a > runtime issue in maturin, which shows up in my the cross build on > amd64-intel for amd64-athlon64... which in the end leads me to build > py-rpds-py on the target machine from ports), the current implementation > of subpackages has to come with the requirement to create slave ports > for each subpackage. > > That's my concern, and that's the reason why I have the opinion, that > portmgr has to keep the lock on subpackages and reject any subpackage > which don't come with a slave port. > > I would be OK to lift this restriction, when the current implementation > is changed to be able to only install the files of a subpacakge on make > install (an implementation could be to require "make install > TARGET_SUBPACAKGES=3Dsub1,sub2,sub3" or "make install-subpackage1 > install-subpacakge2"... as long as recursive dependencies are handled > according to this requirement, those people designing, implementing and > reviewing this can argue about such details). > > Keeping the current implementation (with the restriction of always > introducing slave packages for subpackages) would need a way to denote > that a slave port covers a specific subpackage which would allow package > builders to skip those slave ports (and the subpacakges would need to > have the same package name as the slave port, no idea if this has a > technical disadvantage in terms of having 2 different origins for the > same package name, but it surely would be confusing for people at first > look). > > TLDR: for the use cases you specified in the beginning, I do not see a > benefit, only drawbacks. Can you please provide an example of a benefit > I fail to see (yes, more modularity for qt6-tools may be beneficial for > some people, but the cost/benefit between qt6-tools (something which we > don't provide right now) and "make install" (what we provide right now) > or "build time reduction for package builders" (which would have a > benefit for a lot of use cases) is in my books much more in the > direction of "make install" and "build time reduction" than towards the > modularization of qt6-tools)? > > Bye, > Alexander. > > -- > http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF > http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF > --00000000000019db290610269536 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi Alexander.</div><div><br></div><div>Subpackage mod= ularization of existing ports (i.e. qt6-tools) provides benefits to package= users/builders: smaller dependency footprint, smaller usage on disk (usefu= l for embedded systems), smaller jails. There are no benefits nor regressio= ns for port builders for this specific use case.</div><div><br></div><div>O= n the other hand, moving master/slave ports to subpackages comes with the c= ost of losing the ability to build the slave ports independently. For packa= ge users there is no change and some benefit for package builders.<br></div= ><div>The issue can be mitigated by introducing options to select which sub= package to build (available for make install as well), but, still, a single= subpackage cannot be built independently.<br></div><div>This limitation <i= >can</i> be unacceptable in some cases. For example, bofh@, the php maintai= ner, considers subpackages not the way to go, so the master/slave approach = will stay.<br></div><div><br></div><div>We introduced subpackages in the fr= amework to explore all use cases, by trying and testing its adoption.<br></= div><div>By doing so we have already identified some issues (i.e. USES is n= ot subpackage-aware yet) and interesting new use cases (subpackage with deb= ug symbols).</div><div>As per master/slave merge use case, we will let the = maintainers decide if they want to move forward with subpackage adoptions, = knowing the regression for port builders, but we won't push them in thi= s direction.</div><div><br></div><div>Best regards,<br></div><div>Luca<br><= /div><div><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" c= lass=3D"gmail_attr">On Mon, Jan 29, 2024 at 10:04=E2=80=AFAM Alexander Leid= inger <<a href=3D"mailto:Alexander@leidinger.net">Alexander@leidinger.ne= t</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin= :0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"= >Am 2024-01-27 16:59, schrieb Luca Pizzamiglio:<br> <br> > Hi Stefan.<br> > <br> > Let's start from the beginning, as it seems that things are not cl= ear.<br> > <br> > Subpackages is a feature to create multiple packages from one build<br= > > Those subpackages can depend on the main package.<br> > The main package cannot depend on any subpackages.<br> > This limits the cases where subpackages can be applied. The main <br> > package MUST be independent from its subpackages. Subpackages can only= <br> > add features (like a plugin).<br> > To recall your NLS example (ref <br> > <a href=3D"https://reviews.freebsd.org/D16457#715443" rel=3D"noreferre= r" target=3D"_blank">https://reviews.freebsd.org/D16457#715443</a>): this i= s not a use-case for <br> > subpackages. If the main program/library needs to be compiled <br> > differently with or without NLS, this is not viable for subpackages.<b= r> > If a port needs to be built multiple times to create different <br> > subpackages, this is not a viable case for subpackages.<br> > A good candidate is qt6-tools: this port contains multiple tools <br> > (designer, linguist, help, and so on). Those tools could be put in <br= > > different subpackages.<br> > <br> > I hope this explanation helps to clarify and address some of your <br> > concerns.<br> <br> As I read this, lang/php is the best example of a port where subpackages <b= r> has a benefit (in the sense it matches your description above to the <br> letter, the main port independent from the subpacakges, and what can be <br= > build as subpackages is a plugin/extension).<br> <br> > OPTIONS and SUBPACKAGES<br> > Do we have to convert OPTIONS to SUBPACKAGES? No.<br> > Can a SUBPACKAGE be built only if an OPTION is enabled? Yes.<br> > The only viable use cases for SUBPACKAGES being enabled/disabled by <b= r> > OPTIONS is limited to those portions of the port that do not affect th= e <br> > main package.<br> > Examples are: additional libraries, additional data files, and so on.<= br> > <br> > Consolidate master/slave ports in one bigger ports<br> > About this topic, I guess your concerns are mainly related to potentia= l <br> > explosion of build time of the consolidated ports.<br> > This is a justified concern.<br> > In those cases, we are suggesting to convert slave ports in SUBPACKAGE= S <br> > enabled via OPTIONS.<br> > This will allow port builders to configure the bigger ports to not <br= > > build all SUBPACKAGES, but only the needed ones. This should restore <= br> > the previous build time.<br> > <br> > However, as for the php case, the maintainer is going to evaluate if <= br> > the consolidation makes sense or not.<br> > If a consolidation is going to result more problematic than beneficial= , <br> > it can be reverted and subpackages not adopted for the use case.<br> <br> If I understand the sum of all the above correct, you suggest to remove <br= > slave ports and to use subpackages instead (where this makes sense in <br> terms of the current implementation of subpackages). Or worded <br> differently to the same effect (as I only care about the effect and not <br= > the intention), when someone converts a port to subpackages, the <br> corresponding slave packages shall be removed (or for new ports: slave <br> ports shall not be introduced in this case).<br> <br> Removing slave ports means we can not depend upon specific parts anymore <b= r> when installing from ports, as the subpackages can not be targeted for <br> install directly and my example of a subpackages aware php results in <br> security implications of to much being installed and active if installed <b= r> via make install in /usr/ports/something/with_webinterface. -> best <br> example of lang/php to use subpacakges is the best example of why not to <b= r> use subpackages / shows what is a regression in the features we provide <br= > in our ports collection.<br> <br> While qt6-tools may be a good example where subpackages makes sense, we <br= > can not depend on subpacakges for "make install", and as if the p= ort <br> would be converted to have subpackages but no slave ports are <br> introduced, pkg install and make install would differ in the amount of <br> installed files.<br> <br> > For port builders<br> > <br> > Port builders can experience longer build times in the future, as <br> > master/slave ports could be consolidated in one single larger ports.<b= r> > If this is the case, the larger ports should provide OPTIONS to not <b= r> > build unneeded subpackages.<br> > If no OPTION is available, please work with the maintainer to introduc= e <br> > one.<br> <br> I fail to see the benefit:<br> We either lose the possibility to target parts of a package (when slave <br= > ports are removed / not introduced) on make install (with a similar <br> amount of build time for the ports tree as it is right now), or get <br> higher build times for the package builders. In both cases we do not <br> gain something significant.<br> <br> If we want to keep the (useful in some cases) feature to install from <br> ports (there is the case of py39-rpds-py failing to build in my <br> poudriere which I tried to debug with the author of py-maturin due to a <br= > runtime issue in maturin, which shows up in my the cross build on <br> amd64-intel for amd64-athlon64... which in the end leads me to build <br> py-rpds-py on the target machine from ports), the current implementation <b= r> of subpackages has to come with the requirement to create slave ports <br> for each subpackage.<br> <br> That's my concern, and that's the reason why I have the opinion, th= at <br> portmgr has to keep the lock on subpackages and reject any subpackage <br> which don't come with a slave port.<br> <br> I would be OK to lift this restriction, when the current implementation <br= > is changed to be able to only install the files of a subpacakge on make <br= > install (an implementation could be to require "make install <br> TARGET_SUBPACAKGES=3Dsub1,sub2,sub3" or "make install-subpackage1= <br> install-subpacakge2"... as long as recursive dependencies are handled = <br> according to this requirement, those people designing, implementing and <br= > reviewing this can argue about such details).<br> <br> Keeping the current implementation (with the restriction of always <br> introducing slave packages for subpackages) would need a way to denote <br> that a slave port covers a specific subpackage which would allow package <b= r> builders to skip those slave ports (and the subpacakges would need to <br> have the same package name as the slave port, no idea if this has a <br> technical disadvantage in terms of having 2 different origins for the <br> same package name, but it surely would be confusing for people at first <br= > look).<br> <br> TLDR: for the use cases you specified in the beginning, I do not see a <br> benefit, only drawbacks. Can you please provide an example of a benefit <br= > I fail to see (yes, more modularity for qt6-tools may be beneficial for <br= > some people, but the cost/benefit between qt6-tools (something which we <br= > don't provide right now) and "make install" (what we provide = right now) <br> or "build time reduction for package builders" (which would have = a <br> benefit for a lot of use cases) is in my books much more in the <br> direction of "make install" and "build time reduction" = than towards the <br> modularization of qt6-tools)?<br> <br> Bye,<br> Alexander.<br> <br> -- <br> <a href=3D"http://www.Leidinger.net" rel=3D"noreferrer" target=3D"_blank">h= ttp://www.Leidinger.net</a> Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF= <br> <a href=3D"http://www.FreeBSD.org" rel=3D"noreferrer" target=3D"_blank">htt= p://www.FreeBSD.org</a>=C2=A0 =C2=A0 netchild@FreeBSD.org=C2=A0 : PGP 0x8F3= 1830F9F2772BF<br> </blockquote></div> --00000000000019db290610269536--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB88xy-DesXEH9natthizUYOayw=nmGXXm-WzYgYVOkaE=81kA>