Date: Sat, 27 Jan 2024 16:59:53 +0100 From: Luca Pizzamiglio <pizzamig@freebsd.org> To: Stefan Esser <se@freebsd.org> Cc: FreeBSD ports <freebsd-ports@freebsd.org>, portmgr <portmgr@freebsd.org>, FreeBSD Core Team <core@freebsd.org> Subject: Re: This is going to break port building without poudriere! Message-ID: <CAB88xy-AkwJWhUtdfjernOumzshahJ_KRH1wOqRROceFbq3=GA@mail.gmail.com> In-Reply-To: <609374cf-7f1f-4643-8379-f368b23ccb09@freebsd.org> References: <CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg@mail.gmail.com> <cd0c0cb0-6035-45b4-b3e8-d99115e6c013@FreeBSD.org> <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com> <609374cf-7f1f-4643-8379-f368b23ccb09@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000084de2c060fef8147 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Stefan. Let's start from the beginning, as it seems that things are not clear. Subpackages is a feature to create multiple packages from one build. Those subpackages can depend on the main package. The main package cannot depend on any subpackages. This limits the cases where subpackages can be applied. The main package MUST be independent from its subpackages. Subpackages can only add features (like a plugin). To recall your NLS example (ref https://reviews.freebsd.org/D16457#715443): this is not a use-case for subpackages. If the main program/library needs to be compiled differently with or without NLS, this is not viable for subpackages. If a port needs to be built multiple times to create different subpackages, this is not a viable case for subpackages. A good candidate is qt6-tools: this port contains multiple tools (designer, linguist, help, and so on). Those tools could be put in different subpackages. I hope this explanation helps to clarify and address some of your concerns. *OPTIONS and SUBPACKAGES* Do we have to convert OPTIONS to SUBPACKAGES? No. Can a SUBPACKAGE be built only if an OPTION is enabled? Yes. The only viable use cases for SUBPACKAGES being enabled/disabled by OPTIONS is limited to those portions of the port that do not affect the main package. Examples are: additional libraries, additional data files, and so on. *Consolidate master/slave ports in one bigger ports* About this topic, I guess your concerns are mainly related to potential explosion of build time of the consolidated ports. This is a justified concern. In those cases, we are suggesting to convert slave ports in SUBPACKAGES enabled via OPTIONS. This will allow port builders to configure the bigger ports to not build all SUBPACKAGES, but only the needed ones. This should restore the previous build time. However, as for the php case, the maintainer is going to evaluate if the consolidation makes sense or not. If a consolidation is going to result more problematic than beneficial, it can be reverted and subpackages not adopted for the use case. *For port builders* Port builders can experience longer build times in the future, as master/slave ports could be consolidated in one single larger ports. If this is the case, the larger ports should provide OPTIONS to not build unneeded subpackages. If no OPTION is available, please work with the maintainer to introduce one= . Probably I didn't address all of your concerns, but I hope you gave you a clearer picture/explanation. Best regards, Luca On Fri, Jan 26, 2024 at 5:08=E2=80=AFPM Stefan Esser <se@freebsd.org> wrote= : > Am 25.01.24 um 20:57 schrieb Luca Pizzamiglio: > > Hi Stefan, > > > > I did reply to your first email, but not to your second one. > > I preferred (in agreement with portmgr@) to open the discussion with > everyone, > > instead of keeping it just between you and me. > > > > As you can read in the email you have linked, I didn't ignore your > comments. > > You have mentioned some of the issues without providing a solution. > > > This implementation will break port dependencies, since there is no > way > > a port can depend on a specific sub-package - there even is no way = a > > non-default sub-package can be built without manual selection of th= e > > options that activate its creation. > > > > A port can depend on a specific sub-package. The category/origin~subpkg > is the > > chosen format. > > Non-default subpackages shouldn't exist. In the aforementioned email, I > > explicitly say that IF a subpackage is enabled by an option, the option > must be > > enabled by default. > > Your previous comments contributed to making this point clearer. > > If there should be no "non-default subpackages", then selecting subpackag= es > to install by OPTIONs makes no sense. But there are many scenarios where > not all subpackages can be built at the same time due to conlflicting > dependencies or requirements (often controlled by OPTIONs). > > > Dependencies stated in the port Makefile are converted into package > > dependencies that can be resolved by the "pkg" command, but that > cannot > > be directly used to build and install the requested sub-package fro= m > > a port. > > > > I don't know where you get this idea, but it's not how it works. > > If a port has a subpackage dependency, the category/origin~subpkg is th= e > chosen > > format to express that dependency. > > Yes, but how will this dependency be processed in the ports system? > > Will the port at the given origin be built (all subpackages) and all the > available subpackages will be installed? > > How do you deal with the case, where o ne of those subpackages conflicts > with an already installed port? > > What actually works is the mapping of the origin~subpackage into a packag= e > dependency that can be resolved by the pkg command. That's what I am > complaining about: only sucgh package dependencies continue to work, but > port dependencies don't in the same way as without subpackages. > > > The related port (category/origin) has to be built and installed. > > By running `make install`, the whole port is installed, subpackages as > well. > > The only "issue" I see is that via 'make install' you cannot install > only the > > subpackage, but the entire port only. > > However this is not different than before. > > Except that before there was not one port that built potentially tens of > subpackages. > > > Has there been a general consensus that support for direct port > building > > (without poudriere) will be abandoned? > > > > Port building is not abandoned, and it's actively supported, as it's th= e > > foundation of poudriere. > > The case of poudriere is not generic port building, since all dependencie= s > are installed from packages in poudriere. Poudriere never installs from a > port, only from packages, and it can deal with subpackage dependencies. > > Port building on the base system uses a completely different approach tha= n > package building in poudriere. > > > If a regression has been introduced, by me or anyone else, it has to be > fixed. > > I may have overlooked some use cases, but AFAIK I didn't introduce any > > regression (confirmed by the exp-run) > > The exp-run is performed by building packages with poudriere. What has th= is > to do with the port building on the base system that I'm worried about? > > Besides, you only showed that you did not break the ports system without > subpackages by adding features for the support of subpackages to > bsd.port.mk. > The relevant test case would have been an exp-run with several relevant > ports converted to subpackages. > > But even then, port building outside poudriere is different and may be > severely broken without any issues reported by an exp-run, even one that > tests actual subpackages. > > > Ports that do not create sub-packages can still be depended on by > other > > ports, but as critical dependencies have been depended to > sub-packages > > a ever large fraction of ports will only build in poudriere. > > > > Again, no. > > If you run make install on any port, it will be built. The package and > the > > subpackages are all installed. > > Yes, and that is wrong! > > > You cannot install a portion of a port, and it has never been possible. > > No, but with subpackages it will become possible to install a portion of > a port from a package. And since the goal is to convert ports that share > a distfile to subpackages, this will become a common case for large ports= . > > > Nothing is broken here and I fail to understand where you get the idea > of this > > behavior. > > Because I consider just the above as breaking important features of the > current ports system (the ability to selectively install just the ports > that I need and not all that can potentially be built from one distfile). > > > I wrote tests and examples to implement the feature and get them workin= g > before > > adding support to poudriere. > > I have seen your tests and they are only relevant for the package > dependency > case (dependencies resolved by the pkg command). > > > This change does also obviously break port management tools like > portmaster, > > which took me significant effort to adapt to FLAVOR support (which > also had > > been implemented without consideration for other tools than > poudriere), and > > which I have been maintaining since then. > > > > Yes, portmaster need to be able to parse the new dependencies format, b= y > > removing the ~subpkg. By removing the suffix, you get the category/port > and > > everything is fine. > > No, I get a whole lot of subpackages that I did not ask for! > > > As announced, we are keeping subpackages adoption blocked with a > git-hook to > > give time to maintainers to add support to subpkg and to introduce the > feature > > slowly. > > The damage will be to other users than the committers of these ports, and > even developers that depend on them might not notice any issues, since th= ey > are using poudriere to test their ports. > > > portmgr@ doesn't officially support neither portmaster nor portupgrade. > We > > simply lack the manpower to do it. > > I do not expect PortMgr to support either of these tools. But I do expect > PortMgr to not break features of the port system due to neglect of buildi= ng > ports on the base system. I know that there have been discussions about > abandoning the base system builds (especially since in some cases ports d= o > build in a clean jail, but not with a previous version installed, e.g. > because > of preferring installed header files above those in the source director<)= . > > But most ports build just fine on the base system and this is the fastest > way > for a developer to get some required functionality with free choice of th= e > OPTIONs (when different from those of a pre-built package, for example). > > > The reason given for sub-packages support is reduced build time for > some > > packages that share a common distfile (e.g. qt5) when building > official > > packages. > > > > As explained, in the short term there are this benefit and getting rid > of many > > master/slave ports (hard to maintain). > > This does only work if there is e.g. a port that can be compiled for CLI > and > GUI, for example, but that does not need special build options to include > the required hooks to the added functionality. > > I have seen the devel/appstream port as of > https://reviews.freebsd.org/D43445, > and it clearly shows the weakness of the subpackages implementation. You > did > not get rid of a master/slave port combination, but you have now combined > the > CLI and Qt6 versions (while there previously was one port for the CLI > version > and one flavored port for Qt5/Qt6). > > And devel/appstream/Makefile has become a lot more complex and the > complexity > of having both a slave port and subpackages makes it even worse. > > If I build the CLI version after subpackages, I get all of Qt6 as a > dependency! > > And Qt6 is not only fetched and built, but also installed (when building > the > port as a dependency on the base system), whether I need it or not. > > > But there are several use cases in the future. > > For instance, work is in place to provide debug symbols as subpackage, > in a > > similar way as pkg-base. > > There is nothing wrong with such use-cases, and I do not object the suppo= rt > of subpackages at all. > > I had just been hoping that my comments to the earlier review would lead = to > a better design - which would have easily been possible (and I had worked > out a concept several years ago, which I had discussed with some members > of PortMgr and possibly also in mails to the ports maillist). > > > But this comes at a high cost for all builds outside the package > build > > cluster, since now lots of unnecessary sub-trees will be compiled a= nd > > installed, if only one program (i.e. sub-package) is desired. > > > > This is the reason why OPTIONS with SUBPACAKGES have been introduced, t= o > reduce > > build time for port builders. > > But that does not work well if there is no way to deduce the required > OPTIONS from the required subpackages. And it requires manual interventio= n. > > There should have been variables like e.g. SUBPKG_DEFINE; SUBPKG_DEFAULT, > SUBPKG_SELECT which control the building and installation of subpackages, > and OPTIONs might depend on the selected set of subpackages (e.g., if a > ~docs subpackage is selected, the DOCS option could be acticated). > > > You are pessimizing the build for thousands of users to spare a few > > cycles for a very small percentage of packages built once in a whil= e > on > > the build cluster. > > > > I have already explained this point above. > > I did not see a valid explanation. Thousands (or more) port builders will > waste cycles and space for subpackages that are not actually wanted, sinc= e > there is no mechanism that allows to easily restrict the amount built > (without resorting to OPTIONs, but those are sticky and the next port to > be built may require subpackages controlled by other options, this is not > a workable conept). > > > I had pointed out other issues with this approach in the review > comment > > and in the private mail I sent 2024-01-02 after finding that this > version > > has been committed without formal acceptance of your review D40549. > > > > As I replied to you already, there has been no formal acceptance in > > phabricator, but there was consensus in portmgr@ to land it. > > I apologize for not having used the appropriate reviews channel, I > totally > > agree that it has not been a good behavior from my side, as I'm not > providing a > > good example. > > As explained before, I had mostly given up on FreeBSD in late 2022, mostl= y > because of frustration about PortMgr. I had stopped reading most of the > FreeBSD mail lists and might have missed an early announcement of the > plan to commit your Phabricator review. As you know, I had hoped to be > able to stear the subpackages development in a better direction as part > of the PortMgr team, but there was an objection of the author of the > original subpackages review (that your commit is based on), probably > because of my earlier comment on his approach. > > > You are breaking use-cases of a large number of users that still > build > > ports without poudriere. This is especially causing a barrier to > entry > > of new port developers, since this will require them to setup > poudriere > > before they can begin port development. (This will only become an > issue > > over time, as more and more dependencies will have been converted t= o > > sub-packages, but then there will be no way back.) > > > > I still fail to see what is going to break. > > Non-poudriere users can experience longer build time for dependencies, > but > > dependencies can be installed via packages, at least most of the time. > > This "dependencies can be installed via packages" is just the sentiment o= f > PortMgr that has no interest for port building in the base system. > > Some dependencies can be installed from packages, but in many cases that > leads to lots of already installed programs to be substituted with packag= es > built with different OPTIONs. Mixing packages and ports is not working > well, > and the only sane possibilities are a local repository built with custom > options using poudriere, or just building everything from a port on the > base system. > > > Your announcement mentions some of the issues, but does not offer a= ny > > actual solution. > > > > > > A sane implementation of sub-packages would not make their creation > > depend on OPTION settings, with no way to determine the required > OPTION > > setting for a non-default sub-package (i.e., only default > sub-packages > > can be depended on). > > > > I'm not going to repeat all the issues pointed out in > > > > https://reviews.freebsd.org/D16457#715443 > > <https://reviews.freebsd.org/D16457#715443>; > > > > but had on multiple occasions pointed out that a sane mechanism wou= ld > > start with "SUB_PACKAGE_DEFINE" and "SUB_PACKAGE_DEFAULT" variables > and > > with a mechanism to determine the required OPTION values from the > actual > > set of sub-packages to be built (instead of the opposite, to make t= he > > selection of sub-packages depend on the OPTIONs). > > > > I clearly addressed this topic in the aforementioned email. > > Sorry, I did not see these points addressed. Please give specific pointer= s. > > IMHO, all of the issues in my comment to review D16457 are still open. > > > We chose not to follow this approach, and I motivated it (changing > > configuration of other ports it's NOT something to have IMO). > > What are you talking about ("changing configuration of other ports") ??? > > > We clearly disagree on this point. > > I have opened up the discussion to get feedback from the rest of the > community > > as well. > > So far, you are the only one strongly against this approach. > > Maybe this is because other developers will only notice the effects of th= is > change when many ports have been converted to subpackages? > > This will make it much harder for new ports contributors to work on their > first port, since port builds outside poudriere will become penalized mor= e > and more over time, as subpackages are introduced (and become > dependencies). > > I have deep insight into the ports framework since I needed to understand > it > when I took over portmaster and added FLAVOR support to it. And as a > certified > Common Criteria evaluator (plus other standards) I might have a different > view > on aspects of software development. > > I have mostly given up my attempts to help improve the ports system, for > lack > of support (or at least not being ignored when I ask for a review) by > PortMgr. > There are a lot of sometimes trivial changes, but my past experience with > PortMgr is that I cannot hope for any kind of review to suggested changes= , > since everybody is busy working in his field of interest and does not car= e > for any outside contributions (and my attempt to contribute from within > PortMgr has been blocked, too). > > Best regards, STefan > --00000000000084de2c060fef8147 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi Stefan.</div><div><br></div><div>Let's start f= rom the beginning, as it seems that things are not clear.</div><div><br></d= iv><div>Subpackages is a feature to create multiple packages from one build= .</div><div>Those subpackages can depend on the main package.</div><div>The= main package cannot depend on any subpackages.</div><div>This limits the c= ases where subpackages can be applied. The main package MUST be independent= from its subpackages. Subpackages can only add features (like a plugin).<b= r></div><div>To recall your NLS example (ref <a href=3D"https://reviews.fre= ebsd.org/D16457#715443">https://reviews.freebsd.org/D16457#715443</a>): thi= s is not a use-case for subpackages. If the main program/library needs to b= e compiled differently with or without NLS, this is not viable for subpacka= ges.</div><div>If a port needs to be built multiple times to create differe= nt subpackages, this is not a viable case for subpackages.</div><div>A good= candidate is qt6-tools: this port contains multiple tools (designer, lingu= ist, help, and so on). Those tools could be put in different subpackages.</= div><div><br></div><div>I hope this explanation helps to clarify and addres= s some of your concerns.</div><div><br></div><div><b>OPTIONS and SUBPACKAGE= S</b></div><div>Do we have to convert OPTIONS to SUBPACKAGES? No.</div><div= >Can a SUBPACKAGE be built only if an OPTION is enabled? Yes.</div><div>The= only viable use cases for SUBPACKAGES being enabled/disabled by OPTIONS is= limited to those portions of the port that do not affect the main package.= </div><div>Examples are: additional libraries, additional data files, and s= o on.</div><div><b><br></b></div><div><b>Consolidate master/slave ports in = one bigger ports</b></div><div>About this topic, I guess your concerns are = mainly related to potential explosion of build time of the consolidated por= ts.</div><div>This is a justified concern.</div><div>In those cases, we are= suggesting to convert slave ports in SUBPACKAGES enabled via OPTIONS.</div= ><div>This will allow port builders to configure the bigger ports to not bu= ild all SUBPACKAGES, but only the needed ones. This should restore the prev= ious build time.</div><div><br></div><div>However, as for the php case, the= maintainer is going to evaluate if the consolidation makes sense or not.</= div><div>If a consolidation is going to result more problematic than benefi= cial, it can be reverted and subpackages not adopted for the use case.<br><= /div><div><br></div><div><b>For port builders<br></b></div><div>Port builde= rs can experience longer build times in the future, as master/slave ports c= ould be consolidated in one single larger ports.</div><div>If this is the c= ase, the larger ports should provide OPTIONS to not build unneeded subpacka= ges.</div><div>If no OPTION is available, please work with the maintainer t= o introduce one.<br></div><div><br></div><div>Probably I didn't address= all of your concerns, but I hope you gave you a clearer picture/explanatio= n.</div><div><br></div><div>Best regards,</div><div>Luca<br></div><div><div= ><div><br></div><div><br></div></div></div></div><br><div class=3D"gmail_qu= ote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, Jan 26, 2024 at 5:08=E2= =80=AFPM Stefan Esser <<a href=3D"mailto:se@freebsd.org">se@freebsd.org<= /a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0= px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">A= m 25.01.24 um 20:57 schrieb Luca Pizzamiglio:<br> > Hi Stefan,<br> > <br> > I did reply to your first email, but not to your second one.<br> > I preferred (in agreement with portmgr@) to open the discussion with e= veryone, <br> > instead of keeping it just between you and me.<br> > <br> > As you can read in the email you have linked, I didn't ignore your= comments.<br> <br> You have mentioned some of the issues without providing a solution.<br> <br> >=C2=A0 =C2=A0 =C2=A0This implementation will break port dependencies, s= ince there is no way<br> >=C2=A0 =C2=A0 =C2=A0a port can depend on a specific sub-package - there= even is no way a<br> >=C2=A0 =C2=A0 =C2=A0non-default sub-package can be built without manual= selection of the<br> >=C2=A0 =C2=A0 =C2=A0options that activate its creation.<br> > <br> > A port can depend on a specific sub-package. The category/origin~subpk= g is the <br> > chosen format.<br> > Non-default subpackages shouldn't exist. In the aforementioned ema= il, I <br> > explicitly say that IF a subpackage is enabled by an option, the optio= n must be <br> > enabled by default.<br> > Your previous comments contributed to making this point clearer.<br> <br> If there should be no "non-default subpackages", then selecting s= ubpackages<br> to install by OPTIONs makes no sense. But there are many scenarios where<br= > not all subpackages can be built at the same time due to conlflicting<br> dependencies or requirements (often controlled by OPTIONs).<br> <br> >=C2=A0 =C2=A0 =C2=A0Dependencies stated in the port Makefile are conver= ted into package<br> >=C2=A0 =C2=A0 =C2=A0dependencies that can be resolved by the "pkg&= quot; command, but that cannot<br> >=C2=A0 =C2=A0 =C2=A0be directly used to build and install the requested= sub-package from<br> >=C2=A0 =C2=A0 =C2=A0a port.<br> > <br> > I don't know where you get this idea, but it's not how it work= s.<br> > If a port has a subpackage dependency, the category/origin~subpkg is t= he chosen <br> > format to express that dependency.<br> <br> Yes, but how will this dependency be processed in the ports system?<br> <br> Will the port at the given origin be built (all subpackages) and all the<br= > available subpackages will be installed?<br> <br> How do you deal with the case, where o ne of those subpackages conflicts<br= > with an already installed port?<br> <br> What actually works is the mapping of the origin~subpackage into a package<= br> dependency that can be resolved by the pkg command. That's what I am<br= > complaining about: only sucgh package dependencies continue to work, but<br= > port dependencies don't in the same way as without subpackages.<br> <br> > The related port (category/origin) has to be built and installed.<br> > By running `make install`, the whole port is installed, subpackages as= well.<br> > The only "issue" I see is that via 'make install' yo= u cannot install only the <br> > subpackage, but the entire port only.<br> > However this is not different than before.<br> <br> Except that before there was not one port that built potentially tens of<br= > subpackages.<br> <br> >=C2=A0 =C2=A0 =C2=A0Has there been a general consensus that support for= direct port building<br> >=C2=A0 =C2=A0 =C2=A0(without poudriere) will be abandoned?<br> > <br> > Port building is not abandoned, and it's actively supported, as it= 's the <br> > foundation of poudriere.<br> <br> The case of poudriere is not generic port building, since all dependencies<= br> are installed from packages in poudriere. Poudriere never installs from a<b= r> port, only from packages, and it can deal with subpackage dependencies.<br> <br> Port building on the base system uses a completely different approach than<= br> package building in poudriere.<br> <br> > If a regression has been introduced, by me or anyone else, it has to b= e fixed.<br> > I may have overlooked some use cases, but AFAIK I didn't introduce= any <br> > regression (confirmed by the exp-run)<br> <br> The exp-run is performed by building packages with poudriere. What has this= <br> to do with the port building on the base system that I'm worried about?= <br> <br> Besides, you only showed that you did not break the ports system without<br= > subpackages by adding features for the support of subpackages to <a href=3D= "http://bsd.port.mk" rel=3D"noreferrer" target=3D"_blank">bsd.port.mk</a>.<= br> The relevant test case would have been an exp-run with several relevant<br> ports converted to subpackages.<br> <br> But even then, port building outside poudriere is different and may be<br> severely broken without any issues reported by an exp-run, even one that<br= > tests actual subpackages.<br> <br> >=C2=A0 =C2=A0 =C2=A0Ports that do not create sub-packages can still be = depended on by other<br> >=C2=A0 =C2=A0 =C2=A0ports, but as critical dependencies have been depen= ded to sub-packages<br> >=C2=A0 =C2=A0 =C2=A0a ever large fraction of ports will only build in p= oudriere.<br> > <br> > Again, no.<br> > If you run make install on any port, it will be built. The package and= the <br> > subpackages are all installed.<br> <br> Yes, and that is wrong!<br> <br> > You cannot install a portion of a port, and it has never been possible= .<br> <br> No, but with subpackages it will become possible to install a portion of<br= > a port from a package. And since the goal is to convert ports that share<br= > a distfile to subpackages, this will become a common case for large ports.<= br> <br> > Nothing is broken here and I fail to understand where you get the idea= of this <br> > behavior.<br> <br> Because I consider just the above as breaking important features of the<br> current ports system (the ability to selectively install just the ports<br> that I need and not all that can potentially be built from one distfile).<b= r> <br> > I wrote tests and examples to implement the feature and get them worki= ng before <br> > adding support to poudriere.<br> <br> I have seen your tests and they are only relevant for the package dependenc= y<br> case (dependencies resolved by the pkg command).<br> <br> >=C2=A0 =C2=A0 =C2=A0This change does also obviously break port manageme= nt tools like portmaster,<br> >=C2=A0 =C2=A0 =C2=A0which took me significant effort to adapt to FLAVOR= support (which also had<br> >=C2=A0 =C2=A0 =C2=A0been implemented without consideration for other to= ols than poudriere), and<br> >=C2=A0 =C2=A0 =C2=A0which I have been maintaining since then.<br> > <br> > Yes, portmaster need to be able to parse the new dependencies format, = by <br> > removing the ~subpkg. By removing the suffix, you get the category/por= t and <br> > everything is fine.<br> <br> No, I get a whole lot of subpackages that I did not ask for!<br> <br> > As announced, we are keeping subpackages adoption blocked with a git-h= ook to <br> > give time to maintainers to add support to subpkg and to introduce the= feature <br> > slowly.<br> <br> The damage will be to other users than the committers of these ports, and<b= r> even developers that depend on them might not notice any issues, since they= <br> are using poudriere to test their ports.<br> <br> > portmgr@ doesn't officially support neither portmaster nor portupg= rade. We <br> > simply lack the manpower to do it.<br> <br> I do not expect PortMgr to support either of these tools. But I do expect<b= r> PortMgr to not break features of the port system due to neglect of building= <br> ports on the base system. I know that there have been discussions about<br> abandoning the base system builds (especially since in some cases ports do<= br> build in a clean jail, but not with a previous version installed, e.g. beca= use<br> of preferring installed header files above those in the source director<= ).<br> <br> But most ports build just fine on the base system and this is the fastest w= ay<br> for a developer to get some required functionality with free choice of the<= br> OPTIONs (when different from those of a pre-built package, for example).<br= > <br> >=C2=A0 =C2=A0 =C2=A0The reason given for sub-packages support is reduce= d build time for some<br> >=C2=A0 =C2=A0 =C2=A0packages that share a common distfile (e.g. qt5) wh= en building official<br> >=C2=A0 =C2=A0 =C2=A0packages.<br> > <br> > As explained, in the short term there are this benefit and getting rid= of many <br> > master/slave ports (hard to maintain).<br> <br> This does only work if there is e.g. a port that can be compiled for CLI an= d<br> GUI, for example, but that does not need special build options to include<b= r> the required hooks to the added functionality.<br> <br> I have seen the devel/appstream port as of <a href=3D"https://reviews.freeb= sd.org/D43445" rel=3D"noreferrer" target=3D"_blank">https://reviews.freebsd= .org/D43445</a>,<br> and it clearly shows the weakness of the subpackages implementation. You di= d<br> not get rid of a master/slave port combination, but you have now combined t= he<br> CLI and Qt6 versions (while there previously was one port for the CLI versi= on<br> and one flavored port for Qt5/Qt6).<br> <br> And devel/appstream/Makefile has become a lot more complex and the complexi= ty<br> of having both a slave port and subpackages makes it even worse.<br> <br> If I build the CLI version after subpackages, I get all of Qt6 as a depende= ncy!<br> <br> And Qt6 is not only fetched and built, but also installed (when building th= e<br> port as a dependency on the base system), whether I need it or not.<br> <br> > But there are several use cases in the future.<br> > For instance, work is in place to provide debug symbols as subpackage,= in a <br> > similar way as pkg-base.<br> <br> There is nothing wrong with such use-cases, and I do not object the support= <br> of subpackages at all.<br> <br> I had just been hoping that my comments to the earlier review would lead to= <br> a better design - which would have easily been possible (and I had worked<b= r> out a concept several years ago, which I had discussed with some members<br= > of PortMgr and possibly also in mails to the ports maillist).<br> <br> >=C2=A0 =C2=A0 =C2=A0But this comes at a high cost for all builds outsid= e the package build<br> >=C2=A0 =C2=A0 =C2=A0cluster, since now lots of unnecessary sub-trees wi= ll be compiled and<br> >=C2=A0 =C2=A0 =C2=A0installed, if only one program (i.e. sub-package) i= s desired.<br> > <br> > This is the reason why OPTIONS with SUBPACAKGES have been introduced, = to reduce <br> > build time for port builders.<br> <br> But that does not work well if there is no way to deduce the required<br> OPTIONS from the required subpackages. And it requires manual intervention.= <br> <br> There should have been variables like e.g. SUBPKG_DEFINE; SUBPKG_DEFAULT,<b= r> SUBPKG_SELECT which control the building and installation of subpackages,<b= r> and OPTIONs might depend on the selected set of subpackages (e.g., if a<br> ~docs subpackage is selected, the DOCS option could be acticated).<br> <br> >=C2=A0 =C2=A0 =C2=A0You are pessimizing the build for thousands of user= s to spare a few<br> >=C2=A0 =C2=A0 =C2=A0cycles for a very small percentage of packages buil= t once in a while on<br> >=C2=A0 =C2=A0 =C2=A0the build cluster.<br> > <br> > I have already explained this point above.<br> <br> I did not see a valid explanation. Thousands (or more) port builders will<b= r> waste cycles and space for subpackages that are not actually wanted, since<= br> there is no mechanism that allows to easily restrict the amount built<br> (without resorting to OPTIONs, but those are sticky and the next port to<br= > be built may require subpackages controlled by other options, this is not<b= r> a workable conept).<br> <br> >=C2=A0 =C2=A0 =C2=A0I had pointed out other issues with this approach i= n the review comment<br> >=C2=A0 =C2=A0 =C2=A0and in the private mail I sent 2024-01-02 after fin= ding that this version<br> >=C2=A0 =C2=A0 =C2=A0has been committed without formal acceptance of you= r review D40549.<br> > <br> > As I replied to you already, there has been no formal acceptance in <b= r> > phabricator, but there was consensus in portmgr@ to land it.<br> > I apologize for not having used the appropriate reviews channel, I tot= ally <br> > agree that it has not been a good behavior from my side, as I'm no= t providing a <br> > good example.<br> <br> As explained before, I had mostly given up on FreeBSD in late 2022, mostly<= br> because of frustration about PortMgr. I had stopped reading most of the<br> FreeBSD mail lists and might have missed an early announcement of the<br> plan to commit your Phabricator review. As you know, I had hoped to be<br> able to stear the subpackages development in a better direction as part<br> of the PortMgr team, but there was an objection of the author of the<br> original subpackages review (that your commit is based on), probably<br> because of my earlier comment on his approach.<br> <br> >=C2=A0 =C2=A0 =C2=A0You are breaking use-cases of a large number of use= rs that still build<br> >=C2=A0 =C2=A0 =C2=A0ports without poudriere. This is especially causing= a barrier to entry<br> >=C2=A0 =C2=A0 =C2=A0of new port developers, since this will require the= m to setup poudriere<br> >=C2=A0 =C2=A0 =C2=A0before they can begin port development. (This will = only become an issue<br> >=C2=A0 =C2=A0 =C2=A0over time, as more and more dependencies will have = been converted to<br> >=C2=A0 =C2=A0 =C2=A0sub-packages, but then there will be no way back.)<= br> > <br> > I still fail to see what is going to break.<br> > Non-poudriere users can experience longer build time for dependencies,= but <br> > dependencies can be installed via packages, at least most of the time.= <br> <br> This "dependencies can be installed via packages" is just the sen= timent of<br> PortMgr that has no interest for port building in the base system.<br> <br> Some dependencies can be installed from packages, but in many cases that<br= > leads to lots of already installed programs to be substituted with packages= <br> built with different OPTIONs. Mixing packages and ports is not working well= ,<br> and the only sane possibilities are a local repository built with custom<br= > options using poudriere, or just building everything from a port on the<br> base system.<br> <br> >=C2=A0 =C2=A0 =C2=A0Your announcement mentions some of the issues, but = does not offer any<br> >=C2=A0 =C2=A0 =C2=A0actual solution.<br> > <br> > <br> >=C2=A0 =C2=A0 =C2=A0A sane implementation of sub-packages would not mak= e their creation<br> >=C2=A0 =C2=A0 =C2=A0depend on OPTION settings, with no way to determine= the required OPTION<br> >=C2=A0 =C2=A0 =C2=A0setting for a non-default sub-package (i.e., only d= efault sub-packages<br> >=C2=A0 =C2=A0 =C2=A0can be depended on).<br> > <br> >=C2=A0 =C2=A0 =C2=A0I'm not going to repeat all the issues pointed = out in<br> > <br> >=C2=A0 =C2=A0 =C2=A0<a href=3D"https://reviews.freebsd.org/D16457#71544= 3" rel=3D"noreferrer" target=3D"_blank">https://reviews.freebsd.org/D16457#= 715443</a><br> >=C2=A0 =C2=A0 =C2=A0<<a href=3D"https://reviews.freebsd.org/D16457#7= 15443" rel=3D"noreferrer" target=3D"_blank">https://reviews.freebsd.org/D16= 457#715443</a>><br> > <br> >=C2=A0 =C2=A0 =C2=A0but had on multiple occasions pointed out that a sa= ne mechanism would<br> >=C2=A0 =C2=A0 =C2=A0start with "SUB_PACKAGE_DEFINE" and "= ;SUB_PACKAGE_DEFAULT" variables and<br> >=C2=A0 =C2=A0 =C2=A0with a mechanism to determine the required OPTION v= alues from the actual<br> >=C2=A0 =C2=A0 =C2=A0set of sub-packages to be built (instead of the opp= osite, to make the<br> >=C2=A0 =C2=A0 =C2=A0selection of sub-packages depend on the OPTIONs).<b= r> > <br> > I clearly addressed this topic in the aforementioned email.<br> <br> Sorry, I did not see these points addressed. Please give specific pointers.= <br> <br> IMHO, all of the issues in my comment to review D16457 are still open.<br> <br> > We chose not to follow this approach, and I motivated it (changing <br= > > configuration of other ports it's NOT something to have IMO).<br> <br> What are you talking about ("changing configuration of other ports&quo= t;) ???<br> <br> > We clearly disagree on this point.<br> > I have opened up the discussion to get feedback from the rest of the c= ommunity <br> > as well.<br> > So far, you are the only one strongly against this approach.<br> <br> Maybe this is because other developers will only notice the effects of this= <br> change when many ports have been converted to subpackages?<br> <br> This will make it much harder for new ports contributors to work on their<b= r> first port, since port builds outside poudriere will become penalized more<= br> and more over time, as subpackages are introduced (and become dependencies)= .<br> <br> I have deep insight into the ports framework since I needed to understand i= t<br> when I took over portmaster and added FLAVOR support to it. And as a certif= ied<br> Common Criteria evaluator (plus other standards) I might have a different v= iew<br> on aspects of software development.<br> <br> I have mostly given up my attempts to help improve the ports system, for la= ck<br> of support (or at least not being ignored when I ask for a review) by PortM= gr.<br> There are a lot of sometimes trivial changes, but my past experience with<b= r> PortMgr is that I cannot hope for any kind of review to suggested changes,<= br> since everybody is busy working in his field of interest and does not care<= br> for any outside contributions (and my attempt to contribute from within<br> PortMgr has been blocked, too).<br> <br> Best regards, STefan<br> </blockquote></div> --00000000000084de2c060fef8147--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB88xy-AkwJWhUtdfjernOumzshahJ_KRH1wOqRROceFbq3=GA>