Date: Thu, 25 Jan 2024 20:57:49 +0100 From: Luca Pizzamiglio <pizzamig@freebsd.org> To: Stefan Esser <se@freebsd.org> Cc: freebsd-ports <ports@freebsd.org>, portmgr <portmgr@freebsd.org>, FreeBSD Core Team <core@freebsd.org> Subject: Re: This is going to break port building without poudriere! (was: Subpackage explanations) Message-ID: <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com> In-Reply-To: <cd0c0cb0-6035-45b4-b3e8-d99115e6c013@FreeBSD.org> References: <CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg@mail.gmail.com> <cd0c0cb0-6035-45b4-b3e8-d99115e6c013@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000cdfc6e060fca984e Content-Type: text/plain; charset="UTF-8" Hi Stefan, I did reply to your first email, but not to your second one. I preferred (in agreement with portmgr@) to open the discussion with everyone, instead of keeping it just between you and me. As you can read in the email you have linked, I didn't ignore your comments. This implementation will break port dependencies, since there is no way > a port can depend on a specific sub-package - there even is no way a > non-default sub-package can be built without manual selection of the > options that activate its creation. > A port can depend on a specific sub-package. The category/origin~subpkg is the chosen format. Non-default subpackages shouldn't exist. In the aforementioned email, I explicitly say that IF a subpackage is enabled by an option, the option must be enabled by default. Your previous comments contributed to making this point clearer. > Dependencies stated in the port Makefile are converted into package > dependencies that can be resolved by the "pkg" command, but that cannot > be directly used to build and install the requested sub-package from > a port. > I don't know where you get this idea, but it's not how it works. If a port has a subpackage dependency, the category/origin~subpkg is the chosen format to express that dependency. The related port (category/origin) has to be built and installed. By running `make install`, the whole port is installed, subpackages as well. The only "issue" I see is that via 'make install' you cannot install only the subpackage, but the entire port only. However this is not different than before. Has there been a general consensus that support for direct port building > (without poudriere) will be abandoned? > Port building is not abandoned, and it's actively supported, as it's the foundation of poudriere. If a regression has been introduced, by me or anyone else, it has to be fixed. I may have overlooked some use cases, but AFAIK I didn't introduce any regression (confirmed by the exp-run) > Ports that do not create sub-packages can still be depended on by other > ports, but as critical dependencies have been depended to sub-packages > a ever large fraction of ports will only build in poudriere. > Again, no. If you run make install on any port, it will be built. The package and the subpackages are all installed. You cannot install a portion of a port, and it has never been possible. Nothing is broken here and I fail to understand where you get the idea of this behavior. I wrote tests and examples to implement the feature and get them working before adding support to poudriere. This change does also obviously break port management tools like portmaster, > which took me significant effort to adapt to FLAVOR support (which also had > been implemented without consideration for other tools than poudriere), and > which I have been maintaining since then. > Yes, portmaster need to be able to parse the new dependencies format, by removing the ~subpkg. By removing the suffix, you get the category/port and everything is fine. As announced, we are keeping subpackages adoption blocked with a git-hook to give time to maintainers to add support to subpkg and to introduce the feature slowly. portmgr@ doesn't officially support neither portmaster nor portupgrade. We simply lack the manpower to do it. The reason given for sub-packages support is reduced build time for some > packages that share a common distfile (e.g. qt5) when building official > packages. > As explained, in the short term there are this benefit and getting rid of many master/slave ports (hard to maintain) But there are several use cases in the future. For instance, work is in place to provide debug symbols as subpackage, in a similar way as pkg-base. > But this comes at a high cost for all builds outside the package build > cluster, since now lots of unnecessary sub-trees will be compiled and > installed, if only one program (i.e. sub-package) is desired. > This is the reason why OPTIONS with SUBPACAKGES have been introduced, to reduce build time for port builders. > You are pessimizing the build for thousands of users to spare a few > cycles for a very small percentage of packages built once in a while on > the build cluster. > I have already explained this point above. I had pointed out other issues with this approach in the review comment > and in the private mail I sent 2024-01-02 after finding that this version > has been committed without formal acceptance of your review D40549. > As I replied to you already, there has been no formal acceptance in phabricator, but there was consensus in portmgr@ to land it. I apologize for not having used the appropriate reviews channel, I totally agree that it has not been a good behavior from my side, as I'm not providing a good example. You are breaking use-cases of a large number of users that still build > ports without poudriere. This is especially causing a barrier to entry > of new port developers, since this will require them to setup poudriere > before they can begin port development. (This will only become an issue > over time, as more and more dependencies will have been converted to > sub-packages, but then there will be no way back.) > I still fail to see what is going to break. Non-poudriere users can experience longer build time for dependencies, but dependencies can be installed via packages, at least most of the time. Your announcement mentions some of the issues, but does not offer any > actual solution. > A sane implementation of sub-packages would not make their creation > depend on OPTION settings, with no way to determine the required OPTION > setting for a non-default sub-package (i.e., only default sub-packages > can be depended on). > > I'm not going to repeat all the issues pointed out in > > https://reviews.freebsd.org/D16457#715443 > > but had on multiple occasions pointed out that a sane mechanism would > start with "SUB_PACKAGE_DEFINE" and "SUB_PACKAGE_DEFAULT" variables and > with a mechanism to determine the required OPTION values from the actual > set of sub-packages to be built (instead of the opposite, to make the > selection of sub-packages depend on the OPTIONs). > I clearly addressed this topic in the aforementioned email. We chose not to follow this approach, and I motivated it (changing configuration of other ports it's NOT something to have IMO). We clearly disagree on this point. I have opened up the discussion to get feedback from the rest of the community as well. So far, you are the only one strongly against this approach. Best regards, pizzamig --000000000000cdfc6e060fca984e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Hi Stefan,</div><div><br></div><div>= I did reply to your first email, but not to your second one.<br> I preferre= d (in agreement with portmgr@) to open the discussion with everyone, instea= d of keeping it just between you and me.<br></div><div><br></div><div>As yo= u can read in the email you have linked, I didn't ignore your comments.= </div></div><div><br></div><div class=3D"gmail_quote"><blockquote class=3D"= gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex"> This implementation will break port dependencies, since there is no way<br> a port can depend on a specific sub-package - there even is no way a<br> non-default sub-package can be built without manual selection of the<br> options that activate its creation.<br></blockquote><div>A port can depend = on a specific sub-package. The category/origin~subpkg is the chosen format.= </div><div>Non-default subpackages shouldn't exist. In the aforemention= ed email, I explicitly say that IF a subpackage is enabled by an option, th= e option must be enabled by default.<br>Your previous comments contributed = to making this point clearer.</div><div>=C2=A0<br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex"> Dependencies stated in the port Makefile are converted into package<br> dependencies that can be resolved by the "pkg" command, but that = cannot<br> be directly used to build and install the requested sub-package from<br> a port.<br></blockquote><div>I don't know where you get this idea, but = it's not how it works.<br></div></div><div>If a port has a subpackage d= ependency, the category/origin~subpkg is the chosen format to express that = dependency.<br> The related port (category/origin) has to be built and inst= alled.</div><div>By running `make install`, the whole port is installed, su= bpackages as well.<br></div><div> The only "issue" I see is that = via 'make install' you cannot install only the subpackage, but the = entire port only.<br></div><div>However this is not different than before.<= br></div><div><br></div><div class=3D"gmail_quote"><blockquote class=3D"gma= il_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,2= 04,204);padding-left:1ex"> Has there been a general consensus that support for direct port building<br= > (without poudriere) will be abandoned?<br></blockquote><div>Port building i= s not abandoned, and it's actively supported, as it's the foundatio= n of poudriere.</div><div>If a regression has been introduced, by me or any= one else, it has to be fixed.</div><div>I may have overlooked some use case= s, but AFAIK I didn't introduce any regression (confirmed by the exp-ru= n)<br></div><div>=C2=A0<br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le= ft:1ex"> Ports that do not create sub-packages can still be depended on by other<br> ports, but as critical dependencies have been depended to sub-packages<br> a ever large fraction of ports will only build in poudriere. <br></blockquo= te><div>Again, no.</div><div>If you run make install on any port, it will b= e built. The package and the subpackages are all installed.</div><div>You c= annot install a portion of a port, and it has never been possible.</div><di= v>Nothing is broken here and I fail to understand where you get the idea of= this behavior.</div><div>I wrote tests and examples to implement the featu= re and get them working before adding support to poudriere.<br></div><div><= br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e= x;border-left:1px solid rgb(204,204,204);padding-left:1ex"> This change does also obviously break port management tools like portmaster= ,<br> which took me significant effort to adapt to FLAVOR support (which also had= <br> been implemented without consideration for other tools than poudriere), and= <br> which I have been maintaining since then.<br></blockquote><div>Yes, portmas= ter need to be able to parse the new dependencies format, by removing the ~= subpkg. By removing the suffix, you get the category/port and everything is= fine.<br></div><div>As announced, we are keeping subpackages adoption bloc= ked with a git-hook to give time to maintainers to add support to subpkg an= d to introduce the feature slowly.<br></div><div>portmgr@ doesn't offic= ially support neither portmaster nor portupgrade. We simply lack the manpow= er to do it.<br></div><div><br></div><blockquote class=3D"gmail_quote" styl= e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin= g-left:1ex"> The reason given for sub-packages support is reduced build time for some<br= > packages that share a common distfile (e.g. qt5) when building official<br> packages.<br></blockquote><div>As explained, in the short term there are th= is benefit and getting rid of many master/slave ports (hard to maintain)<br= ></div><div>But there are several use cases in the future.</div><div>For in= stance, work is in place to provide debug symbols as subpackage, in a simil= ar way as pkg-base.<br></div><div>=C2=A0<br></div><blockquote class=3D"gmai= l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20= 4,204);padding-left:1ex"> But this comes at a high cost for all builds outside the package build<br> cluster, since now lots of unnecessary sub-trees will be compiled and<br> installed, if only one program (i.e. sub-package) is desired.<br></blockquo= te><div>This is the reason why OPTIONS with SUBPACAKGES have been introduce= d, to reduce build time for port builders.<br></div><div>=C2=A0<br></div><b= lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le= ft:1px solid rgb(204,204,204);padding-left:1ex"> You are pessimizing the build for thousands of users to spare a few<br> cycles for a very small percentage of packages built once in a while on<br> the build cluster.<br></blockquote><div>I have already explained this point= above.<br></div><div><br></div><blockquote class=3D"gmail_quote" style=3D"= margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef= t:1ex"> I had pointed out other issues with this approach in the review comment<br> and in the private mail I sent 2024-01-02 after finding that this version<b= r> has been committed without formal acceptance of your review D40549.<br></bl= ockquote><div>As I replied to you already, there has been no formal accepta= nce in phabricator, but there was consensus in portmgr@ to land it.</div><d= iv>I apologize for not having used the appropriate reviews channel, I total= ly agree that it has not been a good behavior from my side, as I'm not = providing a good example.<br></div><div><br></div><blockquote class=3D"gmai= l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20= 4,204);padding-left:1ex"> You are breaking use-cases of a large number of users that still build<br> ports without poudriere. This is especially causing a barrier to entry<br> of new port developers, since this will require them to setup poudriere<br> before they can begin port development. (This will only become an issue<br> over time, as more and more dependencies will have been converted to<br> sub-packages, but then there will be no way back.)<br></blockquote><div>I s= till fail to see what is going to break.</div>Non-poudriere users can exper= ience longer build time for dependencies, but dependencies can be installed= via packages, at least most of the time.</div><div class=3D"gmail_quote"><= br></div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"> Your announcement mentions some of the issues, but does not offer any<br> actual solution.<br></blockquote><div><br></div><blockquote class=3D"gmail_= quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,= 204);padding-left:1ex"> A sane implementation of sub-packages would not make their creation<br> depend on OPTION settings, with no way to determine the required OPTION<br> setting for a non-default sub-package (i.e., only default sub-packages<br> can be depended on).<br> <br> I'm not going to repeat all the issues pointed out in<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"https://reviews.freebsd.org/D16457#7= 15443" rel=3D"noreferrer" target=3D"_blank">https://reviews.freebsd.org/D16= 457#715443</a><br> <br> but had on multiple occasions pointed out that a sane mechanism would<br> start with "SUB_PACKAGE_DEFINE" and "SUB_PACKAGE_DEFAULT&quo= t; variables and<br> with a mechanism to determine the required OPTION values from the actual<br= > set of sub-packages to be built (instead of the opposite, to make the<br> selection of sub-packages depend on the OPTIONs).<br></blockquote><div>I cl= early addressed this topic in the aforementioned email.</div><div>We chose = not to follow this approach, and I motivated it (changing configuration of = other ports it's NOT something to have IMO).<br></div><div>We clearly d= isagree on this point.</div><div>I have opened up the discussion to get fee= dback from the rest of the community as well.<br></div><div>So far, you are= the only one strongly against this approach.</div><div><br></div><div>Best= regards,<br></div><div>pizzamig<br></div><div><br></div></div></div> --000000000000cdfc6e060fca984e--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w>