Date: Fri, 26 Jan 2024 17:08:04 +0100 From: Stefan Esser <se@freebsd.org> To: Luca Pizzamiglio <pizzamig@freebsd.org> Cc: FreeBSD ports <freebsd-ports@freebsd.org>, portmgr <portmgr@FreeBSD.org>, FreeBSD Core Team <core@freebsd.org> Subject: Re: This is going to break port building without poudriere! Message-ID: <609374cf-7f1f-4643-8379-f368b23ccb09@freebsd.org> In-Reply-To: <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com> References: <CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg@mail.gmail.com> <cd0c0cb0-6035-45b4-b3e8-d99115e6c013@FreeBSD.org> <CAB88xy8gTC4UJK0fOiHnVCFf0AGtLoHfHdOAF29zChQ8=5SV6w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Am 25.01.24 um 20:57 schrieb Luca Pizzamiglio: > Hi Stefan, > > I did reply to your first email, but not to your second one. > I preferred (in agreement with portmgr@) to open the discussion with everyone, > instead of keeping it just between you and me. > > As you can read in the email you have linked, I didn't ignore your comments. You have mentioned some of the issues without providing a solution. > This implementation will break port dependencies, since there is no way > a port can depend on a specific sub-package - there even is no way a > non-default sub-package can be built without manual selection of the > options that activate its creation. > > A port can depend on a specific sub-package. The category/origin~subpkg is the > chosen format. > Non-default subpackages shouldn't exist. In the aforementioned email, I > explicitly say that IF a subpackage is enabled by an option, the option must be > enabled by default. > Your previous comments contributed to making this point clearer. If there should be no "non-default subpackages", then selecting subpackages to install by OPTIONs makes no sense. But there are many scenarios where not all subpackages can be built at the same time due to conlflicting dependencies or requirements (often controlled by OPTIONs). > Dependencies stated in the port Makefile are converted into package > dependencies that can be resolved by the "pkg" command, but that cannot > be directly used to build and install the requested sub-package from > a port. > > I don't know where you get this idea, but it's not how it works. > If a port has a subpackage dependency, the category/origin~subpkg is the chosen > format to express that dependency. Yes, but how will this dependency be processed in the ports system? Will the port at the given origin be built (all subpackages) and all the available subpackages will be installed? How do you deal with the case, where o ne of those subpackages conflicts with an already installed port? What actually works is the mapping of the origin~subpackage into a package dependency that can be resolved by the pkg command. That's what I am complaining about: only sucgh package dependencies continue to work, but port dependencies don't in the same way as without subpackages. > The related port (category/origin) has to be built and installed. > By running `make install`, the whole port is installed, subpackages as well. > The only "issue" I see is that via 'make install' you cannot install only the > subpackage, but the entire port only. > However this is not different than before. Except that before there was not one port that built potentially tens of subpackages. > Has there been a general consensus that support for direct port building > (without poudriere) will be abandoned? > > Port building is not abandoned, and it's actively supported, as it's the > foundation of poudriere. The case of poudriere is not generic port building, since all dependencies are installed from packages in poudriere. Poudriere never installs from a port, only from packages, and it can deal with subpackage dependencies. Port building on the base system uses a completely different approach than package building in poudriere. > If a regression has been introduced, by me or anyone else, it has to be fixed. > I may have overlooked some use cases, but AFAIK I didn't introduce any > regression (confirmed by the exp-run) The exp-run is performed by building packages with poudriere. What has this to do with the port building on the base system that I'm worried about? Besides, you only showed that you did not break the ports system without subpackages by adding features for the support of subpackages to bsd.port.mk. The relevant test case would have been an exp-run with several relevant ports converted to subpackages. But even then, port building outside poudriere is different and may be severely broken without any issues reported by an exp-run, even one that tests actual subpackages. > Ports that do not create sub-packages can still be depended on by other > ports, but as critical dependencies have been depended to sub-packages > a ever large fraction of ports will only build in poudriere. > > Again, no. > If you run make install on any port, it will be built. The package and the > subpackages are all installed. Yes, and that is wrong! > You cannot install a portion of a port, and it has never been possible. No, but with subpackages it will become possible to install a portion of a port from a package. And since the goal is to convert ports that share a distfile to subpackages, this will become a common case for large ports. > Nothing is broken here and I fail to understand where you get the idea of this > behavior. Because I consider just the above as breaking important features of the current ports system (the ability to selectively install just the ports that I need and not all that can potentially be built from one distfile). > I wrote tests and examples to implement the feature and get them working before > adding support to poudriere. I have seen your tests and they are only relevant for the package dependency case (dependencies resolved by the pkg command). > This change does also obviously break port management tools like portmaster, > which took me significant effort to adapt to FLAVOR support (which also had > been implemented without consideration for other tools than poudriere), and > which I have been maintaining since then. > > Yes, portmaster need to be able to parse the new dependencies format, by > removing the ~subpkg. By removing the suffix, you get the category/port and > everything is fine. No, I get a whole lot of subpackages that I did not ask for! > As announced, we are keeping subpackages adoption blocked with a git-hook to > give time to maintainers to add support to subpkg and to introduce the feature > slowly. The damage will be to other users than the committers of these ports, and even developers that depend on them might not notice any issues, since they are using poudriere to test their ports. > portmgr@ doesn't officially support neither portmaster nor portupgrade. We > simply lack the manpower to do it. I do not expect PortMgr to support either of these tools. But I do expect PortMgr to not break features of the port system due to neglect of building ports on the base system. I know that there have been discussions about abandoning the base system builds (especially since in some cases ports do build in a clean jail, but not with a previous version installed, e.g. because of preferring installed header files above those in the source director<). But most ports build just fine on the base system and this is the fastest way for a developer to get some required functionality with free choice of the OPTIONs (when different from those of a pre-built package, for example). > The reason given for sub-packages support is reduced build time for some > packages that share a common distfile (e.g. qt5) when building official > packages. > > As explained, in the short term there are this benefit and getting rid of many > master/slave ports (hard to maintain). This does only work if there is e.g. a port that can be compiled for CLI and GUI, for example, but that does not need special build options to include the required hooks to the added functionality. I have seen the devel/appstream port as of https://reviews.freebsd.org/D43445, and it clearly shows the weakness of the subpackages implementation. You did not get rid of a master/slave port combination, but you have now combined the CLI and Qt6 versions (while there previously was one port for the CLI version and one flavored port for Qt5/Qt6). And devel/appstream/Makefile has become a lot more complex and the complexity of having both a slave port and subpackages makes it even worse. If I build the CLI version after subpackages, I get all of Qt6 as a dependency! And Qt6 is not only fetched and built, but also installed (when building the port as a dependency on the base system), whether I need it or not. > But there are several use cases in the future. > For instance, work is in place to provide debug symbols as subpackage, in a > similar way as pkg-base. There is nothing wrong with such use-cases, and I do not object the support of subpackages at all. I had just been hoping that my comments to the earlier review would lead to a better design - which would have easily been possible (and I had worked out a concept several years ago, which I had discussed with some members of PortMgr and possibly also in mails to the ports maillist). > But this comes at a high cost for all builds outside the package build > cluster, since now lots of unnecessary sub-trees will be compiled and > installed, if only one program (i.e. sub-package) is desired. > > This is the reason why OPTIONS with SUBPACAKGES have been introduced, to reduce > build time for port builders. But that does not work well if there is no way to deduce the required OPTIONS from the required subpackages. And it requires manual intervention. There should have been variables like e.g. SUBPKG_DEFINE; SUBPKG_DEFAULT, SUBPKG_SELECT which control the building and installation of subpackages, and OPTIONs might depend on the selected set of subpackages (e.g., if a ~docs subpackage is selected, the DOCS option could be acticated). > You are pessimizing the build for thousands of users to spare a few > cycles for a very small percentage of packages built once in a while on > the build cluster. > > I have already explained this point above. I did not see a valid explanation. Thousands (or more) port builders will waste cycles and space for subpackages that are not actually wanted, since there is no mechanism that allows to easily restrict the amount built (without resorting to OPTIONs, but those are sticky and the next port to be built may require subpackages controlled by other options, this is not a workable conept). > I had pointed out other issues with this approach in the review comment > and in the private mail I sent 2024-01-02 after finding that this version > has been committed without formal acceptance of your review D40549. > > As I replied to you already, there has been no formal acceptance in > phabricator, but there was consensus in portmgr@ to land it. > I apologize for not having used the appropriate reviews channel, I totally > agree that it has not been a good behavior from my side, as I'm not providing a > good example. As explained before, I had mostly given up on FreeBSD in late 2022, mostly because of frustration about PortMgr. I had stopped reading most of the FreeBSD mail lists and might have missed an early announcement of the plan to commit your Phabricator review. As you know, I had hoped to be able to stear the subpackages development in a better direction as part of the PortMgr team, but there was an objection of the author of the original subpackages review (that your commit is based on), probably because of my earlier comment on his approach. > You are breaking use-cases of a large number of users that still build > ports without poudriere. This is especially causing a barrier to entry > of new port developers, since this will require them to setup poudriere > before they can begin port development. (This will only become an issue > over time, as more and more dependencies will have been converted to > sub-packages, but then there will be no way back.) > > I still fail to see what is going to break. > Non-poudriere users can experience longer build time for dependencies, but > dependencies can be installed via packages, at least most of the time. This "dependencies can be installed via packages" is just the sentiment of PortMgr that has no interest for port building in the base system. Some dependencies can be installed from packages, but in many cases that leads to lots of already installed programs to be substituted with packages built with different OPTIONs. Mixing packages and ports is not working well, and the only sane possibilities are a local repository built with custom options using poudriere, or just building everything from a port on the base system. > Your announcement mentions some of the issues, but does not offer any > actual solution. > > > A sane implementation of sub-packages would not make their creation > depend on OPTION settings, with no way to determine the required OPTION > setting for a non-default sub-package (i.e., only default sub-packages > can be depended on). > > I'm not going to repeat all the issues pointed out in > > https://reviews.freebsd.org/D16457#715443 > <https://reviews.freebsd.org/D16457#715443>; > > but had on multiple occasions pointed out that a sane mechanism would > start with "SUB_PACKAGE_DEFINE" and "SUB_PACKAGE_DEFAULT" variables and > with a mechanism to determine the required OPTION values from the actual > set of sub-packages to be built (instead of the opposite, to make the > selection of sub-packages depend on the OPTIONs). > > I clearly addressed this topic in the aforementioned email. Sorry, I did not see these points addressed. Please give specific pointers. IMHO, all of the issues in my comment to review D16457 are still open. > We chose not to follow this approach, and I motivated it (changing > configuration of other ports it's NOT something to have IMO). What are you talking about ("changing configuration of other ports") ??? > We clearly disagree on this point. > I have opened up the discussion to get feedback from the rest of the community > as well. > So far, you are the only one strongly against this approach. Maybe this is because other developers will only notice the effects of this change when many ports have been converted to subpackages? This will make it much harder for new ports contributors to work on their first port, since port builds outside poudriere will become penalized more and more over time, as subpackages are introduced (and become dependencies). I have deep insight into the ports framework since I needed to understand it when I took over portmaster and added FLAVOR support to it. And as a certified Common Criteria evaluator (plus other standards) I might have a different view on aspects of software development. I have mostly given up my attempts to help improve the ports system, for lack of support (or at least not being ignored when I ask for a review) by PortMgr. There are a lot of sometimes trivial changes, but my past experience with PortMgr is that I cannot hope for any kind of review to suggested changes, since everybody is busy working in his field of interest and does not care for any outside contributions (and my attempt to contribute from within PortMgr has been blocked, too). Best regards, STefan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?609374cf-7f1f-4643-8379-f368b23ccb09>