Date: Wed, 24 Jan 2024 10:28:48 +0100 From: Luca Pizzamiglio <pizzamig@freebsd.org> To: FreeBSD Ports mailing list <freebsd-ports@freebsd.org>, freebsd-ports <ports@freebsd.org> Subject: Subpackage explanations Message-ID: <CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--000000000000628b99060fadb173 Content-Type: text/plain; charset="UTF-8" Hi porters! At the beginning of January, we merged the support to subpackages in the framework. Subpackage is the feature to create multiple packages from one build of one port. In other words, now it's possible to group files into multiple packages. This means that from one port it's possible to split the build into several packages. Some additional details are available in this lighting talk at EuroBSD 2023 (https://youtu.be/e-FUYbGNdBg?t=824). *Use cases we want to tackle* The first use case we want to get rid of is master/slave ports when slave ports could be built with the master port. An example (already merged) is https://reviews.freebsd.org/D43445: devel/appstream will create also the -compose and -qt6 binaries as subpackage in one build. Another use case is ports that could be split in smaller units. An example is devel/qt6-tools: this port contains all qt6 related utilities. The package could be split into smaller subpackages, one per tool. Ports that depend on one of the qt6 tools could then install only the smaller subpackage, instead of the whole collection of tools. Those first use cases should provide some benefit to build times (i.e. extract/patch/configure will happen only once, build dependencies are smaller, then faster to install) on the package clusters, but they could increase build times for folks building ports. *Use cases we don't want to tackle (yet)* Subpackages enable the adoption of micro-subpackages, a typical pattern for Linux distributions that split a package in smaller ones: one with docs (-doc), one with static libraries and headers (-dev), one with manpages (-man), one with examples (-examples), and so on. While technically this could be easy to do, it's a major change for users and there needs to be a discussion in the community on whether it is a desirable feature. For this reason, we are not proceeding with this *SUBPACKAGES and OPTIONS* Subpackages can affect build time negatively for users that don't use default packages, but they build packages directly from ports. Let's take the appstream port as an example. Before subpackages, you could selectively build appstream and anything else. After subpackages, the appstream port is going to build appstream, appstream-compose and appstream-qt6 (and the related dependencies). This can considerably increase build time for some folks. One way to mitigate it is to add a mechanism to make subpackages optional, via OPTIONS (currently supported). An OPTION for -compose and one for -qt6 can enable/disable those subpackages, restoring the shorter build time. However, optional subpackages can cause unexpected results when other ports depend on subpackages. Let's imagine a port depending on appstream-compose, but appstream has the compose OPTION disabled. There are two possible approaches: 1. the build of the port is going to fail as appstream is not building the dependency (the subpackage is disabled by the OPTION) 2. the build is going to change the configuration of appstream, enabling the compose OPTION, to build the dependency. We chose the approach #1: even if the build fails, it shows which package fails to be installed, providing insights on how to fix it. Approach #2 introduces (hidden) side effects and surprises, like unexpected dependencies or custom configuration ignored. The benefit is that the dependent port is successfully built, ignoring the effect of changing configuration of the dependencies. As both approaches have their own pros and cons, anyone can have a different opinion on that. To limit the effects on the selected approach: * all options enabling subpkg has to be ON by default, to not introduce broken dependencies * we encourage to limit the introduction of those optional subpackages, limiting their adoption only for cases where the related subpackage adds a relevant set of dependencies Additionally, the approach #1 could be improved by detecting such use cases and providing a meaningful error message. *Adoption and documentation* Subpackages are already in the framework, but while we start to adopt them for a couple of ports, we are still finding a few details that needs to be clarified (i.e. MOVED). Subpackages' adoption is blocked atm by a git hook (server side). The hook can be unblocked via portmgr approval (push option allow-subpkg). We encourage port committers to start thinking which of their ports is suitable to use subpackages or not. We don't have official documentation yet, I'm going to work on that in the next few weeks. On phabricator, there is a patch (https://reviews.freebsd.org/D43079) containing two fake ports, used to build and test subpackages. They provide the best example for now of how subpackages look like. *Tooling support* poudriere already supports subpackages. portupgrade, portmaster and synth don't support subpackages. We kindly ask the maintainers of those projects to add the support to subpackages. Support in portlint is in progress. A git commit hook will be developed to ensure that all subpackages are built by default. Best regards, pizzamig on behalf of portmgr --000000000000628b99060fadb173 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr">Hi porters!<div><br></div><div>At the beg= inning of January, we merged the support to subpackages in the framework.</= div><div>Subpackage is the feature to create multiple packages from one build of one port. In = other=20 words, now it's possible to group files into multiple packages. This=20 means that from one port it's possible to split the build into several = packages.</div><div>Some additional details are available in this lighting = talk at EuroBSD 2023 (<a href=3D"https://youtu.be/e-FUYbGNdBg?t=3D824" targ= et=3D"_blank">https://youtu.be/e-FUYbGNdBg?t=3D824</a>).</div><div><br></di= v><div><b>Use cases we want to tackle</b></div><div>The first use case we w= ant to get rid of is master/slave ports when slave ports could be built wit= h the master port.</div><div>An example (already merged) is=C2=A0<a href=3D= "https://reviews.freebsd.org/D43445" target=3D"_blank">https://reviews.free= bsd.org/D43445</a>: devel/appstream will create also the -compose and -qt6 = binaries as subpackage in one build. <br></div><div><br></div><div>Another use case is ports that could be split in smaller units. An example is=20 devel/qt6-tools: this port contains all qt6 related utilities. The=20 package could be split into smaller subpackages, one per tool.</div><div>Po= rts that depend on one of the qt6 tools could then install only the=20 smaller subpackage, instead of the whole collection of tools.</div><div><br= ></div><div>Those first use cases should provide some benefit to build times (i.e.=20 extract/patch/configure will happen only once, build dependencies are=20 smaller, then faster to install) on the package clusters, but they could in= crease build times for folks building ports.<br></div><div><br></div><div><= b>Use cases we don't want to tackle (yet)</b><br></div><div>Subpackages enable the adoption of micro-subpackages, a typical pattern for Linux=20 distributions that split a package in smaller ones: one with docs=20 (-doc), one with static libraries and headers (-dev), one with manpages=20 (-man), one with examples (-examples), and so on.</div><div>While=20 technically this could be easy to do, it's a major change for users and= =20 there needs to be a discussion in the community on whether it is a desirabl= e feature. For this reason, we are not proceeding with this <br></div><br><= div><b>SUBPACKAGES and OPTIONS</b></div><div>Subpackages can affect build t= ime negatively for users that don't use default packages, but they buil= d packages directly from ports.<br></div><div>Let's take the appstream = port as an example.</div><div>Before subpackages, you could selectively bui= ld appstream and anything else.</div><div>After subpackages, the appstream port is going to build appstream,=20 appstream-compose and appstream-qt6 (and the related dependencies).</div><d= iv>This can considerably increase build time for some folks.</div><div>One way to mitigate it is to add a mechanism to make subpackages optional,=20 via OPTIONS (currently supported). An OPTION for -compose and one for=20 -qt6 can enable/disable those subpackages, restoring the shorter build time= .</div><div>However, optional subpackages can cause unexpected results when= other ports depend on subpackages.</div><div>Let's imagine a port depe= nding on appstream-compose, but appstream has the compose OPTION disabled.<= br></div><div>There are two possible approaches:</div><div>1. the build of = the port is going to fail as appstream is not building the dependency (the = subpackage is disabled by the OPTION)<br></div><div>2. the build is going t= o change the configuration of appstream, enabling the compose OPTION, to bu= ild the dependency.<br></div><div><br></div><div>We chose the approach #1: even if the build fails, it shows which package=20 fails to be installed, providing insights on how to fix it.<br></div><div>A= pproach #2 introduces (hidden) side effects and surprises, like unexpected=20 dependencies or custom configuration ignored. The benefit is that=20 the dependent port is successfully built, ignoring the effect of changing c= onfiguration of the=20 dependencies.<br></div><div>As both approaches have their own pros and cons= , anyone can have a different opinion on that.<br>To limit the effects on t= he selected approach:<br>* all options enabling subpkg has to be ON by defa= ult, to not introduce broken dependencies<br></div><div>* we encourage to limit the introduction of those optional subpackages, limi= ting their adoption only for cases where the related subpackage adds a relevant=20 set of dependencies<br></div><div>Additionally, the approach #1 could be im= proved by detecting such use cases and providing a meaningful error message= .</div><br><div><div><b>Adoption and documentation<br></b></div><div>Subpac= kages are already in the framework, but while we start to adopt them for a=20 couple of ports, we are still finding a few details that needs to be=20 clarified (i.e. MOVED).<br></div><div>Subpackages' adoption is blocked= =20 atm by a git hook (server side). The hook can be unblocked via portmgr=20 approval (push option allow-subpkg).</div><div>We encourage port committers= to start thinking which of their ports is suitable to use subpackages or n= ot.<br></div><div>We don't have official documentation yet, I'm goi= ng to work on that in the next few weeks.<br></div>On phabricator, there is= a patch (<a href=3D"https://reviews.freebsd.org/D43079" target=3D"_blank">= https://reviews.freebsd.org/D43079</a>) containing two fake ports, used to build and test subpackages.</div><div>T= hey=20 provide the best example for now of how subpackages look like.</div><div><b= r></div><div><b>Tooling support<br></b></div><div>poudriere already support= s subpackages.</div><div>portupgrade, portmaster and synth don't support subpackages. We kindly ask the=20 maintainers of those projects to add the support to subpackages.</div><div>= Support in portlint is in progress.</div><div>A git commit hook will be dev= eloped to ensure that all subpackages are built by default.<br></div><div><= div><br></div><div><div>Best regards,<br></div><div>pizzamig on behalf of p= ortmgr</div></div></div></div></div> --000000000000628b99060fadb173--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB88xy-8hAknWJDRBjbJo2%2Bw878ZMosKcvQbpKVzwq%2BH7%2Bzuyg>