From nobody Tue Jan 30 09:42:20 2024 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TPKw11Tjjz58Vn4 for ; Tue, 30 Jan 2024 09:42:41 +0000 (UTC) (envelope-from luca.pizzamiglio@gmail.com) Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TPKvz46qgz472R; Tue, 30 Jan 2024 09:42:39 +0000 (UTC) (envelope-from luca.pizzamiglio@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of luca.pizzamiglio@gmail.com designates 209.85.166.45 as permitted sender) smtp.mailfrom=luca.pizzamiglio@gmail.com Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-7bed8faf6ebso132198039f.2; Tue, 30 Jan 2024 01:42:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706607757; x=1707212557; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5mc46pwGMcJbldxZK4HnPjvx/B6cLH9ugtmTrEum4jk=; b=Oko9DggRqdxaccx6rebLKrE2k9jdAeB+cv+mOLL82k7PnqN4MX+Clf3qSPb+wugSHE OJyvP039E1N8ljiUIBvDtIXYKwrzv6kFrD9l8MGzH03jZcs3nXL+trXNoPS+GbJVgjzF RbVIOsqPcznpW7Yd0eMXxSds1AABbmIJQ/i1+6b3WuY4SR6vJQUHFhcyifaaaGufTfOh io/FMcSrlXxTUdJQD53HteAC2MKpp0e+wHIjNrGoReaaFK9DbiBgwWoOClrldr2Y9P0U z6YxaqFLPFu6UDPk+UhZjbB4Hq0JcNJiOq9cvcVYDHcGJ690goOAW0xCdtIPNEiG0k7B ewhA== X-Gm-Message-State: AOJu0YyWiEz1043fbH4jFIekxcVhNpLkP1KnRdg4juVpfGqJSxu3jnP4 FD1JRM4jL339klXAnlCgv/qXR+pr3kMcWzUREnEXbZvfUqgjHIaqWiVY7QR5LKY= X-Google-Smtp-Source: AGHT+IHOdGEZHserP9MLyvmg0gy7Stu6XJ6rdwYNTGA2lMaOj9wPTlUYEixCq6rpf82IjZvh/XGusw== X-Received: by 2002:a05:6e02:1d15:b0:362:8bdf:45cc with SMTP id i21-20020a056e021d1500b003628bdf45ccmr9279190ila.17.1706607757065; Tue, 30 Jan 2024 01:42:37 -0800 (PST) Received: from mail-il1-f175.google.com (mail-il1-f175.google.com. [209.85.166.175]) by smtp.gmail.com with ESMTPSA id k2-20020a92c9c2000000b00363789a0b23sm1845923ilq.6.2024.01.30.01.42.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 30 Jan 2024 01:42:36 -0800 (PST) Received: by mail-il1-f175.google.com with SMTP id e9e14a558f8ab-363890fe589so3585585ab.0; Tue, 30 Jan 2024 01:42:36 -0800 (PST) X-Received: by 2002:a05:6e02:f46:b0:363:812d:d6a3 with SMTP id y6-20020a056e020f4600b00363812dd6a3mr3963416ilj.20.1706607756562; Tue, 30 Jan 2024 01:42:36 -0800 (PST) List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 References: <609374cf-7f1f-4643-8379-f368b23ccb09@freebsd.org> <1e27bce44f2ba85f0097fbc6aba1dac1@Leidinger.net> In-Reply-To: <1e27bce44f2ba85f0097fbc6aba1dac1@Leidinger.net> From: Luca Pizzamiglio Date: Tue, 30 Jan 2024 10:42:20 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: This is going to break port building without poudriere! To: Alexander Leidinger Cc: FreeBSD ports , portmgr , FreeBSD Core Team Content-Type: multipart/alternative; boundary="00000000000019db290610269536" X-Spamd-Bar: / X-Spamd-Result: default: False [-0.92 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.98)[0.983]; FORGED_SENDER(0.30)[pizzamig@freebsd.org,lucapizzamiglio@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; SUBJECT_ENDS_EXCLAIM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TAGGED_FROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-ports@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[pizzamig@freebsd.org,lucapizzamiglio@gmail.com]; RCPT_COUNT_THREE(0.00)[4]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.166.45:from]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_IN_DNSWL_NONE(0.00)[209.85.166.45:from,209.85.166.175:received] X-Rspamd-Queue-Id: 4TPKvz46qgz472R --00000000000019db290610269536 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Alexander. Subpackage modularization of existing ports (i.e. qt6-tools) provides benefits to package users/builders: smaller dependency footprint, smaller usage on disk (useful for embedded systems), smaller jails. There are no benefits nor regressions for port builders for this specific use case. On the other hand, moving master/slave ports to subpackages comes with the cost of losing the ability to build the slave ports independently. For package users there is no change and some benefit for package builders. The issue can be mitigated by introducing options to select which subpackage to build (available for make install as well), but, still, a single subpackage cannot be built independently. This limitation *can* be unacceptable in some cases. For example, bofh@, the php maintainer, considers subpackages not the way to go, so the master/slave approach will stay. We introduced subpackages in the framework to explore all use cases, by trying and testing its adoption. By doing so we have already identified some issues (i.e. USES is not subpackage-aware yet) and interesting new use cases (subpackage with debug symbols). As per master/slave merge use case, we will let the maintainers decide if they want to move forward with subpackage adoptions, knowing the regression for port builders, but we won't push them in this direction. Best regards, Luca On Mon, Jan 29, 2024 at 10:04=E2=80=AFAM Alexander Leidinger < Alexander@leidinger.net> wrote: > Am 2024-01-27 16:59, schrieb Luca Pizzamiglio: > > > Hi Stefan. > > > > Let's start from the beginning, as it seems that things are not clear. > > > > Subpackages is a feature to create multiple packages from one build > > Those subpackages can depend on the main package. > > The main package cannot depend on any subpackages. > > This limits the cases where subpackages can be applied. The main > > package MUST be independent from its subpackages. Subpackages can only > > add features (like a plugin). > > To recall your NLS example (ref > > https://reviews.freebsd.org/D16457#715443): this is not a use-case for > > subpackages. If the main program/library needs to be compiled > > differently with or without NLS, this is not viable for subpackages. > > If a port needs to be built multiple times to create different > > subpackages, this is not a viable case for subpackages. > > A good candidate is qt6-tools: this port contains multiple tools > > (designer, linguist, help, and so on). Those tools could be put in > > different subpackages. > > > > I hope this explanation helps to clarify and address some of your > > concerns. > > As I read this, lang/php is the best example of a port where subpackages > has a benefit (in the sense it matches your description above to the > letter, the main port independent from the subpacakges, and what can be > build as subpackages is a plugin/extension). > > > OPTIONS and SUBPACKAGES > > Do we have to convert OPTIONS to SUBPACKAGES? No. > > Can a SUBPACKAGE be built only if an OPTION is enabled? Yes. > > The only viable use cases for SUBPACKAGES being enabled/disabled by > > OPTIONS is limited to those portions of the port that do not affect the > > main package. > > Examples are: additional libraries, additional data files, and so on. > > > > Consolidate master/slave ports in one bigger ports > > About this topic, I guess your concerns are mainly related to potential > > explosion of build time of the consolidated ports. > > This is a justified concern. > > In those cases, we are suggesting to convert slave ports in SUBPACKAGES > > enabled via OPTIONS. > > This will allow port builders to configure the bigger ports to not > > build all SUBPACKAGES, but only the needed ones. This should restore > > the previous build time. > > > > However, as for the php case, the maintainer is going to evaluate if > > the consolidation makes sense or not. > > If a consolidation is going to result more problematic than beneficial, > > it can be reverted and subpackages not adopted for the use case. > > If I understand the sum of all the above correct, you suggest to remove > slave ports and to use subpackages instead (where this makes sense in > terms of the current implementation of subpackages). Or worded > differently to the same effect (as I only care about the effect and not > the intention), when someone converts a port to subpackages, the > corresponding slave packages shall be removed (or for new ports: slave > ports shall not be introduced in this case). > > Removing slave ports means we can not depend upon specific parts anymore > when installing from ports, as the subpackages can not be targeted for > install directly and my example of a subpackages aware php results in > security implications of to much being installed and active if installed > via make install in /usr/ports/something/with_webinterface. -> best > example of lang/php to use subpacakges is the best example of why not to > use subpackages / shows what is a regression in the features we provide > in our ports collection. > > While qt6-tools may be a good example where subpackages makes sense, we > can not depend on subpacakges for "make install", and as if the port > would be converted to have subpackages but no slave ports are > introduced, pkg install and make install would differ in the amount of > installed files. > > > For port builders > > > > Port builders can experience longer build times in the future, as > > master/slave ports could be consolidated in one single larger ports. > > If this is the case, the larger ports should provide OPTIONS to not > > build unneeded subpackages. > > If no OPTION is available, please work with the maintainer to introduce > > one. > > I fail to see the benefit: > We either lose the possibility to target parts of a package (when slave > ports are removed / not introduced) on make install (with a similar > amount of build time for the ports tree as it is right now), or get > higher build times for the package builders. In both cases we do not > gain something significant. > > If we want to keep the (useful in some cases) feature to install from > ports (there is the case of py39-rpds-py failing to build in my > poudriere which I tried to debug with the author of py-maturin due to a > runtime issue in maturin, which shows up in my the cross build on > amd64-intel for amd64-athlon64... which in the end leads me to build > py-rpds-py on the target machine from ports), the current implementation > of subpackages has to come with the requirement to create slave ports > for each subpackage. > > That's my concern, and that's the reason why I have the opinion, that > portmgr has to keep the lock on subpackages and reject any subpackage > which don't come with a slave port. > > I would be OK to lift this restriction, when the current implementation > is changed to be able to only install the files of a subpacakge on make > install (an implementation could be to require "make install > TARGET_SUBPACAKGES=3Dsub1,sub2,sub3" or "make install-subpackage1 > install-subpacakge2"... as long as recursive dependencies are handled > according to this requirement, those people designing, implementing and > reviewing this can argue about such details). > > Keeping the current implementation (with the restriction of always > introducing slave packages for subpackages) would need a way to denote > that a slave port covers a specific subpackage which would allow package > builders to skip those slave ports (and the subpacakges would need to > have the same package name as the slave port, no idea if this has a > technical disadvantage in terms of having 2 different origins for the > same package name, but it surely would be confusing for people at first > look). > > TLDR: for the use cases you specified in the beginning, I do not see a > benefit, only drawbacks. Can you please provide an example of a benefit > I fail to see (yes, more modularity for qt6-tools may be beneficial for > some people, but the cost/benefit between qt6-tools (something which we > don't provide right now) and "make install" (what we provide right now) > or "build time reduction for package builders" (which would have a > benefit for a lot of use cases) is in my books much more in the > direction of "make install" and "build time reduction" than towards the > modularization of qt6-tools)? > > Bye, > Alexander. > > -- > http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF > http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF > --00000000000019db290610269536 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Alexander.

Subpackage mod= ularization of existing ports (i.e. qt6-tools) provides benefits to package= users/builders: smaller dependency footprint, smaller usage on disk (usefu= l for embedded systems), smaller jails. There are no benefits nor regressio= ns for port builders for this specific use case.

O= n the other hand, moving master/slave ports to subpackages comes with the c= ost of losing the ability to build the slave ports independently. For packa= ge users there is no change and some benefit for package builders.
The issue can be mitigated by introducing options to select which sub= package to build (available for make install as well), but, still, a single= subpackage cannot be built independently.
This limitation can be unacceptable in some cases. For example, bofh@, the php maintai= ner, considers subpackages not the way to go, so the master/slave approach = will stay.

We introduced subpackages in the fr= amework to explore all use cases, by trying and testing its adoption.
By doing so we have already identified some issues (i.e. USES is n= ot subpackage-aware yet) and interesting new use cases (subpackage with deb= ug symbols).
As per master/slave merge use case, we will let the = maintainers decide if they want to move forward with subpackage adoptions, = knowing the regression for port builders, but we won't push them in thi= s direction.

Best regards,
Luca
<= /div>


On Mon, Jan 29, 2024 at 10:04=E2=80=AFAM Alexander Leid= inger <Alexander@leidinger.ne= t> wrote:
Am 2024-01-27 16:59, schrieb Luca Pizzamiglio:

> Hi Stefan.
>
> Let's start from the beginning, as it seems that things are not cl= ear.
>
> Subpackages is a feature to create multiple packages from one build > Those subpackages can depend on the main package.
> The main package cannot depend on any subpackages.
> This limits the cases where subpackages can be applied. The main
> package MUST be independent from its subpackages. Subpackages can only=
> add features (like a plugin).
> To recall your NLS example (ref
> https://reviews.freebsd.org/D16457#715443): this i= s not a use-case for
> subpackages. If the main program/library needs to be compiled
> differently with or without NLS, this is not viable for subpackages. > If a port needs to be built multiple times to create different
> subpackages, this is not a viable case for subpackages.
> A good candidate is qt6-tools: this port contains multiple tools
> (designer, linguist, help, and so on). Those tools could be put in > different subpackages.
>
> I hope this explanation helps to clarify and address some of your
> concerns.

As I read this, lang/php is the best example of a port where subpackages has a benefit (in the sense it matches your description above to the
letter, the main port independent from the subpacakges, and what can be build as subpackages is a plugin/extension).

> OPTIONS and SUBPACKAGES
> Do we have to convert OPTIONS to SUBPACKAGES? No.
> Can a SUBPACKAGE be built only if an OPTION is enabled? Yes.
> The only viable use cases for SUBPACKAGES being enabled/disabled by > OPTIONS is limited to those portions of the port that do not affect th= e
> main package.
> Examples are: additional libraries, additional data files, and so on.<= br> >
> Consolidate master/slave ports in one bigger ports
> About this topic, I guess your concerns are mainly related to potentia= l
> explosion of build time of the consolidated ports.
> This is a justified concern.
> In those cases, we are suggesting to convert slave ports in SUBPACKAGE= S
> enabled via OPTIONS.
> This will allow port builders to configure the bigger ports to not > build all SUBPACKAGES, but only the needed ones. This should restore <= br> > the previous build time.
>
> However, as for the php case, the maintainer is going to evaluate if <= br> > the consolidation makes sense or not.
> If a consolidation is going to result more problematic than beneficial= ,
> it can be reverted and subpackages not adopted for the use case.

If I understand the sum of all the above correct, you suggest to remove slave ports and to use subpackages instead (where this makes sense in
terms of the current implementation of subpackages). Or worded
differently to the same effect (as I only care about the effect and not the intention), when someone converts a port to subpackages, the
corresponding slave packages shall be removed (or for new ports: slave
ports shall not be introduced in this case).

Removing slave ports means we can not depend upon specific parts anymore when installing from ports, as the subpackages can not be targeted for
install directly and my example of a subpackages aware php results in
security implications of to much being installed and active if installed via make install in /usr/ports/something/with_webinterface. -> best
example of lang/php to use subpacakges is the best example of why not to use subpackages / shows what is a regression in the features we provide in our ports collection.

While qt6-tools may be a good example where subpackages makes sense, we can not depend on subpacakges for "make install", and as if the p= ort
would be converted to have subpackages but no slave ports are
introduced, pkg install and make install would differ in the amount of
installed files.

> For port builders
>
> Port builders can experience longer build times in the future, as
> master/slave ports could be consolidated in one single larger ports. > If this is the case, the larger ports should provide OPTIONS to not > build unneeded subpackages.
> If no OPTION is available, please work with the maintainer to introduc= e
> one.

I fail to see the benefit:
We either lose the possibility to target parts of a package (when slave ports are removed / not introduced) on make install (with a similar
amount of build time for the ports tree as it is right now), or get
higher build times for the package builders. In both cases we do not
gain something significant.

If we want to keep the (useful in some cases) feature to install from
ports (there is the case of py39-rpds-py failing to build in my
poudriere which I tried to debug with the author of py-maturin due to a runtime issue in maturin, which shows up in my the cross build on
amd64-intel for amd64-athlon64... which in the end leads me to build
py-rpds-py on the target machine from ports), the current implementation of subpackages has to come with the requirement to create slave ports
for each subpackage.

That's my concern, and that's the reason why I have the opinion, th= at
portmgr has to keep the lock on subpackages and reject any subpackage
which don't come with a slave port.

I would be OK to lift this restriction, when the current implementation is changed to be able to only install the files of a subpacakge on make install (an implementation could be to require "make install
TARGET_SUBPACAKGES=3Dsub1,sub2,sub3" or "make install-subpackage1=
install-subpacakge2"... as long as recursive dependencies are handled =
according to this requirement, those people designing, implementing and reviewing this can argue about such details).

Keeping the current implementation (with the restriction of always
introducing slave packages for subpackages) would need a way to denote
that a slave port covers a specific subpackage which would allow package builders to skip those slave ports (and the subpacakges would need to
have the same package name as the slave port, no idea if this has a
technical disadvantage in terms of having 2 different origins for the
same package name, but it surely would be confusing for people at first look).

TLDR: for the use cases you specified in the beginning, I do not see a
benefit, only drawbacks. Can you please provide an example of a benefit I fail to see (yes, more modularity for qt6-tools may be beneficial for some people, but the cost/benefit between qt6-tools (something which we don't provide right now) and "make install" (what we provide = right now)
or "build time reduction for package builders" (which would have = a
benefit for a lot of use cases) is in my books much more in the
direction of "make install" and "build time reduction" = than towards the
modularization of qt6-tools)?

Bye,
Alexander.

--
h= ttp://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF=
htt= p://www.FreeBSD.org=C2=A0 =C2=A0 netchild@FreeBSD.org=C2=A0 : PGP 0x8F3= 1830F9F2772BF
--00000000000019db290610269536--