From nobody Sat Mar  2 15:19:36 2024
X-Original-To: dev-commits-ports-all@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tn7t72vv2z5CPts
	for <dev-commits-ports-all@mlmmj.nyi.freebsd.org>; Sat,  2 Mar 2024 15:19:43 +0000 (UTC)
	(envelope-from daniel.engberg.lists@pyret.net)
Received: from smtp-bc0b.mail.infomaniak.ch (smtp-bc0b.mail.infomaniak.ch [IPv6:2001:1600:7:10::bc0b])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "relay.mail.infomaniak.ch", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Tn7t61xWsz41FJ
	for <dev-commits-ports-all@freebsd.org>; Sat,  2 Mar 2024 15:19:42 +0000 (UTC)
	(envelope-from daniel.engberg.lists@pyret.net)
Authentication-Results: mx1.freebsd.org;
	none
Received: from smtp-4-0000.mail.infomaniak.ch (smtp-4-0000.mail.infomaniak.ch [10.7.10.107])
	by smtp-4-3000.mail.infomaniak.ch (Postfix) with ESMTPS id 4Tn7t05MW7z5RQ;
	Sat,  2 Mar 2024 16:19:36 +0100 (CET)
Received: from unknown by smtp-4-0000.mail.infomaniak.ch (Postfix) with ESMTPA id 4Tn7t02kz5z14H;
	Sat,  2 Mar 2024 16:19:36 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=pyret.net;
	s=20231006; t=1709392776;
	bh=i1Gx41CcYTV/Y9Zn8TTGdk4WwzMJLWTcgIU97ZFpBKc=;
	h=Date:Subject:From:Reply-To:To:Cc:References:In-Reply-To:From;
	b=lllURrRfoLA7S9t1xnmAh8D3Y5jzOdscEhhUAqOhDfxkrR1EW6uWcc/kdAwrA9mTX
	 DiV1i5B21PNme8AlqXSXtEqyCm4Ub6lC9H68rP+gkk1/qw0ERYQFlux6c2DoMZQYWi
	 ZtBfiptcwEyHXxzVvUReJl4V7W6zBFtX6UEblCDl2y89v0/EZ7qE96ois+R2poVPEX
	 I/JHsFkUkUrATvWDK78QvwogVQCJDZSxkjkFtprnhIWOCzyYFAl5Kqw+a9WaptcoDW
	 BNsuQppVyAkCGUdgtAZITnptIsTbKS9dsMy1m/fsXNb9LgldVBT/YsBRXNyxrXXu6m
	 jRg6fF81JX/wg==
Message-ID: <0aba1eb996aeee8e357e8017719fdccd@mail.infomaniak.com>
Date: Sat, 02 Mar 2024 16:19:36 +0100
Subject: Re: git: c2afef5c2587 - main - Features/testing.mk: Introduce the
 TESTING_UNSAFE knob.
From: Daniel Engberg <daniel.engberg.lists@pyret.net>
Reply-To: Daniel Engberg <daniel.engberg.lists@pyret.net>
To: Gleb Popov <arrowd@freebsd.org>
Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org,
 dev-commits-ports-main@freebsd.org
List-Id: Commit messages for all branches of the ports repository <dev-commits-ports-all.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-all
List-Help: <mailto:dev-commits-ports-all+help@freebsd.org>
List-Post: <mailto:dev-commits-ports-all@freebsd.org>
List-Subscribe: <mailto:dev-commits-ports-all+subscribe@freebsd.org>
List-Unsubscribe: <mailto:dev-commits-ports-all+unsubscribe@freebsd.org>
Sender: owner-dev-commits-ports-all@freebsd.org
X-BeenThere: dev-commits-ports-all@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-WS-User-Origin: eyJpdiI6Ing0c0xOdHpybVBhaFRrMkpOMFdtcVE9PSIsInZhbHVlIjoiWS92ZUhHN1UwUHoxNUFmN2FQYTk0Zz09IiwibWFjIjoiNGJmZmYzNzlmYTZkZThjNzA1NzdkNmE3NmFhZmVmNzlhOTU2MjI2Y2E5ZDYxZDRhNDNhMDY3OWU3YzBiNTkyOCIsInRhZyI6IiJ9
X-WS-User-Mbox: eyJpdiI6Im00UW50UWVJSjRRdCtTZk55eGVpd2c9PSIsInZhbHVlIjoicGNzMTcxZXhIWmdLSmxjMThEQkZ0Zz09IiwibWFjIjoiYmYxMWRlZDZlYmRiMjc1YzFmNTdiNDhiMTM1MjgwYzE0MWQyOGYxODdmY2Y5ZTdjYzcyMGIwZDk3ODJmNzlkZCIsInRhZyI6IiJ9
X-WS-Location: eJxzKUpMKykGAAfpAmU-
X-Mailer: Infomaniak Workspace (1.3.647)
References: <202403011927.421JRqxx065749@gitrepo.freebsd.org>
 <ab6429b3b2004cd8ad9e7ed0b563a5c9@mail.infomaniak.com>
 <CALH631n+xb2bwqZgQc0NM0E=u3PE_Oc7X5v+xbBJYTkDhyqa0A@mail.gmail.com>
In-Reply-To: <CALH631n+xb2bwqZgQc0NM0E=u3PE_Oc7X5v+xbBJYTkDhyqa0A@mail.gmail.com>
X-Infomaniak-Routing: alpha
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:29222, ipnet:2001:1600::/32, country:CH]
X-Rspamd-Queue-Id: 4Tn7t61xWsz41FJ

On 2024-03-02T08:45:20.000+01:00, Gleb Popov <arrowd@freebsd.org> wrote:
>  On Fri, Mar 1, 2024 at 11:20=E2=80=AFPM Daniel Engberg
> <daniel.engberg.lists@pyret.net> wrote:
>=20
> >  =20
> >  Hi,
> >=20
> >  Thanks for adding this, however this won't work on ports that uses TES=
T menu option as far as I can tell which is a relatively common occurence?
> >=20
> >  Se also https://cgit.freebsd.org/ports/tree/Mk/bsd.options.desc.mk#n49=
5
> >  Best regards,
> >  Daniel
> >=20
> =20
> There is a subtle overlap between Features and OPTIONS that I still
> have not fully grasped.
>=20
> A Feature can be thought of as an OPTION that is generic enough to be
> applied to every port. For instance, almost every software can be
> built in the "debug" mode (whatever that means), which makes DEBUG a
> good candidate for a Feature and not an OPTION. However, some software
> may have additional toggles that are not hooked into the standard way
> of enabling debug mode for a given build system.
>=20
> A CMake example - there is a standard CMAKE_BUILD_TYPE variable, but
> the software may also define sort of ENABLE_EXTRA_CHECKS option, which
> further improve debugging experience but is not enabled by
> CMAKE_BUILD_TYPE.
>=20
> In such cases it might have a sense to define the DEBUG option and
> then do DEBUG_CMAKE_ON=3D -DENABLE_EXTRA_CHECKS=3Dyes in the port.
>=20
> In case of LTO a software may require some additional dependency, so
> that just setting WITH_LTO=3Dyes would not be enough. So this is another
> case where the option-corresponding-to-a-feature is required.
>=20
> Note that it is still possible to build WITH_DEBUG=3Dyes but
> WITHOUT=3DDEBUG. It will enable debugging in general, but not enable
> some specific debugging features for the given port. This hints that
> Features and OPTIONS are separate ideas and probably should coexist.
>=20
> Getting back to the TESTING feature, I agree that the TEST option
> doesn't really make sense as it is. At least we should change its
> description and remove the "and/or run" part, because running tests is
> now (and should be) a Feature-level decision.
> Maybe the TEST option doesn't make sense at all now, but like I said I
didn't get my head over it yet. Let's see how it goes.

In my experience and looking at other ports the framework lacks functionali=
ty and/or causes execution to fail which is why the menu option is utilized=
. Just looking at cmake:testing we're already working around the stock TEST=
_* options and functionality available.

Best regards,
Daniel