Date: Wed, 6 Nov 2024 12:13:56 -0700 From: Warner Losh <imp@bsdimp.com> To: Igor Ostapenko <igoro@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: RFC: Add required_klds metadata to Kyua Message-ID: <CANCZdfpPtgRRN9VZjLOObOqPcBvVXokaO_acDWi1ssWdM8nGMA@mail.gmail.com> In-Reply-To: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org> References: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000006efd590626435211
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Wed, Nov 6, 2024, 10:12=E2=80=AFAM Igor Ostapenko <igoro@freebsd.org> wr=
ote:
> Hi FreeBSD developers,
>
> During Kyua's execenv=3Djail feature implementation there were discussion=
s to
> get some help from Kyua side with kernel modules required by specific
> tests.
> The thing got "required_klds" codename.
>
> The problem to solve here is a well-known one: typically we invoke "kyua
> test" and get a lot of "Skipped" due to many kernel module requirements.
> And
> the pain is to know which ones need loading, especially if we target a
> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new
> test is added with new module requirements. And other related issues.
>
> I would like to share the existing ideas and vision before the
> implementation itself. I propose to split it onto several subsequent
> projects to ease reasoning, review, etc.
>
>
> Project A: declaration & skipping
> ---------------------------------
>
> Add a new metadata property to Kyua which is used by a test to define its
> requirements on specific kernel modules expected being loaded. The
> following
> interface is expected:
>
> - Define it on ATF level, e.g. for atf-sh:
>
> atf_set require.klds pf pflog
>
> - Define it on Kyuafile level:
>
> test_program{name=3D"testbinary1", required_klds=3D"pf pflog"}
>
> - Get it listed per test case:
>
> > kyua list -v testbinary1 | grep required_klds
> required_klds =3D pf pflog if_epair
>
> Having it declared we could simplify tests and avoid manual checks for
> modules with the respective "skip test" lib calls. Thus, this new metadat=
a
> would work the same way as the existing required_programs and
> required_files
> ones -- if some of the required modules are not loaded Kyua will skip suc=
h
> test with a respective message including names of the missing modules.
>
> An extra benefit is that we do not need to iterate it like: 1) kyua test,
> 2)
> kldload a-missing-module, 3) repeat. Such iteration is usually needed due
> to
> our tests typically do kldstat sequentially with "skip fast" logic.
> Skipping
> reason information still is not aggregated to make it easy to run a singl=
e
> kldload for all missing modules, but that's another story which is covere=
d
> by the following Project B.
>
> Q1: I have doubts regarding the name of this new metadata. There are
> several
> other options in my mind, it would be appreciated to hear opinions
> regarding
> this:
> a) required_klds -- the original idea based on the vibe that FreeBSD
> Kyua
> fork is mainly for FreeBSD
>
> b) required_mods -- to keep it OS agnostic while Kyua still is treated
> as an OS agnostic tool, and freebsd/kyua GitHub repo actually is th=
e
> main Kyua repo in the world today
>
> c) required_kmods -- a little improvement of the previous option to ma=
ke
> sure it talks about "kernel modules" and leave space for other
> meanings
> in the future, just in case
>
> d) required_klds | required_kmods -- to have some official aliases
> supported, but it's expected to bring confusion instead
>
> e) your variant
>
Love this. Would go with a FreeBSD name, though, because no two oses have
the same module names. Don't care a huge amount.
Would also like an option to never load kld so qemu testing can be a lot
faster...
Warner
> Project B: automatic kldload
> ----------------------------
>
> Having Project A implemented and the existing tests augmented with the
> respective metadata we could ask Kyua to automatically load the required
> kernel modules.
>
> From the implementation perspective, Kyua is expected to have a generic
> concept of a requirement resolver. A resolver may read all metadata and
> decide on its actions. If we decide on "klds" name in Project A then the
> very first resolver implemented would look for required_klds metadata, if
> it's present, then the mentioned kernel modules will be loaded. Such
> resolver could be named "klds", and we can think of future additions like
> "pkgs" resolver and so on.
>
> Currently, I see it as a new kyua command to make it separate from the
> testing process itself. The interface could be as follows:
>
> - List all available requirement resolvers (name and short description):
>
> > kyua rr list
>
> - Run all resolvers or only specified ones. It would deal with the curren=
t
> dir, and later we can think of "-k" option like "kyua test" command ha=
s.
>
> > kyua rr run [resolver1-name resolver2-name ...]
>
> - It's proposed to be a sub-set of commands in case if in the future we
> want
> additions, e.g. dry-run or verify command without actual resolving.
>
> Having this interface the FreeBSD CI could have a separate step to run
> "kyua
> rr run" before the existing step which runs "kyua test". But from
> development perspective it may quickly end up with a need to have a singl=
e
> command for both. The following interface addition could be there:
>
> - Run all or specified requirement resolvers before the actual testing:
>
> > kyua test --rr ["resolver1-name resolver2-name ..."]
>
> Another option is to use the existing Kyua mechanisms and add it as a
> configuration variable, i.e. it can be configured once in the local
> kyua.conf not to specify it with every test command invocation:
>
> # without configuration:
> > kyua -v rr=3D"klds pkgs" test
>
> # with configuration:
> > echo 'rr =3D "klds pkgs"' >> /etc/kyua/kyua.conf
> > kyua test # now it does resolving first
>
>
> Q2: Project B has the same set of questions about naming and style. For
> example, "rr" name for kyua.conf feels non-obvious and probably it could =
be
> named like "requirement_resolvers" to keep it self-explainable.
>
>
> What do you think? Any idea, comment, or suggestion are welcome.
>
>
> Best regards,
> igoro
>
>
--0000000000006efd590626435211
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"auto"><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr" =
class=3D"gmail_attr">On Wed, Nov 6, 2024, 10:12=E2=80=AFAM Igor Ostapenko &=
lt;<a href=3D"mailto:igoro@freebsd.org">igoro@freebsd.org</a>> wrote:<br=
></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex">Hi FreeBSD developers,<br>
<br>
During Kyua's execenv=3Djail feature implementation there were discussi=
ons to<br>
get some help from Kyua side with kernel modules required by specific tests=
.<br>
The thing got "required_klds" codename.<br>
<br>
The problem to solve here is a well-known one: typically we invoke "ky=
ua<br>
test" and get a lot of "Skipped" due to many kernel module r=
equirements. And<br>
the pain is to know which ones need loading, especially if we target a<br>
sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new<b=
r>
test is added with new module requirements. And other related issues.<br>
<br>
I would like to share the existing ideas and vision before the<br>
implementation itself. I propose to split it onto several subsequent<br>
projects to ease reasoning, review, etc.<br>
<br>
<br>
Project A: declaration & skipping<br>
---------------------------------<br>
<br>
Add a new metadata property to Kyua which is used by a test to define its<b=
r>
requirements on specific kernel modules expected being loaded. The followin=
g<br>
interface is expected:<br>
<br>
- Define it on ATF level, e.g. for atf-sh:<br>
<br>
=C2=A0 =C2=A0 =C2=A0atf_set require.klds pf pflog<br>
<br>
- Define it on Kyuafile level:<br>
<br>
=C2=A0 =C2=A0 =C2=A0test_program{name=3D"testbinary1", required_k=
lds=3D"pf pflog"}<br>
<br>
- Get it listed per test case:<br>
<br>
=C2=A0 =C2=A0 =C2=A0> kyua list -v testbinary1 | grep required_klds<br>
=C2=A0 =C2=A0 =C2=A0required_klds =3D pf pflog if_epair<br>
<br>
Having it declared we could simplify tests and avoid manual checks for<br>
modules with the respective "skip test" lib calls. Thus, this new=
metadata<br>
would work the same way as the existing required_programs and required_file=
s<br>
ones -- if some of the required modules are not loaded Kyua will skip such<=
br>
test with a respective message including names of the missing modules.<br>
<br>
An extra benefit is that we do not need to iterate it like: 1) kyua test, 2=
)<br>
kldload a-missing-module, 3) repeat. Such iteration is usually needed due t=
o<br>
our tests typically do kldstat sequentially with "skip fast" logi=
c. Skipping<br>
reason information still is not aggregated to make it easy to run a single<=
br>
kldload for all missing modules, but that's another story which is cove=
red<br>
by the following Project B.<br>
<br>
Q1: I have doubts regarding the name of this new metadata. There are severa=
l<br>
other options in my mind, it would be appreciated to hear opinions regardin=
g<br>
this:<br>
=C2=A0 =C2=A0a) required_klds -- the original idea based on the vibe that F=
reeBSD Kyua<br>
=C2=A0 =C2=A0 =C2=A0 fork is mainly for FreeBSD<br>
<br>
=C2=A0 =C2=A0b) required_mods -- to keep it OS agnostic while Kyua still is=
treated<br>
=C2=A0 =C2=A0 =C2=A0 as an OS agnostic tool, and freebsd/kyua GitHub repo a=
ctually is the<br>
=C2=A0 =C2=A0 =C2=A0 main Kyua repo in the world today<br>
<br>
=C2=A0 =C2=A0c) required_kmods -- a little improvement of the previous opti=
on to make<br>
=C2=A0 =C2=A0 =C2=A0 sure it talks about "kernel modules" and lea=
ve space for other meanings<br>
=C2=A0 =C2=A0 =C2=A0 in the future, just in case<br>
<br>
=C2=A0 =C2=A0d) required_klds | required_kmods -- to have some official ali=
ases<br>
=C2=A0 =C2=A0 =C2=A0 supported, but it's expected to bring confusion in=
stead<br>
<br>
=C2=A0 =C2=A0e) your variant<br></blockquote></div></div><div dir=3D"auto">=
<br></div><div dir=3D"auto">Love this. Would go with a FreeBSD name, though=
, because no two oses have the same module names. Don't care a huge amo=
unt.</div><div dir=3D"auto"><br></div><div dir=3D"auto">Would also like an =
option to never load kld so qemu testing can be a lot faster...</div><div d=
ir=3D"auto"><br></div><div dir=3D"auto">Warner</div><div dir=3D"auto"><br><=
/div><div dir=3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quot=
e"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left=
:1px #ccc solid;padding-left:1ex">
<br>
Project B: automatic kldload<br>
----------------------------<br>
<br>
Having Project A implemented and the existing tests augmented with the<br>
respective metadata we could ask Kyua to automatically load the required<br=
>
kernel modules.<br>
<br>
=C2=A0From the implementation perspective, Kyua is expected to have a gener=
ic<br>
concept of a requirement resolver. A resolver may read all metadata and<br>
decide on its actions. If we decide on "klds" name in Project A t=
hen the<br>
very first resolver implemented would look for required_klds metadata, if<b=
r>
it's present, then the mentioned kernel modules will be loaded. Such<br=
>
resolver could be named "klds", and we can think of future additi=
ons like<br>
"pkgs" resolver and so on.<br>
<br>
Currently, I see it as a new kyua command to make it separate from the<br>
testing process itself. The interface could be as follows:<br>
<br>
- List all available requirement resolvers (name and short description):<br=
>
<br>
=C2=A0 =C2=A0 =C2=A0> kyua rr list<br>
<br>
- Run all resolvers or only specified ones. It would deal with the current<=
br>
=C2=A0 =C2=A0dir, and later we can think of "-k" option like &quo=
t;kyua test" command has.<br>
<br>
=C2=A0 =C2=A0 =C2=A0> kyua rr run [resolver1-name resolver2-name ...]<br=
>
<br>
- It's proposed to be a sub-set of commands in case if in the future we=
want<br>
=C2=A0 =C2=A0additions, e.g. dry-run or verify command without actual resol=
ving.<br>
<br>
Having this interface the FreeBSD CI could have a separate step to run &quo=
t;kyua<br>
rr run" before the existing step which runs "kyua test". But=
from<br>
development perspective it may quickly end up with a need to have a single<=
br>
command for both. The following interface addition could be there:<br>
<br>
- Run all or specified requirement resolvers before the actual testing:<br>
<br>
=C2=A0 =C2=A0 =C2=A0> kyua test --rr ["resolver1-name resolver2-nam=
e ..."]<br>
<br>
Another option is to use the existing Kyua mechanisms and add it as a<br>
configuration variable, i.e. it can be configured once in the local<br>
kyua.conf not to specify it with every test command invocation:<br>
<br>
=C2=A0 =C2=A0 =C2=A0# without configuration:<br>
=C2=A0 =C2=A0 =C2=A0> kyua -v rr=3D"klds pkgs" test<br>
<br>
=C2=A0 =C2=A0 =C2=A0# with configuration:<br>
=C2=A0 =C2=A0 =C2=A0> echo 'rr =3D "klds pkgs"' >&g=
t; /etc/kyua/kyua.conf<br>
=C2=A0 =C2=A0 =C2=A0> kyua test=C2=A0 # now it does resolving first<br>
<br>
<br>
Q2: Project B has the same set of questions about naming and style. For<br>
example, "rr" name for kyua.conf feels non-obvious and probably i=
t could be<br>
named like "requirement_resolvers" to keep it self-explainable.<b=
r>
<br>
<br>
What do you think? Any idea, comment, or suggestion are welcome.<br>
<br>
<br>
Best regards,<br>
igoro<br>
<br>
</blockquote></div></div></div>
--0000000000006efd590626435211--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfpPtgRRN9VZjLOObOqPcBvVXokaO_acDWi1ssWdM8nGMA>