Date: Wed, 6 Nov 2024 12:13:56 -0700 From: Warner Losh <imp@bsdimp.com> To: Igor Ostapenko <igoro@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: RFC: Add required_klds metadata to Kyua Message-ID: <CANCZdfpPtgRRN9VZjLOObOqPcBvVXokaO_acDWi1ssWdM8nGMA@mail.gmail.com> In-Reply-To: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org> References: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
On Wed, Nov 6, 2024, 10:12 AM Igor Ostapenko <igoro@freebsd.org> wrote:
> Hi FreeBSD developers,
>
> During Kyua's execenv=jail feature implementation there were discussions to
> get some help from Kyua side with kernel modules required by specific
> tests.
> The thing got "required_klds" codename.
>
> The problem to solve here is a well-known one: typically we invoke "kyua
> test" and get a lot of "Skipped" due to many kernel module requirements.
> And
> the pain is to know which ones need loading, especially if we target a
> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new
> test is added with new module requirements. And other related issues.
>
> I would like to share the existing ideas and vision before the
> implementation itself. I propose to split it onto several subsequent
> projects to ease reasoning, review, etc.
>
>
> Project A: declaration & skipping
> ---------------------------------
>
> Add a new metadata property to Kyua which is used by a test to define its
> requirements on specific kernel modules expected being loaded. The
> following
> interface is expected:
>
> - Define it on ATF level, e.g. for atf-sh:
>
> atf_set require.klds pf pflog
>
> - Define it on Kyuafile level:
>
> test_program{name="testbinary1", required_klds="pf pflog"}
>
> - Get it listed per test case:
>
> > kyua list -v testbinary1 | grep required_klds
> required_klds = pf pflog if_epair
>
> Having it declared we could simplify tests and avoid manual checks for
> modules with the respective "skip test" lib calls. Thus, this new metadata
> would work the same way as the existing required_programs and
> required_files
> ones -- if some of the required modules are not loaded Kyua will skip such
> test with a respective message including names of the missing modules.
>
> An extra benefit is that we do not need to iterate it like: 1) kyua test,
> 2)
> kldload a-missing-module, 3) repeat. Such iteration is usually needed due
> to
> our tests typically do kldstat sequentially with "skip fast" logic.
> Skipping
> reason information still is not aggregated to make it easy to run a single
> kldload for all missing modules, but that's another story which is covered
> by the following Project B.
>
> Q1: I have doubts regarding the name of this new metadata. There are
> several
> other options in my mind, it would be appreciated to hear opinions
> regarding
> this:
> a) required_klds -- the original idea based on the vibe that FreeBSD
> Kyua
> fork is mainly for FreeBSD
>
> b) required_mods -- to keep it OS agnostic while Kyua still is treated
> as an OS agnostic tool, and freebsd/kyua GitHub repo actually is the
> main Kyua repo in the world today
>
> c) required_kmods -- a little improvement of the previous option to make
> sure it talks about "kernel modules" and leave space for other
> meanings
> in the future, just in case
>
> d) required_klds | required_kmods -- to have some official aliases
> supported, but it's expected to bring confusion instead
>
> e) your variant
>
Love this. Would go with a FreeBSD name, though, because no two oses have
the same module names. Don't care a huge amount.
Would also like an option to never load kld so qemu testing can be a lot
faster...
Warner
> Project B: automatic kldload
> ----------------------------
>
> Having Project A implemented and the existing tests augmented with the
> respective metadata we could ask Kyua to automatically load the required
> kernel modules.
>
> From the implementation perspective, Kyua is expected to have a generic
> concept of a requirement resolver. A resolver may read all metadata and
> decide on its actions. If we decide on "klds" name in Project A then the
> very first resolver implemented would look for required_klds metadata, if
> it's present, then the mentioned kernel modules will be loaded. Such
> resolver could be named "klds", and we can think of future additions like
> "pkgs" resolver and so on.
>
> Currently, I see it as a new kyua command to make it separate from the
> testing process itself. The interface could be as follows:
>
> - List all available requirement resolvers (name and short description):
>
> > kyua rr list
>
> - Run all resolvers or only specified ones. It would deal with the current
> dir, and later we can think of "-k" option like "kyua test" command has.
>
> > kyua rr run [resolver1-name resolver2-name ...]
>
> - It's proposed to be a sub-set of commands in case if in the future we
> want
> additions, e.g. dry-run or verify command without actual resolving.
>
> Having this interface the FreeBSD CI could have a separate step to run
> "kyua
> rr run" before the existing step which runs "kyua test". But from
> development perspective it may quickly end up with a need to have a single
> command for both. The following interface addition could be there:
>
> - Run all or specified requirement resolvers before the actual testing:
>
> > kyua test --rr ["resolver1-name resolver2-name ..."]
>
> Another option is to use the existing Kyua mechanisms and add it as a
> configuration variable, i.e. it can be configured once in the local
> kyua.conf not to specify it with every test command invocation:
>
> # without configuration:
> > kyua -v rr="klds pkgs" test
>
> # with configuration:
> > echo 'rr = "klds pkgs"' >> /etc/kyua/kyua.conf
> > kyua test # now it does resolving first
>
>
> Q2: Project B has the same set of questions about naming and style. For
> example, "rr" name for kyua.conf feels non-obvious and probably it could be
> named like "requirement_resolvers" to keep it self-explainable.
>
>
> What do you think? Any idea, comment, or suggestion are welcome.
>
>
> Best regards,
> igoro
>
>
[-- Attachment #2 --]
<div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 6, 2024, 10:12 AM Igor Ostapenko <<a href="mailto:igoro@freebsd.org">igoro@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi FreeBSD developers,<br>
<br>
During Kyua's execenv=jail feature implementation there were discussions to<br>
get some help from Kyua side with kernel modules required by specific tests.<br>
The thing got "required_klds" codename.<br>
<br>
The problem to solve here is a well-known one: typically we invoke "kyua<br>
test" and get a lot of "Skipped" due to many kernel module requirements. And<br>
the pain is to know which ones need loading, especially if we target a<br>
sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new<br>
test is added with new module requirements. And other related issues.<br>
<br>
I would like to share the existing ideas and vision before the<br>
implementation itself. I propose to split it onto several subsequent<br>
projects to ease reasoning, review, etc.<br>
<br>
<br>
Project A: declaration & skipping<br>
---------------------------------<br>
<br>
Add a new metadata property to Kyua which is used by a test to define its<br>
requirements on specific kernel modules expected being loaded. The following<br>
interface is expected:<br>
<br>
- Define it on ATF level, e.g. for atf-sh:<br>
<br>
atf_set require.klds pf pflog<br>
<br>
- Define it on Kyuafile level:<br>
<br>
test_program{name="testbinary1", required_klds="pf pflog"}<br>
<br>
- Get it listed per test case:<br>
<br>
> kyua list -v testbinary1 | grep required_klds<br>
required_klds = pf pflog if_epair<br>
<br>
Having it declared we could simplify tests and avoid manual checks for<br>
modules with the respective "skip test" lib calls. Thus, this new metadata<br>
would work the same way as the existing required_programs and required_files<br>
ones -- if some of the required modules are not loaded Kyua will skip such<br>
test with a respective message including names of the missing modules.<br>
<br>
An extra benefit is that we do not need to iterate it like: 1) kyua test, 2)<br>
kldload a-missing-module, 3) repeat. Such iteration is usually needed due to<br>
our tests typically do kldstat sequentially with "skip fast" logic. Skipping<br>
reason information still is not aggregated to make it easy to run a single<br>
kldload for all missing modules, but that's another story which is covered<br>
by the following Project B.<br>
<br>
Q1: I have doubts regarding the name of this new metadata. There are several<br>
other options in my mind, it would be appreciated to hear opinions regarding<br>
this:<br>
a) required_klds -- the original idea based on the vibe that FreeBSD Kyua<br>
fork is mainly for FreeBSD<br>
<br>
b) required_mods -- to keep it OS agnostic while Kyua still is treated<br>
as an OS agnostic tool, and freebsd/kyua GitHub repo actually is the<br>
main Kyua repo in the world today<br>
<br>
c) required_kmods -- a little improvement of the previous option to make<br>
sure it talks about "kernel modules" and leave space for other meanings<br>
in the future, just in case<br>
<br>
d) required_klds | required_kmods -- to have some official aliases<br>
supported, but it's expected to bring confusion instead<br>
<br>
e) your variant<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Love this. Would go with a FreeBSD name, though, because no two oses have the same module names. Don't care a huge amount.</div><div dir="auto"><br></div><div dir="auto">Would also like an option to never load kld so qemu testing can be a lot faster...</div><div dir="auto"><br></div><div dir="auto">Warner</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Project B: automatic kldload<br>
----------------------------<br>
<br>
Having Project A implemented and the existing tests augmented with the<br>
respective metadata we could ask Kyua to automatically load the required<br>
kernel modules.<br>
<br>
From the implementation perspective, Kyua is expected to have a generic<br>
concept of a requirement resolver. A resolver may read all metadata and<br>
decide on its actions. If we decide on "klds" name in Project A then the<br>
very first resolver implemented would look for required_klds metadata, if<br>
it's present, then the mentioned kernel modules will be loaded. Such<br>
resolver could be named "klds", and we can think of future additions like<br>
"pkgs" resolver and so on.<br>
<br>
Currently, I see it as a new kyua command to make it separate from the<br>
testing process itself. The interface could be as follows:<br>
<br>
- List all available requirement resolvers (name and short description):<br>
<br>
> kyua rr list<br>
<br>
- Run all resolvers or only specified ones. It would deal with the current<br>
dir, and later we can think of "-k" option like "kyua test" command has.<br>
<br>
> kyua rr run [resolver1-name resolver2-name ...]<br>
<br>
- It's proposed to be a sub-set of commands in case if in the future we want<br>
additions, e.g. dry-run or verify command without actual resolving.<br>
<br>
Having this interface the FreeBSD CI could have a separate step to run "kyua<br>
rr run" before the existing step which runs "kyua test". But from<br>
development perspective it may quickly end up with a need to have a single<br>
command for both. The following interface addition could be there:<br>
<br>
- Run all or specified requirement resolvers before the actual testing:<br>
<br>
> kyua test --rr ["resolver1-name resolver2-name ..."]<br>
<br>
Another option is to use the existing Kyua mechanisms and add it as a<br>
configuration variable, i.e. it can be configured once in the local<br>
kyua.conf not to specify it with every test command invocation:<br>
<br>
# without configuration:<br>
> kyua -v rr="klds pkgs" test<br>
<br>
# with configuration:<br>
> echo 'rr = "klds pkgs"' >> /etc/kyua/kyua.conf<br>
> kyua test # now it does resolving first<br>
<br>
<br>
Q2: Project B has the same set of questions about naming and style. For<br>
example, "rr" name for kyua.conf feels non-obvious and probably it could be<br>
named like "requirement_resolvers" to keep it self-explainable.<br>
<br>
<br>
What do you think? Any idea, comment, or suggestion are welcome.<br>
<br>
<br>
Best regards,<br>
igoro<br>
<br>
</blockquote></div></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfpPtgRRN9VZjLOObOqPcBvVXokaO_acDWi1ssWdM8nGMA>