Date: Thu, 22 Feb 2024 20:57:54 +0000 From: igor.ostapenko@pm.me To: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Add jail execution environment support to the FreeBSD test suite Message-ID: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me>
next in thread | raw e-mail | index | archive | help
Hi FreeBSD developers,
There is a proposal to improve the FreeBSD test suite.
1 The Problem
The FreeBSD test suite is based on the Kyua framework. The latter supports
running tests in parallel. However, some tests cannot be run in parallel an=
d
are marked with is_exclusive=3D"true" metadata, which makes Kyua run such t=
ests
in sequence.
Many tests are not meant to be exclusive conceptually, they are so for very
simple technical reasons. For instance, some network related tests are base=
d
on jail and vnet usage. It's convenient for such tests and it provides a lo=
t
of isolation already not to conflict with other tests. But they are still
marked as exclusive due to the shared space of jail names, routing, etc.
The project seeks more tests, and it's kind of a trend for new tests like
jail/vnet based ones to be created as is_exclusive=3D"true" from the very
beginning. It only piles up the suite with exclusive tests, e.g. new tests
from my side faced a fair question from a reviewer whether they could be
re-designed for a parallel run. [1]
If such tests were 100% isolated they would be able to run in parallel and
decrease the test time for CI runs and for the runs within the development
process.
And the problem is that trying to add more isolation by a test itself looks=
to
be a doable task from a glance, but it would add a lot of complexity to a t=
est
code, or could be found as an impossible task in a specific case.
2 The Idea
The idea is not new. A test could be running in a jail -- it provides the
required isolation with minimum or zero effort from a test.
3 The Implementation
There is a lot of work done already and the working patch passed the initia=
l
review (thanks to markj@ and ngie@). [2]
It adds a new concept to the Kyua framework -- an execution environment. Tw=
o
new metadata were added for that: execenv and execenv_jail.
execenv is a switch to select an environment. If a test's metadata defines
execenv=3D"jail" then Kyua will create a temporary jail, run such test with=
in
it, and remove the jail. If execenv=3D"host" is provided or execenv metadat=
a is
undefined then Kyua will run such test as it does today.
execenv_jail metadata takes effect only in case of execenv=3D"jail". It all=
ows a
test to request specific parameters for its jail. These parameters are simp=
ly
arguments to jail(8), e.g. execenv_jail=3D"vnet allow.raw_sockets".
4 The Adoption
ATF based tests can easily define this new metadata via Kyuafile or directl=
y,
e.g. for atf-sh based tests:
=09test_head()
=09{
=09=09atf_set descr "Test foo in case of bar"
=09=09atf_set require.user root
=09=09atf_set execenv jail
=09=09atf_set execenv.jail vnet allow.raw_sockets
=09}
Non-ATF based ones will do it via Kyuafile. Our test suite does it through =
a
Makefile:
=09TEST_METADATA+=3D execenv=3D"jail"
=09TEST_METADATA+=3D execenv_jail=3D"vnet allow.raw_sockets"
The patch got some little evolution, I started with a single execenv_jail
metadata, and during the patch discussion and review, I ended up with two
knobs: execenv and execenv_jail. It turned out to be a cleaner and less tri=
cky
interface such way. The evolution reasoning can be found in the history of =
the
respective Differential. [2]
5 MFC Concerns
For now, I see at least one issue from the usual project workflow perspecti=
ve.
Let's imagine that the Kyua framework got this execenv feature committed to
15-CURRENT, we started to convert existing tests and create new ones to use
execenv=3D"jail". If some feature or a bug fix needs to be ported back to
14-STABLE or 13-STABLE, then "old" Kyua without execenv feature will fail t=
o
run such tests:
=09kyua: E: Load of 'Kyuafile' failed: Failed to load Lua file 'Kyuafile': =
Kyuafile:9: Unknown metadata property execenv.
>From a combinatorics perspective, the first three options pop up to deal wi=
th
that:
a) Patch Kyua the same way for the supported STABLE branches so it will be
able to run back ported tests based on execenv=3D"jail" (it's not system=
ABI
change after all)
b) Exclusively patch Kyua framework for the supported STABLE branches to
simply skip such tests (does not look to provide much benefit)
c) Do not back port tests, only the fix/feature itself (kind of a bad idea)
6 The Demo
My test environment showed promising run time numbers for almost the whole
test suite (ZFS excluded). One of the tests yielded 36 min with test
parallelism improvement versus 1 h 25 min without. In my case with 8 cores,
the suite runs about 2 times faster with the improvement. [3]
7 Action Points
My current vision of the plan looks as follows:
- [ ] community: Review, testing, comments -- probably we want to change th=
e
design
- [ ] committers: Help with the main commit -- it should hit freebsd/kyua
GitHub fork first [4], then vendor branch, and merge to
main after
- [ ] igoro: Provide the subsequent PRs to separate FreeBSD specifics and f=
ix
existing Kyua tests
- [ ] igoro: Provide the PRs to add brand new tests of Kyua itself to cover
the new feature
- [ ] igoro: Provide the respective documentation updates
- [ ] igoro: Migrate some of the existing tests for the start, e.g. netpfil=
/pf
- [ ] committers: Help with review and respective commits/merges
The plan is not strict, it depends on the discussion and interest of
volunteers.
I hope that this proposal is found valuable for the project. If so, any hel=
p
is appreciated.
[1] New tests exclusivity concern: https://reviews.freebsd.org/D42314
[2] The Kyua patch: https://reviews.freebsd.org/D42350
[3] The whole test suite demo: https://reviews.freebsd.org/D42410
[4] The respective PR to the fork: https://github.com/freebsd/kyua/pull/224
Best regards, Igor.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=>