Date: Wed, 06 Nov 2024 21:27:05 +0100 From: Kristof Provost <kp@FreeBSD.org> To: =?utf-8?q?Olivier_Cochard-Labb=C3=A9?= <olivier@freebsd.org> Cc: Igor Ostapenko <igoro@freebsd.org>, freebsd-testing@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: RFC: Add required_klds metadata to Kyua Message-ID: <9975A26D-6AB4-433F-B4B2-515F956BF366@FreeBSD.org> In-Reply-To: <CA%2Bq%2BTcpLeVqHhtL7VHvWsijQO9sxewY5opeUQ895G43EC24qgg@mail.gmail.com> References: <9fc6fc72-4c2c-4b09-bdb5-122a49b45295@FreeBSD.org> <CA%2Bq%2BTcpLeVqHhtL7VHvWsijQO9sxewY5opeUQ895G43EC24qgg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--=_MailMate_C07615AC-1A59-4556-90E2-ECBD1E2270CC_= Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 6 Nov 2024, at 21:13, Olivier Cochard-Labbé wrote: > On Wed, Nov 6, 2024 at 7:12 PM Igor Ostapenko <igoro@freebsd.org> > wrote: > >> >> >> The problem to solve here is a well-known one: typically we invoke >> "kyua >> test" and get a lot of "Skipped" due to many kernel module >> requirements. >> And >> the pain is to know which ones need loading, especially if we target >> a >> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a >> new >> test is added with new module requirements. And other related issues. >> >> >> Hello Igor, > > Thanks again for working on this topic > And there are even more challenging cases: what about tests that > assess the > behavior of different modules and need to unload them as well? > For example, sys/netipsec/tunnel/* or sys/opencrypto/, which involve > loading and unloading modules between tests." > That’s a good point. I ran into that a few days ago when I looked at using the new execenv=jail option to parallelise the netipsec tests, and discovered that that wasn’t straightforward. I wonder if an `forbidden_klds` or something is the answer there. At first to skip a test if the forbidden module is present, later on perhaps to enforce unloading of the module so the test can run. That may not be straightforward through, because module state is global, and if we only support loading required modules that’s easy (gather the list and load, or load when starting a test that requires a new module), but if we support unloading them it’ll affect test scheduling. (i.e. we can’t load modules while a `forbidden_klds` test is running, and can’t unload while a `require_klds` test is running). That said, perhaps we don’t need to worry about that just yet. It’s a minority of tests that forbid modules, so those can just keep doing what they’re doing already (i.e. run exclusively, do the checks in the test script itself). It shouldn’t stop us from making the majority of tests better. Best regards, Kristof --=_MailMate_C07615AC-1A59-4556-90E2-ECBD1E2270CC_= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE html> <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"= > </head> <body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty= le=3D"white-space: normal;"> <p dir=3D"auto">On 6 Nov 2024, at 21:13, Olivier Cochard-Labb=C3=A9 wrote= :</p> </div><div class=3D"plaintext" style=3D"white-space: normal;"><blockquote= style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #136= BCE; color: #136BCE;"><p dir=3D"auto">On Wed, Nov 6, 2024 at 7:12=E2=80=AF= PM Igor Ostapenko <igoro@freebsd.org> wrote:</p> <blockquote style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px= solid #136BCE; border-left-color: #4B89CF; color: #4B89CF;"><p dir=3D"au= to">The problem to solve here is a well-known one: typically we invoke "k= yua <br> test" and get a lot of "Skipped" due to many kernel module requirements. <br> And <br> the pain is to know which ones need loading, especially if we target a <br> sub-tree of /usr/tests. Also, the FreeBSD CI needs periodic sync if a new= <br> test is added with new module requirements. And other related issues.</p>= <p dir=3D"auto">Hello Igor,</p> </blockquote><p dir=3D"auto">Thanks again for working on this topic <br> And there are even more challenging cases: what about tests that assess t= he <br> behavior of different modules and need to unload them as well? <br> For example, sys/netipsec/tunnel/* or sys/opencrypto/, which involve <br> loading and unloading modules between tests."</p> <br></blockquote></div> <div class=3D"markdown" style=3D"white-space: normal;"> <p dir=3D"auto">That=E2=80=99s a good point. I ran into that a few days a= go when I looked at using the new execenv=3Djail option to parallelise th= e netipsec tests, and discovered that that wasn=E2=80=99t straightforward= =2E</p> <p dir=3D"auto">I wonder if an <code style=3D"padding: 0 0.25em; backgrou= nd-color: #E4E4E4;">forbidden_klds</code> or something is the answer ther= e. At first to skip a test if the forbidden module is present, later on p= erhaps to enforce unloading of the module so the test can run.</p> <p dir=3D"auto">That may not be straightforward through, because module s= tate is global, and if we only support loading required modules that=E2=80= =99s easy (gather the list and load, or load when starting a test that re= quires a new module), but if we support unloading them it=E2=80=99ll affe= ct test scheduling. (i.e. we can=E2=80=99t load modules while a <code sty= le=3D"padding: 0 0.25em; background-color: #E4E4E4;">forbidden_klds</code= > test is running, and can=E2=80=99t unload while a <code style=3D"paddin= g: 0 0.25em; background-color: #E4E4E4;">require_klds</code> test is runn= ing).</p> <p dir=3D"auto">That said, perhaps we don=E2=80=99t need to worry about t= hat just yet. It=E2=80=99s a minority of tests that forbid modules, so th= ose can just keep doing what they=E2=80=99re doing already (i.e. run excl= usively, do the checks in the test script itself). It shouldn=E2=80=99t s= top us from making the majority of tests better.</p> <p dir=3D"auto">Best regards,<br> Kristof</p> </div> </div> </body> </html> --=_MailMate_C07615AC-1A59-4556-90E2-ECBD1E2270CC_=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9975A26D-6AB4-433F-B4B2-515F956BF366>