Date: Fri, 23 Feb 2024 10:41:49 +0100 From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= <olivier@freebsd.org> To: igor.ostapenko@pm.me Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Add jail execution environment support to the FreeBSD test suite Message-ID: <CA%2Bq%2BTcp5rLVucgsAmBQMRtb-tPzxs0p2Ojz4U6nPyTaVqxjfhg@mail.gmail.com> In-Reply-To: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me> References: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000350b630612095fe2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Feb 22, 2024 at 9:58=E2=80=AFPM <igor.ostapenko@pm.me> wrote: > > There is a proposal to improve the FreeBSD test suite. > > > Thanks a lot for working on improving the parallel tests! At work, we are using: 1. a Nanobsd based, so a lot of WITHOUT_ : WITHOUT_JAIL, WITHOUT_PF, no VNET (a nightmare to debug network stack with it), etc.; 2. The standard full test suite (about 8200 tests) is reduced to about 7400 tests with this lighter nanobsd; 3. We have been using kyua parallelism=3D8 for years, and I had to skip onl= y about 10-20 tests that were not compliant with parallel mode. It toke about 23 minutes to run them (in a bhyve VM): Test cases: 7429 total, 905 skipped, 29 expected failures, 1 broken, 4 failed Start time: 2024-02-21T00:15:42.527291Z End time: 2024-02-21T00:38:57.036211Z Because we were using WITHOUT_JAIL, to improve the time spent, one of my ideas was to divide the tests into groups of 4 or 8 and run as many bhyve VMs in parallel. I assumed that by running regression tests in a jail, I was testing the jail system as much as the test object (and for our use case, I don=E2=80= =99t care about the jail because we are not using it). So using the bhyve VM I had the feeling the tests were more confined. Now if WITH_JAIL becomes mandatory to be able to run the test, I will just have to modify my build system (like by adding a WITHOUT_JAIL just at installworld time and not during buildworld). Regards, Olivier --000000000000350b630612095fe2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:"courier new",monospace"><br></div></div><div class=3D"g= mail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Feb 22, 2024 at 9= :58=E2=80=AFPM <<a href=3D"mailto:igor.ostapenko@pm.me">igor.ostapenko@p= m.me</a>> wrote:</div><blockquote class=3D"gmail_quote" style=3D"margin:= 0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> There is a proposal to improve the FreeBSD test suite.<br> <br><br></blockquote><div><br></div><div class=3D"gmail_default" style=3D"f= ont-family:"courier new",monospace">Thanks a lot for working on i= mproving the parallel tests!</div><div class=3D"gmail_default" style=3D"fon= t-family:"courier new",monospace"><br></div><div class=3D"gmail_d= efault" style=3D"font-family:"courier new",monospace">At work, we= are using:</div><div class=3D"gmail_default" style=3D"font-family:"co= urier new",monospace">1. a Nanobsd based, so a lot of WITHOUT_ : WITHO= UT_JAIL, WITHOUT_PF, no VNET (a nightmare to debug network stack with it), = etc.;</div><div class=3D"gmail_default" style=3D"font-family:"courier = new",monospace">2. The standard full test suite (about 8200 tests) is = reduced to about 7400 tests with this lighter nanobsd;</div><div class=3D"g= mail_default" style=3D"font-family:"courier new",monospace">3. We= have been using kyua parallelism=3D8 for years, and I had to skip only abo= ut 10-20 tests that were not compliant with parallel mode.</div><div class= =3D"gmail_default" style=3D"font-family:"courier new",monospace">= <br></div><div class=3D"gmail_default" style=3D"font-family:"courier n= ew",monospace">It toke about 23 minutes to run them (in a bhyve VM):</= div><div class=3D"gmail_default" style=3D"font-family:"courier new&quo= t;,monospace"><br></div><div class=3D"gmail_default" style=3D"font-family:&= quot;courier new",monospace">Test cases: 7429 total, 905 skipped, 29 e= xpected failures, 1 broken, 4 failed<br>Start time: 2024-02-21T00:15:42.527= 291Z<br>End time: =C2=A0 2024-02-21T00:38:57.036211Z</div><div class=3D"gma= il_default" style=3D"font-family:"courier new",monospace"><br></d= iv><div class=3D"gmail_default" style=3D"font-family:"courier new"= ;,monospace">Because we were using WITHOUT_JAIL, to improve the time spent,= one of my ideas was to divide the tests into groups of 4 or 8 and run as m= any bhyve VMs in parallel.</div><div class=3D"gmail_default" style=3D"font-= family:"courier new",monospace">I assumed that by running regress= ion tests in a jail, I was testing the jail system as much as the test obje= ct (and for our use case, I don=E2=80=99t care about the jail because we ar= e not using it).</div><div class=3D"gmail_default" style=3D"font-family:&qu= ot;courier new",monospace">So using the bhyve VM I had the feeling the= tests were more confined.</div><div class=3D"gmail_default" style=3D"font-= family:"courier new",monospace"><br></div><div class=3D"gmail_def= ault" style=3D"font-family:"courier new",monospace">Now if WITH_J= AIL becomes mandatory to be able to run the test, I will just have to modif= y my build system (like by adding a WITHOUT_JAIL just at installworld time = and not during buildworld).</div><div class=3D"gmail_default" style=3D"font= -family:"courier new",monospace"><br></div><div class=3D"gmail_de= fault" style=3D"font-family:"courier new",monospace">Regards,</di= v><div class=3D"gmail_default" style=3D"font-family:"courier new"= ,monospace">Olivier</div></div></div> --000000000000350b630612095fe2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bq%2BTcp5rLVucgsAmBQMRtb-tPzxs0p2Ojz4U6nPyTaVqxjfhg>