Date: Mon, 26 Feb 2024 13:25:08 -0700 From: Warner Losh <imp@bsdimp.com> To: Igor Ostapenko <igor.ostapenko@pm.me> Cc: Brooks Davis <brooks@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Add jail execution environment support to the FreeBSD test suite Message-ID: <CANCZdfpSFKmYBE_x1esgCU=9P47tsFV9wkDukLRvNxocOmjMGQ@mail.gmail.com> In-Reply-To: <8OSrqDr54xqBgG5cQG_M2Puk2zSxQUletfKhYUElMb8PNnydh6xTDuPgX4HzICWQO6VbC3q27yAOWstIdJ8iZsfN03gJBf9-l03D_qwYcH8=@pm.me> References: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me> <ZdjIHZFy8rF-PDxB@spindle.one-eyed-alien.net> <8OSrqDr54xqBgG5cQG_M2Puk2zSxQUletfKhYUElMb8PNnydh6xTDuPgX4HzICWQO6VbC3q27yAOWstIdJ8iZsfN03gJBf9-l03D_qwYcH8=@pm.me>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000005f95b806124eb547 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Feb 26, 2024 at 12:58=E2=80=AFPM Igor Ostapenko <igor.ostapenko@pm.= me> wrote: > On Friday, February 23rd, 2024 at 6:30 PM, Brooks Davis < > brooks@freebsd.org> wrote: > > > > > 2 The Idea > > > > > > The idea is not new. A test could be running in a jail -- it provides > the > > > required isolation with minimum or zero effort from a test. > > > > > > This generally sounds good. One minor concern I have is how this would > > interact with the ability to run the test suite in a jail. This is > > imperfectly supported today (IIRC ~350 failures on amd64), but it's > > quite userful for testing sweeping userspace-only tests like libsys and > > I'd love to see support expanded and improved (failures fixed or tests > > skipped, poudriere jail support, etc). > > > > Thanks for your consideration. > > As I understand it's a question to the specific tests, which needs revisi= ng > and tuning for a non-prison0 case. I saw such issues with existing tests, > frequently it was about jail restrictions. And by design, a root within a > jail > cannot alter restrictions of its parent jail, i.e. it's up to the creator > of > such jail. In contrast, execenv=3Djail asks Kyua to create a temporary ja= il > for > a test execution, and Kyua can be asked to configure it for the needs of = a > test via execenv_jail. But when Kyua itself is running within an existing > jail > then expectations of the test suite should be lifted a level above, > especially for usual execenv=3Dhost based tests. Thus, it's out of Kyua's > control to help with that. And this proposal does not change or improve > this > case. > > Just a quick idea here. Probably, the test suite could help a creator of > such > jail above Kyua itself with its metadata. For example, a test is not > intended > to be run within a jail and it works fine if Kyua runs within prison0, bu= t > if > Kyua itself runs in a jail it starts failing due to missing, for instance= , > allow.mlock allowence, see jail(8). A test could define > execenv_jail=3D"allow.mlock" metadata without asking for execenv=3Djail, = i.e. > it's > like a hint for a non-prison0 case. And Kyua could aggregate and provide > such > information to ease creation of a jail above the Kyua itself. > > On the other end of the spectrum, we could think of a new feature of > jail(8) > to be able to ask it for a jail with maximum possible set of allowed > things. > Just thinking out loud. > So I run the test suite in jail too today. To test how well qemu user-mode stuff is working. I see closer to 800 failures IIRC... Some of these are easy (like all the geom tests that need /dev/md.ctl to create the md they test with), but others are harder and whack-a-mole: mknode tests, for example, will always fail (at least with the default jail settings) and so should be fixed. There's several tests that would work in a native jail that would fail in a qemu jail because the system calls are missing, of which I think the jail system calls are in the list... I also run it on amd64 machines in a i386 jail and on aarch64 machines with a armv7 jail too. But I've done it a lot less, and don't have good numbers for doing that. I suspect they will be slightly more than Brooks is seeing since our 32-bit emulation is imperfect. So long as we keep the concepts of 'run this in the current env vs creating a new jail' and 'test expected to fail inside a jail vs work in un-jailed proc' separate, I think we'll be fine. And it might be nice to also have a 'don't run tests that need temp jails' option as well that I could mash for times I know they will fail to even launch. Cool stuff. Warner --0000000000005f95b806124eb547 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Mon, Feb 26, 2024 at 12:58=E2=80= =AFPM Igor Ostapenko <<a href=3D"mailto:igor.ostapenko@pm.me">igor.ostap= enko@pm.me</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex">On Friday, February 23rd, 2024 at 6:30 PM, Brooks Davis <<a h= ref=3D"mailto:brooks@freebsd.org" target=3D"_blank">brooks@freebsd.org</a>&= gt; wrote:<br> > <br> > > 2 The Idea<br> > > <br> > > The idea is not new. A test could be running in a jail -- it prov= ides the<br> > > required isolation with minimum or zero effort from a test.<br> > <br> > <br> > This generally sounds good. One minor concern I have is how this would= <br> > interact with the ability to run the test suite in a jail. This is<br> > imperfectly supported today (IIRC ~350 failures on amd64), but it'= s<br> > quite userful for testing sweeping userspace-only tests like libsys an= d<br> > I'd love to see support expanded and improved (failures fixed or t= ests<br> > skipped, poudriere jail support, etc).<br> > <br> <br> Thanks for your consideration.<br> <br> As I understand it's a question to the specific tests, which needs revi= sing<br> and tuning for a non-prison0 case. I saw such issues with existing tests,<b= r> frequently it was about jail restrictions. And by design, a root within a j= ail<br> cannot alter restrictions of its parent jail, i.e. it's up to the creat= or of<br> such jail. In contrast, execenv=3Djail asks Kyua to create a temporary jail= for<br> a test execution, and Kyua can be asked to configure it for the needs of a<= br> test via execenv_jail. But when Kyua itself is running within an existing j= ail<br> then expectations of the test suite should be lifted a level above,<br> especially for usual execenv=3Dhost based tests. Thus, it's out of Kyua= 's<br> control to help with that. And this proposal does not change or improve thi= s<br> case.<br> <br> Just a quick idea here. Probably, the test suite could help a creator of su= ch<br> jail above Kyua itself with its metadata. For example, a test is not intend= ed<br> to be run within a jail and it works fine if Kyua runs within prison0, but = if<br> Kyua itself runs in a jail it starts failing due to missing, for instance,<= br> allow.mlock allowence, see jail(8). A test could define<br> execenv_jail=3D"allow.mlock" metadata without asking for execenv= =3Djail, i.e. it's<br> like a hint for a non-prison0 case. And Kyua could aggregate and provide su= ch<br> information to ease creation of a jail above the Kyua itself.<br> <br> On the other end of the spectrum, we could think of a new feature of jail(8= )<br> to be able to ask it for a jail with maximum possible set of allowed things= .<br> Just thinking out loud.<br></blockquote><div><br></div><div>So I run the te= st suite in jail too today. To test how well qemu user-mode stuff</div><div= >is working. I see closer to 800 failures IIRC... Some of these are easy (l= ike</div><div>all the geom tests that need /dev/md.ctl to create the md the= y test with), but</div><div>others are harder and whack-a-mole: mknode test= s, for example, will always</div><div>fail (at least with the default jail = settings) and so should be fixed. There's several</div><div>tests that = would work in a native jail that would fail in a qemu jail because the</div= ><div>system calls are missing, of which I think the jail system calls are = in the list...</div><div><br></div><div>I also run it on amd64 machines in = a i386 jail and on aarch64=C2=A0machines with a armv7</div><div>jail too. B= ut I've done it a lot less, and don't have good numbers for doing t= hat. I</div><div>suspect they will be slightly more than Brooks is seeing s= ince our 32-bit emulation</div><div>is imperfect.</div><div><br></div><div>= So long as we keep the concepts of 'run this in the current env vs crea= ting a new jail'</div><div>and 'test expected to fail inside a jail= vs work in un-jailed proc' separate, I think</div><div>we'll be fi= ne. And it might be nice to also have a 'don't run tests that need = temp jails'</div><div>option as well that I could mash for times I know= they will fail to even launch.</div><div><br></div><div>Cool stuff.</div><= div><br></div><div>Warner</div></div></div> --0000000000005f95b806124eb547--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfpSFKmYBE_x1esgCU=9P47tsFV9wkDukLRvNxocOmjMGQ>