From nobody Mon Feb 26 20:25:08 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkBv63XLKz5C6Fp for ; Mon, 26 Feb 2024 20:25:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkBv60Ztfz46st for ; Mon, 26 Feb 2024 20:25:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a3e891b5e4eso417467266b.0 for ; Mon, 26 Feb 2024 12:25:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1708979120; x=1709583920; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Upmp3LZFiuTvKwuBBHjxQeiY6cJ24tOCy/aQ2ZlsQtE=; b=v5PuHKXP6STe8//N9FvQp9p4fAuwovVrWl14J3V5lu90TJsvNRumrG+mrDLRZ/wKuE xb7HchN+48F6pZO2A7H8MC3TgfpyrnHlKzTN7GBo5f/nkLj4bQtPNesGlQQJRnUB5yIK 96/7dYsdofDUI5jeQVbgo+f7XX8x4JCICRIDnLLkcPrz/74Jr8sxQr3Vm7U27vAboggL RYUrs8f+5qPprEUYr+4AsHq8E9uiSwNJ06uVMGMtW4IIqxZoxoSkrqBerH9p32XAOlJK O0yoQao79SVXca5PfKFTJL8CPC8mXCM6DGa6DLOZKoicXgdbJ7w2+j0Tx7P82SXCIsPC jr+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708979120; x=1709583920; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Upmp3LZFiuTvKwuBBHjxQeiY6cJ24tOCy/aQ2ZlsQtE=; b=tythAg5CU4Txo9MrwLzFECh2y2mD5RosjuYoRvkKrjiXNCTlKiXqX9tTt+drQJOZyb u/jyKuMXG+Jv3D2teYSJg+oau8fEZqqvrPqUmCvtEoV72Zb0hB3b3mTJ3dy++dNPyzjM Q4r3aEIPPrSaQoZBD5WZuhy9ixX0q8pM+5QK/g3f+apjN11lCQUtyDVtuSJHGB6Z0Jn2 W/jvG3tPCX/ejXXIJASB7sO7RoWzPW3Sqgm3liSSwAgaZJdY6eqKfYs+qHDtmAR6rsHB 8VpuNpKvBkoMU8DL7MDUNCAw4/jQmFR9sEnZiCvjH927Ui3x7mSbz3CfI7k75lCYWx9A JKTg== X-Forwarded-Encrypted: i=1; AJvYcCWOyTC3wXGMJhhWca9YscSjsaQ6CJNYlLQT6l2A++FFcju8/25IRFH8cjIKTHT/nCFTLahTNVKFVcC1sA55Kyve9cFZZi+gKEOB9/w= X-Gm-Message-State: AOJu0Yy0IiAz7Wpp1uxHA31z4/0GgR6ITn2YWADqw5gLnUbSWd8Wzybv Hp6fboAK65wCN/xiqBUTJmtbV+D5aYAkI3WnxySbZ0mtQQ6tuH8GIFDJZeRA/n5YKmG7v/pFbAU YI94jjtDK//WWQaR/nYOiBhVQW1Lm1mQ+KCLqyA== X-Google-Smtp-Source: AGHT+IGGMEukCMaWqGIB/puUKTBsWi+j905x8InI7y0yTrb/M4QxJUs6DgHRTH3DUb3Kesnm/ZaZvoKqYUan46xVXWg= X-Received: by 2002:a17:906:3951:b0:a3e:a5b9:d69d with SMTP id g17-20020a170906395100b00a3ea5b9d69dmr5459188eje.74.1708979119947; Mon, 26 Feb 2024 12:25:19 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me> <8OSrqDr54xqBgG5cQG_M2Puk2zSxQUletfKhYUElMb8PNnydh6xTDuPgX4HzICWQO6VbC3q27yAOWstIdJ8iZsfN03gJBf9-l03D_qwYcH8=@pm.me> In-Reply-To: <8OSrqDr54xqBgG5cQG_M2Puk2zSxQUletfKhYUElMb8PNnydh6xTDuPgX4HzICWQO6VbC3q27yAOWstIdJ8iZsfN03gJBf9-l03D_qwYcH8=@pm.me> From: Warner Losh Date: Mon, 26 Feb 2024 13:25:08 -0700 Message-ID: Subject: Re: Add jail execution environment support to the FreeBSD test suite To: Igor Ostapenko Cc: Brooks Davis , "freebsd-hackers@freebsd.org" Content-Type: multipart/alternative; boundary="0000000000005f95b806124eb547" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4TkBv60Ztfz46st --0000000000005f95b806124eb547 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Feb 26, 2024 at 12:58=E2=80=AFPM Igor Ostapenko wrote: > On Friday, February 23rd, 2024 at 6:30 PM, Brooks Davis < > brooks@freebsd.org> wrote: > > > > > 2 The Idea > > > > > > The idea is not new. A test could be running in a jail -- it provides > the > > > required isolation with minimum or zero effort from a test. > > > > > > This generally sounds good. One minor concern I have is how this would > > interact with the ability to run the test suite in a jail. This is > > imperfectly supported today (IIRC ~350 failures on amd64), but it's > > quite userful for testing sweeping userspace-only tests like libsys and > > I'd love to see support expanded and improved (failures fixed or tests > > skipped, poudriere jail support, etc). > > > > Thanks for your consideration. > > As I understand it's a question to the specific tests, which needs revisi= ng > and tuning for a non-prison0 case. I saw such issues with existing tests, > frequently it was about jail restrictions. And by design, a root within a > jail > cannot alter restrictions of its parent jail, i.e. it's up to the creator > of > such jail. In contrast, execenv=3Djail asks Kyua to create a temporary ja= il > for > a test execution, and Kyua can be asked to configure it for the needs of = a > test via execenv_jail. But when Kyua itself is running within an existing > jail > then expectations of the test suite should be lifted a level above, > especially for usual execenv=3Dhost based tests. Thus, it's out of Kyua's > control to help with that. And this proposal does not change or improve > this > case. > > Just a quick idea here. Probably, the test suite could help a creator of > such > jail above Kyua itself with its metadata. For example, a test is not > intended > to be run within a jail and it works fine if Kyua runs within prison0, bu= t > if > Kyua itself runs in a jail it starts failing due to missing, for instance= , > allow.mlock allowence, see jail(8). A test could define > execenv_jail=3D"allow.mlock" metadata without asking for execenv=3Djail, = i.e. > it's > like a hint for a non-prison0 case. And Kyua could aggregate and provide > such > information to ease creation of a jail above the Kyua itself. > > On the other end of the spectrum, we could think of a new feature of > jail(8) > to be able to ask it for a jail with maximum possible set of allowed > things. > Just thinking out loud. > So I run the test suite in jail too today. To test how well qemu user-mode stuff is working. I see closer to 800 failures IIRC... Some of these are easy (like all the geom tests that need /dev/md.ctl to create the md they test with), but others are harder and whack-a-mole: mknode tests, for example, will always fail (at least with the default jail settings) and so should be fixed. There's several tests that would work in a native jail that would fail in a qemu jail because the system calls are missing, of which I think the jail system calls are in the list... I also run it on amd64 machines in a i386 jail and on aarch64 machines with a armv7 jail too. But I've done it a lot less, and don't have good numbers for doing that. I suspect they will be slightly more than Brooks is seeing since our 32-bit emulation is imperfect. So long as we keep the concepts of 'run this in the current env vs creating a new jail' and 'test expected to fail inside a jail vs work in un-jailed proc' separate, I think we'll be fine. And it might be nice to also have a 'don't run tests that need temp jails' option as well that I could mash for times I know they will fail to even launch. Cool stuff. Warner --0000000000005f95b806124eb547 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Feb 26, 2024 at 12:58=E2=80= =AFPM Igor Ostapenko <igor.ostap= enko@pm.me> wrote:
On Friday, February 23rd, 2024 at 6:30 PM, Brooks Davis <brooks@freebsd.org&= gt; wrote:
>
> > 2 The Idea
> >
> > The idea is not new. A test could be running in a jail -- it prov= ides the
> > required isolation with minimum or zero effort from a test.
>
>
> This generally sounds good. One minor concern I have is how this would=
> interact with the ability to run the test suite in a jail. This is
> imperfectly supported today (IIRC ~350 failures on amd64), but it'= s
> quite userful for testing sweeping userspace-only tests like libsys an= d
> I'd love to see support expanded and improved (failures fixed or t= ests
> skipped, poudriere jail support, etc).
>

Thanks for your consideration.

As I understand it's a question to the specific tests, which needs revi= sing
and tuning for a non-prison0 case. I saw such issues with existing tests, frequently it was about jail restrictions. And by design, a root within a j= ail
cannot alter restrictions of its parent jail, i.e. it's up to the creat= or of
such jail. In contrast, execenv=3Djail asks Kyua to create a temporary jail= for
a test execution, and Kyua can be asked to configure it for the needs of a<= br> test via execenv_jail. But when Kyua itself is running within an existing j= ail
then expectations of the test suite should be lifted a level above,
especially for usual execenv=3Dhost based tests. Thus, it's out of Kyua= 's
control to help with that. And this proposal does not change or improve thi= s
case.

Just a quick idea here. Probably, the test suite could help a creator of su= ch
jail above Kyua itself with its metadata. For example, a test is not intend= ed
to be run within a jail and it works fine if Kyua runs within prison0, but = if
Kyua itself runs in a jail it starts failing due to missing, for instance,<= br> allow.mlock allowence, see jail(8). A test could define
execenv_jail=3D"allow.mlock" metadata without asking for execenv= =3Djail, i.e. it's
like a hint for a non-prison0 case. And Kyua could aggregate and provide su= ch
information to ease creation of a jail above the Kyua itself.

On the other end of the spectrum, we could think of a new feature of jail(8= )
to be able to ask it for a jail with maximum possible set of allowed things= .
Just thinking out loud.

So I run the te= st suite in jail too today. To test how well qemu user-mode stuff
is working. I see closer to 800 failures IIRC... Some of these are easy (l= ike
all the geom tests that need /dev/md.ctl to create the md the= y test with), but
others are harder and whack-a-mole: mknode test= s, for example, will always
fail (at least with the default jail = settings) and so should be fixed. There's several
tests that = would work in a native jail that would fail in a qemu jail because the
system calls are missing, of which I think the jail system calls are = in the list...

I also run it on amd64 machines in = a i386 jail and on aarch64=C2=A0machines with a armv7
jail too. B= ut I've done it a lot less, and don't have good numbers for doing t= hat. I
suspect they will be slightly more than Brooks is seeing s= ince our 32-bit emulation
is imperfect.

= So long as we keep the concepts of 'run this in the current env vs crea= ting a new jail'
and 'test expected to fail inside a jail= vs work in un-jailed proc' separate, I think
we'll be fi= ne. And it might be nice to also have a 'don't run tests that need = temp jails'
option as well that I could mash for times I know= they will fail to even launch.

Cool stuff.
<= div>
Warner
--0000000000005f95b806124eb547--