From nobody Mon Apr 29 22:16:58 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VSyP46WCPz5JyKG for ; Mon, 29 Apr 2024 22:17:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VSyP44V58z43nf for ; Mon, 29 Apr 2024 22:17:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12e.google.com with SMTP id 2adb3069b0e04-51acb95b892so6189404e87.2 for ; Mon, 29 Apr 2024 15:17:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1714429030; x=1715033830; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=mNySv+uUrjebTettjux8qBQO6EUCmQdUsyyvUiDvhAo=; b=v612OmqeeIrPw6jCLFV+lArC0SIaVNDEaPmJgna7c4+d+f5ER+dfxWq9eSbVvUn7yB TtViK7N2NWTiP7COHT8hq9lHmX7mpE/GTXNhbCzMY4guafuh0PlJUX1le62C8bWk51yW hXb8fMnfdO2mXMOBXFLaTYB8VtI70r9TBlA0g8Uraut53tBADd9UkwGl+PFQdh5EQf9E Er9cSnmrsszVc6zIdkZpDNxZxgoV8oOEhLkJ0dhLJKz9Sv2agFBjU8AGF7DeTSnBX+8M 2FBaGeqzMfR4BWAxAFbhBSwqM0SBsSKEoztPBmwJ80jnyo0O8RaVbg3Y9C42DDmr8tB5 MsRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714429030; x=1715033830; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mNySv+uUrjebTettjux8qBQO6EUCmQdUsyyvUiDvhAo=; b=YmS2iQIiKclzAtjQwEGyqdIX8CXgOuN+AzsgeL8OEKRoJ87FkHNJSE+w9eDuzbkV+q 8q8R8hWjCvNbJ2fFF0AO13/t0Ez5M5aLLWMZ0e2wKWAdFwud5038fNBri6I98QsbXS4c bfqpj2/ERm7aAhMPHkAdnljBhvmorLtJN0mEm6q6hQq0IsqJAwEgEqBjSvq1RmkuXxYL FhSTsiNrjw49GEJpgXWD9qmkBTEx433UfTOSY+OpOdKz0FX0Ad9I9XkatBgBfWuMM0kE uYFpQKqSwKSCw6B8+zWKXBcn1vj3GW5NrSgLrMGgoYGUx4U0jkyOGsAFOy58mnfvZBfE IpKA== X-Forwarded-Encrypted: i=1; AJvYcCXpMNxb00Wi1dn+tFbjKB/deKvEcJ1xCSpmIL5C9SI22mf4NmV8PfxBx2IRNoJzi7CZF5cYhoLpsRONwt286xVkekfLwqY7hnkf3rg= X-Gm-Message-State: AOJu0YyP4lfziDHKzG6CYmuvUzboPoM+Gx43YwD6Gqx2lp5HFzufo1dd 84+s1+X63NBtHhEq4dqWCZrKcEB5Mb76djJbfeHMBVDyTOcfeiR18P1RfomxVDKoJCSj5yGpBK8 4Fk/CWfn0hg8UK6tjhoeB4ViiS+zMaGLRSMCB7e2LWPOj6pLHZ6U= X-Google-Smtp-Source: AGHT+IFmHydug+7Qc0MAzW3Nc8qhtiNudCYSenBNHrKdEK/tReqTt0A4w4QzJBPP/alEnTsGpD4+/5a9NP2NanMzVQ0= X-Received: by 2002:ac2:4c12:0:b0:515:cbf1:9fda with SMTP id t18-20020ac24c12000000b00515cbf19fdamr8614553lfq.61.1714429029921; Mon, 29 Apr 2024 15:17:09 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Mon, 29 Apr 2024 16:16:58 -0600 Message-ID: Subject: Re: How to run tests without installing? To: Brooks Davis Cc: Christian Weisgerber , freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000525a9b0617439d65" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4VSyP44V58z43nf --000000000000525a9b0617439d65 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Apr 29, 2024 at 4:03=E2=80=AFPM Brooks Davis w= rote: > On Mon, Apr 29, 2024 at 12:34:51PM +0200, Christian Weisgerber wrote: > > How can I run the regression tests on my work-in-progress without > > installing it first? > > > > Say I'm changing something in sh. Can I run the tests on the > > compiled sh in /usr/obj, without having to install my potentially > > broken work into the system? Running "make tests" in src/bin/sh > > doesn't seem to actually test anything. > > Generally speaking you can't. This is one of the problematic things > about the current test framework. The best you can do for something > like sh where you really don't want to install a broken one is probably > installing in a jail and running tests there. > For library tests, though, you can run the individual test just like kyua would, but all bets are off if it relies on anything other than the code in the binary (and associated shared libraries). Even then, it's a royal pain. I usually stop one step short of a jail when I've needed to do this: I just make installworld DESTDIR=3D$HOME/mumble and then do sudo chroot $HOME/mumble. Though it doesn't work at all well for some tests (they hate being run in a chroot), but they tend to be networking tests anyway... But the same tests that don't work well in a chroot work even less well in a jail. There's a crazy lot of them that require different /dev/ entries, which is why I don't do the chroot thing that often... tl;dr: It's easier not to test, and that's actually a problem. Warner --000000000000525a9b0617439d65 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Apr 29, 2024 at 4:03=E2=80=AF= PM Brooks Davis <brooks@freebsd.or= g> wrote:
On Mon, Apr 29, 2024 at 12:34:51PM +0200, Christian Weisgerber wrote:
> How can I run the regression tests on my work-in-progress without
> installing it first?
>
> Say I'm changing something in sh.=C2=A0 Can I run the tests on the=
> compiled sh in /usr/obj, without having to install my potentially
> broken work into the system?=C2=A0 Running "make tests" in s= rc/bin/sh
> doesn't seem to actually test anything.

Generally speaking you can't.=C2=A0 This is one of the problematic thin= gs
about the current test framework.=C2=A0 The best you can do for something like sh where you really don't want to install a broken one is probably=
installing in a jail and running tests there.

For library tests, though, you can run the individual test just like= kyua
would, but all bets are off if it relies on anything other = than the code in
the binary (and associated shared libraries). Ev= en then, it's a royal
pain.

I usuall= y stop one step short of a jail when I've needed to do this:
= I just make installworld DESTDIR=3D$HOME/mumble and then do
sudo = chroot $HOME/mumble. Though it doesn't work at all well
for s= ome tests (they hate being run in a chroot), but they tend to
be = networking tests anyway... But the same tests that don't work
well in a chroot work even less well in a jail. There's a crazy lot
of them that require different /dev/ entries, which is why I don= 9;t do
the chroot thing that often...

tl= ;dr: It's easier not to test, and that's actually a problem.
<= div>
Warner
--000000000000525a9b0617439d65--