Date: Mon, 25 Nov 2013 21:52:39 -0800 From: Garrett Cooper <yanegomi@gmail.com> To: Julio Merino <julio@meroh.net> Cc: freebsd-testing@freebsd.org, rpaulo@freebsd.org, Simon Gerraty <sjg@juniper.net> Subject: Re: Migrating tools/regression/ tests (and dealing with TAP) Message-ID: <9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE@gmail.com> In-Reply-To: <CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA@mail.gmail.com> References: <CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Nov 25, 2013, at 10:29 AM, Julio Merino <julio@meroh.net> wrote:
> Hello all,
>=20
> Here come sample patches to move existing tests in =
tools/regression/bin
> to the new layout ('tests' subdirectories within the corresponding =
tools)
> and to hook these tests into the test suite.
>=20
> The patches are quite straightforward: I just did an 'svn mv' of the
> existing shell scripts files, wrote a Makefile using plain.test.mk =
(here
> is where things get interesting) and applied minor changes to get all
> things sorted out (e.g. mtree directory registration).
>=20
> Consider this a non-working proof of concept. Again: this does NOT =
work
> (well, it does, but yields invalid results). The problem is that
> TAP-based test programs (like the ones to be run by prove) do NOT =
report
> 1 on failure and thus the plain Kyua interface is unsuitable because =
it
> always reports everything as passed.
>=20
> There are two solutions:
>=20
> 1) Patch every existing TAP-based test to propagate a 1 error code if
> any of its tests fails. Doable, yes, but it will take a while.
> (Mind you: this is the approach we tried in NetBSD and not all old
> tests are gone yet, years later... it's a significant amount of work
> to manually tweak all test programs!)
>=20
> I'd rather keep existing code as is *for now*; it simplifies the
> review of the changes and it doesn't immediately break people's
> knowledge of existing test code. (If there is a need for a rewrite,
> I'd suggest to go and use the ATF interface instead. But that's =
even
> more work.)
>=20
> 2) Add a TAP-compliant backend to Kyua. This would allow us to plug
> pretty much all existing tests in tools/regression/ without changes
> to the code. I believe this is a worthy goal for the transitional
> step and an easy first cut.
>=20
> But this means I've gotta implement this backend (fun!). I've been
> reading the protocol and seems reasonably simple, so I expect to =
have
> something workable this week.
>=20
> I'm choosing option 2 for now while leaving open the option of =
rewriting
> tests using ATF *only after* they have been hooked into the build.
> (What I'm shooting for, if you haven't guessed yet, is having a single
> tests tree instead of 3 in src ASAP... Keeping multiple trees is not =
a
> pretty situation for long.
>=20
> Once the new Kyua backend is ready, the proposed patches in this mail
> should change minimally: basically do s/plain/prove/ in the Makefiles.
> This is why I'm sending this mail out already: you can get a pretty
> good idea of what I'd like to do and your feedback would be useful!
>=20
> There is a "tricky" side change to be made to bsd.progs.mk so that it =
is
> possible to have a directory with SCRIPTS but no PROGS. make =
tinderbox
> is happy.
>=20
> Comments?
>=20
> Thanks!
> <migrate-bin-tests.sh><handle-scripts.diff>
The big concern is that you=92re shuffling around tests that are =
being actively developed by other developers. Have you provided a heads =
up to Jilles first at least..?
Thanks,
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE>