Date: Mon, 25 Nov 2013 13:29:33 -0500 From: Julio Merino <julio@meroh.net> To: freebsd-testing@freebsd.org Cc: Garrett Cooper <yanegomi@gmail.com>, rpaulo@freebsd.org, Simon Gerraty <sjg@juniper.net> Subject: Migrating tools/regression/ tests (and dealing with TAP) Message-ID: <CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hello all,
Here come sample patches to move existing tests in tools/regression/bin
to the new layout ('tests' subdirectories within the corresponding tools)
and to hook these tests into the test suite.
The patches are quite straightforward: I just did an 'svn mv' of the
existing shell scripts files, wrote a Makefile using plain.test.mk (here
is where things get interesting) and applied minor changes to get all
things sorted out (e.g. mtree directory registration).
Consider this a non-working proof of concept. Again: this does NOT work
(well, it does, but yields invalid results). The problem is that
TAP-based test programs (like the ones to be run by prove) do NOT report
1 on failure and thus the plain Kyua interface is unsuitable because it
always reports everything as passed.
There are two solutions:
1) Patch every existing TAP-based test to propagate a 1 error code if
any of its tests fails. Doable, yes, but it will take a while.
(Mind you: this is the approach we tried in NetBSD and not all old
tests are gone yet, years later... it's a significant amount of work
to manually tweak all test programs!)
I'd rather keep existing code as is *for now*; it simplifies the
review of the changes and it doesn't immediately break people's
knowledge of existing test code. (If there is a need for a rewrite,
I'd suggest to go and use the ATF interface instead. But that's even
more work.)
2) Add a TAP-compliant backend to Kyua. This would allow us to plug
pretty much all existing tests in tools/regression/ without changes
to the code. I believe this is a worthy goal for the transitional
step and an easy first cut.
But this means I've gotta implement this backend (fun!). I've been
reading the protocol and seems reasonably simple, so I expect to have
something workable this week.
I'm choosing option 2 for now while leaving open the option of rewriting
tests using ATF *only after* they have been hooked into the build.
(What I'm shooting for, if you haven't guessed yet, is having a single
tests tree instead of 3 in src ASAP... Keeping multiple trees is not a
pretty situation for long.
Once the new Kyua backend is ready, the proposed patches in this mail
should change minimally: basically do s/plain/prove/ in the Makefiles.
This is why I'm sending this mail out already: you can get a pretty
good idea of what I'd like to do and your feedback would be useful!
There is a "tricky" side change to be made to bsd.progs.mk so that it is
possible to have a directory with SCRIPTS but no PROGS. make tinderbox
is happy.
Comments?
Thanks!
[-- Attachment #2 --]
Make bsd.progs.mk work in directories with SCRIPTS but no PROGS.
This change fixes some cases where bsd.progs.mk would fail to handle
directories with SCRIPTS but no PROGS. In particular, "install" did
not handle such scripts nor dependent files when bsd.subdir.mk was
added to the mix.
This is "make tinderbox" clean.
Index: share/mk/bsd.progs.mk
===================================================================
--- share/mk/bsd.progs.mk (revision 258552)
+++ share/mk/bsd.progs.mk (working copy)
@@ -73,7 +73,7 @@
# handle being called [bsd.]progs.mk
.include <bsd.prog.mk>
-.ifndef PROG
+.ifndef _RECURSING_PROGS
# tell progs.mk we might want to install things
PROGS_TARGETS+= cleandepend cleandir cleanobj depend install
@@ -84,11 +84,13 @@
.endif
$p ${p}_p: .PHONY .MAKE
- (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} PROG=$p ${x.$p})
+ (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} _RECURSING_PROGS= \
+ SUBDIR= PROG=$p ${x.$p})
.for t in ${PROGS_TARGETS:O:u}
$p.$t: .PHONY .MAKE
- (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} PROG=$p ${x.$p} ${@:E})
+ (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} _RECURSING_PROGS= \
+ SUBDIR= PROG=$p ${x.$p} ${@:E})
.endfor
.endfor
@@ -96,4 +98,18 @@
$t: ${PROGS:%=%.$t}
.endfor
+SCRIPTS_TARGETS+= cleandepend cleandir cleanobj depend install
+
+.for p in ${SCRIPTS}
+.for t in ${SCRIPTS_TARGETS:O:u}
+$p.$t: .PHONY .MAKE
+ (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} _RECURSING_PROGS= \
+ SUBDIR= SCRIPT=$p ${x.$p} ${@:E})
+.endfor
+.endfor
+
+.for t in ${SCRIPTS_TARGETS:O:u}
+$t: ${SCRIPTS:%=%.$t}
+.endfor
+
.endif
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA>