Date: Mon, 25 Nov 2013 13:29:33 -0500 From: Julio Merino <julio@meroh.net> To: freebsd-testing@freebsd.org Cc: Garrett Cooper <yanegomi@gmail.com>, rpaulo@freebsd.org, Simon Gerraty <sjg@juniper.net> Subject: Migrating tools/regression/ tests (and dealing with TAP) Message-ID: <CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--089e012281acdc2e8004ec048c0f
Content-Type: text/plain; charset=ISO-8859-1
Hello all,
Here come sample patches to move existing tests in tools/regression/bin
to the new layout ('tests' subdirectories within the corresponding tools)
and to hook these tests into the test suite.
The patches are quite straightforward: I just did an 'svn mv' of the
existing shell scripts files, wrote a Makefile using plain.test.mk (here
is where things get interesting) and applied minor changes to get all
things sorted out (e.g. mtree directory registration).
Consider this a non-working proof of concept. Again: this does NOT work
(well, it does, but yields invalid results). The problem is that
TAP-based test programs (like the ones to be run by prove) do NOT report
1 on failure and thus the plain Kyua interface is unsuitable because it
always reports everything as passed.
There are two solutions:
1) Patch every existing TAP-based test to propagate a 1 error code if
any of its tests fails. Doable, yes, but it will take a while.
(Mind you: this is the approach we tried in NetBSD and not all old
tests are gone yet, years later... it's a significant amount of work
to manually tweak all test programs!)
I'd rather keep existing code as is *for now*; it simplifies the
review of the changes and it doesn't immediately break people's
knowledge of existing test code. (If there is a need for a rewrite,
I'd suggest to go and use the ATF interface instead. But that's even
more work.)
2) Add a TAP-compliant backend to Kyua. This would allow us to plug
pretty much all existing tests in tools/regression/ without changes
to the code. I believe this is a worthy goal for the transitional
step and an easy first cut.
But this means I've gotta implement this backend (fun!). I've been
reading the protocol and seems reasonably simple, so I expect to have
something workable this week.
I'm choosing option 2 for now while leaving open the option of rewriting
tests using ATF *only after* they have been hooked into the build.
(What I'm shooting for, if you haven't guessed yet, is having a single
tests tree instead of 3 in src ASAP... Keeping multiple trees is not a
pretty situation for long.
Once the new Kyua backend is ready, the proposed patches in this mail
should change minimally: basically do s/plain/prove/ in the Makefiles.
This is why I'm sending this mail out already: you can get a pretty
good idea of what I'd like to do and your feedback would be useful!
There is a "tricky" side change to be made to bsd.progs.mk so that it is
possible to have a directory with SCRIPTS but no PROGS. make tinderbox
is happy.
Comments?
Thanks!
--089e012281acdc2e8004ec048c0f
Content-Type: text/x-diff; charset=us-ascii; name="handle-scripts.diff"
Content-Disposition: attachment; filename="handle-scripts.diff"
Content-Transfer-Encoding: base64
X-Attachment-Id: 6c1840eb6ca50d84_0.2
TWFrZSBic2QucHJvZ3MubWsgd29yayBpbiBkaXJlY3RvcmllcyB3aXRoIFNDUklQVFMgYnV0IG5v
IFBST0dTLg0KDQpUaGlzIGNoYW5nZSBmaXhlcyBzb21lIGNhc2VzIHdoZXJlIGJzZC5wcm9ncy5t
ayB3b3VsZCBmYWlsIHRvIGhhbmRsZQ0KZGlyZWN0b3JpZXMgd2l0aCBTQ1JJUFRTIGJ1dCBubyBQ
Uk9HUy4gIEluIHBhcnRpY3VsYXIsICJpbnN0YWxsIiBkaWQNCm5vdCBoYW5kbGUgc3VjaCBzY3Jp
cHRzIG5vciBkZXBlbmRlbnQgZmlsZXMgd2hlbiBic2Quc3ViZGlyLm1rIHdhcw0KYWRkZWQgdG8g
dGhlIG1peC4NCg0KVGhpcyBpcyAibWFrZSB0aW5kZXJib3giIGNsZWFuLg0KDQpJbmRleDogc2hh
cmUvbWsvYnNkLnByb2dzLm1rDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQotLS0gc2hhcmUvbWsvYnNkLnByb2dzLm1r
CShyZXZpc2lvbiAyNTg1NTIpDQorKysgc2hhcmUvbWsvYnNkLnByb2dzLm1rCSh3b3JraW5nIGNv
cHkpDQpAQCAtNzMsNyArNzMsNyBAQA0KICMgaGFuZGxlIGJlaW5nIGNhbGxlZCBbYnNkLl1wcm9n
cy5taw0KIC5pbmNsdWRlIDxic2QucHJvZy5taz4NCiANCi0uaWZuZGVmIFBST0cNCisuaWZuZGVm
IF9SRUNVUlNJTkdfUFJPR1MNCiAjIHRlbGwgcHJvZ3MubWsgd2UgbWlnaHQgd2FudCB0byBpbnN0
YWxsIHRoaW5ncw0KIFBST0dTX1RBUkdFVFMrPSBjbGVhbmRlcGVuZCBjbGVhbmRpciBjbGVhbm9i
aiBkZXBlbmQgaW5zdGFsbA0KIA0KQEAgLTg0LDExICs4NCwxMyBAQA0KIC5lbmRpZg0KIA0KICRw
ICR7cH1fcDogLlBIT05ZIC5NQUtFDQotCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtN
QUtFRklMRX0gUFJPRz0kcCAke3guJHB9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1m
ICR7TUFLRUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAk
e3guJHB9KQ0KIA0KIC5mb3IgdCBpbiAke1BST0dTX1RBUkdFVFM6Tzp1fQ0KICRwLiR0OiAuUEhP
TlkgLk1BS0UNCi0JKGNkICR7LkNVUkRJUn0gJiYgJHtNQUtFfSAtZiAke01BS0VGSUxFfSBQUk9H
PSRwICR7eC4kcH0gJHtAOkV9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1mICR7TUFL
RUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAke3guJHB9
ICR7QDpFfSkNCiAuZW5kZm9yDQogLmVuZGZvcg0KIA0KQEAgLTk2LDQgKzk4LDE4IEBADQogJHQ6
ICR7UFJPR1M6JT0lLiR0fQ0KIC5lbmRmb3INCiANCitTQ1JJUFRTX1RBUkdFVFMrPSBjbGVhbmRl
cGVuZCBjbGVhbmRpciBjbGVhbm9iaiBkZXBlbmQgaW5zdGFsbA0KKw0KKy5mb3IgcCBpbiAke1ND
UklQVFN9DQorLmZvciB0IGluICR7U0NSSVBUU19UQVJHRVRTOk86dX0NCiskcC4kdDogLlBIT05Z
IC5NQUtFDQorCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtNQUtFRklMRX0gX1JFQ1VS
U0lOR19QUk9HUz0gXA0KKwkgICAgU1VCRElSPSBTQ1JJUFQ9JHAgJHt4LiRwfSAke0A6RX0pDQor
LmVuZGZvcg0KKy5lbmRmb3INCisNCisuZm9yIHQgaW4gJHtTQ1JJUFRTX1RBUkdFVFM6Tzp1fQ0K
KyR0OiAke1NDUklQVFM6JT0lLiR0fQ0KKy5lbmRmb3INCisNCiAuZW5kaWYNCg==
--089e012281acdc2e8004ec048c0f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA>