Date: Mon, 25 Nov 2013 13:29:33 -0500 From: Julio Merino <julio@meroh.net> To: freebsd-testing@freebsd.org Cc: Garrett Cooper <yanegomi@gmail.com>, rpaulo@freebsd.org, Simon Gerraty <sjg@juniper.net> Subject: Migrating tools/regression/ tests (and dealing with TAP) Message-ID: <CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--089e012281acdc2e8004ec048c0f Content-Type: text/plain; charset=ISO-8859-1 Hello all, Here come sample patches to move existing tests in tools/regression/bin to the new layout ('tests' subdirectories within the corresponding tools) and to hook these tests into the test suite. The patches are quite straightforward: I just did an 'svn mv' of the existing shell scripts files, wrote a Makefile using plain.test.mk (here is where things get interesting) and applied minor changes to get all things sorted out (e.g. mtree directory registration). Consider this a non-working proof of concept. Again: this does NOT work (well, it does, but yields invalid results). The problem is that TAP-based test programs (like the ones to be run by prove) do NOT report 1 on failure and thus the plain Kyua interface is unsuitable because it always reports everything as passed. There are two solutions: 1) Patch every existing TAP-based test to propagate a 1 error code if any of its tests fails. Doable, yes, but it will take a while. (Mind you: this is the approach we tried in NetBSD and not all old tests are gone yet, years later... it's a significant amount of work to manually tweak all test programs!) I'd rather keep existing code as is *for now*; it simplifies the review of the changes and it doesn't immediately break people's knowledge of existing test code. (If there is a need for a rewrite, I'd suggest to go and use the ATF interface instead. But that's even more work.) 2) Add a TAP-compliant backend to Kyua. This would allow us to plug pretty much all existing tests in tools/regression/ without changes to the code. I believe this is a worthy goal for the transitional step and an easy first cut. But this means I've gotta implement this backend (fun!). I've been reading the protocol and seems reasonably simple, so I expect to have something workable this week. I'm choosing option 2 for now while leaving open the option of rewriting tests using ATF *only after* they have been hooked into the build. (What I'm shooting for, if you haven't guessed yet, is having a single tests tree instead of 3 in src ASAP... Keeping multiple trees is not a pretty situation for long. Once the new Kyua backend is ready, the proposed patches in this mail should change minimally: basically do s/plain/prove/ in the Makefiles. This is why I'm sending this mail out already: you can get a pretty good idea of what I'd like to do and your feedback would be useful! There is a "tricky" side change to be made to bsd.progs.mk so that it is possible to have a directory with SCRIPTS but no PROGS. make tinderbox is happy. Comments? Thanks! --089e012281acdc2e8004ec048c0f Content-Type: text/x-diff; charset=us-ascii; name="handle-scripts.diff" Content-Disposition: attachment; filename="handle-scripts.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: 6c1840eb6ca50d84_0.2 TWFrZSBic2QucHJvZ3MubWsgd29yayBpbiBkaXJlY3RvcmllcyB3aXRoIFNDUklQVFMgYnV0IG5v IFBST0dTLg0KDQpUaGlzIGNoYW5nZSBmaXhlcyBzb21lIGNhc2VzIHdoZXJlIGJzZC5wcm9ncy5t ayB3b3VsZCBmYWlsIHRvIGhhbmRsZQ0KZGlyZWN0b3JpZXMgd2l0aCBTQ1JJUFRTIGJ1dCBubyBQ Uk9HUy4gIEluIHBhcnRpY3VsYXIsICJpbnN0YWxsIiBkaWQNCm5vdCBoYW5kbGUgc3VjaCBzY3Jp cHRzIG5vciBkZXBlbmRlbnQgZmlsZXMgd2hlbiBic2Quc3ViZGlyLm1rIHdhcw0KYWRkZWQgdG8g dGhlIG1peC4NCg0KVGhpcyBpcyAibWFrZSB0aW5kZXJib3giIGNsZWFuLg0KDQpJbmRleDogc2hh cmUvbWsvYnNkLnByb2dzLm1rDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQotLS0gc2hhcmUvbWsvYnNkLnByb2dzLm1r CShyZXZpc2lvbiAyNTg1NTIpDQorKysgc2hhcmUvbWsvYnNkLnByb2dzLm1rCSh3b3JraW5nIGNv cHkpDQpAQCAtNzMsNyArNzMsNyBAQA0KICMgaGFuZGxlIGJlaW5nIGNhbGxlZCBbYnNkLl1wcm9n cy5taw0KIC5pbmNsdWRlIDxic2QucHJvZy5taz4NCiANCi0uaWZuZGVmIFBST0cNCisuaWZuZGVm IF9SRUNVUlNJTkdfUFJPR1MNCiAjIHRlbGwgcHJvZ3MubWsgd2UgbWlnaHQgd2FudCB0byBpbnN0 YWxsIHRoaW5ncw0KIFBST0dTX1RBUkdFVFMrPSBjbGVhbmRlcGVuZCBjbGVhbmRpciBjbGVhbm9i aiBkZXBlbmQgaW5zdGFsbA0KIA0KQEAgLTg0LDExICs4NCwxMyBAQA0KIC5lbmRpZg0KIA0KICRw ICR7cH1fcDogLlBIT05ZIC5NQUtFDQotCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtN QUtFRklMRX0gUFJPRz0kcCAke3guJHB9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1m ICR7TUFLRUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAk e3guJHB9KQ0KIA0KIC5mb3IgdCBpbiAke1BST0dTX1RBUkdFVFM6Tzp1fQ0KICRwLiR0OiAuUEhP TlkgLk1BS0UNCi0JKGNkICR7LkNVUkRJUn0gJiYgJHtNQUtFfSAtZiAke01BS0VGSUxFfSBQUk9H PSRwICR7eC4kcH0gJHtAOkV9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1mICR7TUFL RUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAke3guJHB9 ICR7QDpFfSkNCiAuZW5kZm9yDQogLmVuZGZvcg0KIA0KQEAgLTk2LDQgKzk4LDE4IEBADQogJHQ6 ICR7UFJPR1M6JT0lLiR0fQ0KIC5lbmRmb3INCiANCitTQ1JJUFRTX1RBUkdFVFMrPSBjbGVhbmRl cGVuZCBjbGVhbmRpciBjbGVhbm9iaiBkZXBlbmQgaW5zdGFsbA0KKw0KKy5mb3IgcCBpbiAke1ND UklQVFN9DQorLmZvciB0IGluICR7U0NSSVBUU19UQVJHRVRTOk86dX0NCiskcC4kdDogLlBIT05Z IC5NQUtFDQorCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtNQUtFRklMRX0gX1JFQ1VS U0lOR19QUk9HUz0gXA0KKwkgICAgU1VCRElSPSBTQ1JJUFQ9JHAgJHt4LiRwfSAke0A6RX0pDQor LmVuZGZvcg0KKy5lbmRmb3INCisNCisuZm9yIHQgaW4gJHtTQ1JJUFRTX1RBUkdFVFM6Tzp1fQ0K KyR0OiAke1NDUklQVFM6JT0lLiR0fQ0KKy5lbmRmb3INCisNCiAuZW5kaWYNCg== --089e012281acdc2e8004ec048c0f--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADyfeQU1-4pxjbrFQyJW_-HvoWTQY6TMYv%2BVCwBr5B06HW9=WA>