From owner-freebsd-testing@FreeBSD.ORG Mon Nov 25 18:29:57 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 32B8279D for ; Mon, 25 Nov 2013 18:29:57 +0000 (UTC) Received: from mail-la0-f42.google.com (mail-la0-f42.google.com [209.85.215.42]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 44E3A2E3D for ; Mon, 25 Nov 2013 18:29:56 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id ec20so3423883lab.29 for ; Mon, 25 Nov 2013 10:29:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc :content-type; bh=d+rOrBz3vFyR0kZO7eEb0bEjqqQDeIGaTrG/PJ6rpUY=; b=HcEhJjeSPYlN4QedY91qFxIw0tD2EcSX8z6fci4k2VEF7FdCHArmuzRXDcDbMzBbz/ /w8dsrC0OAYQ3T89DFerlq/JgCCSQgJ05a6EyRwQeGrf6BwquazL4KkM4LIVVl5WM9sK 7IMOKbkXQOD9s92BtfbNenltvcI0dIaHzFd5NVoQfrIq6XitbcZxtlbmwXdpVMhwaDYK VfGoXOmGIXhh0Wruh0dYNlNMN8DhZGKJXVfVj8HMz+2PaJHiiqpS+EJfHiKA/MMKRwW9 MdXAChYSUs0tRNzTmLQv1azwoFUjCfIU/17RR6PQbHHeTihXf8UXoDpHBKtvhyD367ct tBnw== X-Gm-Message-State: ALoCoQmpB0UJclyvh9l/oeNHAZ71N4dzRG1/+7Ynf/gsHLnbnOexnOGBv950e4DiKSsElkj1P9we X-Received: by 10.152.120.135 with SMTP id lc7mr2073991lab.38.1385404194098; Mon, 25 Nov 2013 10:29:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Mon, 25 Nov 2013 10:29:33 -0800 (PST) X-Originating-IP: [108.176.158.82] From: Julio Merino Date: Mon, 25 Nov 2013 13:29:33 -0500 Message-ID: Subject: Migrating tools/regression/ tests (and dealing with TAP) To: freebsd-testing@freebsd.org Content-Type: multipart/mixed; boundary=089e012281acdc2e8004ec048c0f X-Content-Filtered-By: Mailman/MimeDel 2.1.16 Cc: Garrett Cooper , rpaulo@freebsd.org, Simon Gerraty X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 18:29:57 -0000 --089e012281acdc2e8004ec048c0f Content-Type: text/plain; charset=ISO-8859-1 Hello all, Here come sample patches to move existing tests in tools/regression/bin to the new layout ('tests' subdirectories within the corresponding tools) and to hook these tests into the test suite. The patches are quite straightforward: I just did an 'svn mv' of the existing shell scripts files, wrote a Makefile using plain.test.mk (here is where things get interesting) and applied minor changes to get all things sorted out (e.g. mtree directory registration). Consider this a non-working proof of concept. Again: this does NOT work (well, it does, but yields invalid results). The problem is that TAP-based test programs (like the ones to be run by prove) do NOT report 1 on failure and thus the plain Kyua interface is unsuitable because it always reports everything as passed. There are two solutions: 1) Patch every existing TAP-based test to propagate a 1 error code if any of its tests fails. Doable, yes, but it will take a while. (Mind you: this is the approach we tried in NetBSD and not all old tests are gone yet, years later... it's a significant amount of work to manually tweak all test programs!) I'd rather keep existing code as is *for now*; it simplifies the review of the changes and it doesn't immediately break people's knowledge of existing test code. (If there is a need for a rewrite, I'd suggest to go and use the ATF interface instead. But that's even more work.) 2) Add a TAP-compliant backend to Kyua. This would allow us to plug pretty much all existing tests in tools/regression/ without changes to the code. I believe this is a worthy goal for the transitional step and an easy first cut. But this means I've gotta implement this backend (fun!). I've been reading the protocol and seems reasonably simple, so I expect to have something workable this week. I'm choosing option 2 for now while leaving open the option of rewriting tests using ATF *only after* they have been hooked into the build. (What I'm shooting for, if you haven't guessed yet, is having a single tests tree instead of 3 in src ASAP... Keeping multiple trees is not a pretty situation for long. Once the new Kyua backend is ready, the proposed patches in this mail should change minimally: basically do s/plain/prove/ in the Makefiles. This is why I'm sending this mail out already: you can get a pretty good idea of what I'd like to do and your feedback would be useful! There is a "tricky" side change to be made to bsd.progs.mk so that it is possible to have a directory with SCRIPTS but no PROGS. make tinderbox is happy. Comments? Thanks! --089e012281acdc2e8004ec048c0f Content-Type: text/x-diff; charset=us-ascii; name="handle-scripts.diff" Content-Disposition: attachment; filename="handle-scripts.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: 6c1840eb6ca50d84_0.2 TWFrZSBic2QucHJvZ3MubWsgd29yayBpbiBkaXJlY3RvcmllcyB3aXRoIFNDUklQVFMgYnV0IG5v IFBST0dTLg0KDQpUaGlzIGNoYW5nZSBmaXhlcyBzb21lIGNhc2VzIHdoZXJlIGJzZC5wcm9ncy5t ayB3b3VsZCBmYWlsIHRvIGhhbmRsZQ0KZGlyZWN0b3JpZXMgd2l0aCBTQ1JJUFRTIGJ1dCBubyBQ Uk9HUy4gIEluIHBhcnRpY3VsYXIsICJpbnN0YWxsIiBkaWQNCm5vdCBoYW5kbGUgc3VjaCBzY3Jp cHRzIG5vciBkZXBlbmRlbnQgZmlsZXMgd2hlbiBic2Quc3ViZGlyLm1rIHdhcw0KYWRkZWQgdG8g dGhlIG1peC4NCg0KVGhpcyBpcyAibWFrZSB0aW5kZXJib3giIGNsZWFuLg0KDQpJbmRleDogc2hh cmUvbWsvYnNkLnByb2dzLm1rDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQotLS0gc2hhcmUvbWsvYnNkLnByb2dzLm1r CShyZXZpc2lvbiAyNTg1NTIpDQorKysgc2hhcmUvbWsvYnNkLnByb2dzLm1rCSh3b3JraW5nIGNv cHkpDQpAQCAtNzMsNyArNzMsNyBAQA0KICMgaGFuZGxlIGJlaW5nIGNhbGxlZCBbYnNkLl1wcm9n cy5taw0KIC5pbmNsdWRlIDxic2QucHJvZy5taz4NCiANCi0uaWZuZGVmIFBST0cNCisuaWZuZGVm IF9SRUNVUlNJTkdfUFJPR1MNCiAjIHRlbGwgcHJvZ3MubWsgd2UgbWlnaHQgd2FudCB0byBpbnN0 YWxsIHRoaW5ncw0KIFBST0dTX1RBUkdFVFMrPSBjbGVhbmRlcGVuZCBjbGVhbmRpciBjbGVhbm9i aiBkZXBlbmQgaW5zdGFsbA0KIA0KQEAgLTg0LDExICs4NCwxMyBAQA0KIC5lbmRpZg0KIA0KICRw ICR7cH1fcDogLlBIT05ZIC5NQUtFDQotCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtN QUtFRklMRX0gUFJPRz0kcCAke3guJHB9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1m ICR7TUFLRUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAk e3guJHB9KQ0KIA0KIC5mb3IgdCBpbiAke1BST0dTX1RBUkdFVFM6Tzp1fQ0KICRwLiR0OiAuUEhP TlkgLk1BS0UNCi0JKGNkICR7LkNVUkRJUn0gJiYgJHtNQUtFfSAtZiAke01BS0VGSUxFfSBQUk9H PSRwICR7eC4kcH0gJHtAOkV9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1mICR7TUFL RUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAke3guJHB9 ICR7QDpFfSkNCiAuZW5kZm9yDQogLmVuZGZvcg0KIA0KQEAgLTk2LDQgKzk4LDE4IEBADQogJHQ6 ICR7UFJPR1M6JT0lLiR0fQ0KIC5lbmRmb3INCiANCitTQ1JJUFRTX1RBUkdFVFMrPSBjbGVhbmRl cGVuZCBjbGVhbmRpciBjbGVhbm9iaiBkZXBlbmQgaW5zdGFsbA0KKw0KKy5mb3IgcCBpbiAke1ND UklQVFN9DQorLmZvciB0IGluICR7U0NSSVBUU19UQVJHRVRTOk86dX0NCiskcC4kdDogLlBIT05Z IC5NQUtFDQorCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtNQUtFRklMRX0gX1JFQ1VS U0lOR19QUk9HUz0gXA0KKwkgICAgU1VCRElSPSBTQ1JJUFQ9JHAgJHt4LiRwfSAke0A6RX0pDQor LmVuZGZvcg0KKy5lbmRmb3INCisNCisuZm9yIHQgaW4gJHtTQ1JJUFRTX1RBUkdFVFM6Tzp1fQ0K KyR0OiAke1NDUklQVFM6JT0lLiR0fQ0KKy5lbmRmb3INCisNCiAuZW5kaWYNCg== --089e012281acdc2e8004ec048c0f--