From owner-freebsd-testing@FreeBSD.ORG Tue Nov 26 05:52:43 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 40004FCA; Tue, 26 Nov 2013 05:52:43 +0000 (UTC) Received: from mail-pd0-x230.google.com (mail-pd0-x230.google.com [IPv6:2607:f8b0:400e:c02::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1421223D0; Tue, 26 Nov 2013 05:52:43 +0000 (UTC) Received: by mail-pd0-f176.google.com with SMTP id w10so6941085pde.7 for ; Mon, 25 Nov 2013 21:52:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=wj9gE4CvitlpBXiHLA85YzFnwHr/hPpTrwZ+NmcLzDM=; b=mCNZHiSsHb+hqTz0eoVeDbHC/2SbUPXJn6S7OEMBOT0vy4nO93OoxUr62GVm6MZyEl bMgy/30nq5gX+Rk/80Uus9w9dkGAJTiD312+gcnyh1+T3RYAqWuzUMQzRnZeIyw4Uy+o obv8zNF5GNoCFFl2ltBV3dwDacR3wjRfCGV0lAvuoJqkmG0/lUre4ODSx0/pzpR0z9y1 nUWvl6YrVVAE0EoMi+OhdcDRA7xyOmCyEeJxE3SkSahS1o/hlnXfhJLA/ipZ/5jo/ELM HCK1pVzX5K54EWILj/mIotL13F1bPdxe2tKkKcQRqmffGLWDqcY4/IxatSgnNlvt3Bf/ Lscw== X-Received: by 10.67.30.70 with SMTP id kc6mr31106006pad.32.1385445162698; Mon, 25 Nov 2013 21:52:42 -0800 (PST) Received: from [192.168.20.5] (c-50-181-163-89.hsd1.wa.comcast.net. [50.181.163.89]) by mx.google.com with ESMTPSA id sy10sm88109904pac.15.2013.11.25.21.52.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 25 Nov 2013 21:52:41 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: Migrating tools/regression/ tests (and dealing with TAP) From: Garrett Cooper In-Reply-To: Date: Mon, 25 Nov 2013 21:52:39 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE@gmail.com> References: To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@freebsd.org, rpaulo@freebsd.org, Simon Gerraty X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 05:52:43 -0000 On Nov 25, 2013, at 10:29 AM, Julio Merino wrote: > Hello all, >=20 > Here come sample patches to move existing tests in = tools/regression/bin > to the new layout ('tests' subdirectories within the corresponding = tools) > and to hook these tests into the test suite. >=20 > The patches are quite straightforward: I just did an 'svn mv' of the > existing shell scripts files, wrote a Makefile using plain.test.mk = (here > is where things get interesting) and applied minor changes to get all > things sorted out (e.g. mtree directory registration). >=20 > Consider this a non-working proof of concept. Again: this does NOT = work > (well, it does, but yields invalid results). The problem is that > TAP-based test programs (like the ones to be run by prove) do NOT = report > 1 on failure and thus the plain Kyua interface is unsuitable because = it > always reports everything as passed. >=20 > There are two solutions: >=20 > 1) Patch every existing TAP-based test to propagate a 1 error code if > any of its tests fails. Doable, yes, but it will take a while. > (Mind you: this is the approach we tried in NetBSD and not all old > tests are gone yet, years later... it's a significant amount of work > to manually tweak all test programs!) >=20 > I'd rather keep existing code as is *for now*; it simplifies the > review of the changes and it doesn't immediately break people's > knowledge of existing test code. (If there is a need for a rewrite, > I'd suggest to go and use the ATF interface instead. But that's = even > more work.) >=20 > 2) Add a TAP-compliant backend to Kyua. This would allow us to plug > pretty much all existing tests in tools/regression/ without changes > to the code. I believe this is a worthy goal for the transitional > step and an easy first cut. >=20 > But this means I've gotta implement this backend (fun!). I've been > reading the protocol and seems reasonably simple, so I expect to = have > something workable this week. >=20 > I'm choosing option 2 for now while leaving open the option of = rewriting > tests using ATF *only after* they have been hooked into the build. > (What I'm shooting for, if you haven't guessed yet, is having a single > tests tree instead of 3 in src ASAP... Keeping multiple trees is not = a > pretty situation for long. >=20 > Once the new Kyua backend is ready, the proposed patches in this mail > should change minimally: basically do s/plain/prove/ in the Makefiles. > This is why I'm sending this mail out already: you can get a pretty > good idea of what I'd like to do and your feedback would be useful! >=20 > There is a "tricky" side change to be made to bsd.progs.mk so that it = is > possible to have a directory with SCRIPTS but no PROGS. make = tinderbox > is happy. >=20 > Comments? >=20 > Thanks! > The big concern is that you=92re shuffling around tests that are = being actively developed by other developers. Have you provided a heads = up to Jilles first at least..? Thanks, -Garrett