From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:34:31 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2D60C48E for ; Sun, 24 Nov 2013 22:34:31 +0000 (UTC) Received: from mail-yh0-x22d.google.com (mail-yh0-x22d.google.com [IPv6:2607:f8b0:4002:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E6E772996 for ; Sun, 24 Nov 2013 22:34:30 +0000 (UTC) Received: by mail-yh0-f45.google.com with SMTP id v1so1275066yhn.4 for ; Sun, 24 Nov 2013 14:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=IZAz+zqZfwkRR9wHDFlctwDVMh5gCy9Mb5DzkOWA/2w=; b=R3Y0LdaoJXOH3/ADqgFDn3U8PBmp11dxtiChVaoj7MQIBgFNq6S7Aw4RsLUjoo2c/m J2gH5b+5p9RUY1ebSvNP+Jl2dCB8iPiPPp6F3FiRK7m3Sw26VQgExuISHJKdeyMR8CE9 GQzmRW65lUjUNfaRuHanVnk7JH0LK5L04+XiWMOBiYr8sb+j12xnFRDRWS+gymx+ZhPV 0Jg3dGbv0IVwxPY1DR3CirjUDYj5vmeEAcyIOkWRmkooQHmfqN6xHRQ0m3qUILkugHNz RVu3hn8OvIJOPaQ5z2qZobDXPYNdqf4oj8LcZqE8OJlKpVxrQz8oTi/p2QlyAHHiweYf as4w== X-Received: by 10.236.120.74 with SMTP id o50mr24489498yhh.45.1385332468903; Sun, 24 Nov 2013 14:34:28 -0800 (PST) Received: from ?IPv6:2601:8:ab80:7d6:608b:3f59:7890:c8fe? ([2601:8:ab80:7d6:608b:3f59:7890:c8fe]) by mx.google.com with ESMTPSA id m29sm70010611yho.14.2013.11.24.14.34.28 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 24 Nov 2013 14:34:28 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: Adding new "test interfaces" to work with Kyua? From: Garrett Cooper In-Reply-To: Date: Sun, 24 Nov 2013 14:34:26 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:34:31 -0000 On Nov 24, 2013, at 2:29 PM, Julio Merino wrote: > On Sun, Nov 24, 2013 at 5:09 PM, Garrett Cooper = wrote: >> Trying to reconcile my diffs and get things sane again, I = realize that due to refactoring (and especially after I installed = devel/kyua) that prove.test.mk no longer functions for me off my branch. = Are there directions for how I can program in a new test interface so I = can integrate prove testing into Kyua? >=20 > What kind of interface do prove-based test programs have? Are they > binaries/scripts that you can just run and assume that a 0 exit code > means passed and 1 means failed? If yes, you'd just funnel them > through the 'plain' interface and they should just work as a first > cut. (I believe this approach works for most test programs out > there.) >=20 > But there are some details here: >=20 > https://code.google.com/p/kyua/wiki/TestersDesign >=20 > and more in the kyua-tester manual page and its references. It=92s a bit more complex than that. nose and prove both implement = collection and execution test features, meaning that it goes and figures = out what to execute (for nose it=92s a variety of parameters =97 = filename, mode, etc =97 that determine whether or not it=92s testable; = for prove it=92s all files with a .t extension), then executes it. I=92d like to take the error messages and make them more meaningful = though, because it doesn=92t help if something fails and I need to = diagnose why it failed, but lost the info I need to debug the problem... > That said... I'm not sure now if I have detailed the "protocol" in > detail anywhere... and if not, that's something for me to start doing > :P That would be nice :). Thanks! -Garrett=