From owner-freebsd-testing@FreeBSD.ORG Mon Nov 25 17:21:02 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9DA294E4 for ; Mon, 25 Nov 2013 17:21:02 +0000 (UTC) Received: from mail-la0-f42.google.com (mail-la0-f42.google.com [209.85.215.42]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2B07829D1 for ; Mon, 25 Nov 2013 17:21:01 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id ec20so3271490lab.15 for ; Mon, 25 Nov 2013 09:21:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=b6ID2bn73fL+m+yG7JhBbHY28F0TwxKhJyXN50Cxzrg=; b=hBup9m06wzUIhXjdH+a6j0ZvnoHfRKI5JHmUzgyKbiFbL+83YXnRmDHOLv7Behpeod yMrxldtK07SabmsznG5e0XLnM5NWSSIfOh7q1O95a7hMf9zyhWAeF0oEo5xPI0bdIBy0 +qZtisisSPuKYzcDRX5k2bPgDwbw2VAN8G7BS4HIOO2MEI/qqZQbOCyxj9mwiqF2+va4 IZqhSZlpCEuEhPFa+gO5HoF4AanjeDyfte4c/DYmhs+nCUwJGX+2iIF9gJcc8zSndvG+ VsXtk04CwavednQJM6Q1qHxddKykDHHi4wbuiuSYUDmFtVgmvGruKw71DWWfR4awIs5+ o59A== X-Gm-Message-State: ALoCoQkHkKLHl8cXt2D/BlqKIGySjJZM0AhekwNAlZpI9NraqwWC9I0UY6Y25epS9u42UBMBELY1 X-Received: by 10.152.116.109 with SMTP id jv13mr2328103lab.30.1385399645848; Mon, 25 Nov 2013 09:14:05 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Mon, 25 Nov 2013 09:13:45 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> From: Julio Merino Date: Mon, 25 Nov 2013 12:13:45 -0500 Message-ID: Subject: Re: Adding new "test interfaces" to work with Kyua? To: Garrett Cooper Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 17:21:02 -0000 On Sun, Nov 24, 2013 at 5:34 PM, Garrett Cooper wro= te: > On Nov 24, 2013, at 2:29 PM, Julio Merino wrote: > >> On Sun, Nov 24, 2013 at 5:09 PM, Garrett Cooper = wrote: >>> Trying to reconcile my diffs and get things sane again, I realiz= e that due to refactoring (and especially after I installed devel/kyua) tha= t prove.test.mk no longer functions for me off my branch. Are there directi= ons for how I can program in a new test interface so I can integrate prove = testing into Kyua? >> >> What kind of interface do prove-based test programs have? Are they >> binaries/scripts that you can just run and assume that a 0 exit code >> means passed and 1 means failed? If yes, you'd just funnel them >> through the 'plain' interface and they should just work as a first >> cut. (I believe this approach works for most test programs out >> there.) >> >> But there are some details here: >> >> https://code.google.com/p/kyua/wiki/TestersDesign >> >> and more in the kyua-tester manual page and its references. > > It=92s a bit more complex than that. nose and prove both implement collec= tion and execution test features, meaning that it goes and figures out what= to execute (for nose it=92s a variety of parameters =97 filename, mode, et= c =97 that determine whether or not it=92s testable; for prove it=92s all f= iles with a .t extension), then executes it. Bleh. Turns out that prove-based test programs do not return 1 on failure. Time to write a prove tester given that I have some spare time this week :-P --=20 Julio Merino / @jmmv