From owner-freebsd-testing@FreeBSD.ORG Tue Mar 3 00:07:34 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 79ACE4FB; Tue, 3 Mar 2015 00:07:34 +0000 (UTC) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0AD51D76; Tue, 3 Mar 2015 00:07:34 +0000 (UTC) Received: by wggx12 with SMTP id x12so36750207wgg.6; Mon, 02 Mar 2015 16:07:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=IondlNoKiOXx8gyoAiaCjic3H1bl7XclcSt8gV2EWlQ=; b=eF73sKiIoOPGOSkAsp+Jnb7ETePZd9VAclUnECqAUQvb2OaQIQ2UKIfqIfC3qkrHty 06UVfBa1drBWxDOSMK6puKkOTidw8uiCs6waF8ExrALVleE1bKZZkdiIpOLmqK0y3TPr ABJ9KfQoakjF2vquVHP0HgvVE8gwwtvo4K6SIDQNDuGSIIZJmIjpmqtxZV/wvg4N6uYd EPRoKcpv4MLQq+a1/BwhuILn4WTx64LeuxD3bLsJbpLNmFsoZXgEXxPQhltv5NN4b3/3 5Kurkb3C6/aVqFk7m59H4hrozDLeZhcZqPvsMXtUeaTeurX3okkJrqiLoYqtUzqANuat heWA== MIME-Version: 1.0 X-Received: by 10.180.9.71 with SMTP id x7mr41226863wia.0.1425341252446; Mon, 02 Mar 2015 16:07:32 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.17.129 with HTTP; Mon, 2 Mar 2015 16:07:32 -0800 (PST) In-Reply-To: References: <20150214215750.GA5065@rodrigc-laptop1> Date: Mon, 2 Mar 2015 17:07:32 -0700 X-Google-Sender-Auth: dAVkArvgapQ9o6wVeSNxlbV-PPk Message-ID: Subject: Re: Python unittest backend for Kyua From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Mar 2015 00:07:34 -0000 On Mon, Mar 2, 2015 at 4:42 PM, Julio Merino wrote: > >> On Feb 15, 2015, at 00:16, Alan Somers wrote: >> >> Not much further than when I made that post, and no. In fact, I >> hardly use Python at all. Professionally, my company is all about >> Ruby, and I would be very interested in a Ruby test/unit tester for >> Kyua. However, Ruby's test/unit's implementation is much messier than >> Python's unittest, so I thought that writing a unittest tester would >> be good practice for writing a test/unit tester. Indeed it was. I >> found unittest very easy to interface to; it hardly took me any time >> to write that tester. But in the end I stopped working on it because >> jmmv and I disagreed about test isolation. Jmmv's primary interest is >> in writing system-level tests. He basically sees a unittest tester as >> an alternative to atf-sh. But my primary interest is in tying >> together separate components' test suites and get a consistent view of >> all results. So naturally jmmv wants the same level of isolation as >> atf-sh provides, but I want the same level of isolation as unittest >> provides. In fact, atf-sh style isolation is bad for my use case, >> because it can cause tests that were originally written for unittest >> to fail. It also drastically increases runtime because the Python >> interpreter must be restarted between each test. > > As a matter of fact, I have changed my thoughts on this. I would like Ky= ua to better support "unittest"-style test programs because that's what pre= tty much all testing libraries implement and what other people know... and = well, this model is just much faster and equally useful for the vast majori= ty of the cases. Well, maybe I should dust off my old branch and get back to work. But I don't know how my python tester would work with your executor branch. If I understand correctly, the executor branch dispenses with the *-tester binaries and calls the test programs directly from the kyua executable, correct? The python tester relied on the tester being a separate binary. The tester was actually implemented in Python, which made it very easy for the tester to interact with individual test cases. How would I accomplish that on the executor branch? -Alan