From owner-freebsd-testing@FreeBSD.ORG Mon Mar 2 23:42:54 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 87D66CEE for ; Mon, 2 Mar 2015 23:42:54 +0000 (UTC) Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 443F2B56 for ; Mon, 2 Mar 2015 23:42:53 +0000 (UTC) Received: by ykp131 with SMTP id 131so14984202ykp.11 for ; Mon, 02 Mar 2015 15:42:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=jNGInw8Sj+RxxPCbX57iE20faQuiF5B6yAlUH1xavCg=; b=GTjuF1deFZzR5prcy5O9p3K2+JA7wB0N4/8w04vljOgVnahllJEtCAC8I+GaXZzNeZ 4vpsk/zKtmNrT748iQg8HgmGwdtcMhEgU6VI6EmxMPUrJrsAMuI2c7rjH/PwY79IJKie iyLtdNT/oO98l5cPQIfof76WIu6dfVnNDj2p4pFlkdXXqrHZPkqWNfr232NgV8OoFrV8 kg0p9KN9n1LwZf2YzErw0q7LfY1xPPvUReMH0TGVgsVI8QlXRgEDCHdQdA4bQB/vrDCw ylG8l5yTMIo+RUWTnN2Qcb8bUeiq4K3KDpc1/MbSeSp4lSpr9JDLEUCUZjR1AB5c4wOB nA3w== X-Gm-Message-State: ALoCoQk6qSRJcDMR9au2X18GNEr3abBrotchxZV4/FwEWlmop1cbQ0DTh0Ef0WNSxBAPE14dLmh+ X-Received: by 10.236.15.170 with SMTP id f30mr27544722yhf.198.1425339766879; Mon, 02 Mar 2015 15:42:46 -0800 (PST) Received: from dhcp-172-26-107-241.nyc.corp.google.com ([172.26.107.241]) by mx.google.com with ESMTPSA id l32sm11232953yhq.44.2015.03.02.15.42.46 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 02 Mar 2015 15:42:46 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Python unittest backend for Kyua From: Julio Merino In-Reply-To: Date: Mon, 2 Mar 2015 18:42:45 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20150214215750.GA5065@rodrigc-laptop1> To: Alan Somers X-Mailer: Apple Mail (2.2070.6) Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Mar 2015 23:42:54 -0000 > On Feb 15, 2015, at 00:16, Alan Somers wrote: >=20 > Not much further than when I made that post, and no. In fact, I > hardly use Python at all. Professionally, my company is all about > Ruby, and I would be very interested in a Ruby test/unit tester for > Kyua. However, Ruby's test/unit's implementation is much messier than > Python's unittest, so I thought that writing a unittest tester would > be good practice for writing a test/unit tester. Indeed it was. I > found unittest very easy to interface to; it hardly took me any time > to write that tester. But in the end I stopped working on it because > jmmv and I disagreed about test isolation. Jmmv's primary interest is > in writing system-level tests. He basically sees a unittest tester as > an alternative to atf-sh. But my primary interest is in tying > together separate components' test suites and get a consistent view of > all results. So naturally jmmv wants the same level of isolation as > atf-sh provides, but I want the same level of isolation as unittest > provides. In fact, atf-sh style isolation is bad for my use case, > because it can cause tests that were originally written for unittest > to fail. It also drastically increases runtime because the Python > interpreter must be restarted between each test. As a matter of fact, I have changed my thoughts on this. I would like = Kyua to better support "unittest"-style test programs because that's = what pretty much all testing libraries implement and what other people = know... and well, this model is just much faster and equally useful for = the vast majority of the cases. In fact, ATF's interface is overkill... and super-complicated to = implement in the run-time engine. Note that the "unittest" attempt I did in shtk 1.6 does not follow any = of the ATF conventions and I think it's more manageable than atf: = https://groups.google.com/d/msg/kyua-discuss/yhcDuchVtrs/D6C_LU6FXOUJ > I've never used nose or pytest. Personally, I'm a little leery of > test discovery, because it's contrary to how Kyua normally works. > Normally, you have to explicitly specify each test file in the > Kyuafile (or the makefile that builds the Kyuafile). Having some > tests be automatically discovered and others specified in Kyuafiles > might be confusing. I'm leaning against automatic discovery. But I > could be convinced otherwise. In the case of Ruby, all of our tests > are executed through Rake. So my preferred method would be to add a > Rake task that would generate the appropriate Kyuafile. I think auto-discovery within the test program is great although, = unfortunately, this is something that ATF does not provide. (It'd be = baked in but I have not thought about how hard that would be.) I do not like the idea of automatic test program discovery, but = Kyuafiles offer you enough functionality on their own (I think) to do = this yourself. See the Kyuafile.top file shipped in Kyua. Cheers=