From owner-freebsd-testing@FreeBSD.ORG Sat Feb 14 21:57:46 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C640225; Sat, 14 Feb 2015 21:57:46 +0000 (UTC) Received: from mail-pd0-f176.google.com (mail-pd0-f176.google.com [209.85.192.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D153BE0; Sat, 14 Feb 2015 21:57:46 +0000 (UTC) Received: by pdev10 with SMTP id v10so26682688pde.7; Sat, 14 Feb 2015 13:57:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=D5ehDf7qjXNqUmbGvunj7dClD6kNzH6KrgqnvyWS1cM=; b=ykNPYMgzWWPpjIBPCnWY+tfriqFvrccBAc61oBI0q2SPOZc4F8pyXhb8blNUFxJpnJ 0lDEPQFK/luYKgO1krpMJVaxk0T7vCWFpqg6AKR9JSISOs+noxmhlJSTQb0nkl+gnRxz 1BO+7Ya/GiPtBqeNyHsVI2OC+Lv2F9Eh2gI1EixoUn6Ypq4tHWBaMthXCSTI61J5tqjW zHJdv2aIysJgW8bm4E4XLVgj1NF5BoDG4C8m0HN4Ckg8tJ3tNgJQSwLLuKJQw82nU7nq mtFDQsHJzIWZrDXVEuy0bTmtXU4jbjkfg4bpJcQnu16JMAsAcwXADEMRzs8wa5oFq53E Tyqw== X-Received: by 10.68.134.228 with SMTP id pn4mr234516pbb.31.1423951059395; Sat, 14 Feb 2015 13:57:39 -0800 (PST) Received: from rodrigc-laptop1 (c-24-6-186-207.hsd1.ca.comcast.net. [24.6.186.207]) by mx.google.com with ESMTPSA id w17sm10305313pdj.83.2015.02.14.13.57.36 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 Feb 2015 13:57:37 -0800 (PST) Sender: Craig Rodrigues Date: Sat, 14 Feb 2015 13:57:50 -0800 From: Craig Rodrigues To: Alan Somers Subject: Re: Python unittest backend for Kyua Message-ID: <20150214215750.GA5065@rodrigc-laptop1> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fUYQa+Pmc3FrFX/N" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Feb 2015 21:57:46 -0000 --fUYQa+Pmc3FrFX/N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 26, 2013 at 09:47:29PM -0700, Alan Somers wrote: > I wrote a Kyua backend for Python programs that use the unittest > module from the standard library. It's incomplete (no man page, no > tests, no test case isolation, and no tested version of python other > than 2.7), but I'm posting it to discuss the approach. Do you think > it's a good start? >=20 > Rather than use unittest's CLI, I wrote the backend to dynamically > load the file under test and interrogate the test using unittest as a > library. This allowed more intimate access. For example, the > unittest CLI doesn't have a "list" command, but the backend can still > list tests. >=20 > Does it look good so far? >=20 > -Alan Alan, I saw this post of yours from a while back. I was just wondering, how far did you get with it? Do you use it in your company? I looked at three Python testing frameworks: (1) unittest https://docs.python.org/2/library/unittest.html (2) nose https://nose.readthedocs.org (3) pytest http://pytest.org I found that pytest was the easiest to set up and did what I needed: -> test discovery -> generate XML test result output compatible with Jenkins -> tests do not need to inherit from any base classes, and just requires a file with "test_*" methods The test discovery makes it somewhat "kyua-like". Do you have any opions/experience with this? My experience is that running the FreeBSD tests in kyua work great. However, for a full product, I've had a really hard time getting QA people to be able to get comfortable writing new ATF tests and Kyuafiles. I've had less problems withe the Python frameworks. -- Craig=20 --fUYQa+Pmc3FrFX/N Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlTfxN4ACgkQ0gqKKjmYR53a/QCgiJc0f3SrlB5FJ3JM/5DCFQDM z/kAoJwiTdg1v5DAnCA7eA/MDpjI3FQL =Ftmb -----END PGP SIGNATURE----- --fUYQa+Pmc3FrFX/N--