From owner-freebsd-testing@FreeBSD.ORG Wed Nov 27 14:02:44 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AE72E3C for ; Wed, 27 Nov 2013 14:02:44 +0000 (UTC) Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 08E702216 for ; Wed, 27 Nov 2013 14:02:43 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id c11so5456810lbj.33 for ; Wed, 27 Nov 2013 06:02:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=2ulRR/gUrhqNgYOHTexyJbXdVNWTgRYneRpHSPAJuKM=; b=QZSA/KseUd5oC5VJDRPB4HiFxT7X/ix97vua+p2/hfEgqDVcUws+TEBLLK1fRwjT0z ig/xnOREUCn0sXYreayrIG5vq7r4QdzWkghywF838NoLfmGEpI+tnlulljqzxslip4L8 RSU8wXJJ7cPv8c8a3gMIDQi4yr/kQTXKgbkfieHKsnRtveWOhuyev4DPFeKUHGe2IzJ5 YrQg7kJry/JB6j7AscRkaHN7Fq4U/vGwslEbVgx7tXdklYM8p31+grnV8Lt0INa2dTh1 71HvwWxnzOP3+JHyhl5Zl7OCUP4SkodR8saWMdSwKNxMXfRkydr6CrO2BH8zrbzV9126 06SQ== X-Gm-Message-State: ALoCoQnVUByOpn+ZjM4RC5VSQEebbXShitM3qPx20OMGuLx7OB21Y1YAr88tr/LMEKMD7NyT6fR9 X-Received: by 10.112.210.66 with SMTP id ms2mr3507lbc.51.1385560606031; Wed, 27 Nov 2013 05:56:46 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Wed, 27 Nov 2013 05:56:25 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: References: From: Julio Merino Date: Wed, 27 Nov 2013 08:56:25 -0500 Message-ID: Subject: Re: Python unittest backend for Kyua To: kyua-discuss@googlegroups.com Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 14:02:44 -0000 On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers wrote: > I wrote a Kyua backend for Python programs that use the unittest > module from the standard library. It's incomplete (no man page, no > tests, no test case isolation, and no tested version of python other > than 2.7), but I'm posting it to discuss the approach. Do you think > it's a good start? > > Rather than use unittest's CLI, I wrote the backend to dynamically > load the file under test and interrogate the test using unittest as a > library. This allowed more intimate access. For example, the > unittest CLI doesn't have a "list" command, but the backend can still > list tests. > > Does it look good so far? I like the idea a lot (and I had not considered it earlier). The code is a good start, but see below. One concern I have is regarding how you will implement isolation features. The more I think about them, the more I think that they are not specific to the ATF interface at all: i.e. all tests should be executed with the same level of isolation. Now, in your case, it seems you'd have to reimplement these features in Python -- which means that there will be two different implementations for the same thing and this can easily lead to inconsistencies. (atf had 3 versions of this in the very beginning, one for each supported language, and it wasn't pretty!) But having a pure Python binary has its benefits, if only for simplicity. The alternatives would be to either 1) hook the isolation C code into a Python module and call it from your tester, or 2) write the tester in C using all existing code (including CLI parsing for a consistent interface, isolation features, etc.) and just call into the Python interpreter to list tests or run one of them. What do you think? -- Julio Merino / @jmmv