From owner-freebsd-testing@FreeBSD.ORG Wed Nov 27 23:47:27 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0A465496 for ; Wed, 27 Nov 2013 23:47:27 +0000 (UTC) Received: from mail-wg0-x234.google.com (mail-wg0-x234.google.com [IPv6:2a00:1450:400c:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 94B7A68D for ; Wed, 27 Nov 2013 23:47:26 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id x13so7546879wgg.7 for ; Wed, 27 Nov 2013 15:47:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=EMY8i9Fre+AMadk59LGFWIIS8Vtmw9BrxmhfgcZpcAM=; b=LrsT3VEnzM5gQUwikjFmlK2W9moAzq1/IAtPGEdsrEJNxjIqUWHzTdrsJffRFcNc2n JHviVro4Zt3PPfIiTRp1CWut7SJmcC3J6AW12BpIGK/jGAT0meh6Wq3Y1BnjhSloWYUt 9kYdiVS5mjClPLF8nn9hJ0bTnra56mxRDyMzHacU1t80fvOXFwLZuptfIjaZJJZvB+5/ 6cjn8X939svcrDxe4A5mH+hu+yDUhqSfnyGbkn5Rt/0wKYVLBbpC+m8qEETh/+I8ehmH Rb2xUG4wx+wiqPf1pVZb3rR7E9r2mCo16jKxezRAOXmI8xVxTsK5l/7uL+c8kmk7wRIA Os8w== MIME-Version: 1.0 X-Received: by 10.180.98.229 with SMTP id el5mr285106wib.29.1385596044863; Wed, 27 Nov 2013 15:47:24 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.171.35 with HTTP; Wed, 27 Nov 2013 15:47:24 -0800 (PST) In-Reply-To: References: Date: Wed, 27 Nov 2013 16:47:24 -0700 X-Google-Sender-Auth: drNDbV3P_tiVaMUTTym03lI5JLU Message-ID: Subject: Re: Python unittest backend for Kyua From: Alan Somers To: kyua-discuss@googlegroups.com Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 23:47:27 -0000 On Wed, Nov 27, 2013 at 6:56 AM, Julio Merino wrote: > On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers wrote: >> I wrote a Kyua backend for Python programs that use the unittest >> module from the standard library. It's incomplete (no man page, no >> tests, no test case isolation, and no tested version of python other >> than 2.7), but I'm posting it to discuss the approach. Do you think >> it's a good start? >> >> Rather than use unittest's CLI, I wrote the backend to dynamically >> load the file under test and interrogate the test using unittest as a >> library. This allowed more intimate access. For example, the >> unittest CLI doesn't have a "list" command, but the backend can still >> list tests. >> >> Does it look good so far? > > I like the idea a lot (and I had not considered it earlier). The code > is a good start, but see below. > > One concern I have is regarding how you will implement isolation > features. The more I think about them, the more I think that they are > not specific to the ATF interface at all: i.e. all tests should be > executed with the same level of isolation. > > Now, in your case, it seems you'd have to reimplement these features > in Python -- which means that there will be two different > implementations for the same thing and this can easily lead to > inconsistencies. (atf had 3 versions of this in the very beginning, > one for each supported language, and it wasn't pretty!) But having a > pure Python binary has its benefits, if only for simplicity. The other benefit of pure Python is that Python doesn't become a build-time dependency of kyua-testers. As long as the tester is pure python, then Python is only a runtime dependency, and it can be ignored if you don't have any Python test programs. > > The alternatives would be to either 1) hook the isolation C code into > a Python module and call it from your tester, or 2) write the tester > in C using all existing code (including CLI parsing for a consistent > interface, isolation features, etc.) and just call into the Python > interpreter to list tests or run one of them. I've never programmed around the Python/C interface. I'll take a look and see how hard it would be. Is there any list of which isolation features a tester needs? The ones I know of are: * Separate process for each test case * Catch segfaults and other test deaths * setuid * setgid * timeout * chdir * sanitize ENV variables * core size * reset signal handlers * umask -Alan > > What do you think? > > -- > Julio Merino / @jmmv > > -- > You received this message because you are subscribed to the Google Groups "kyua-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kyua-discuss+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.