Date: Wed, 27 Nov 2013 16:47:24 -0700 From: Alan Somers <asomers@freebsd.org> To: kyua-discuss@googlegroups.com Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Python unittest backend for Kyua Message-ID: <CAOtMX2hNJ2SxosijpQyBV6u0equ1dxsxs1_fxKgV8f_W_OFDaQ@mail.gmail.com> In-Reply-To: <CADyfeQU64LnutLBOZv1n-Wet_byMFWxSAyh-S-LaAFebyuq1_g@mail.gmail.com> References: <CAOtMX2g35ybAzFHriVuQqYMGq5Snm0%2BEcUWNhQgEr%2Bqx1xYpcA@mail.gmail.com> <CADyfeQU64LnutLBOZv1n-Wet_byMFWxSAyh-S-LaAFebyuq1_g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Nov 27, 2013 at 6:56 AM, Julio Merino <julio@meroh.net> wrote: > On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers <asomers@freebsd.org> wrote: >> I wrote a Kyua backend for Python programs that use the unittest >> module from the standard library. It's incomplete (no man page, no >> tests, no test case isolation, and no tested version of python other >> than 2.7), but I'm posting it to discuss the approach. Do you think >> it's a good start? >> >> Rather than use unittest's CLI, I wrote the backend to dynamically >> load the file under test and interrogate the test using unittest as a >> library. This allowed more intimate access. For example, the >> unittest CLI doesn't have a "list" command, but the backend can still >> list tests. >> >> Does it look good so far? > > I like the idea a lot (and I had not considered it earlier). The code > is a good start, but see below. > > One concern I have is regarding how you will implement isolation > features. The more I think about them, the more I think that they are > not specific to the ATF interface at all: i.e. all tests should be > executed with the same level of isolation. > > Now, in your case, it seems you'd have to reimplement these features > in Python -- which means that there will be two different > implementations for the same thing and this can easily lead to > inconsistencies. (atf had 3 versions of this in the very beginning, > one for each supported language, and it wasn't pretty!) But having a > pure Python binary has its benefits, if only for simplicity. The other benefit of pure Python is that Python doesn't become a build-time dependency of kyua-testers. As long as the tester is pure python, then Python is only a runtime dependency, and it can be ignored if you don't have any Python test programs. > > The alternatives would be to either 1) hook the isolation C code into > a Python module and call it from your tester, or 2) write the tester > in C using all existing code (including CLI parsing for a consistent > interface, isolation features, etc.) and just call into the Python > interpreter to list tests or run one of them. I've never programmed around the Python/C interface. I'll take a look and see how hard it would be. Is there any list of which isolation features a tester needs? The ones I know of are: * Separate process for each test case * Catch segfaults and other test deaths * setuid * setgid * timeout * chdir * sanitize ENV variables * core size * reset signal handlers * umask -Alan > > What do you think? > > -- > Julio Merino / @jmmv > > -- > You received this message because you are subscribed to the Google Groups "kyua-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kyua-discuss+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2hNJ2SxosijpQyBV6u0equ1dxsxs1_fxKgV8f_W_OFDaQ>