Date: Sat, 14 Feb 2015 22:16:07 -0700 From: Alan Somers <asomers@freebsd.org> To: Craig Rodrigues <rodrigc@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Python unittest backend for Kyua Message-ID: <CAOtMX2jFi89wt1fC5OAAGZhEDHKpLZza=tAXU7WM5y-pwL4Yqw@mail.gmail.com> In-Reply-To: <20150214215750.GA5065@rodrigc-laptop1> References: <CAOtMX2g35ybAzFHriVuQqYMGq5Snm0%2BEcUWNhQgEr%2Bqx1xYpcA@mail.gmail.com> <20150214215750.GA5065@rodrigc-laptop1>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Feb 14, 2015 at 2:57 PM, Craig Rodrigues <rodrigc@freebsd.org> wrote: > On Tue, Nov 26, 2013 at 09:47:29PM -0700, Alan Somers wrote: >> I wrote a Kyua backend for Python programs that use the unittest >> module from the standard library. It's incomplete (no man page, no >> tests, no test case isolation, and no tested version of python other >> than 2.7), but I'm posting it to discuss the approach. Do you think >> it's a good start? >> >> Rather than use unittest's CLI, I wrote the backend to dynamically >> load the file under test and interrogate the test using unittest as a >> library. This allowed more intimate access. For example, the >> unittest CLI doesn't have a "list" command, but the backend can still >> list tests. >> >> Does it look good so far? >> >> -Alan > > Alan, > > I saw this post of yours from a while back. > I was just wondering, how far did you get with it? Do you use > it in your company? Not much further than when I made that post, and no. In fact, I hardly use Python at all. Professionally, my company is all about Ruby, and I would be very interested in a Ruby test/unit tester for Kyua. However, Ruby's test/unit's implementation is much messier than Python's unittest, so I thought that writing a unittest tester would be good practice for writing a test/unit tester. Indeed it was. I found unittest very easy to interface to; it hardly took me any time to write that tester. But in the end I stopped working on it because jmmv and I disagreed about test isolation. Jmmv's primary interest is in writing system-level tests. He basically sees a unittest tester as an alternative to atf-sh. But my primary interest is in tying together separate components' test suites and get a consistent view of all results. So naturally jmmv wants the same level of isolation as atf-sh provides, but I want the same level of isolation as unittest provides. In fact, atf-sh style isolation is bad for my use case, because it can cause tests that were originally written for unittest to fail. It also drastically increases runtime because the Python interpreter must be restarted between each test. > > I looked at three Python testing frameworks: > (1) unittest https://docs.python.org/2/library/unittest.html > (2) nose https://nose.readthedocs.org > (3) pytest http://pytest.org > > I found that pytest was the easiest to set up and did what I needed: > -> test discovery > -> generate XML test result output compatible with Jenkins The way I wrote my bindings, this is a non-issue. With my tester, kyua itself is still responsible for producing the reports. > -> tests do not need to inherit from any base classes, > and just requires a file with "test_*" methods > > The test discovery makes it somewhat "kyua-like". > > > Do you have any opions/experience with this? I've never used nose or pytest. Personally, I'm a little leery of test discovery, because it's contrary to how Kyua normally works. Normally, you have to explicitly specify each test file in the Kyuafile (or the makefile that builds the Kyuafile). Having some tests be automatically discovered and others specified in Kyuafiles might be confusing. I'm leaning against automatic discovery. But I could be convinced otherwise. In the case of Ruby, all of our tests are executed through Rake. So my preferred method would be to add a Rake task that would generate the appropriate Kyuafile. > > My experience is that running the FreeBSD tests in kyua work great. > However, for a full product, I've had a really hard time getting > QA people to be able to get comfortable writing new ATF tests and Kyuafiles. > I've had less problems withe the Python frameworks. My main advice is to think about whether you really want to write tests using an existing framework, or whether by the time you finish Kyua-ifying it won't really be the same framework anymore. I think that there's space for both. What my experience taught me is that one interface can't satisfy both needs. -Alan > > -- > Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2jFi89wt1fC5OAAGZhEDHKpLZza=tAXU7WM5y-pwL4Yqw>