Date: Mon, 2 Mar 2015 18:42:45 -0500 From: Julio Merino <jmmv@meroh.net> To: Alan Somers <asomers@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Python unittest backend for Kyua Message-ID: <C288DCCA-6031-4C03-8163-8826FBCE12A7@meroh.net> In-Reply-To: <CAOtMX2jFi89wt1fC5OAAGZhEDHKpLZza=tAXU7WM5y-pwL4Yqw@mail.gmail.com> References: <CAOtMX2g35ybAzFHriVuQqYMGq5Snm0%2BEcUWNhQgEr%2Bqx1xYpcA@mail.gmail.com> <20150214215750.GA5065@rodrigc-laptop1> <CAOtMX2jFi89wt1fC5OAAGZhEDHKpLZza=tAXU7WM5y-pwL4Yqw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Feb 15, 2015, at 00:16, Alan Somers <asomers@freebsd.org> wrote: >=20 > Not much further than when I made that post, and no. In fact, I > hardly use Python at all. Professionally, my company is all about > Ruby, and I would be very interested in a Ruby test/unit tester for > Kyua. However, Ruby's test/unit's implementation is much messier than > Python's unittest, so I thought that writing a unittest tester would > be good practice for writing a test/unit tester. Indeed it was. I > found unittest very easy to interface to; it hardly took me any time > to write that tester. But in the end I stopped working on it because > jmmv and I disagreed about test isolation. Jmmv's primary interest is > in writing system-level tests. He basically sees a unittest tester as > an alternative to atf-sh. But my primary interest is in tying > together separate components' test suites and get a consistent view of > all results. So naturally jmmv wants the same level of isolation as > atf-sh provides, but I want the same level of isolation as unittest > provides. In fact, atf-sh style isolation is bad for my use case, > because it can cause tests that were originally written for unittest > to fail. It also drastically increases runtime because the Python > interpreter must be restarted between each test. As a matter of fact, I have changed my thoughts on this. I would like = Kyua to better support "unittest"-style test programs because that's = what pretty much all testing libraries implement and what other people = know... and well, this model is just much faster and equally useful for = the vast majority of the cases. In fact, ATF's interface is overkill... and super-complicated to = implement in the run-time engine. Note that the "unittest" attempt I did in shtk 1.6 does not follow any = of the ATF conventions and I think it's more manageable than atf: = https://groups.google.com/d/msg/kyua-discuss/yhcDuchVtrs/D6C_LU6FXOUJ > I've never used nose or pytest. Personally, I'm a little leery of > test discovery, because it's contrary to how Kyua normally works. > Normally, you have to explicitly specify each test file in the > Kyuafile (or the makefile that builds the Kyuafile). Having some > tests be automatically discovered and others specified in Kyuafiles > might be confusing. I'm leaning against automatic discovery. But I > could be convinced otherwise. In the case of Ruby, all of our tests > are executed through Rake. So my preferred method would be to add a > Rake task that would generate the appropriate Kyuafile. I think auto-discovery within the test program is great although, = unfortunately, this is something that ATF does not provide. (It'd be = baked in but I have not thought about how hard that would be.) I do not like the idea of automatic test program discovery, but = Kyuafiles offer you enough functionality on their own (I think) to do = this yourself. See the Kyuafile.top file shipped in Kyua. Cheers=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C288DCCA-6031-4C03-8163-8826FBCE12A7>