Date: Wed, 5 Apr 2017 11:44:51 -0700 From: "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> To: Craig Rodrigues <rodrigc@freebsd.org> Cc: "freebsd-testing@freebsd.org" <testing@freebsd.org> Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 Message-ID: <A66D2B01-59CB-4F4A-8A6D-0C79F15B5D49@gmail.com> In-Reply-To: <CAG=rPVegLMBhv3hXsabhs_QZ1069HPmPsRS9i2605GZ3qP9izQ@mail.gmail.com> References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> <CAG=rPVegLMBhv3hXsabhs_QZ1069HPmPsRS9i2605GZ3qP9izQ@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
(Apologies for taking so long to reply; the longer the reply, the longer it takes for me to write a thoughtful response sometimes ;)..)
> On Jan 17, 2017, at 16:09, Craig Rodrigues <rodrigc@freebsd.org> wrote:
>
> On Tue, Jan 17, 2017 at 3:01 PM, Ngie Cooper (yaneurabeya) <
> yaneurabeya@gmail.com> wrote:
>
>> Hello all,
>>
>> I had an initial discussion with a handful of other test
>> stakeholders and due to the number of caveats with ATF/Kyua and a variety
>> of issues in contributing back to the ATF/Kyua projects (time on
>> maintainer’s end, legal issues, technical issues, etc), I'm seriously
>> considering replacing parts of ATF/Kyua with GoogleTest and shunit2. In
>> particular, I want to do these things [better]:
>
> I'll provide my perspective on your proposal.
> For me, the nice thing about ATF/Kyua was that Julio pushed unit tests to
> live along-side the code in FreeBSD,
> and provided a consistent structure by which these tests could run and
> provide test output.
> The fact that he was able to modify the Makefile system to get this working
> in FreeBSD *and* NetBSD
> is amazing. I never would have had the energy to do that!!
>
> I was hoping that ATF/Kyua would attract interest from projects other than
> FreeBSD and NetBSD,
> so that there would be an ecosystem of projects that would use this stuff
> and drive this forward.
> I updated the Mac Homebrew port of Kyua a few times, and also tried to
> submit a Debian package
> for ATF and Kyua but got stalled there.
> Unfortunately, ATF/Kyua never really took off outside of FreeBSD and NetBSD.
Yes. That’s a good summary of where we’ve come from.
> Also, the implementation details of modern C++ and Lua for Kyua, and shell
> script and C for ATF
> is a high barrier of entry for people wanting to work on the code and
> extend it.
> There are enough developers in FreeBSD who could probably pick up the C and
> ATF parts.
> For the C++ and Lua parts, I think the barrier is quite high, and there are
> not that many developers
> in the FreeBSD community with the skills or interest to work on that.
I’d have to look at how easy divorcing libatf-c from libatf-c++ would be, but I suspect it’s non-trivial. The ATF runners in kyua seem to depend on both libraries (although they can be obtained from ports):
$ ldd /usr/local/tests/kyua/engine/atf_* /usr/local/tests/kyua/bootstrap/atf_helpers | grep c++
libatf-c++.so.2 => /usr/local/lib/libatf-c++.so.2 (0x80117b000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x8015ae000)
libatf-c++.so.2 => /usr/local/lib/libatf-c++.so.2 (0x80119f000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x8015d2000)
libatf-c++.so.2 => /usr/local/lib/libatf-c++.so.2 (0x8011a2000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x8015d5000)
libatf-c++.so.2 => /usr/local/lib/libatf-c++.so.2 (0x8011db000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x80160e000)
libatf-c++.so.2 => /usr/local/lib/libatf-c++.so.2 (0x800823000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x800c56000)
> I have a selfish perspective in that I like to invest time in libraries and
> technologies that
> help me get jobs, even in companies that are not FreeBSD.
> In my last job at a company making a FreeBSD appliance, I tried to push
> adoption of ATF/Kyua
> but wasn't able to convince folks to pick it up. The developers gave me
> feedback that the ATF/Kyua
> documentation didn't give them confidence. It didn't help that I couldn't
> point to other projects
> outside of FreeBSD/NetBSD using this stuff.
> At least for me, I cannot say that ATF/Kyua is a marketable skillset which
> opens doors for jobs. :(
>
> Moving forward, if you try to steer towards libraries and technologies that
> have active
> user communities *even outside of FreeBSD*, that would be nice.
> Testing software isn't unique to FreeBSD, and there is a lot of good work
> going on in these
> other communities.
>
> You mentioned that GoogleTest is actively worked on by Google,
> but there at last one non-Google community (LLVM) is using it, and has a
> selfish interest to
> keep it going. That is one plus in my opinion.
Bingo. Hence, one of the motivators behind GoogleTest (it has more mindshare in the open source community than ATF/Kyua does).
> GoogleTest uses a 2-clause BSD license, so that is OK.
> GoogleTest requires signing a Contributor License Agreement for people who
> want to submit patches upstream to them. That's no different from Kyua.
> That's a bit annoying, but maybe something we can live with.
Yes.
> So I have no problem with GoogleTest on those points.
>
> My questions for you are:
> (1) GoogleTest is very C++-focused. ATF has API's for C, C++, and
> shell.
> How successful will you be at testing a pile of C code with a
> C++-focused library?
> It is technically possible (
> https://meekrosoft.wordpress.com/2009/11/09/unit-testing-c-code-with-the-googletest-framework/
> )
> but what are the gotchas?
That I need to quite frankly investigate. It might come down to the fact that tests need to be written in C++ — which I’m not really keen on, but if the C++ is simple, and the boilerplate is simple enough, then the C++ shouldn’t need to get too fancy, and hence will look at lot like the existing, simple GoogleTest examples.
> (2) What do we do with existing tests? Do they need to be rewritten to
> the GoogleTest API?
> Or do we have infrastructure that allows existing ATF tests to run
> with Google Test?
Same thing as before:
1. Write new tests in the preferred target framework.
2. Don’t rewrite existing tests, unless there’s a good reason to do so (needless refactoring introduces risk).
As far as the second question (“Or do we…”), yes, my goal was to write equivalent macros, if at all possible, for using GoogleTest in lieu of ATF. The ownership for this task falls on me. I might need to write some compatibility functions/macros to ease the transition away from ATF, if my investigation demonstrates it's feasible/possible.
> (3) If this is adopted, are new tests written in GoogleTest API, or in
> the ATF API?
The GoogleTest APIs.
> (4) Kyua is the tool currently used for running the tests, and
> providing test result output.
> Kyua has test-case isolation, and the potential to parallelize the
> execution of test-cases.
> Are the Kyua features of test-case isolation and parallelization
> of test-cases must-haves,
> or can we live without them?
> Do you want to move away from using Kyua, or modify it to work
> with GoogleTest?
As a first step, I want to mock ATF via GoogleTest. Although contributing back to Google (as a corporation — thanks $lawyers) has proved an impedance, we can always patch the port to add in the helper.
> If you want to move away from Kyua, how do you want people to run
> the GoogleTests and
> get the test results?
I really don’t want to change the inputs/outputs if at all possible — just add a bit more “window dressing” to denote setup/teardown fixtures and other gaps that ATF doesn’t have.
If it doesn’t prove fruitful, I’ll send out an update email describing the work I’ve done, the roadblocks/tradeoffs I’ve made in getting to the decision, and ask for some consensus in terms of driving the effort forward, further.
> I have no objection to your proposal. Using a more mainstream testing
> framework with a larger user community,
> and some momentum to keep things going would be nice.
Thanks!
-Ngie
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQIcBAEBCgAGBQJY5TsjAAoJEPWDqSZpMIYVdY0QAI5eemX4HJVupRXPyKfx0bJc
uZ5fY2TnBj+A3y4FycPKsr+hj47SZs/IGVoosn8U4i8KCJKebPQn7Ck9YD+cRUyp
U8ihpvVxTbTCPVm4B+GatdN1dIr8jBAAHWZVyR/Uwi/OFu3U+r86/E2wJQEsjutM
VXVhzEj9PhxqhmAxHhN1p9ZRmE3hJNhlu+zy3YWn4pR8Xk4C/k0edZTuozyZCOkc
LX0Q36m+XIgPjlskMTp8xbQ2X8hnPgXz1YAJwO6hdl0F4Loucyy4FBvlgoYu5RGp
QBEukBt77io/wekCLdYICHdxg7x2LMBKlOAz+mPbL+O0Q/CrP4BCnGu1FYMHCqdH
bajZzme8ojRGTJiBJ5lzavoJjx8lj7SzVOrot986Wmwmta2mh4kmiIEJhbILkqW+
ULJG7TBudfMmJPjJJYB5oh1oliLySf6dDokPlqUeokyWHOj0Pky0uIp3nSJKlvfp
67XEsL2Z9sFG2CVTeLvNq9vCZ0rogdN1kPM3u1nPzNLXa99gIXx0UflfrH/CMi5e
NvDPLq+P7g1AIoTL4ioEVc4u4UbrObUrKvY5QSy+vIzv6Hs6b259Ox8IDYvQiMIQ
0FBcOwq4d3c+tMB6k78AQ9k/vgJJqR56rpJESgHnhUq4c6wmTbLzX3rx/dKdh9mb
O9Dy/2Bj3YD8IOHGEOgn
=Ej3a
-----END PGP SIGNATURE-----
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A66D2B01-59CB-4F4A-8A6D-0C79F15B5D49>