Date: Wed, 5 Apr 2017 11:44:51 -0700 From: "Ngie Cooper (yaneurabeya)" <yaneurabeya@gmail.com> To: Craig Rodrigues <rodrigc@freebsd.org> Cc: "freebsd-testing@freebsd.org" <testing@freebsd.org> Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 Message-ID: <A66D2B01-59CB-4F4A-8A6D-0C79F15B5D49@gmail.com> In-Reply-To: <CAG=rPVegLMBhv3hXsabhs_QZ1069HPmPsRS9i2605GZ3qP9izQ@mail.gmail.com> References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> <CAG=rPVegLMBhv3hXsabhs_QZ1069HPmPsRS9i2605GZ3qP9izQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 (Apologies for taking so long to reply; the longer the reply, the longer = it takes for me to write a thoughtful response sometimes ;)..) > On Jan 17, 2017, at 16:09, Craig Rodrigues <rodrigc@freebsd.org> = wrote: >=20 > On Tue, Jan 17, 2017 at 3:01 PM, Ngie Cooper (yaneurabeya) < > yaneurabeya@gmail.com> wrote: >=20 >> Hello all, >>=20 >> I had an initial discussion with a handful of other test >> stakeholders and due to the number of caveats with ATF/Kyua and a = variety >> of issues in contributing back to the ATF/Kyua projects (time on >> maintainer=E2=80=99s end, legal issues, technical issues, etc), I'm = seriously >> considering replacing parts of ATF/Kyua with GoogleTest and shunit2. = In >> particular, I want to do these things [better]: >=20 > I'll provide my perspective on your proposal. > For me, the nice thing about ATF/Kyua was that Julio pushed unit tests = to > live along-side the code in FreeBSD, > and provided a consistent structure by which these tests could run and > provide test output. > The fact that he was able to modify the Makefile system to get this = working > in FreeBSD *and* NetBSD > is amazing. I never would have had the energy to do that!! >=20 > I was hoping that ATF/Kyua would attract interest from projects other = than > FreeBSD and NetBSD, > so that there would be an ecosystem of projects that would use this = stuff > and drive this forward. > I updated the Mac Homebrew port of Kyua a few times, and also tried to > submit a Debian package > for ATF and Kyua but got stalled there. > Unfortunately, ATF/Kyua never really took off outside of FreeBSD and = NetBSD. Yes. That=E2=80=99s a good summary of where we=E2=80=99ve come from. > Also, the implementation details of modern C++ and Lua for Kyua, and = shell > script and C for ATF > is a high barrier of entry for people wanting to work on the code and > extend it. > There are enough developers in FreeBSD who could probably pick up the = C and > ATF parts. > For the C++ and Lua parts, I think the barrier is quite high, and = there are > not that many developers > in the FreeBSD community with the skills or interest to work on that. I=E2=80=99d have to look at how easy divorcing libatf-c from libatf-c++ = would be, but I suspect it=E2=80=99s non-trivial. The ATF runners in = kyua seem to depend on both libraries (although they can be obtained = from ports): $ ldd /usr/local/tests/kyua/engine/atf_* = /usr/local/tests/kyua/bootstrap/atf_helpers | grep c++ libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x80117b000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015ae000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x80119f000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d2000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x8011a2000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d5000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x8011db000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x80160e000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x800823000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x800c56000) > I have a selfish perspective in that I like to invest time in = libraries and > technologies that > help me get jobs, even in companies that are not FreeBSD. > In my last job at a company making a FreeBSD appliance, I tried to = push > adoption of ATF/Kyua > but wasn't able to convince folks to pick it up. The developers gave = me > feedback that the ATF/Kyua > documentation didn't give them confidence. It didn't help that I = couldn't > point to other projects > outside of FreeBSD/NetBSD using this stuff. > At least for me, I cannot say that ATF/Kyua is a marketable skillset = which > opens doors for jobs. :( >=20 > Moving forward, if you try to steer towards libraries and technologies = that > have active > user communities *even outside of FreeBSD*, that would be nice. > Testing software isn't unique to FreeBSD, and there is a lot of good = work > going on in these > other communities. >=20 > You mentioned that GoogleTest is actively worked on by Google, > but there at last one non-Google community (LLVM) is using it, and has = a > selfish interest to > keep it going. That is one plus in my opinion. Bingo. Hence, one of the motivators behind GoogleTest (it has more = mindshare in the open source community than ATF/Kyua does). > GoogleTest uses a 2-clause BSD license, so that is OK. > GoogleTest requires signing a Contributor License Agreement for people = who > want to submit patches upstream to them. That's no different from = Kyua. > That's a bit annoying, but maybe something we can live with. Yes. > So I have no problem with GoogleTest on those points. >=20 > My questions for you are: > (1) GoogleTest is very C++-focused. ATF has API's for C, C++, and > shell. > How successful will you be at testing a pile of C code with a > C++-focused library? > It is technically possible ( > = https://meekrosoft.wordpress.com/2009/11/09/unit-testing-c-code-with-the-g= oogletest-framework/ > ) > but what are the gotchas? That I need to quite frankly investigate. It might come down to the fact = that tests need to be written in C++ =E2=80=94 which I=E2=80=99m not = really keen on, but if the C++ is simple, and the boilerplate is simple = enough, then the C++ shouldn=E2=80=99t need to get too fancy, and hence = will look at lot like the existing, simple GoogleTest examples. > (2) What do we do with existing tests? Do they need to be rewritten = to > the GoogleTest API? > Or do we have infrastructure that allows existing ATF tests to = run > with Google Test? Same thing as before: 1. Write new tests in the preferred target framework. 2. Don=E2=80=99t rewrite existing tests, unless there=E2=80=99s a good = reason to do so (needless refactoring introduces risk). As far as the second question (=E2=80=9COr do we=E2=80=A6=E2=80=9D), = yes, my goal was to write equivalent macros, if at all possible, for = using GoogleTest in lieu of ATF. The ownership for this task falls on = me. I might need to write some compatibility functions/macros to ease = the transition away from ATF, if my investigation demonstrates it's = feasible/possible. > (3) If this is adopted, are new tests written in GoogleTest API, or = in > the ATF API? The GoogleTest APIs. > (4) Kyua is the tool currently used for running the tests, and > providing test result output. > Kyua has test-case isolation, and the potential to parallelize = the > execution of test-cases. > Are the Kyua features of test-case isolation and = parallelization > of test-cases must-haves, > or can we live without them? > Do you want to move away from using Kyua, or modify it to work > with GoogleTest? As a first step, I want to mock ATF via GoogleTest. Although = contributing back to Google (as a corporation =E2=80=94 thanks $lawyers) = has proved an impedance, we can always patch the port to add in the = helper. > If you want to move away from Kyua, how do you want people to = run > the GoogleTests and > get the test results? I really don=E2=80=99t want to change the inputs/outputs if at all = possible =E2=80=94 just add a bit more =E2=80=9Cwindow dressing=E2=80=9D = to denote setup/teardown fixtures and other gaps that ATF doesn=E2=80=99t = have. If it doesn=E2=80=99t prove fruitful, I=E2=80=99ll send out an update = email describing the work I=E2=80=99ve done, the roadblocks/tradeoffs = I=E2=80=99ve made in getting to the decision, and ask for some consensus = in terms of driving the effort forward, further. > I have no objection to your proposal. Using a more mainstream testing > framework with a larger user community, > and some momentum to keep things going would be nice. Thanks! -Ngie --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJY5TsjAAoJEPWDqSZpMIYVdY0QAI5eemX4HJVupRXPyKfx0bJc uZ5fY2TnBj+A3y4FycPKsr+hj47SZs/IGVoosn8U4i8KCJKebPQn7Ck9YD+cRUyp U8ihpvVxTbTCPVm4B+GatdN1dIr8jBAAHWZVyR/Uwi/OFu3U+r86/E2wJQEsjutM VXVhzEj9PhxqhmAxHhN1p9ZRmE3hJNhlu+zy3YWn4pR8Xk4C/k0edZTuozyZCOkc LX0Q36m+XIgPjlskMTp8xbQ2X8hnPgXz1YAJwO6hdl0F4Loucyy4FBvlgoYu5RGp QBEukBt77io/wekCLdYICHdxg7x2LMBKlOAz+mPbL+O0Q/CrP4BCnGu1FYMHCqdH bajZzme8ojRGTJiBJ5lzavoJjx8lj7SzVOrot986Wmwmta2mh4kmiIEJhbILkqW+ ULJG7TBudfMmJPjJJYB5oh1oliLySf6dDokPlqUeokyWHOj0Pky0uIp3nSJKlvfp 67XEsL2Z9sFG2CVTeLvNq9vCZ0rogdN1kPM3u1nPzNLXa99gIXx0UflfrH/CMi5e NvDPLq+P7g1AIoTL4ioEVc4u4UbrObUrKvY5QSy+vIzv6Hs6b259Ox8IDYvQiMIQ 0FBcOwq4d3c+tMB6k78AQ9k/vgJJqR56rpJESgHnhUq4c6wmTbLzX3rx/dKdh9mb O9Dy/2Bj3YD8IOHGEOgn =Ej3a -----END PGP SIGNATURE----- --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A66D2B01-59CB-4F4A-8A6D-0C79F15B5D49>