From owner-freebsd-testing@freebsd.org Wed Apr 5 18:44:57 2017 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48FCBD308E5 for ; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 224457DA for ; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1EBCCD308E2; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) Delivered-To: testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E63AD308E1 for ; Wed, 5 Apr 2017 18:44:57 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pg0-x22b.google.com (mail-pg0-x22b.google.com [IPv6:2607:f8b0:400e:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1ACF7CA; Wed, 5 Apr 2017 18:44:53 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mail-pg0-x22b.google.com with SMTP id 21so12550274pgg.1; Wed, 05 Apr 2017 11:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:mime-version:from:in-reply-to:date:cc:message-id:references :to; bh=1U5g6l6GXsxRgT2dvzWyEWJnUpHvPgL/G3gBeDUXVfk=; b=rsVhjr6uV7wtSbwTL9X72IMGMguVRlBOak9Yew8EliQ/N3M1zrW6VwYDgxeNJBKvA1 2F7kaaGb1IgKpB6o8SpLIA3h/1BJAUVuzH+NkOM921NUTES81foH/mljaU1Yyf1bAP9Q H8GQsJpCVKHpDBmdTjbawRibMeaqa4Ojp/bsdZi2CBGLF/RGhIBDiyWEM27SxkLiS2WJ cP234kD7aIOrOEGypylnwh3IDCWWA/bO9Rln/NlkADfDR8X1Zw7COCBfoMRt4nw1vbt3 P9Mab6xg308ruXzohPo6hU/SmSfXgzgWVvKqiMYqlL8rEa5U+rwyPbtCon1GhAmhsNek /Dsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:mime-version:from:in-reply-to:date:cc :message-id:references:to; bh=1U5g6l6GXsxRgT2dvzWyEWJnUpHvPgL/G3gBeDUXVfk=; b=nRQpZUt2EdAiYdv2Z4FCAyWpIWeVjD7Uv6vq8JJaEDJhSw9EvwRfqSSoSdn8nBDmaA o731z4liCrZbJ+WPNiDX0j62mDaRInphzZisxRooEIeQ5NvmLenT1o7IBhCanuPUpK03 J8ndWjE0a6jm26tFWyg8rMVzLQXh5r3xff7aJKRJt8NU+HR0DOR8+5mJzxX53toJ8lTs 5r5sIEnLrEIPj8H4jgCMp0YHZ5OH7r51oPg8Q2MIV9msUQdbXxcIhS0MbeJwVW670m6a emxAF8/Fv02qk0XkvzCxiMGGinroKLxi/qmXRP5lw0hN9Fpbu1TwSr7siCXD3Xdl8SLh u+bg== X-Gm-Message-State: AFeK/H3OdJGN59I5W7xBO4KkR9OaZQvhyZpipKMlf09yIFXBb0xTqJdHR9l+RlZCSXpNbg== X-Received: by 10.84.129.3 with SMTP id 3mr38305065plb.150.1491417893024; Wed, 05 Apr 2017 11:44:53 -0700 (PDT) Received: from pinklady.local (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id b128sm38939794pfg.54.2017.04.05.11.44.52 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 05 Apr 2017 11:44:52 -0700 (PDT) Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: multipart/signed; boundary="Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail From: "Ngie Cooper (yaneurabeya)" In-Reply-To: Date: Wed, 5 Apr 2017 11:44:51 -0700 Cc: "freebsd-testing@freebsd.org" Message-Id: References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> To: Craig Rodrigues X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Apr 2017 18:44:57 -0000 --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 (Apologies for taking so long to reply; the longer the reply, the longer = it takes for me to write a thoughtful response sometimes ;)..) > On Jan 17, 2017, at 16:09, Craig Rodrigues = wrote: >=20 > On Tue, Jan 17, 2017 at 3:01 PM, Ngie Cooper (yaneurabeya) < > yaneurabeya@gmail.com> wrote: >=20 >> Hello all, >>=20 >> I had an initial discussion with a handful of other test >> stakeholders and due to the number of caveats with ATF/Kyua and a = variety >> of issues in contributing back to the ATF/Kyua projects (time on >> maintainer=E2=80=99s end, legal issues, technical issues, etc), I'm = seriously >> considering replacing parts of ATF/Kyua with GoogleTest and shunit2. = In >> particular, I want to do these things [better]: >=20 > I'll provide my perspective on your proposal. > For me, the nice thing about ATF/Kyua was that Julio pushed unit tests = to > live along-side the code in FreeBSD, > and provided a consistent structure by which these tests could run and > provide test output. > The fact that he was able to modify the Makefile system to get this = working > in FreeBSD *and* NetBSD > is amazing. I never would have had the energy to do that!! >=20 > I was hoping that ATF/Kyua would attract interest from projects other = than > FreeBSD and NetBSD, > so that there would be an ecosystem of projects that would use this = stuff > and drive this forward. > I updated the Mac Homebrew port of Kyua a few times, and also tried to > submit a Debian package > for ATF and Kyua but got stalled there. > Unfortunately, ATF/Kyua never really took off outside of FreeBSD and = NetBSD. Yes. That=E2=80=99s a good summary of where we=E2=80=99ve come from. > Also, the implementation details of modern C++ and Lua for Kyua, and = shell > script and C for ATF > is a high barrier of entry for people wanting to work on the code and > extend it. > There are enough developers in FreeBSD who could probably pick up the = C and > ATF parts. > For the C++ and Lua parts, I think the barrier is quite high, and = there are > not that many developers > in the FreeBSD community with the skills or interest to work on that. I=E2=80=99d have to look at how easy divorcing libatf-c from libatf-c++ = would be, but I suspect it=E2=80=99s non-trivial. The ATF runners in = kyua seem to depend on both libraries (although they can be obtained = from ports): $ ldd /usr/local/tests/kyua/engine/atf_* = /usr/local/tests/kyua/bootstrap/atf_helpers | grep c++ libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x80117b000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015ae000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x80119f000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d2000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x8011a2000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d5000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x8011db000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x80160e000) libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 = (0x800823000) libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x800c56000) > I have a selfish perspective in that I like to invest time in = libraries and > technologies that > help me get jobs, even in companies that are not FreeBSD. > In my last job at a company making a FreeBSD appliance, I tried to = push > adoption of ATF/Kyua > but wasn't able to convince folks to pick it up. The developers gave = me > feedback that the ATF/Kyua > documentation didn't give them confidence. It didn't help that I = couldn't > point to other projects > outside of FreeBSD/NetBSD using this stuff. > At least for me, I cannot say that ATF/Kyua is a marketable skillset = which > opens doors for jobs. :( >=20 > Moving forward, if you try to steer towards libraries and technologies = that > have active > user communities *even outside of FreeBSD*, that would be nice. > Testing software isn't unique to FreeBSD, and there is a lot of good = work > going on in these > other communities. >=20 > You mentioned that GoogleTest is actively worked on by Google, > but there at last one non-Google community (LLVM) is using it, and has = a > selfish interest to > keep it going. That is one plus in my opinion. Bingo. Hence, one of the motivators behind GoogleTest (it has more = mindshare in the open source community than ATF/Kyua does). > GoogleTest uses a 2-clause BSD license, so that is OK. > GoogleTest requires signing a Contributor License Agreement for people = who > want to submit patches upstream to them. That's no different from = Kyua. > That's a bit annoying, but maybe something we can live with. Yes. > So I have no problem with GoogleTest on those points. >=20 > My questions for you are: > (1) GoogleTest is very C++-focused. ATF has API's for C, C++, and > shell. > How successful will you be at testing a pile of C code with a > C++-focused library? > It is technically possible ( > = https://meekrosoft.wordpress.com/2009/11/09/unit-testing-c-code-with-the-g= oogletest-framework/ > ) > but what are the gotchas? That I need to quite frankly investigate. It might come down to the fact = that tests need to be written in C++ =E2=80=94 which I=E2=80=99m not = really keen on, but if the C++ is simple, and the boilerplate is simple = enough, then the C++ shouldn=E2=80=99t need to get too fancy, and hence = will look at lot like the existing, simple GoogleTest examples. > (2) What do we do with existing tests? Do they need to be rewritten = to > the GoogleTest API? > Or do we have infrastructure that allows existing ATF tests to = run > with Google Test? Same thing as before: 1. Write new tests in the preferred target framework. 2. Don=E2=80=99t rewrite existing tests, unless there=E2=80=99s a good = reason to do so (needless refactoring introduces risk). As far as the second question (=E2=80=9COr do we=E2=80=A6=E2=80=9D), = yes, my goal was to write equivalent macros, if at all possible, for = using GoogleTest in lieu of ATF. The ownership for this task falls on = me. I might need to write some compatibility functions/macros to ease = the transition away from ATF, if my investigation demonstrates it's = feasible/possible. > (3) If this is adopted, are new tests written in GoogleTest API, or = in > the ATF API? The GoogleTest APIs. > (4) Kyua is the tool currently used for running the tests, and > providing test result output. > Kyua has test-case isolation, and the potential to parallelize = the > execution of test-cases. > Are the Kyua features of test-case isolation and = parallelization > of test-cases must-haves, > or can we live without them? > Do you want to move away from using Kyua, or modify it to work > with GoogleTest? As a first step, I want to mock ATF via GoogleTest. Although = contributing back to Google (as a corporation =E2=80=94 thanks $lawyers) = has proved an impedance, we can always patch the port to add in the = helper. > If you want to move away from Kyua, how do you want people to = run > the GoogleTests and > get the test results? I really don=E2=80=99t want to change the inputs/outputs if at all = possible =E2=80=94 just add a bit more =E2=80=9Cwindow dressing=E2=80=9D = to denote setup/teardown fixtures and other gaps that ATF doesn=E2=80=99t = have. If it doesn=E2=80=99t prove fruitful, I=E2=80=99ll send out an update = email describing the work I=E2=80=99ve done, the roadblocks/tradeoffs = I=E2=80=99ve made in getting to the decision, and ask for some consensus = in terms of driving the effort forward, further. > I have no objection to your proposal. Using a more mainstream testing > framework with a larger user community, > and some momentum to keep things going would be nice. Thanks! -Ngie --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJY5TsjAAoJEPWDqSZpMIYVdY0QAI5eemX4HJVupRXPyKfx0bJc uZ5fY2TnBj+A3y4FycPKsr+hj47SZs/IGVoosn8U4i8KCJKebPQn7Ck9YD+cRUyp U8ihpvVxTbTCPVm4B+GatdN1dIr8jBAAHWZVyR/Uwi/OFu3U+r86/E2wJQEsjutM VXVhzEj9PhxqhmAxHhN1p9ZRmE3hJNhlu+zy3YWn4pR8Xk4C/k0edZTuozyZCOkc LX0Q36m+XIgPjlskMTp8xbQ2X8hnPgXz1YAJwO6hdl0F4Loucyy4FBvlgoYu5RGp QBEukBt77io/wekCLdYICHdxg7x2LMBKlOAz+mPbL+O0Q/CrP4BCnGu1FYMHCqdH bajZzme8ojRGTJiBJ5lzavoJjx8lj7SzVOrot986Wmwmta2mh4kmiIEJhbILkqW+ ULJG7TBudfMmJPjJJYB5oh1oliLySf6dDokPlqUeokyWHOj0Pky0uIp3nSJKlvfp 67XEsL2Z9sFG2CVTeLvNq9vCZ0rogdN1kPM3u1nPzNLXa99gIXx0UflfrH/CMi5e NvDPLq+P7g1AIoTL4ioEVc4u4UbrObUrKvY5QSy+vIzv6Hs6b259Ox8IDYvQiMIQ 0FBcOwq4d3c+tMB6k78AQ9k/vgJJqR56rpJESgHnhUq4c6wmTbLzX3rx/dKdh9mb O9Dy/2Bj3YD8IOHGEOgn =Ej3a -----END PGP SIGNATURE----- --Apple-Mail=_77D8F390-D83C-45B5-AC95-0F2CDC22B0CD-- From owner-freebsd-testing@freebsd.org Thu Apr 6 03:37:40 2017 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AEF08D2F0FF for ; Thu, 6 Apr 2017 03:37:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8C25DB3A for ; Thu, 6 Apr 2017 03:37:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 8B7B1D2F0FD; Thu, 6 Apr 2017 03:37:40 +0000 (UTC) Delivered-To: testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B160D2F0FC for ; Thu, 6 Apr 2017 03:37:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-yw0-x234.google.com (mail-yw0-x234.google.com [IPv6:2607:f8b0:4002:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48247B39; Thu, 6 Apr 2017 03:37:40 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-yw0-x234.google.com with SMTP id v76so14863468ywg.0; Wed, 05 Apr 2017 20:37:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=G1q4O1EOPNFclC94tg+KoGGHHRiuN+3w+Twc25UK82I=; b=Kv9BgKIh+ixU5UHGJauJnTOb+3veQaLeHMaAzUQkhvk169E/gF32kmG0475vCKCDTR akd/8mlempNPXh/VArM00yZiT3WqDx2jRdhi8axqRuaeJxI41xuX4pFVMrELo/q5AWve yWVWJprCPjSjp3tWvQELqBRqem9Do9DkFUh3VVHbkz7f0qnddmMNb2Qf31S1Uu8nsxpM krc+0q7fLZUvgCFFrPAA+OFVfoNQdxjz+YDrqcZ13Lk5ppqeHWo1oLBjxLn2ar8aPyIz 9YxQ3aq08v94pw72NMWkjaL5rfrn+w1beK8mANuVFB5KvNem9oZU12Qw/lFmRy6plbAJ yX/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=G1q4O1EOPNFclC94tg+KoGGHHRiuN+3w+Twc25UK82I=; b=S7YyxWKzqpDhEsqMd6NTrFur01qzKiiUiDY5M2VzsREC9g0sfx2UmzXFoN8KL9ioSK YbsT1a/un64SHX0Nbjl5O/SbXzrfMc/0OOY1ocQXjyVznbvvQfHQ47A2fBbafQkr+iLD qZTkmv8VsVa0V0e5QDIng29UWpEcOoHbloLns5pNXAlMJW5NgH0/fnQ4qPUqHIzIuCbm is6m1kW7vH3GLTvfDETAFEwrn4LMFcHEfT7v7aEGO2OIB+8ZV6Ppc1JGgIhoPojLI1mN U6Ir+k5V37glxUTbpTlw8e4pjQInDl09rsIVLHpGa/sehenihzApzLNVa1dE2cIdqCcN sMlg== X-Gm-Message-State: AFeK/H1whf21Q5cdXs3qi7uwn76UjD0jqOYL+a324W9A+z4fWdScYNzyHeLLjM1jDWE+WIvphxBMj+WhOueNfg== X-Received: by 10.129.173.68 with SMTP id l4mr20714510ywk.351.1491449858983; Wed, 05 Apr 2017 20:37:38 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.129.20.214 with HTTP; Wed, 5 Apr 2017 20:37:37 -0700 (PDT) In-Reply-To: References: <45D23581-C780-4C55-80CF-19A81813D672@gmail.com> From: Alan Somers Date: Wed, 5 Apr 2017 21:37:37 -0600 X-Google-Sender-Auth: Q2lTpfuaKxHioQtYXg4U7fuOT2E Message-ID: Subject: Re: Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2 To: "Ngie Cooper (yaneurabeya)" Cc: Craig Rodrigues , "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Apr 2017 03:37:40 -0000 On Wed, Apr 5, 2017 at 12:44 PM, Ngie Cooper (yaneurabeya) wrote: > (Apologies for taking so long to reply; the longer the reply, the longer = it takes for me to write a thoughtful response sometimes ;)..) > >> On Jan 17, 2017, at 16:09, Craig Rodrigues wrote: >> >> On Tue, Jan 17, 2017 at 3:01 PM, Ngie Cooper (yaneurabeya) < >> yaneurabeya@gmail.com> wrote: >> >>> Hello all, >>> >>> I had an initial discussion with a handful of other test >>> stakeholders and due to the number of caveats with ATF/Kyua and a varie= ty >>> of issues in contributing back to the ATF/Kyua projects (time on >>> maintainer=E2=80=99s end, legal issues, technical issues, etc), I'm ser= iously >>> considering replacing parts of ATF/Kyua with GoogleTest and shunit2. In >>> particular, I want to do these things [better]: >> >> I'll provide my perspective on your proposal. >> For me, the nice thing about ATF/Kyua was that Julio pushed unit tests t= o >> live along-side the code in FreeBSD, >> and provided a consistent structure by which these tests could run and >> provide test output. >> The fact that he was able to modify the Makefile system to get this work= ing >> in FreeBSD *and* NetBSD >> is amazing. I never would have had the energy to do that!! >> >> I was hoping that ATF/Kyua would attract interest from projects other th= an >> FreeBSD and NetBSD, >> so that there would be an ecosystem of projects that would use this stuf= f >> and drive this forward. >> I updated the Mac Homebrew port of Kyua a few times, and also tried to >> submit a Debian package >> for ATF and Kyua but got stalled there. >> Unfortunately, ATF/Kyua never really took off outside of FreeBSD and Net= BSD. > > Yes. That=E2=80=99s a good summary of where we=E2=80=99ve come from. > >> Also, the implementation details of modern C++ and Lua for Kyua, and she= ll >> script and C for ATF >> is a high barrier of entry for people wanting to work on the code and >> extend it. >> There are enough developers in FreeBSD who could probably pick up the C = and >> ATF parts. >> For the C++ and Lua parts, I think the barrier is quite high, and there = are >> not that many developers >> in the FreeBSD community with the skills or interest to work on that. > > I=E2=80=99d have to look at how easy divorcing libatf-c from libatf-c++ w= ould be, but I suspect it=E2=80=99s non-trivial. The ATF runners in kyua se= em to depend on both libraries (although they can be obtained from ports): > > $ ldd /usr/local/tests/kyua/engine/atf_* /usr/local/tests/kyua/bootstrap/= atf_helpers | grep c++ > libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 (0x80117b000) > libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015ae000) > libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 (0x80119f000) > libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d2000) > libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 (0x8011a2000) > libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x8015d5000) > libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 (0x8011db000) > libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x80160e000) > libatf-c++.so.2 =3D> /usr/local/lib/libatf-c++.so.2 (0x800823000) > libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x800c56000) It's probably easier than you think. Those programs are just Kyua's own tests, and they aren't needed to run atf-c test programs. For example, this atf-c test program doesn't link against libprivateatf-c++: $ ldd /usr/tests/sbin/devd/client_test /usr/tests/sbin/devd/client_test: libprivateatf-c.so.1 =3D> /usr/lib/libprivateatf-c.so.1 (0x80082200= 0) libc.so.7 =3D> /lib/libc.so.7 (0x800a39000) > >> I have a selfish perspective in that I like to invest time in libraries = and >> technologies that >> help me get jobs, even in companies that are not FreeBSD. >> In my last job at a company making a FreeBSD appliance, I tried to push >> adoption of ATF/Kyua >> but wasn't able to convince folks to pick it up. The developers gave me >> feedback that the ATF/Kyua >> documentation didn't give them confidence. It didn't help that I couldn= 't >> point to other projects >> outside of FreeBSD/NetBSD using this stuff. >> At least for me, I cannot say that ATF/Kyua is a marketable skillset whi= ch >> opens doors for jobs. :( >> >> Moving forward, if you try to steer towards libraries and technologies t= hat >> have active >> user communities *even outside of FreeBSD*, that would be nice. >> Testing software isn't unique to FreeBSD, and there is a lot of good wor= k >> going on in these >> other communities. >> >> You mentioned that GoogleTest is actively worked on by Google, >> but there at last one non-Google community (LLVM) is using it, and has a >> selfish interest to >> keep it going. That is one plus in my opinion. > > Bingo. Hence, one of the motivators behind GoogleTest (it has more mindsh= are in the open source community than ATF/Kyua does). > >> GoogleTest uses a 2-clause BSD license, so that is OK. >> GoogleTest requires signing a Contributor License Agreement for people w= ho >> want to submit patches upstream to them. That's no different from Kyua. >> That's a bit annoying, but maybe something we can live with. > > Yes. > >> So I have no problem with GoogleTest on those points. >> >> My questions for you are: >> (1) GoogleTest is very C++-focused. ATF has API's for C, C++, and >> shell. >> How successful will you be at testing a pile of C code with a >> C++-focused library? >> It is technically possible ( >> https://meekrosoft.wordpress.com/2009/11/09/unit-testing-c-code-with-the= -googletest-framework/ >> ) >> but what are the gotchas? > > That I need to quite frankly investigate. It might come down to the fact = that tests need to be written in C++ =E2=80=94 which I=E2=80=99m not really= keen on, but if the C++ is simple, and the boilerplate is simple enough, t= hen the C++ shouldn=E2=80=99t need to get too fancy, and hence will look at= lot like the existing, simple GoogleTest examples. This, though, is probably harder than you think. Googletest can indeed be used to test C code -- sometimes. But there are numerous C constructs which are not valid C++. For example, _Noreturn, static array sizes, and anything with identifiers that are C++ reserved words. But I suspect that the hardest thing to do in Googletest will be to replicate atf's test case isolation. It'll be easy to do per-test program, but not per-testcase. For that reason, I don't think we should attempt to convert existing tests, and I don't think that Googletest is necessarily the best framework for new tests of C code. > >> (2) What do we do with existing tests? Do they need to be rewritten t= o >> the GoogleTest API? >> Or do we have infrastructure that allows existing ATF tests to r= un >> with Google Test? > > Same thing as before: > 1. Write new tests in the preferred target framework. > 2. Don=E2=80=99t rewrite existing tests, unless there=E2=80=99s a good re= ason to do so (needless refactoring introduces risk). > > As far as the second question (=E2=80=9COr do we=E2=80=A6=E2=80=9D), yes,= my goal was to write equivalent macros, if at all possible, for using Goog= leTest in lieu of ATF. The ownership for this task falls on me. I might nee= d to write some compatibility functions/macros to ease the transition away = from ATF, if my investigation demonstrates it's feasible/possible. Why bother obfuscating Googletest? It's not going to work exactly the same way as ATF, so users will have to learn the differences anyway. We may as well allow them to use Googletest as it's intended to be used. > >> (3) If this is adopted, are new tests written in GoogleTest API, or in >> the ATF API? > > The GoogleTest APIs. > >> (4) Kyua is the tool currently used for running the tests, and >> providing test result output. >> Kyua has test-case isolation, and the potential to parallelize t= he >> execution of test-cases. >> Are the Kyua features of test-case isolation and parallelization >> of test-cases must-haves, >> or can we live without them? >> Do you want to move away from using Kyua, or modify it to work >> with GoogleTest? > > As a first step, I want to mock ATF via GoogleTest. Although contributing= back to Google (as a corporation =E2=80=94 thanks $lawyers) has proved an = impedance, we can always patch the port to add in the helper. > >> If you want to move away from Kyua, how do you want people to ru= n >> the GoogleTests and >> get the test results? > > I really don=E2=80=99t want to change the inputs/outputs if at all possib= le =E2=80=94 just add a bit more =E2=80=9Cwindow dressing=E2=80=9D to denot= e setup/teardown fixtures and other gaps that ATF doesn=E2=80=99t have. > > If it doesn=E2=80=99t prove fruitful, I=E2=80=99ll send out an update ema= il describing the work I=E2=80=99ve done, the roadblocks/tradeoffs I=E2=80= =99ve made in getting to the decision, and ask for some consensus in terms = of driving the effort forward, further. > >> I have no objection to your proposal. Using a more mainstream testing >> framework with a larger user community, >> and some momentum to keep things going would be nice. > > Thanks! > -Ngie I like the idea of using Googletest, but I don't think it can completely displace atf-C. IMHO, Kyua should have runners for as many test frameworks as people want, and isolation should be up to the framework. Different frameworks can provide different levels of isolation. I think that first we should add a Googletest runner to Kyua, then import Googletest into /contrib and rewrite the paltry number of atf-c++ tests to use Googletest. Then we can delete atf-c++. If there's demand, we can add runners for other C testing frameworks (though personally I'm satisfied with atf-c). BTW, FreeBSD already includes one Googletest program in the base, but it's not connected to the build because it depends on something from ports: cddl/usr.sbin/zfsd/tests/zfsd_unittest.cc. -Alan