Date: Thu, 30 Jul 2015 08:20:43 -0400 From: Craig Rodrigues <rodrigc@crodrigues.org> To: Alan Somers <asomers@freebsd.org> Cc: Jim Harris <jimharris@freebsd.org>, "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: NVMe unit tests Message-ID: <CAG=rPVfVWC=E-qAjFFwxts1KTKKgCwz9Lv7DOj9fipi7NheFtA@mail.gmail.com> In-Reply-To: <CAOtMX2h9iVAO_Nb5R7AS3tSiB4sDcg871MT2qr1tzyee--fnkA@mail.gmail.com> References: <CAJP=Hc8vTMgjYP46rHJh9BH3RV3CKimfFJDBs_r7F_xixOx6Fg@mail.gmail.com> <CAOtMX2h9iVAO_Nb5R7AS3tSiB4sDcg871MT2qr1tzyee--fnkA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jul 29, 2015 at 3:02 PM, Alan Somers <asomers@freebsd.org> wrote: > On Wed, Jul 29, 2015 at 11:59 AM, Jim Harris <jimharris@freebsd.org> > wrote: > > Kyua doesn't natively understand CUnit. So you have two options: run the > tests with the plain test adapter, or convert them all to ATF-C tests. The > main disadvantage to the plain test adapter is that Kyua won't be aware of > the detailed test results. For example, if you have 500 CUnit test cases, > Kyua will only report whether the whole suite passed or failed rather than > whether each test case passed or failed. > In the May/June 2015 issue of FreeBSD Journal ( http://freebsdjournal.com ), there is a good interview with Julio Merino (jmmv). In the interview, Julio says: "If you happen to be an atf-sh user, I'd like ot suggest you have a look at shtk's unittest module, which is my latest attempt at implementing a more modern testing library for the shell that follows the common xUnit idioms. And if you use atf-c++, consider googletest as an alternative, a much more mature C++ testing library". On a similar note, I have used CUnit, and find it to be a perfectly fine library for writing unit tests in C. CUnit and googletest can both generate xUnit XML test result output which can be used as input to continuous integration systems such as Jenkins. Based on the comments from Julio, I have a very hard time encouraging people to write new tests using ATF. I have a harder time encouraging people to migrate existing tests from other test libraries to ATF, if they have stuff already working with things like googletest or CUnit. At my work, I originally tried to steer people to write new tests in ATF and migrate tests from CUnit to ATF, but then gave up, because of pushback from developers. The combination we have that seems to work for us is: CUnit for unit tests in C, pyunit tests in Python for tests which can run in a scripting language, and all tests results reported in Jenkins. This combination may not work for everyone, but it is what seems to be OK for us. If kyua could be extended to natively run CUnit and googletest, that would be very useful. That would allow us to run more types of test libraries and have the test results reported natively in kyua. I don't have the time to work on extending kyua in this way. I'm not sure if Julio does either. In that interview, he states that he is quite busy. The alternative is to run CUnit or googletests directly (even if they are in the FreeBSD src tree), and report the results in the Jenkins test report viewer. That option may not be attractive, but it would work. -- Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG=rPVfVWC=E-qAjFFwxts1KTKKgCwz9Lv7DOj9fipi7NheFtA>