Date: Thu, 30 Jul 2015 08:59:36 -0700 From: Jim Harris <jim.harris@gmail.com> To: Craig Rodrigues <rodrigc@crodrigues.org> Cc: Alan Somers <asomers@freebsd.org>, "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: NVMe unit tests Message-ID: <CAJP=Hc_1DfsDa%2BrNjCv=FjkdXGf_NMXKjFebL=HGwsziRhW8yg@mail.gmail.com> In-Reply-To: <CAG=rPVfVWC=E-qAjFFwxts1KTKKgCwz9Lv7DOj9fipi7NheFtA@mail.gmail.com> References: <CAJP=Hc8vTMgjYP46rHJh9BH3RV3CKimfFJDBs_r7F_xixOx6Fg@mail.gmail.com> <CAOtMX2h9iVAO_Nb5R7AS3tSiB4sDcg871MT2qr1tzyee--fnkA@mail.gmail.com> <CAG=rPVfVWC=E-qAjFFwxts1KTKKgCwz9Lv7DOj9fipi7NheFtA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jul 30, 2015 at 5:20 AM, Craig Rodrigues <rodrigc@crodrigues.org> wrote: > > > On Wed, Jul 29, 2015 at 3:02 PM, Alan Somers <asomers@freebsd.org> wrote: > >> On Wed, Jul 29, 2015 at 11:59 AM, Jim Harris <jimharris@freebsd.org> >> wrote: >> >> Kyua doesn't natively understand CUnit. So you have two options: run the >> tests with the plain test adapter, or convert them all to ATF-C tests. >> The >> main disadvantage to the plain test adapter is that Kyua won't be aware of >> the detailed test results. For example, if you have 500 CUnit test cases, >> Kyua will only report whether the whole suite passed or failed rather than >> whether each test case passed or failed. >> > > In the May/June 2015 issue of FreeBSD Journal ( http://freebsdjournal.com > ), there is > a good interview with Julio Merino (jmmv). In the interview, Julio says: > > "If you happen to be an atf-sh user, I'd like ot suggest you have a look > at shtk's unittest module, which is my latest attempt at implementing a > more modern testing library for the shell that follows the common xUnit > idioms. And if you use atf-c++, consider googletest as an alternative, a > much more mature C++ testing library". > > On a similar note, I have used CUnit, and find it to be a perfectly fine > library > for writing unit tests in C. CUnit and googletest can both generate xUnit > XML test result > output which can be used as input to continuous integration systems such > as Jenkins. > > Based on the comments from Julio, I have a very hard time encouraging > people > to write new tests using ATF. I have a harder time encouraging people to > migrate > existing tests from other test libraries to ATF, if they have stuff > already working with things like > googletest or CUnit. > > At my work, I originally tried to steer people to write new tests in ATF > and migrate tests from CUnit to ATF, but then > gave up, because of pushback from developers. The combination we have that > seems to work for us is: CUnit for unit tests in C, pyunit tests in > Python for tests which can run in a scripting language, and all tests > results reported in Jenkins. This combination may not work for everyone, > but it is what seems to be OK for us. > > If kyua could be extended to natively run CUnit and googletest, that would > be > very useful. That would allow us to run more types of test libraries and > have the test results reported natively in kyua. I don't have the time to > work on extending kyua in this way. > I'm not sure if Julio does either. In that interview, he states that he > is quite busy. > > The alternative is to run CUnit or googletests directly (even if they are > in the FreeBSD src tree), and report the results in the Jenkins test report > viewer. That option may not be attractive, but it would work. > Alan and Craig, Thanks for the feedback. I will integrate these into HEAD soonish, under tests/sys/dev/nvme as Alan suggested. I do not plan to switch these tests from CUnit to something else, but would be willing to add code to spit out XML files with per-test results if that would be helpful. Otherwise the tests currently are organized per nvme source file, so we would at least get per-file granularity on the test results. -Jim > -- > Craig >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJP=Hc_1DfsDa%2BrNjCv=FjkdXGf_NMXKjFebL=HGwsziRhW8yg>