From owner-freebsd-testing@freebsd.org Thu Jul 30 15:59:37 2015 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 517289AE75A for ; Thu, 30 Jul 2015 15:59:37 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 174661D9A; Thu, 30 Jul 2015 15:59:37 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: by oigi136 with SMTP id i136so24120874oig.1; Thu, 30 Jul 2015 08:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=m3G4bdkMQBS0EFE2GcK1UjWaW/vDlA80V1rxEV7St/0=; b=YJkbQlq8ExmzdTqH/R9FknI7poSlHrbU/hH9eP5ciO1VcuoRirOwBBleLjodhQmBCf cQ25QymMJHNHmrvHJRW/kl42Qt41PBl9arRM1HdYqOsp12yKYHkUXGW6TnQu0/wqpoPM axZB4G2adQfWMhmf4BNXfd2WcpUbiGHP8lRR0LMHTxEF5vxsVbJFycFt1zgECXgLm5j8 tcO9nmjOHjD1gaa9+p9JMDSUXZHYgA5Mw5GXAYXIjddZb4UBQKB5RdP4lc+qA7HIG/V/ n0jKYn9IiEkAl02CHSIXzjJ0r+3P0tJIJdyPsAqgWyCt7ZXMupKgv7Y50iPuAOQBk3Xo 91Cg== MIME-Version: 1.0 X-Received: by 10.202.180.133 with SMTP id d127mr44667917oif.104.1438271976498; Thu, 30 Jul 2015 08:59:36 -0700 (PDT) Received: by 10.202.104.65 with HTTP; Thu, 30 Jul 2015 08:59:36 -0700 (PDT) In-Reply-To: References: Date: Thu, 30 Jul 2015 08:59:36 -0700 Message-ID: Subject: Re: NVMe unit tests From: Jim Harris To: Craig Rodrigues Cc: Alan Somers , "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2015 15:59:37 -0000 On Thu, Jul 30, 2015 at 5:20 AM, Craig Rodrigues wrote: > > > On Wed, Jul 29, 2015 at 3:02 PM, Alan Somers wrote: > >> On Wed, Jul 29, 2015 at 11:59 AM, Jim Harris >> wrote: >> >> Kyua doesn't natively understand CUnit. So you have two options: run the >> tests with the plain test adapter, or convert them all to ATF-C tests. >> The >> main disadvantage to the plain test adapter is that Kyua won't be aware of >> the detailed test results. For example, if you have 500 CUnit test cases, >> Kyua will only report whether the whole suite passed or failed rather than >> whether each test case passed or failed. >> > > In the May/June 2015 issue of FreeBSD Journal ( http://freebsdjournal.com > ), there is > a good interview with Julio Merino (jmmv). In the interview, Julio says: > > "If you happen to be an atf-sh user, I'd like ot suggest you have a look > at shtk's unittest module, which is my latest attempt at implementing a > more modern testing library for the shell that follows the common xUnit > idioms. And if you use atf-c++, consider googletest as an alternative, a > much more mature C++ testing library". > > On a similar note, I have used CUnit, and find it to be a perfectly fine > library > for writing unit tests in C. CUnit and googletest can both generate xUnit > XML test result > output which can be used as input to continuous integration systems such > as Jenkins. > > Based on the comments from Julio, I have a very hard time encouraging > people > to write new tests using ATF. I have a harder time encouraging people to > migrate > existing tests from other test libraries to ATF, if they have stuff > already working with things like > googletest or CUnit. > > At my work, I originally tried to steer people to write new tests in ATF > and migrate tests from CUnit to ATF, but then > gave up, because of pushback from developers. The combination we have that > seems to work for us is: CUnit for unit tests in C, pyunit tests in > Python for tests which can run in a scripting language, and all tests > results reported in Jenkins. This combination may not work for everyone, > but it is what seems to be OK for us. > > If kyua could be extended to natively run CUnit and googletest, that would > be > very useful. That would allow us to run more types of test libraries and > have the test results reported natively in kyua. I don't have the time to > work on extending kyua in this way. > I'm not sure if Julio does either. In that interview, he states that he > is quite busy. > > The alternative is to run CUnit or googletests directly (even if they are > in the FreeBSD src tree), and report the results in the Jenkins test report > viewer. That option may not be attractive, but it would work. > Alan and Craig, Thanks for the feedback. I will integrate these into HEAD soonish, under tests/sys/dev/nvme as Alan suggested. I do not plan to switch these tests from CUnit to something else, but would be willing to add code to spit out XML files with per-test results if that would be helpful. Otherwise the tests currently are organized per nvme source file, so we would at least get per-file granularity on the test results. -Jim > -- > Craig >