From owner-freebsd-testing@freebsd.org Thu Jul 30 12:20:44 2015 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FD4E9AD53A for ; Thu, 30 Jul 2015 12:20:44 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-yk0-x229.google.com (mail-yk0-x229.google.com [IPv6:2607:f8b0:4002:c07::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 29E51D24; Thu, 30 Jul 2015 12:20:44 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by ykay190 with SMTP id y190so32231586yka.3; Thu, 30 Jul 2015 05:20:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=o/pLLYVMDeXUnt94KEjupNXL7oVTKWQ1QoQ09ZjMYw0=; b=jLWIravlB3zan+KQ5CRzHk/pEDDkjo0Bc/GlMrL0U80GZnuH72wdiBPC1qCi+LdSG/ x2SNdRnidKA+Ty0avhiwQFZ5g2SRwTfMzMRJyQQyDuMMgeSjC3/a4QFJht6wR1g/fh2W sJqaYoCPnNqrXaGy6uddrWT4DqykWRGbmPWTzKAjL7551LFrjzPqkCP+0hKko9258ph9 44+FYsQ8BSOW6GTodyDFDk4+7yRhSy8f0aU51yL8wbjspCCko2pzYbIy5f+DM/3Qltbf deXtj7Z5ihxE/njW9Mgv8TpexDRT4mdw5eWm2OC6DOaPIxbu6mpekIbX4Kkcb406JQl1 MZfA== MIME-Version: 1.0 X-Received: by 10.13.244.195 with SMTP id d186mr50890352ywf.73.1438258843238; Thu, 30 Jul 2015 05:20:43 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.37.79.2 with HTTP; Thu, 30 Jul 2015 05:20:43 -0700 (PDT) In-Reply-To: References: Date: Thu, 30 Jul 2015 08:20:43 -0400 X-Google-Sender-Auth: -FEJ99I09y-6EE6ekBP4iz0zo0Q Message-ID: Subject: Re: NVMe unit tests From: Craig Rodrigues To: Alan Somers Cc: Jim Harris , "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2015 12:20:44 -0000 On Wed, Jul 29, 2015 at 3:02 PM, Alan Somers wrote: > On Wed, Jul 29, 2015 at 11:59 AM, Jim Harris > wrote: > > Kyua doesn't natively understand CUnit. So you have two options: run the > tests with the plain test adapter, or convert them all to ATF-C tests. The > main disadvantage to the plain test adapter is that Kyua won't be aware of > the detailed test results. For example, if you have 500 CUnit test cases, > Kyua will only report whether the whole suite passed or failed rather than > whether each test case passed or failed. > In the May/June 2015 issue of FreeBSD Journal ( http://freebsdjournal.com ), there is a good interview with Julio Merino (jmmv). In the interview, Julio says: "If you happen to be an atf-sh user, I'd like ot suggest you have a look at shtk's unittest module, which is my latest attempt at implementing a more modern testing library for the shell that follows the common xUnit idioms. And if you use atf-c++, consider googletest as an alternative, a much more mature C++ testing library". On a similar note, I have used CUnit, and find it to be a perfectly fine library for writing unit tests in C. CUnit and googletest can both generate xUnit XML test result output which can be used as input to continuous integration systems such as Jenkins. Based on the comments from Julio, I have a very hard time encouraging people to write new tests using ATF. I have a harder time encouraging people to migrate existing tests from other test libraries to ATF, if they have stuff already working with things like googletest or CUnit. At my work, I originally tried to steer people to write new tests in ATF and migrate tests from CUnit to ATF, but then gave up, because of pushback from developers. The combination we have that seems to work for us is: CUnit for unit tests in C, pyunit tests in Python for tests which can run in a scripting language, and all tests results reported in Jenkins. This combination may not work for everyone, but it is what seems to be OK for us. If kyua could be extended to natively run CUnit and googletest, that would be very useful. That would allow us to run more types of test libraries and have the test results reported natively in kyua. I don't have the time to work on extending kyua in this way. I'm not sure if Julio does either. In that interview, he states that he is quite busy. The alternative is to run CUnit or googletests directly (even if they are in the FreeBSD src tree), and report the results in the Jenkins test report viewer. That option may not be attractive, but it would work. -- Craig