From owner-freebsd-testing@FreeBSD.ORG Tue Nov 4 18:38:07 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C3E2C80; Tue, 4 Nov 2014 18:38:07 +0000 (UTC) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DD709B22; Tue, 4 Nov 2014 18:38:06 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id bj1so14967018pad.15 for ; Tue, 04 Nov 2014 10:38:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:date:subject:to:message-id:mime-version; bh=DqarS2Vjwo1bOdflPQZFp3JNKBxa5WGtb45nvF+eyS0=; b=gTg1C5TNRS2qJKUiJbzAIXDfU0rXG8dpIkdLCGEs1iKdOEil14WvQ1Z0RQUoA1CBPX HdqJ8qSXNV32ecxeE9tU+py+sJWWs0q4UvtQuSRaK8Ck+UusMS1IO94dMR9Vy2CZmkwk TIcUZbqT7PB3lQbhYhv1ZJeDcziDwofRw6CvCUnOVEiBv1mT00US4l5IsgSu+aDs1uue tDsPqKvIoAhuk8mXRGUKpuEF9ZK1vlQNAAPPEn2FwevtEEagEXGlUd5B4JS6Lt/nfbUV Hfootb2a2lGNNGoJOTpe0pqLzAiSdUP+XlKDQyCrq9RDQKLZZEwSweoZguLYMzDZHBym iwRA== X-Received: by 10.68.100.69 with SMTP id ew5mr52409019pbb.84.1415126286494; Tue, 04 Nov 2014 10:38:06 -0800 (PST) Received: from [172.16.0.156] (50-206-19-250-static.hfc.comcastbusiness.net. [50.206.19.250]) by mx.google.com with ESMTPSA id wa3sm1073504pac.18.2014.11.04.10.38.05 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 10:38:05 -0800 (PST) From: Garrett Cooper Content-Type: multipart/signed; boundary="Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC"; protocol="application/pgp-signature"; micalg=pgp-sha512 Date: Tue, 4 Nov 2014 10:38:03 -0800 Subject: Kyua/ATF as a test framework discussion To: "freebsd-testing@freebsd.org" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 18:38:07 -0000 --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 (developers BCCed; please continue discussion on -testing@) Hi all, A few questions came up yesterday at the BSD Vendor Summit in = the Q&A session after rodrigc=92s talk about Kyua/ATF related to testing = on FreeBSD, as related to the testing framework (there are a handful of = other questions I=92ll bring up in other threads ;)..). In particular, = some of the questions that stood out in my mind were: 1. What type of testing does Kyua do? Can it be used in = benchmark testing, stress testing, etc? 2. Is there a structured way of checking for feature before = running my tests? 3. Can I run tests with custom arguments, e.g. point something = at a filesystem path, etc? How do I do that? I=92ll try to answer these questions based on experience and = input I=92ve received from several people both internal and external to = EMC/Isilon. Answer to 1. - Kyua/ATF excels as a unit testing framework and only does = non-distributed testing. - Kyua/ATF should not be used in benchmark testing or stress testing. There is a key component missing from kyua that allows it to run = functional tests (both on FreeBSD and in general). In particular it does = not have per test program setup/cleanup hooks that other frameworks do = (cmake, JUnit, LTP, python unittest), and it does not have a = per-testcase setup hook. This is something that I called out several = years ago when I first looked at importing ATF into the base tree (see = the =93Known Issues=94 section in = https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup = hooks has made some testing at EMC/Isilon painful because of how some = testers have written tests, in particular there are testcases that were = added to functionally test out mountd, mount_nfs, syslogd, etc that I = had to disable because of the unnecessary complexity involved in trying = to manage state in setting up/tearing down daemons, running syslogd, = etc, and because of the external state involved the testcases would work = sometimes if run end-to-end, not other times if executed one-by-one, and = not other times if run end-to-end. Adding these hooks isn=92t impossible and should be the next = step in improving the test Kyua/ATF infrastructure (I ran out of time = before trying to implement this, and I=92m not a C++ whiz so I got lost = trying to figure out how to make everything work). Answer to 2. There isn=92t currently a structured API for asking questions = like =93is this driver loaded?", =93is this sysctl set?=94, =93am I = running on UFS/ZFS?=94, etc. In general test cases from = tools/regression, etc have invented homegrown logic for handling this, = so this isn=92t a new issue. I=92ve done some initial prototyping of a C/shell library for = testing for feature requirements on FreeBSD, but the work isn=92t = complete/in the tree yet (please see the mailing list threads below for = more details): - = https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.html - = https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.html I want to complete this sometime before/around the start of the = 2015, in particular because this blocks several of the testcases at = EMC/Isilon from being upstreamed in a good, working state. Answer to 3. Yes and no(-ish). For ATF tests: You can pass through custom arguments on the command line like = so: % kyua test -v mykey=3Dmyvalue This information is that passed through the atf APIs so you can = query the value with either atf-config or atf_tc_config (unfortunately = the mechanism I=92m used to differs enough from what is currently in = place, so I=92ll need to go look up what the current mechanism/structure = is for this, but basically it involves specifying variables kyua.conf). For non-ATF tests: You have to resort to using environment variables today, like I = did with pjdfstest: = https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest/tests/conf?revisi= on=3D274016&view=3Dmarkup#l5 , as I described in the README file: = https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?revision=3D= 274016&view=3Dmarkup . I=92d _really_ like to avoid doing this though = because then we=92ll have 40 some different ways to do things around the = tree instead of a single, supported way to do things. There was an enhancement I opened on GitHub to add = =93configuration integration=94 for the other test types (Plain, TAP) = [via environment variables or command querying], but that wasn=92t = completed: https://github.com/jmmv/kyua/issues/84 . I=92ll reopen the = discussion. I=92ll try to add/migrate relevant links for things noted in = this email on the wiki. Thank you! --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUWR0LAAoJEMZr5QU6S73eUiEH/1xeeonM4LbOCYkAWujkhZwA rgib/2fzwJnvxOi92lLLGupvRzXJE7Dy3Dru7+FsDXs9KQ1WIUyKdjbR7wi/mXAY kNSzsG8138Oe6oAc/ow2rQxW0mn6PZCA9GAmmZZ/qJ/5EDlGAqLJmiRDUPqL8AG1 m5HtLqFgHlozJyK5mpxrBCtjSOISCsF9oip6HSFT1AFFuNgZSHseUzVFnbpUrcJG NUvkrkQdOn1CeftpmFu41PtAD+tcDcfhlUpYDN8eRcne9Pnr2Ed+3p6z73qs0rMB fFVD5rBy8gd5WkpUvVaOifr238kanenA9UFaXYtIvgzoRbYgoWf4dSOKl2BLHRY= =POKQ -----END PGP SIGNATURE----- --Apple-Mail=_5C25921D-35F7-462F-B10A-B36F2089D9EC--