Date: Tue, 4 Nov 2014 10:38:03 -0800 From: Garrett Cooper <yaneurabeya@gmail.com> To: "freebsd-testing@freebsd.org" <freebsd-testing@FreeBSD.org> Subject: Kyua/ATF as a test framework discussion Message-ID: <C3091631-0E35-40F4-BEF0-12AB68EF6B97@gmail.com>
index | next in thread | raw e-mail
[-- Attachment #1 --] (developers BCCed; please continue discussion on -testing@) Hi all, A few questions came up yesterday at the BSD Vendor Summit in the Q&A session after rodrigc’s talk about Kyua/ATF related to testing on FreeBSD, as related to the testing framework (there are a handful of other questions I’ll bring up in other threads ;)..). In particular, some of the questions that stood out in my mind were: 1. What type of testing does Kyua do? Can it be used in benchmark testing, stress testing, etc? 2. Is there a structured way of checking for feature <X> before running my tests? 3. Can I run tests with custom arguments, e.g. point something at a filesystem path, etc? How do I do that? I’ll try to answer these questions based on experience and input I’ve received from several people both internal and external to EMC/Isilon. Answer to 1. - Kyua/ATF excels as a unit testing framework and only does non-distributed testing. - Kyua/ATF should not be used in benchmark testing or stress testing. There is a key component missing from kyua that allows it to run functional tests (both on FreeBSD and in general). In particular it does not have per test program setup/cleanup hooks that other frameworks do (cmake, JUnit, LTP, python unittest), and it does not have a per-testcase setup hook. This is something that I called out several years ago when I first looked at importing ATF into the base tree (see the “Known Issues” section in https://wiki.freebsd.org/TestingFreeBSD ). Not having setup/cleanup hooks has made some testing at EMC/Isilon painful because of how some testers have written tests, in particular there are testcases that were added to functionally test out mountd, mount_nfs, syslogd, etc that I had to disable because of the unnecessary complexity involved in trying to manage state in setting up/tearing down daemons, running syslogd, etc, and because of the external state involved the testcases would work sometimes if run end-to-end, not other times if executed one-by-one, and not other times if run end-to-end. Adding these hooks isn’t impossible and should be the next step in improving the test Kyua/ATF infrastructure (I ran out of time before trying to implement this, and I’m not a C++ whiz so I got lost trying to figure out how to make everything work). Answer to 2. There isn’t currently a structured API for asking questions like “is this driver loaded?", “is this sysctl set?”, “am I running on UFS/ZFS?”, etc. In general test cases from tools/regression, etc have invented homegrown logic for handling this, so this isn’t a new issue. I’ve done some initial prototyping of a C/shell library for testing for feature requirements on FreeBSD, but the work isn’t complete/in the tree yet (please see the mailing list threads below for more details): - https://lists.freebsd.org/pipermail/freebsd-testing/2014-June/000439.html - https://lists.freebsd.org/pipermail/freebsd-testing/2014-July/000467.html I want to complete this sometime before/around the start of the 2015, in particular because this blocks several of the testcases at EMC/Isilon from being upstreamed in a good, working state. Answer to 3. Yes and no(-ish). For ATF tests: You can pass through custom arguments on the command line like so: % kyua test -v mykey=myvalue This information is that passed through the atf APIs so you can query the value with either atf-config or atf_tc_config (unfortunately the mechanism I’m used to differs enough from what is currently in place, so I’ll need to go look up what the current mechanism/structure is for this, but basically it involves specifying variables kyua.conf). For non-ATF tests: You have to resort to using environment variables today, like I did with pjdfstest: https://svnweb.freebsd.org/base/head/tests/sys/pjdfstest/tests/conf?revision=274016&view=markup#l5 , as I described in the README file: https://svnweb.freebsd.org/base/head/share/doc/pjdfstest/README?revision=274016&view=markup . I’d _really_ like to avoid doing this though because then we’ll have 40 some different ways to do things around the tree instead of a single, supported way to do things. There was an enhancement I opened on GitHub to add “configuration integration” for the other test types (Plain, TAP) [via environment variables or command querying], but that wasn’t completed: https://github.com/jmmv/kyua/issues/84 . I’ll reopen the discussion. I’ll try to add/migrate relevant links for things noted in this email on the wiki. Thank you! [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUWR0LAAoJEMZr5QU6S73eUiEH/1xeeonM4LbOCYkAWujkhZwA rgib/2fzwJnvxOi92lLLGupvRzXJE7Dy3Dru7+FsDXs9KQ1WIUyKdjbR7wi/mXAY kNSzsG8138Oe6oAc/ow2rQxW0mn6PZCA9GAmmZZ/qJ/5EDlGAqLJmiRDUPqL8AG1 m5HtLqFgHlozJyK5mpxrBCtjSOISCsF9oip6HSFT1AFFuNgZSHseUzVFnbpUrcJG NUvkrkQdOn1CeftpmFu41PtAD+tcDcfhlUpYDN8eRcne9Pnr2Ed+3p6z73qs0rMB fFVD5rBy8gd5WkpUvVaOifr238kanenA9UFaXYtIvgzoRbYgoWf4dSOKl2BLHRY= =POKQ -----END PGP SIGNATURE-----help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C3091631-0E35-40F4-BEF0-12AB68EF6B97>