Date: Thu, 6 Mar 2014 10:28:13 -0800 From: Garrett Cooper <yaneurabeya@gmail.com> To: Julio Merino <julio@meroh.net> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, kyua-discuss@googlegroups.com Subject: Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles") Message-ID: <CAGHfRMCeVuuQjL2kiV5mbxVh7npYgbsn9yy7ZyJS%2B7ZT3BVcrg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
On Thu, Mar 6, 2014 at 10:02 AM, Julio Merino <jmmv@freebsd.org> wrote:
> On Thu, Mar 6, 2014 at 12:45 PM, Alan Somers <asomers@freebsd.org> wrote:
>> I suggest the more specific name "allow_sysctl_side_effects". As I
>> continue to add tests, config variables like this will become more
>> important. In particular, I have some tests that create and destroy
>> zpools, some that can cause panics, some that load modules, etc.
>> Before I can upstream all of them, we'll need to agree on a consistent
>> set of config variables that control test case execution. I haven't
>> put much thought into it yet.
>
> Sure; more granularity++. I haven't put any thought on this though so
> please propose something :)
>
> Whatever we decide, 1) these variables need to be documented in
> tests(7), 2) a ready-to-enable template in
> ports/devel/kyua-cli/files/kyua.conf.in about them would be beneficial
> and 3) src/projects/jmmv/autotest/ should switch these on.
>
>> How does NetBSD handle tests that have
>> potentially harmful side effects?
>
> They don't AFAIK. The thing is that not that many tests have
> side-effects because the majority of kernel-level tests use rump to
> avoid this problem and be more reliable/deterministic. The few that
> have side-effects have ad-hoc cleanup routines.
Seems like another standalone tool should be developed to cleanup
variables after exit per test and per test suite.
Setup/cleanup in ATF doesn't seem to follow the model that many other
testing frameworks do when it comes to executing testcases and
managing setup/cleanup. I really wish things were like the
JUnit/unittest model, i.e.
- Test suite setup
- For each testcase:
- Run the testcase setup
- If the setup passed, continue
- Run the testcase
- Run the testcase cleanup
- Test suite teardown
Thanks :)!
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGHfRMCeVuuQjL2kiV5mbxVh7npYgbsn9yy7ZyJS%2B7ZT3BVcrg>