Date: Thu, 6 Mar 2014 14:54:55 -0500 From: Julio Merino <jmmv@freebsd.org> To: Garrett Cooper <yaneurabeya@gmail.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, kyua-discuss@googlegroups.com Subject: Re: Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles") Message-ID: <CAFY7cWAccUdcc1OgyfQDXmJNxkNZLMOPgNzg4sDL68DJzwZpkw@mail.gmail.com> In-Reply-To: <CAGHfRMCeVuuQjL2kiV5mbxVh7npYgbsn9yy7ZyJS%2B7ZT3BVcrg@mail.gmail.com> References: <CAGHfRMCeVuuQjL2kiV5mbxVh7npYgbsn9yy7ZyJS%2B7ZT3BVcrg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Mar 6, 2014 at 1:28 PM, Garrett Cooper <yaneurabeya@gmail.com> wrote: > > Seems like another standalone tool should be developed to cleanup > variables after exit per test and per test suite. Which variables? I can see some benefit in providing helper functions to store and load arbitrary data structures to/from disk with one-liners, but is that what you mean? > Setup/cleanup in ATF doesn't seem to follow the model that many other > testing frameworks do when it comes to executing testcases and > managing setup/cleanup. I really wish things were like the > JUnit/unittest model, i.e. > > - Test suite setup > - For each testcase: > - Run the testcase setup > - If the setup passed, continue > - Run the testcase > - Run the testcase cleanup > - Test suite teardown Sure... but the systems that implement this setup assume that all the steps above run under one process. That's fine in the vast majority of unit tests but the model fails when tests modify system-wide state and crash before they get to the cleanup step. Keep in mind that one of the first design decisions for ATF was to support test cases like the above where system-wide state may be modified and you need to revert the changes. Because test cases can crash at any point, having an out-of-process cleanup routine was a must. This comes with its oddities though. All said, I'd also like a more JUnit-like model because it's easier to understand and satisfies 99% of the cases (assuming you have rump).
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFY7cWAccUdcc1OgyfQDXmJNxkNZLMOPgNzg4sDL68DJzwZpkw>