Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 6 Mar 2014 14:54:55 -0500
From:      Julio Merino <jmmv@freebsd.org>
To:        Garrett Cooper <yaneurabeya@gmail.com>
Cc:        "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, kyua-discuss@googlegroups.com
Subject:   Re: Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles")
Message-ID:  <CAFY7cWAccUdcc1OgyfQDXmJNxkNZLMOPgNzg4sDL68DJzwZpkw@mail.gmail.com>
In-Reply-To: <CAGHfRMCeVuuQjL2kiV5mbxVh7npYgbsn9yy7ZyJS%2B7ZT3BVcrg@mail.gmail.com>
References:  <CAGHfRMCeVuuQjL2kiV5mbxVh7npYgbsn9yy7ZyJS%2B7ZT3BVcrg@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Mar 6, 2014 at 1:28 PM, Garrett Cooper <yaneurabeya@gmail.com> wrote:
>
> Seems like another standalone tool should be developed to cleanup
> variables after exit per test and per test suite.

Which variables?

I can see some benefit in providing helper functions to store and load
arbitrary data structures to/from disk with one-liners, but is that
what you mean?

> Setup/cleanup in ATF doesn't seem to follow the model that many other
> testing frameworks do when it comes to executing testcases and
> managing setup/cleanup. I really wish things were like the
> JUnit/unittest model, i.e.
>
> - Test suite setup
> - For each testcase:
>     - Run the testcase setup
>     - If the setup passed, continue
>     - Run the testcase
>     - Run the testcase cleanup
> - Test suite teardown

Sure... but the systems that implement this setup assume that all the
steps above run under one process. That's fine in the vast majority of
unit tests but the model fails when tests modify system-wide state and
crash before they get to the cleanup step.

Keep in mind that one of the first design decisions for ATF was to
support test cases like the above where system-wide state may be
modified and you need to revert the changes.  Because test cases can
crash at any point, having an out-of-process cleanup routine was a
must.  This comes with its oddities though.

All said, I'd also like a more JUnit-like model because it's easier to
understand and satisfies 99% of the cases (assuming you have rump).



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFY7cWAccUdcc1OgyfQDXmJNxkNZLMOPgNzg4sDL68DJzwZpkw>