Date: Thu, 17 Jan 2008 17:33:31 -0800 From: Alfred Perlstein <alfred@freebsd.org> To: John Baldwin <jhb@freebsd.org> Cc: arch@freebsd.org Subject: Re: Simple framework for regression tests in src/tools/regression Message-ID: <20080118013331.GP99258@elvis.mu.org> In-Reply-To: <200801171731.09873.jhb@freebsd.org> References: <200801171731.09873.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
* John Baldwin <jhb@freebsd.org> [080117 14:28] wrote:
> When I converted the test program for the posix shm stuff over to a regression
> test I added some framework bits to manage the output format prove(1)
> expects. You can find it in src/tools/regression/posixshm/test.[ch]. I was
> curious if we wanted to perhaps make it a library in the base system
> (libbsdtest.a or some such) that our regression tools could use.
>
> The basic organization of the current test.[ch] is that you have a linker set
> of tests which it uses to number the tests and determine the test count. The
> framework provides routines for the test to indicate its result like so:
>
> pass() - indicates test passed
> fail() - indicates test failed with no error message
> fail_err(fmt, ...) - failure with printf(3) error message
> fail_errno(fmt, ...) - failure with error message followed by strerror(3)
> output similar to err(3)
> skip(const char *) - test skipped with reason for skip
> todo(const char *) - test marked as TODO
>
> If a given test doesn't call any of the status indicators above, then the main
> test loop will fail the test with an error of "unknown status". No effort is
> made to handle multiple status indications for a given test. I assume
> prove(1) would handle complaining about such anomalies in the test output.
>
> Tests consist of a void foo(void) function along with a description and each
> test is created via the TEST() macro like so:
>
> void
> mytest(void)
> {
> struct passwd *pwd;
>
> pwd = getpwuid(getuid());
> if (strcmp(pwd->pw_name, "jhb") == 0) {
> fail_err("great evil detected");
> return;
> }
> pass();
> }
> TEST(mytest, "check for evil users");
>
> To invoke the tests, one executes the 'run_tests()' method provided by the
> framework. It outputs the header expected by prove(1) and then executes each
> test in succession. I specifically didn't put main() in the framework in
> case a test utility wants to take optional arguments, etc. A simple test
> utility (like posixshm.c) just invokes run_tests() in its main().
>
> One design decision that I'm sure will be debated is that I went with the
> functions to indicate results (fail, etc.) rather than having the test method
> return a value because it becomes more complicated to handle error messages
> otherwise. If this were C++ the tests could return a std::pair<> of a status
> code and some sort of object with a toString() method or something (or if it
> was python I would return a tuple). However, this is C. :) It's also nicer
> to be able to just pass the va args to vprintf() directly w/o having to print
> them into strings and print them out later, etc.
>
> Anyways, comments, etc.?
There is C++ in the base system.
You can return a pair by returning a structure which is supported in
C.
-Alfred
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080118013331.GP99258>