From owner-freebsd-arch@FreeBSD.ORG Thu Jan 17 22:31:21 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8BD516A418 for ; Thu, 17 Jan 2008 22:31:21 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id 9CAFF13C457 for ; Thu, 17 Jan 2008 22:31:21 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8s) with ESMTP id 228881272-1834499 for ; Thu, 17 Jan 2008 17:29:28 -0500 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id m0HMVDCs085244 for ; Thu, 17 Jan 2008 17:31:13 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: arch@freebsd.org Date: Thu, 17 Jan 2008 17:31:09 -0500 User-Agent: KMail/1.9.7 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200801171731.09873.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 17 Jan 2008 17:31:13 -0500 (EST) X-Virus-Scanned: ClamAV 0.91.2/5493/Thu Jan 17 13:09:26 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Subject: Simple framework for regression tests in src/tools/regression X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2008 22:31:21 -0000 When I converted the test program for the posix shm stuff over to a regression test I added some framework bits to manage the output format prove(1) expects. You can find it in src/tools/regression/posixshm/test.[ch]. I was curious if we wanted to perhaps make it a library in the base system (libbsdtest.a or some such) that our regression tools could use. The basic organization of the current test.[ch] is that you have a linker set of tests which it uses to number the tests and determine the test count. The framework provides routines for the test to indicate its result like so: pass() - indicates test passed fail() - indicates test failed with no error message fail_err(fmt, ...) - failure with printf(3) error message fail_errno(fmt, ...) - failure with error message followed by strerror(3) output similar to err(3) skip(const char *) - test skipped with reason for skip todo(const char *) - test marked as TODO If a given test doesn't call any of the status indicators above, then the main test loop will fail the test with an error of "unknown status". No effort is made to handle multiple status indications for a given test. I assume prove(1) would handle complaining about such anomalies in the test output. Tests consist of a void foo(void) function along with a description and each test is created via the TEST() macro like so: void mytest(void) { struct passwd *pwd; pwd = getpwuid(getuid()); if (strcmp(pwd->pw_name, "jhb") == 0) { fail_err("great evil detected"); return; } pass(); } TEST(mytest, "check for evil users"); To invoke the tests, one executes the 'run_tests()' method provided by the framework. It outputs the header expected by prove(1) and then executes each test in succession. I specifically didn't put main() in the framework in case a test utility wants to take optional arguments, etc. A simple test utility (like posixshm.c) just invokes run_tests() in its main(). One design decision that I'm sure will be debated is that I went with the functions to indicate results (fail, etc.) rather than having the test method return a value because it becomes more complicated to handle error messages otherwise. If this were C++ the tests could return a std::pair<> of a status code and some sort of object with a toString() method or something (or if it was python I would return a tuple). However, this is C. :) It's also nicer to be able to just pass the va args to vprintf() directly w/o having to print them into strings and print them out later, etc. Anyways, comments, etc.? -- John Baldwin