From owner-freebsd-arch@FreeBSD.ORG Fri Jan 18 00:01:34 2008 Return-Path: Delivered-To: arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EC7016A417; Fri, 18 Jan 2008 00:01:34 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from VARK.MIT.EDU (VARK.MIT.EDU [18.95.3.179]) by mx1.freebsd.org (Postfix) with ESMTP id 6099713C43E; Fri, 18 Jan 2008 00:01:34 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from VARK.MIT.EDU (localhost [127.0.0.1]) by VARK.MIT.EDU (8.14.2/8.14.1) with ESMTP id m0HNePSj086159; Thu, 17 Jan 2008 18:40:25 -0500 (EST) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by VARK.MIT.EDU (8.14.2/8.14.1/Submit) id m0HNePNJ086158; Thu, 17 Jan 2008 18:40:25 -0500 (EST) (envelope-from das@FreeBSD.ORG) Date: Thu, 17 Jan 2008 18:40:25 -0500 From: David Schultz To: John Baldwin Message-ID: <20080117234025.GA86029@VARK.MIT.EDU> Mail-Followup-To: John Baldwin , arch@FreeBSD.ORG References: <200801171731.09873.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200801171731.09873.jhb@freebsd.org> Cc: arch@FreeBSD.ORG Subject: Re: Simple framework for regression tests in src/tools/regression X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jan 2008 00:01:34 -0000 On Thu, Jan 17, 2008, John Baldwin wrote: > When I converted the test program for the posix shm stuff over to a regression > test I added some framework bits to manage the output format prove(1) > expects. You can find it in src/tools/regression/posixshm/test.[ch]. I was > curious if we wanted to perhaps make it a library in the base system > (libbsdtest.a or some such) that our regression tools could use. > > The basic organization of the current test.[ch] is that you have a linker set > of tests which it uses to number the tests and determine the test count. The > framework provides routines for the test to indicate its result like so: > > pass() - indicates test passed > fail() - indicates test failed with no error message > fail_err(fmt, ...) - failure with printf(3) error message > fail_errno(fmt, ...) - failure with error message followed by strerror(3) > output similar to err(3) > skip(const char *) - test skipped with reason for skip > todo(const char *) - test marked as TODO I think it would be great if some day we had an easy way to set up tools/regression so that all the tests get run by tinderbox automatically. I don't know if that has any impact on the design of what you propose. It would be nice if the library also provided an option (perhaps via an env variable) to make it possible to choose between a) printing a failure message and continuing b) calling abort() to trap into the debugger Also, add a drop-in replacement for assert() to the list above. Some of my tests call assert() because that happened to be convenient at the time, but that's not the right thing to do for prove(1). Also on my wish list: - A test infrastructure where you can type 'make' on level of the hierarchy and have it run all the tests under that directory; prove -r isn't flexible enough. It should be possible to skip tests that must be run as root and tests that only apply to certain architectures[1]. It should be possible to skip "long-running" tests, and it should be possible to specify some sort of abstract complexity parameter to tests that can run for arbitrary amounts of time. For example, I have a lot of libm tests that could run all day, but you wouldn't want to run them for more than a few seconds unless you had nothing better to do with the machine. - A good way to set up performance regression tests. This, of course, requires that the tests be run repeatedly on the same hardware. But on the library front, this means having a way to output measurements to a file somewhere and compare to previous measurements. (It might also mean convenience functions to gather timing and memory usage data, and routines to output platform configuration data automatically.) The Solaris kernel group has something like this. As I recall, they ran it on the same box nightly, and you could do things like generate a graph of fork latency for the previous few years, with different lines for different VCS branches. Not that we want to go that far, but it's something to think about... [1] A 'skip' library call might not be enough. Some tests might not even compile if they are for a different architecture. Another question is what if a test requires that a particular port be installed before it will run. Currently we simply don't allow dependencies like that, so for example, my libm tests that use ports/math/mpfr simply aren't checked in. But you could imagine being able to type 'make install-test-depends' and having all the right software get installed automagically; otherwise, the test infrastructure would notice that it is missing and refrain from attempting to compile the test.