From owner-freebsd-current Sun May 11 10:11:30 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.5/8.8.5) id KAA04956 for current-outgoing; Sun, 11 May 1997 10:11:30 -0700 (PDT) Received: from bitbox.follo.net (bitbox.follo.net [194.198.43.36]) by hub.freebsd.org (8.8.5/8.8.5) with ESMTP id KAA04951 for ; Sun, 11 May 1997 10:11:27 -0700 (PDT) Received: (from eivind@localhost) by bitbox.follo.net (8.7.6/8.7.3) id TAA06018; Sun, 11 May 1997 19:09:52 +0200 (MET DST) Date: Sun, 11 May 1997 19:09:52 +0200 (MET DST) Message-Id: <199705111709.TAA06018@bitbox.follo.net> From: Eivind Eklund To: Terry Lambert CC: current@FreeBSD.ORG In-reply-to: Terry Lambert's message of Sat, 10 May 1997 12:43:05 -0700 (MST) Subject: Regression tests (was Re: A 3.0-current SNAP building machine has been found!) References: <18077.863238129@time.cdrom.com> <199705101943.MAA04327@phaeton.artisoft.com> Sender: owner-current@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk Cc: to announce removed - PLEASE do this for any other followups in the same thread! > > I suppose that Terry will now suggest some sort of voting system and I > > can't even say that it's such a bad idea (just so long as I don't have > > to write the vote collection and tabulation software :-). > > > > Comments? > > I was actually against using "voting"; I prefer forcing developers to > verify that compiles work and the resulting code does not trivially > fail prior to it being committed. That way the tree would always work; > the best you can get out of "voting" is "provided with high confidence > that it might work". Regression tests. We really should have this integrated into the build system. I have been thinking a bit about this, but haven't come up with any system I feel entirely satisfied with yet, and thus haven't brought it up for discussion (Besides, I have a lot of other FreeBSD-related projects not completed yet, and thus feel a bit timid). For my personal projects, I use one of the following systems (depending on the size/complexity of the project): -DTEST - every source file define a main(), and calling this return OK if the test passes or is not present. If the test fails, the program print a message and exit with failure. Advantages: Extremely simple, requires thought about writing testable code Disadvantages: Often too simple, requires thought about writing testable code -DREGRESS - Each object file exports _regress(), and a new source file is created calling each of these. The auto-generated file is compiled and called. Advantages: Fairly simple, allow dependencies between modules. Still requires thought about testability. Disadvantages: Often too simple. None of the above adapt well to a large established code base. :-( I'd guess we need a combination of something like -DREGRESS, as well as a directory with shell scripts to run to test. These could then do the necessary creation of datafiles and actual testing. They should probably be run in a pre-set order, to allow a later test to depend on a former test and the datafiles created by it. (This might mean that a Makefile for them would be a good idea.) Anyway; being able to do a 'make regress' on the entire FreeBSD source tree and know all new code and a lot of old code got tested would be a _good_ feeling. Eivind.