From owner-freebsd-arch Wed Apr 11 0:12:37 2001 Delivered-To: freebsd-arch@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id A668F37B424 for ; Wed, 11 Apr 2001 00:12:30 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (robert@fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.11.3/8.11.3) with SMTP id f3B7Cwf85038; Wed, 11 Apr 2001 03:12:59 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Wed, 11 Apr 2001 03:12:57 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Peter Jeremy Cc: freebsd-arch@FreeBSD.ORG Subject: Re: Features to facilitate correctness and regression testing In-Reply-To: <20010411161553.U66243@gsmx07.alcatel.com.au> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Wed, 11 Apr 2001, Peter Jeremy wrote: > > It has been > > suggested to me that these would make a great addition to a new > > regression/ CVS sub-tree, that they should be included under > > appropriate existing parts of the tree reflecting what they test, > > and that they would be best in the form of a port. > > I'd prefer to see any test code in the tree near the code being tested > - this (marginally) improves the chances of tests being updated to > cover new functionality. A number of parts of the system already > include stand-alone self-checks which could probably be invoked as > part of an overall regression test. Pushing the tests into a > separate port makes it more difficult for developers to use the > tests and would appear to make it harder to update the tests. > > Any generic test skeleton files probably belong in a "regression" > tree under /usr/src. Where do you think that kernel regression tests should be placed? Often, these tests will be initiated and managed from userland, so they are probably inappropriate to drop in sys/? > Since you are talking about changing the security-related behaviour of > the system, I'd prefer it to require an "options REGRESSION" (and maybe > then set a sysctl), rather than be a KLD. I'm not at all keen on the > idea of having it stashed away as a separately maintained port. One of my primary motivations for looking at importing the features into the base kernel was to avoid both the non-use, bitrot and synchronization problems currently inherent to ports. Ports with kld's suffer breakage in a number of ways, due to lack of synchronization with the base tree, difficulty in maintaining up-to-date kld builds when the system is a moving target, etc. (For example, I rebuild my workstation almost daily on -CURRENT -- and I have to rebuild my vmware kld's about as frequently or risk panics). For many consumers sitting on a release, this may be less of a problem, but for a development tool on the -CURRENT branch, it's a serious problem. Does this mean we need a which provides constants and prototypes for regression-specific interfaces which are enabled using options REGRESSION? (This is how I have it implemented in my local source tree, but I'm interested in whether this seems like a good idea to anyone else :-). > > Is the idea of helper functions for regression testing simply > > philosophically wrong, or does it serve a useful function > > I think it's a valid part of white-box testing. In any piece of > software, there are going to be code paths that are very difficult or > impossible to exercise using the `normal' interfaces. Providing > interfaces that are intended solely for testing the correctness of the > code seems perfectly reasonable. This is no different to the JTAG ports > on virtually every modern LSI chip. Sounds good to me -- I think we agree on almost all points. Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message