From owner-freebsd-arch Tue Apr 10 20:14:44 2001 Delivered-To: freebsd-arch@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 782A937B422 for ; Tue, 10 Apr 2001 20:14:40 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (robert@fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.11.3/8.11.3) with SMTP id f3B3FEf82357 for ; Tue, 10 Apr 2001 23:15:14 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Tue, 10 Apr 2001 23:15:14 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: freebsd-arch@FreeBSD.org Subject: Features to facilitate correctness and regression testing Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG As part of the TrustedBSD work, I'm in the process of developing a small correctness/regression testing suite to (in as much as is possible) exhaustively explore access control implementation correctness. The goal of the suite is to allow me to take a previously identified set of "correct behavior" and verify that, as I introduce changes, correctness is maintained -- something that is very important when introducing new security features. Right now, the suite tests inter-process access control only, but I'm in the process of expanding it to test file system access control. One thing I have noticed in testing inter-process access control is that adding a few helper system calls to the kernel greatly facilitates testing, and allows me to more easily isolate the feature being tested. For example, there are some parts of the process credential (such as the P_SUGID flag) that are modified only implicitly by the userland process. To explore the full interaction space, it is necessary to jump through a large number of hoops that introduce many additional variables into the testing process, whereas I'd like to be able to fashion test suites that test individual features or services (in as much as possible) in isolation. This type of scenario raises a number of questions: 1) To what extent are we interested in including correctness and regression testing suites in the base development tree? It has been suggested to me that these would make a great addition to a new regression/ CVS sub-tree, that they should be included under appropriate existing parts of the tree reflecting what they test, and that they would be best in the form of a port. In any case, I'm very interested in the idea of a "make regression" being easily accessible to developers, to allow them to run fairly intensive correctness tests after making changes. 2) Some helper functions may allow processes to modify system state in such a way as to potentially violate normal limits on system behavior: for example, while not inherently evil, my __setsugid() system call allows a process to turn off the P_SUGID bit if it is currently set. This greatly facilitates setting up test scenarios, but it's not a feature that we necessarily want to introduce in the distributed system. This is area where some discussion is needed: should these features be included in the base system, then be optionally compiled? (options REGRESSION for kernel or NO_REGRESSION=yes in userland) Should these features be independently distributed -- for example, a regression.ko built from src/sys/regression, or even a port? Is the idea of helper functions for regression testing simply philosophically wrong, or does it serve a useful function (I believe the latter, as it allows you to reduce variables and more carefully isolate pieces of behavior for testing, but I'd be interested in the opinions of others). 3) Is it called regression or correctness testing? :-) I've been going by the definition at: http://www.data-dimensions.com/Testers'Network/regression1.htm Also, thanks to Doug Barton for doing some initial discussing of this stuff with me. Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message