Date: Tue, 10 Apr 2001 23:15:14 -0400 (EDT) From: Robert Watson <rwatson@FreeBSD.org> To: freebsd-arch@FreeBSD.org Subject: Features to facilitate correctness and regression testing Message-ID: <Pine.NEB.3.96L.1010410230318.81804A-100000@fledge.watson.org>
next in thread | raw e-mail | index | archive | help
As part of the TrustedBSD work, I'm in the process of developing a small
correctness/regression testing suite to (in as much as is possible)
exhaustively explore access control implementation correctness. The goal
of the suite is to allow me to take a previously identified set of
"correct behavior" and verify that, as I introduce changes, correctness is
maintained -- something that is very important when introducing new
security features. Right now, the suite tests inter-process access
control only, but I'm in the process of expanding it to test file system
access control.
One thing I have noticed in testing inter-process access control is that
adding a few helper system calls to the kernel greatly facilitates
testing, and allows me to more easily isolate the feature being tested.
For example, there are some parts of the process credential (such as the
P_SUGID flag) that are modified only implicitly by the userland process.
To explore the full interaction space, it is necessary to jump through a
large number of hoops that introduce many additional variables into the
testing process, whereas I'd like to be able to fashion test suites that
test individual features or services (in as much as possible) in
isolation. This type of scenario raises a number of questions:
1) To what extent are we interested in including correctness and
regression testing suites in the base development tree? It has been
suggested to me that these would make a great addition to a new
regression/ CVS sub-tree, that they should be included under
appropriate existing parts of the tree reflecting what they test,
and that they would be best in the form of a port. In any case, I'm
very interested in the idea of a "make regression" being easily
accessible to developers, to allow them to run fairly intensive
correctness tests after making changes.
2) Some helper functions may allow processes to modify system state in
such a way as to potentially violate normal limits on system behavior:
for example, while not inherently evil, my __setsugid() system call
allows a process to turn off the P_SUGID bit if it is currently set.
This greatly facilitates setting up test scenarios, but it's not
a feature that we necessarily want to introduce in the distributed
system. This is area where some discussion is needed: should these
features be included in the base system, then be optionally
compiled? (options REGRESSION for kernel or NO_REGRESSION=yes in
userland) Should these features be independently distributed -- for
example, a regression.ko built from src/sys/regression, or even
a port? Is the idea of helper functions for regression testing simply
philosophically wrong, or does it serve a useful function (I believe
the latter, as it allows you to reduce variables and more carefully
isolate pieces of behavior for testing, but I'd be interested in the
opinions of others).
3) Is it called regression or correctness testing? :-) I've been going
by the definition at:
http://www.data-dimensions.com/Testers'Network/regression1.htm
Also, thanks to Doug Barton for doing some initial discussing of this
stuff with me.
Robert N M Watson FreeBSD Core Team, TrustedBSD Project
robert@fledge.watson.org NAI Labs, Safeport Network Services
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1010410230318.81804A-100000>