Date: Fri, 18 Oct 2002 03:45:13 -0400 (EDT) From: Chris BeHanna <behanna@zbzoom.net> To: FreeBSD-Stable <stable@freebsd.org> Subject: Re: freebsd test matrix Message-ID: <20021018034039.F1275-100000@topperwein.dyndns.org> In-Reply-To: <20021017234320.GA1664@panix.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 17 Oct 2002, David Kleiner wrote:
> On Thu, Oct 17, 2002 at 09:04:54AM +0200, Fischer, Oliver wrote:
> > Chris BeHanna wrote:
> > >
> > > I'm currently doing QA on a (very) large software project, and all
> > >of those things are important. Some of the testing uses existing
> > >industry test suites and benchmarking tools, and some of it (much of
> > >it) is custom. Being able to compile, install, and boot is just the
> > >tip of the iceberg.
> >
> > I agree, it is only the top of it. I have an idea how to do this for a
> > lot of programms, but no one how to do it with the more system specific.
> > Does anyone know how commercial os vendors do it?
>
> As something I am involved with (not to mention any names) - and I
> don't know if any of it has been looked at in the open-source U*x
> world:
>
> nightly build cycles with basic (short stress, basic api sanity,
> some standards compliance)
>
> bi-weekly - more involved api, abi, stress, standards
>
> beta-candidates, release candidates and so on - even more
> longer-term test suites.
>
> As we, most likely, cannot use anything out of the Commercial World,
> all this has to be built from close to scratch. I may be wrong here
> but it is a likelier scenario.
>
> Ideas, anyone?
There are some standardized test suites that can be used (e.g.,
POSIX compliance, SFS, SVVS, gABI). Some test standards-compliance,
while others stress the system (SFS, for example, is the gold standard
for testing filesystem performance).
The thing that bites you in the butt is test harnesses.
Invariably, the commonly-available (free) harnesses will lack features
desired by the QA team; therefore, they end up rolling their own. In
a volunteer project like this, that's a big, but necessary, first step
to take.
Ideally, each developer will have a set of unit tests for what
(s)he is working on, that could, with some massaging, be plugged into
a larger harness to do nightly regression testing (perhaps via
tinderbox). The trouble is keeping the tests up-to-date. Invariably,
what the tests are testing ends up changing underneath them, leading
to spurious failures in the test report.
--
Chris BeHanna http://www.pennasoft.com
Principal Consultant
PennaSoft Corporation
chris@pennasoft.com
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021018034039.F1275-100000>