Date: Fri, 27 Aug 1999 02:05:51 +0000 (GMT) From: Terry Lambert <tlambert@primenet.com> To: bmcgover@cisco.com (Brian McGovern) Cc: hackers@FreeBSD.ORG Subject: Re: Looking for good QA tests... Message-ID: <199908270205.TAA22584@usr06.primenet.com> In-Reply-To: <199908261449.KAA02505@bmcgover-pc.cisco.com> from "Brian McGovern" at Aug 26, 99 10:49:50 am
next in thread | previous in thread | raw e-mail | index | archive | help
> However, I'm now at the point where I'd like to start collecting materials to
> do this. By "materials", I mean both test scenarios and code for performing
> these tests.
>
> While I now have your mind spinning, I'd like to lay down some
> limits. Firstly, my current resources for testing is relatively
> small. I have four machines that I can use most of the time.
> Therefore, we can't do "everything, all the time", so tests
> should be limited to cover as broad a range of items as we
> have time for. My estimation is we will get approximately 48
> hours per release candidate as we move towards Jordan cutting
> a release. In between, we might get a bit more slack, but not
> a lot. Secondly, these tests can not take a lot of person hours
> to perform. _I_ can give approximately 1-2 man hours per test
> cycle. Therefore, these tests need to be automated, have good
> reporting (especially if they fail), and be easy to set up and
> run. Lastly, these tests should be tests, and not benchmarks.
> I can't stress this enough. Knowing how fasts my disks are is
> useless compared to a bug in the network interface that causes
> the machine to panic.
>
> Now, as more people sign on to both write code/test plans, as
> well as do the testing, we might be able to expand the types
> of tests we can run. My plans are to wrap up whatever I'm
> given in the form of code in to a port/package set, and have
> it distributed so that anyone with a spare machine can come
> onboard and help out.
1) Download the NIST/PCTS. This should be one of the tests
that is run against FreeBSD before it is released. We
will preferrably pass, and include the test results on
the CDROM, even if we can't afford to pay a testing lab
to run the exact same code and get legal license to the
trademark (that can come later).
2) The test framework I put together for monitoring memory
pool leaks through code exercises that utilize the pools
being monitored is a Good Thing(tm). I believe that
Michael Hancock ported it to -current. My initial tests
that I provided with this framework were able to exercise
file operations on varios file systems, and detect namei
buffer leaks on any mounted FS that they were run on.
The test code did full branch path coverage on that
particular memory pool.
This would be a good framework for similar kernel memory
allocation and deallovation testing. I believe that Doug
Rabson also had a copy of this code when he was reviewing
my VFS megapatch (which included nameifree(), which leaked
on NFS under certain obscure circumstances, and I didn't
have NFS resources to test with).
3) Back when UNIX International closed shop, I saved all of
their TET and E/TET ("(Extended) Test Environment Toolkit").
This code is normally used for SVID III testing by USL
and other parties, but it is a very good validation
testing framework, capable of being scripted for test
cases for both positive and negative results.
This code should be available from many sources; originally,
DigiBoard (ftp.digibd.com at the time) put up a mirror of
my rescued copy of the U.I. archive (this is actually where
the draft copy of Spec. 1170 that went all over the place
came from: their copy of my copy of U.I.'s archive).
4) I suggest lmbench be incorporated, as well. Not because
it is any good as a comparative macro benchmark between
heterogeneous systems, but because comparison between
FreeBSD releases with prior releases results on the same
hardware could be useful to indicate when pessimizations
occur.
Any or all of these would be a nice start. You might also want to
contact Doug Ambrisko. He may be interested, assuming that these
tests achieve formal standing for acceptance testing.
Terry Lambert
terry@lambert.org
---
Any opinions in this posting are my own and not those of my present
or previous employers.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199908270205.TAA22584>