From owner-freebsd-hackers Thu Aug 26 19: 7: 9 1999 Delivered-To: freebsd-hackers@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id E74E915D89 for ; Thu, 26 Aug 1999 19:06:47 -0700 (PDT) (envelope-from tlambert@usr06.primenet.com) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id TAA17064; Thu, 26 Aug 1999 19:05:39 -0700 (MST) Received: from usr06.primenet.com(206.165.6.206) via SMTP by smtp04.primenet.com, id smtpdAAAf2aytH; Thu Aug 26 19:05:33 1999 Received: (from tlambert@localhost) by usr06.primenet.com (8.8.5/8.8.5) id TAA22584; Thu, 26 Aug 1999 19:05:51 -0700 (MST) From: Terry Lambert Message-Id: <199908270205.TAA22584@usr06.primenet.com> Subject: Re: Looking for good QA tests... To: bmcgover@cisco.com (Brian McGovern) Date: Fri, 27 Aug 1999 02:05:51 +0000 (GMT) Cc: hackers@FreeBSD.ORG In-Reply-To: <199908261449.KAA02505@bmcgover-pc.cisco.com> from "Brian McGovern" at Aug 26, 99 10:49:50 am X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > However, I'm now at the point where I'd like to start collecting materials to > do this. By "materials", I mean both test scenarios and code for performing > these tests. > > While I now have your mind spinning, I'd like to lay down some > limits. Firstly, my current resources for testing is relatively > small. I have four machines that I can use most of the time. > Therefore, we can't do "everything, all the time", so tests > should be limited to cover as broad a range of items as we > have time for. My estimation is we will get approximately 48 > hours per release candidate as we move towards Jordan cutting > a release. In between, we might get a bit more slack, but not > a lot. Secondly, these tests can not take a lot of person hours > to perform. _I_ can give approximately 1-2 man hours per test > cycle. Therefore, these tests need to be automated, have good > reporting (especially if they fail), and be easy to set up and > run. Lastly, these tests should be tests, and not benchmarks. > I can't stress this enough. Knowing how fasts my disks are is > useless compared to a bug in the network interface that causes > the machine to panic. > > Now, as more people sign on to both write code/test plans, as > well as do the testing, we might be able to expand the types > of tests we can run. My plans are to wrap up whatever I'm > given in the form of code in to a port/package set, and have > it distributed so that anyone with a spare machine can come > onboard and help out. 1) Download the NIST/PCTS. This should be one of the tests that is run against FreeBSD before it is released. We will preferrably pass, and include the test results on the CDROM, even if we can't afford to pay a testing lab to run the exact same code and get legal license to the trademark (that can come later). 2) The test framework I put together for monitoring memory pool leaks through code exercises that utilize the pools being monitored is a Good Thing(tm). I believe that Michael Hancock ported it to -current. My initial tests that I provided with this framework were able to exercise file operations on varios file systems, and detect namei buffer leaks on any mounted FS that they were run on. The test code did full branch path coverage on that particular memory pool. This would be a good framework for similar kernel memory allocation and deallovation testing. I believe that Doug Rabson also had a copy of this code when he was reviewing my VFS megapatch (which included nameifree(), which leaked on NFS under certain obscure circumstances, and I didn't have NFS resources to test with). 3) Back when UNIX International closed shop, I saved all of their TET and E/TET ("(Extended) Test Environment Toolkit"). This code is normally used for SVID III testing by USL and other parties, but it is a very good validation testing framework, capable of being scripted for test cases for both positive and negative results. This code should be available from many sources; originally, DigiBoard (ftp.digibd.com at the time) put up a mirror of my rescued copy of the U.I. archive (this is actually where the draft copy of Spec. 1170 that went all over the place came from: their copy of my copy of U.I.'s archive). 4) I suggest lmbench be incorporated, as well. Not because it is any good as a comparative macro benchmark between heterogeneous systems, but because comparison between FreeBSD releases with prior releases results on the same hardware could be useful to indicate when pessimizations occur. Any or all of these would be a nice start. You might also want to contact Doug Ambrisko. He may be interested, assuming that these tests achieve formal standing for acceptance testing. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message