Date: Tue, 22 Oct 2002 17:24:49 -0400 From: David Kleiner <kleiner@panix.com> To: "R. David Murray" <bitz@bitdance.com> Cc: Matthew Whelan <muttley@gotadsl.co.uk>, FreeBSD-Stable <stable@FreeBSD.ORG> Subject: Re: freebsd test matrix Message-ID: <20021022212449.GA8816@panix.com> In-Reply-To: <20021022072923.N83973-100000@twirl.bitdance.com> References: <20021022012432.3866.MUTTLEY@gotadsl.co.uk> <20021022072923.N83973-100000@twirl.bitdance.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 22, 2002 at 07:32:19AM -0400, R. David Murray wrote: > On Tue, 22 Oct 2002, Matthew Whelan wrote: > > Not quite what I meant... there always comes a point when testing where > > the rate at which you find bugs drops below the cost/benefit threshold > > of removing them. It's no good delivering a bug-free product 3 years > > after its usefulness has expired. It's also no good delivering that > > piece of perfection for triple the price anyone's willing to pay. > > Ah, I see. I was coming at it from the "write the tests first" > viewpoint. In that situation, you have your test suite to validate > changes against, and when anyone finds new bugs, you add a test > that covers it so you won't recreate after some later change. > So no, I wasn't advocating spending infinate amounts of time > looking for bugs. Release early, release often <grin>. > Oh, but then it's the whole other set of system tests - regression verification. If blah fails on test cases 4,5,6 out of 40 and there is a verifiable case then you just add it to the test suite, with reference to open PR's, or a new regression caused by new check-in in blah.z Since the OpenGroup test suites cost $$$, unless there is a sponsor willing and able to pay for it, all those test suites ought to be done from scratch. So do the SPEC-like benchmarks - I am wandering away, since benchmarking is not the same as system validation. Then there is stress test - system-wide and component (fs, vm, net...) - and then, again, on a larger scale somebody has to buy the boxes and put them somewhere. Is there a case for loosely-coupled p2p network of systems running freebsd validation suites? [ trimmed ] David To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021022212449.GA8816>