From owner-freebsd-stable Tue Oct 22 14:24:58 2002 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A573F37B401 for ; Tue, 22 Oct 2002 14:24:56 -0700 (PDT) Received: from mail3.panix.com (mail3.panix.com [166.84.1.74]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8D39043E4A for ; Tue, 22 Oct 2002 14:24:55 -0700 (PDT) (envelope-from kleiner@panix.com) Received: from panix5.panix.com (panix5.panix.com [166.84.1.5]) by mail3.panix.com (Postfix) with ESMTP id 8797398BB8; Tue, 22 Oct 2002 17:24:49 -0400 (EDT) Received: (from kleiner@localhost) by panix5.panix.com (8.11.6/8.8.8/PanixN1.0) id g9MLOnL11634; Tue, 22 Oct 2002 17:24:49 -0400 (EDT) Date: Tue, 22 Oct 2002 17:24:49 -0400 From: David Kleiner To: "R. David Murray" Cc: Matthew Whelan , FreeBSD-Stable Subject: Re: freebsd test matrix Message-ID: <20021022212449.GA8816@panix.com> References: <20021022012432.3866.MUTTLEY@gotadsl.co.uk> <20021022072923.N83973-100000@twirl.bitdance.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20021022072923.N83973-100000@twirl.bitdance.com> User-Agent: Mutt/1.4i Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Oct 22, 2002 at 07:32:19AM -0400, R. David Murray wrote: > On Tue, 22 Oct 2002, Matthew Whelan wrote: > > Not quite what I meant... there always comes a point when testing where > > the rate at which you find bugs drops below the cost/benefit threshold > > of removing them. It's no good delivering a bug-free product 3 years > > after its usefulness has expired. It's also no good delivering that > > piece of perfection for triple the price anyone's willing to pay. > > Ah, I see. I was coming at it from the "write the tests first" > viewpoint. In that situation, you have your test suite to validate > changes against, and when anyone finds new bugs, you add a test > that covers it so you won't recreate after some later change. > So no, I wasn't advocating spending infinate amounts of time > looking for bugs. Release early, release often . > Oh, but then it's the whole other set of system tests - regression verification. If blah fails on test cases 4,5,6 out of 40 and there is a verifiable case then you just add it to the test suite, with reference to open PR's, or a new regression caused by new check-in in blah.z Since the OpenGroup test suites cost $$$, unless there is a sponsor willing and able to pay for it, all those test suites ought to be done from scratch. So do the SPEC-like benchmarks - I am wandering away, since benchmarking is not the same as system validation. Then there is stress test - system-wide and component (fs, vm, net...) - and then, again, on a larger scale somebody has to buy the boxes and put them somewhere. Is there a case for loosely-coupled p2p network of systems running freebsd validation suites? [ trimmed ] David To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message