From owner-freebsd-arch@FreeBSD.ORG Sun Jan 18 22:29:13 2009 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBD8E1065676 for ; Sun, 18 Jan 2009 22:29:13 +0000 (UTC) (envelope-from pho@holm.cc) Received: from relay02.pair.com (relay02.pair.com [209.68.5.16]) by mx1.freebsd.org (Postfix) with SMTP id A687B8FC38 for ; Sun, 18 Jan 2009 22:29:13 +0000 (UTC) (envelope-from pho@holm.cc) Received: (qmail 71640 invoked from network); 18 Jan 2009 22:29:09 -0000 Received: from 87.58.145.190 (HELO x2.osted.lan) (87.58.145.190) by relay02.pair.com with SMTP; 18 Jan 2009 22:29:09 -0000 X-pair-Authenticated: 87.58.145.190 Received: from x2.osted.lan (localhost.osted.lan [127.0.0.1]) by x2.osted.lan (8.14.2/8.14.2) with ESMTP id n0IMT8BT042953; Sun, 18 Jan 2009 23:29:08 +0100 (CET) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.2/8.14.2/Submit) id n0IMT84b042952; Sun, 18 Jan 2009 23:29:08 +0100 (CET) (envelope-from pho) Date: Sun, 18 Jan 2009 23:29:08 +0100 From: Peter Holm To: Alfred Perlstein Message-ID: <20090118222908.GA42845@x2.osted.lan> References: <20090118082145.GA18067@x2.osted.lan> <86iqocstjm.fsf@ds4.des.no> <49733419.5000407@FreeBSD.org> <20090118203134.GF60686@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090118203134.GF60686@elvis.mu.org> User-Agent: Mutt/1.4.2.3i Cc: Dag-Erling Sm??rgrav , freebsd-arch@freebsd.org Subject: Re: stress2 is now in projects X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jan 2009 22:29:14 -0000 On Sun, Jan 18, 2009 at 12:31:34PM -0800, Alfred Perlstein wrote: > * Kris Kennaway [090118 05:52] wrote: > > Dag-Erling Sm??rgrav wrote: > > >Peter Holm writes: > > >>The key functionality of this test suite is that it runs a random > > >>number of test programs for a random period, in random incarnations > > >>and in random sequence. > > > > > >In other words, it's non-deterministic and non-reproducable. > > > > > >You should at the very least allow the user to specify the random seed. > > > > > >DES > > > > I doubt this will help at all since the test suite is (by design) > > massively parallel, so you're at the mercy of small timing changes. > > If the start and stop times of the scripts were recorded one could > synch with the original potentially between runs, at least on the > same hardware it ran. > > Basically, replay the suite based on time instead of random. > During the more than 10 years I have used this test suite with FreeBSD I have always prioritized the ability to panic the kernel. I have never looked much into a method of re-creating a test stream. As I see it, it is a *slight* inconvenience that a panic or deadlock is not 100% reproducible in time. A fix for any problem still needs a thorough test (measured in days), IMHO. Several methods exists, as I see it, to create a test stream. One could for example generate the random work list first and then execute the tests after that. After a panic you would the have the work list that created the problem. But would re-running the work list reproduce the panic? I seriously doubt that. But there is only one way to find out. It should not be that hard to create deterministic execution of the the tests. - Peter > -Alfred