From owner-freebsd-arch@FreeBSD.ORG  Mon Nov 19 08:09:43 2012
Return-Path: <owner-freebsd-arch@FreeBSD.ORG>
Delivered-To: freebsd-arch@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id B0AA8FF8;
 Mon, 19 Nov 2012 08:09:43 +0000 (UTC)
 (envelope-from phk@phk.freebsd.dk)
Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222])
 by mx1.freebsd.org (Postfix) with ESMTP id 6344C8FC08;
 Mon, 19 Nov 2012 08:09:43 +0000 (UTC)
Received: from critter.freebsd.dk (unknown [192.168.61.3])
 by phk.freebsd.dk (Postfix) with ESMTP id 321EA8A3FC;
 Mon, 19 Nov 2012 08:09:42 +0000 (UTC)
Received: from critter.freebsd.dk (localhost [127.0.0.1])
 by critter.freebsd.dk (8.14.5/8.14.5) with ESMTP id qAJ89f4u031882;
 Mon, 19 Nov 2012 08:09:41 GMT (envelope-from phk@phk.freebsd.dk)
To: Marcel Moolenaar <marcel@xcllnt.net>
Subject: Re: [RFC] test layout/standardization for FreeBSD
In-reply-to: <F3302426-CA5D-4B61-8F3A-081395B393D9@xcllnt.net>
From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
References: <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com>
 <7099.1352886181@critter.freebsd.dk>
 <F3302426-CA5D-4B61-8F3A-081395B393D9@xcllnt.net>
Date: Mon, 19 Nov 2012 08:09:41 +0000
Message-ID: <31881.1353312581@critter.freebsd.dk>
Cc: Garrett Cooper <yanegomi@gmail.com>, George Neville-Neil <gnn@freebsd.org>,
 "freebsd-arch@FreeBSD.org Arch" <freebsd-arch@freebsd.org>,
 Matthew Fleming <mdf@freebsd.org>
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Nov 2012 08:09:43 -0000

--------
In message <F3302426-CA5D-4B61-8F3A-081395B393D9@xcllnt.net>, Marcel Moolenaar 
writes:

>For example: If a test is not reliable on a heavily loaded machine,
>then the test is ipso facto not 100% deterministic. So, B implies
>A. How is this different from a meta data perspective? If we test
>a RNG, are we ever going to be 100% deterministic?

Some tests are not 100% deterministic no matter what.  That is
an important distinction.

>Also, the estimated duration for tests is very platform specific.

Only if CPU or I/O bound.  Tests for things like dynamic route-expiry,
and other timeout driven parts of the system will have long realtime
clock, but short cpu-clock.

For a comprehensive nightly test-run, you want such test-cases,
for a quick sanitycheck on your kernel rewrite, you probably
don't.

I fully agree with your "crawl before we run" approach, but testing
is a quite mature, if underappreciated, discipline, and we would do
well to learn from others mistakes.

All I'm asking at this point, is that we make a space for storing
this metadata in a machine-readable format.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.