From owner-freebsd-arch@FreeBSD.ORG  Wed Nov 14 09:43:09 2012
Return-Path: <owner-freebsd-arch@FreeBSD.ORG>
Delivered-To: freebsd-arch@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id 627126E9;
 Wed, 14 Nov 2012 09:43:09 +0000 (UTC)
 (envelope-from phk@phk.freebsd.dk)
Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222])
 by mx1.freebsd.org (Postfix) with ESMTP id 124E18FC15;
 Wed, 14 Nov 2012 09:43:08 +0000 (UTC)
Received: from critter.freebsd.dk (unknown [192.168.61.3])
 by phk.freebsd.dk (Postfix) with ESMTP id F40BB8A412;
 Wed, 14 Nov 2012 09:43:01 +0000 (UTC)
Received: from critter.freebsd.dk (localhost [127.0.0.1])
 by critter.freebsd.dk (8.14.5/8.14.5) with ESMTP id qAE9h10L007100;
 Wed, 14 Nov 2012 09:43:01 GMT (envelope-from phk@phk.freebsd.dk)
To: Garrett Cooper <yanegomi@gmail.com>
Subject: Re: [RFC] test layout/standardization for FreeBSD
In-reply-to: <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com>
From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
References: <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com>
Date: Wed, 14 Nov 2012 09:43:01 +0000
Message-ID: <7099.1352886181@critter.freebsd.dk>
Cc: George Neville-Neil <gnn@freebsd.org>, Marcel Moolenaar <marcel@xcllnt.net>,
 Matthew Fleming <mdf@freebsd.org>,
 "freebsd-arch@FreeBSD.org Arch" <freebsd-arch@freebsd.org>
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
 <mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 14 Nov 2012 09:43:09 -0000

--------
In message <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com>
, Garrett Cooper writes:

>I asked for feedback from some of the stakeholders of
>the FreeBSD test automation effort [...]

Can I throw in something here ?

A very important thing is to have systematic metadata about test-cases,
in particular:

A) Is the test 100% deterministic, or is there a risk of random failure ?

B) Is the test reliable on a heavily loaded machine ?

C) Estimated duration of test

etc.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.