Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 14 Nov 2012 10:34:11 +0000
From:      "Poul-Henning Kamp" <phk@phk.freebsd.dk>
To:        Garrett Cooper <yanegomi@gmail.com>
Cc:        George Neville-Neil <gnn@freebsd.org>, Marcel Moolenaar <marcel@xcllnt.net>, Matthew Fleming <mdf@freebsd.org>, "freebsd-arch@FreeBSD.org Arch" <freebsd-arch@freebsd.org>
Subject:   Re: [RFC] test layout/standardization for FreeBSD
Message-ID:  <7285.1352889251@critter.freebsd.dk>
In-Reply-To: <CAGH67wRFyFrkZWxGEQBA%2BY7e1v7JQoFiQyEp0w2qzrtU4KWuyQ@mail.gmail.com>
References:  <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com> <7099.1352886181@critter.freebsd.dk> <CAGH67wRFyFrkZWxGEQBA%2BY7e1v7JQoFiQyEp0w2qzrtU4KWuyQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
--------
In message <CAGH67wRFyFrkZWxGEQBA+Y7e1v7JQoFiQyEp0w2qzrtU4KWuyQ@mail.gmail.com>
, Garrett Cooper writes:

>> A very important thing is to have systematic metadata about test-cases,
>> in particular:
>>
>> A) Is the test 100% deterministic, or is there a risk of random failure ?
>>
>> B) Is the test reliable on a heavily loaded machine ?
>>
>> C) Estimated duration of test

>- These are important points and to be clear based on discussion prior to
>and up to the vendor summit, the goal was to provide deterministic
>unittests for the first prototype [...]

That's very laudable, but deterministic tests are not always possible,
and therefore your design should be able to handle non-deterministic tests
even if you are going to frown upon them for all eternity.

>- Point C cannot be accurately answered as it depends on what tests are are
>in a tree, run via `make test[kernel,world]`, etc.

Point C is very important, because if you want to leave space for
comprehensive tests, you will have tests that take many minutes to
execute, and again, you may not like them, but please make space for
them in your design.

>(going into B) further) Other items such as fault tolerance, stress, etc
>need to be worked into the overall test ecosystem of FreeBSD [...]

That is not what I'm talking about here, I'm talking about recording
for each and every unit-test if it can be expected to work on a system
under high load or not.

The point I am trying to drive home here, is that a test-case without
metadata as to its behaviour is not a good idea.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7285.1352889251>