From owner-freebsd-arch@FreeBSD.ORG Wed Nov 14 10:34:14 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2AD3297C; Wed, 14 Nov 2012 10:34:14 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id D18E88FC14; Wed, 14 Nov 2012 10:34:13 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id 910F88A412; Wed, 14 Nov 2012 10:34:12 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.5/8.14.5) with ESMTP id qAEAYBn6007286; Wed, 14 Nov 2012 10:34:11 GMT (envelope-from phk@phk.freebsd.dk) To: Garrett Cooper Subject: Re: [RFC] test layout/standardization for FreeBSD In-reply-to: From: "Poul-Henning Kamp" References: <7099.1352886181@critter.freebsd.dk> Date: Wed, 14 Nov 2012 10:34:11 +0000 Message-ID: <7285.1352889251@critter.freebsd.dk> Cc: George Neville-Neil , Marcel Moolenaar , Matthew Fleming , "freebsd-arch@FreeBSD.org Arch" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 10:34:14 -0000 -------- In message , Garrett Cooper writes: >> A very important thing is to have systematic metadata about test-cases, >> in particular: >> >> A) Is the test 100% deterministic, or is there a risk of random failure ? >> >> B) Is the test reliable on a heavily loaded machine ? >> >> C) Estimated duration of test >- These are important points and to be clear based on discussion prior to >and up to the vendor summit, the goal was to provide deterministic >unittests for the first prototype [...] That's very laudable, but deterministic tests are not always possible, and therefore your design should be able to handle non-deterministic tests even if you are going to frown upon them for all eternity. >- Point C cannot be accurately answered as it depends on what tests are are >in a tree, run via `make test[kernel,world]`, etc. Point C is very important, because if you want to leave space for comprehensive tests, you will have tests that take many minutes to execute, and again, you may not like them, but please make space for them in your design. >(going into B) further) Other items such as fault tolerance, stress, etc >need to be worked into the overall test ecosystem of FreeBSD [...] That is not what I'm talking about here, I'm talking about recording for each and every unit-test if it can be expected to work on a system under high load or not. The point I am trying to drive home here, is that a test-case without metadata as to its behaviour is not a good idea. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.