From owner-freebsd-arch@FreeBSD.ORG Wed Nov 14 10:09:59 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1C8D1D10; Wed, 14 Nov 2012 10:09:59 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id AD92B8FC13; Wed, 14 Nov 2012 10:09:58 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so270001oag.13 for ; Wed, 14 Nov 2012 02:09:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=C7ElJ5eNl1knoT5lEASi6Om+WBT8CwAZAKFJ7Q7YqSE=; b=v5Nf36WN50mRFZ9Z36UhNlTddGmZXfyj57Mhkgob3rwedOeQk6oltds6zD0fHgccTk HIqCGWCiMmZ09eypyPyvpspU9DmxluvqFBsKJFYkH04nHhlS9gl04TPFLzRnBSRJBwKB Rfh5q7h3FDTs4mlkYTe1QDXPibNkPom1SkQeibxX7LNKMO49dsSiZPd2UtIn8iFfWwgf fPcdseCoBOrxyw8bbBacazpP/G1oohCUIl97+sM7gSSLjT8+iXEc62SiBQydHmhkg0i9 /VY9g0+CKNBCsFzP0kY7UhJTvyY8n3Co56JZycaezMznIFsqD6XdugCwkfRlA63UmzVH Nt0A== MIME-Version: 1.0 Received: by 10.60.170.114 with SMTP id al18mr20166671oec.56.1352887797940; Wed, 14 Nov 2012 02:09:57 -0800 (PST) Received: by 10.76.143.33 with HTTP; Wed, 14 Nov 2012 02:09:57 -0800 (PST) In-Reply-To: <7099.1352886181@critter.freebsd.dk> References: <7099.1352886181@critter.freebsd.dk> Date: Wed, 14 Nov 2012 02:09:57 -0800 Message-ID: Subject: Re: [RFC] test layout/standardization for FreeBSD From: Garrett Cooper To: Poul-Henning Kamp Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: George Neville-Neil , Marcel Moolenaar , Matthew Fleming , "freebsd-arch@FreeBSD.org Arch" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 10:09:59 -0000 On Wed, Nov 14, 2012 at 1:43 AM, Poul-Henning Kamp wrote: > -------- > In message 40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com> > , Garrett Cooper writes: > > >I asked for feedback from some of the stakeholders of > >the FreeBSD test automation effort [...] > > Can I throw in something here ? > > A very important thing is to have systematic metadata about test-cases, > in particular: > > A) Is the test 100% deterministic, or is there a risk of random failure ? > > B) Is the test reliable on a heavily loaded machine ? > > C) Estimated duration of test > > etc. > - These are important points and to be clear based on discussion prior to and up to the vendor summit, the goal was to provide deterministic unittests for the first prototype (which covers point A and implicitly point B); the goal from an end-user perspective as to how they would be affected today is that unless make test[kernel,world] was actually called, it would just built and install the tests as necessary - Point C cannot be accurately answered as it depends on what tests are are in a tree, run via `make test[kernel,world]`, etc. (going into B) further) Other items such as fault tolerance, stress, etc need to be worked into the overall test ecosystem of FreeBSD (I recognize the need for integrating pho's stress2 project into FreeBSD proper so it's run on a regular basis, the need for adding to stress2, etc), but it wasn't in the scope of the initial work being done for the ATF FreeBSD port / in the short to medium term for test automation on FreeBSD and as such hasn't been planned in the overall "automation roadmap". Going further, as far as items B/C are concerned I would argue that they are highly dependent upon the test matter, how the testcases are implemented, etc. From my perspective the goal is to have (from a functional perspective) simple testcases that exercise functionality to ensure that requirements noted in manpages (because that's the user facing documentation) are adhered to and not regressed, and then move on to other more critical items requiring attention (again, stress, fault tolerance, etc). Less deterministic items need to be factored out of standard runs and/or decorated (like how one can do in python) such that they aren't necessarily executed by default, but rather when explicitly requested, either as individual tests or as a group of tests. As the phrase goes, "Rome wasn't built in a day". Thanks, -Garrett