Date: Wed, 14 Nov 2012 02:09:57 -0800 From: Garrett Cooper <yanegomi@gmail.com> To: Poul-Henning Kamp <phk@phk.freebsd.dk> Cc: George Neville-Neil <gnn@freebsd.org>, Marcel Moolenaar <marcel@xcllnt.net>, Matthew Fleming <mdf@freebsd.org>, "freebsd-arch@FreeBSD.org Arch" <freebsd-arch@freebsd.org> Subject: Re: [RFC] test layout/standardization for FreeBSD Message-ID: <CAGH67wRFyFrkZWxGEQBA%2BY7e1v7JQoFiQyEp0w2qzrtU4KWuyQ@mail.gmail.com> In-Reply-To: <7099.1352886181@critter.freebsd.dk> References: <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com> <7099.1352886181@critter.freebsd.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Nov 14, 2012 at 1:43 AM, Poul-Henning Kamp <phk@phk.freebsd.dk>wrote: > -------- > In message <CAGH67wSoU08N8= > 40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com> > , Garrett Cooper writes: > > >I asked for feedback from some of the stakeholders of > >the FreeBSD test automation effort [...] > > Can I throw in something here ? > > A very important thing is to have systematic metadata about test-cases, > in particular: > > A) Is the test 100% deterministic, or is there a risk of random failure ? > > B) Is the test reliable on a heavily loaded machine ? > > C) Estimated duration of test > > etc. > - These are important points and to be clear based on discussion prior to and up to the vendor summit, the goal was to provide deterministic unittests for the first prototype (which covers point A and implicitly point B); the goal from an end-user perspective as to how they would be affected today is that unless make test[kernel,world] was actually called, it would just built and install the tests as necessary - Point C cannot be accurately answered as it depends on what tests are are in a tree, run via `make test[kernel,world]`, etc. (going into B) further) Other items such as fault tolerance, stress, etc need to be worked into the overall test ecosystem of FreeBSD (I recognize the need for integrating pho's stress2 project into FreeBSD proper so it's run on a regular basis, the need for adding to stress2, etc), but it wasn't in the scope of the initial work being done for the ATF FreeBSD port / in the short to medium term for test automation on FreeBSD and as such hasn't been planned in the overall "automation roadmap". Going further, as far as items B/C are concerned I would argue that they are highly dependent upon the test matter, how the testcases are implemented, etc. From my perspective the goal is to have (from a functional perspective) simple testcases that exercise functionality to ensure that requirements noted in manpages (because that's the user facing documentation) are adhered to and not regressed, and then move on to other more critical items requiring attention (again, stress, fault tolerance, etc). Less deterministic items need to be factored out of standard runs and/or decorated (like how one can do in python) such that they aren't necessarily executed by default, but rather when explicitly requested, either as individual tests or as a group of tests. As the phrase goes, "Rome wasn't built in a day". Thanks, -Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGH67wRFyFrkZWxGEQBA%2BY7e1v7JQoFiQyEp0w2qzrtU4KWuyQ>