Date: Thu, 15 Nov 2012 14:06:52 -0800 From: Marcel Moolenaar <marcel@xcllnt.net> To: Poul-Henning Kamp <phk@phk.freebsd.dk> Cc: Garrett Cooper <yanegomi@gmail.com>, George Neville-Neil <gnn@freebsd.org>, Matthew Fleming <mdf@freebsd.org>, "freebsd-arch@FreeBSD.org Arch" <freebsd-arch@freebsd.org> Subject: Re: [RFC] test layout/standardization for FreeBSD Message-ID: <F3302426-CA5D-4B61-8F3A-081395B393D9@xcllnt.net> In-Reply-To: <7099.1352886181@critter.freebsd.dk> References: <CAGH67wSoU08N8=40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com> <7099.1352886181@critter.freebsd.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Nov 14, 2012, at 1:43 AM, Poul-Henning Kamp <phk@phk.freebsd.dk> = wrote: > -------- > In message = <CAGH67wSoU08N8=3D40RE3j0na4B6-VhZFwhMdkw-6CYhoxKKHumg@mail.gmail.com> > , Garrett Cooper writes: >=20 >> I asked for feedback from some of the stakeholders of >> the FreeBSD test automation effort [...] >=20 > Can I throw in something here ? >=20 > A very important thing is to have systematic metadata about = test-cases, > in particular: >=20 > A) Is the test 100% deterministic, or is there a risk of random = failure ? >=20 > B) Is the test reliable on a heavily loaded machine ? >=20 > C) Estimated duration of test I can't disagree, but I would argue that it's not more important than having testcases. I'm not trying to be flippant, but we need to get off the ground first before we try to retract the landing gear. I would argue that if we build this stuff out, we either hit the problems that meta data would solve or we don't. If we do, we also have real examples to work with. If we don't then we didn't "waste" time. Since at this point in time it's just an academic exercise that doesn't yield anything concrete, we're more likely than not wasting time. For example: If a test is not reliable on a heavily loaded machine, then the test is ipso facto not 100% deterministic. So, B implies A. How is this different from a meta data perspective? If we test a RNG, are we ever going to be 100% deterministic? Also, the estimated duration for tests is very platform specific. What exactly does it mean when the actual runtime is shorter or longer than the estimated time? What's the error margin we're comfortable with? Why that error margin? Did the test FAIL if it took too long or ran too short? If not, then what is it for? In short: I'm not sure we can discuss anything concrete just yet. So let's wait until we can. I don't expect a lot of difficulty collecting statistics and data. So let's do that first. The easy cases will show up soon I think and we'll sort those out first. Think functional tests vs performance tests and white-box vs black-box. Eventually we'll have buckets and we'll find that meta data is a lot easier to define and use on a per bucket basis.... Maybe not... Let's just see... $0.02 --=20 Marcel Moolenaar marcel@xcllnt.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F3302426-CA5D-4B61-8F3A-081395B393D9>