Date: Thu, 27 Feb 2014 10:20:29 -0700 From: Alan Somers <asomers@freebsd.org> To: Peter Holm <peter@holm.cc> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: 100% repeatability? Message-ID: <CAOtMX2gE1TuAXOm7CdsG24PW-NSz5zgWHpxUN9p8dZVcCUs9%2Bw@mail.gmail.com> In-Reply-To: <20140227081759.GA29517@x2.osted.lan> References: <20140227081759.GA29517@x2.osted.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Feb 27, 2014 at 1:17 AM, Peter Holm <peter@holm.cc> wrote:
> I looked at creating a test scenario for the following piece of code
> from src/sys/kern/kern_fork.c:
>
> 881 if ((nprocs >= maxproc - 10 && priv_check_cred(td->td_ucred,
> 882 PRIV_MAXPROC, 0) != 0) || nprocs >= maxproc) {
> 883 error = EAGAIN;
> 884 goto fail;
> 885 }
>
> I came up with a test scenario for this:
>
> http://people.freebsd.org/~pho/kern_fork_test.c
>
> But it is not 100% repeatable, as I would expect many other kernel
> tests will be.
> The test case runs as expected, until for example root decides to do a
> parallel buildworld.
> Is this a show stopper?
First of all, let me say thank you for writing ATF tests. It's great
that the community is starting to use it. As for the test:
1) nlistf and memf should be explicitly initialized, and may as well
be private variables in getprocs().
2) global variables are unfortunate. Would it be possible for
getprocs() to return maxprocs and nprocs as
return arguments instead of global variables? Alternatively, it could
return (maxprocs - nprocs), since that's the only way they're used.
3) It's confusing to have a testcase with the same name as a global
variable. It doesn't confuse the compiler, since ATF_TC() and friends
mangle the name. But it does confuse cscope and humans.
4) Regarding repeatability: non-repeatability is very very bad. It
leads to spurious failures appearing in kyua's results, which cause
casual testers to ignore the entire suite. If they come to expect a
few spurious failures, then they'll ignore real failures. Serious
testers learn to ignore the intermittently failing tests, which isn't
good either. It simply takes too much effort to debug each and every
failure; people won't do it. I would not commit a test that isn't
repeatable.
If nprocs and maxproc were jail-specific variables, then you could
simply run this test inside of a jail. However, they look global to
me. I think that the only good way to test the feature would be to
run the test inside of a private bhyve instance with an extremely
minimal set of services. That way you could ensure that no unexpected
processes would be started. It would be much more work, though. You
would also need a way to communicate to the guest OS. Perhaps an
emulated serial port?
-Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2gE1TuAXOm7CdsG24PW-NSz5zgWHpxUN9p8dZVcCUs9%2Bw>