From owner-freebsd-testing@FreeBSD.ORG Thu Feb 27 17:20:32 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF9A3F52 for ; Thu, 27 Feb 2014 17:20:31 +0000 (UTC) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8B1F61DC5 for ; Thu, 27 Feb 2014 17:20:31 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id hi5so915845wib.1 for ; Thu, 27 Feb 2014 09:20:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/SXg85GMVfob3u14MLPkgcJPEI4NoCfmgrLjHZDfs34=; b=ZaxEfnG3FCu6IRC79/qacKeOkKVw6Fcaifa3Y99Bpka643GNNQ9f21+UyGxBbVB5sS 3ofuN6I+KiU9JtN4ADS82ilkdOTfHuvLngskgKgHndnMIjUdTcU1Bc9Hb1zreFcB6rwB CA4Tf9rDz13P+AcvVbsY92uN0Shn4L3gp511unX5HcQ2sidmSn62RV4UpcXY/Y4LdZ+D f+CPN/ljKCgAgx8n7AoitP0XQSZZ6rzM5CKYoN60I3xM5wvhsFJPxgnMUbatNrDbHlYe 4mphTfJvzpyvGfBddcKd2S7ExZPo1GqEFr4zI2zm4n+vp8DJ81jiuoH/zjhb3SflcoBf 99sQ== MIME-Version: 1.0 X-Received: by 10.194.10.162 with SMTP id j2mr2347109wjb.92.1393521629890; Thu, 27 Feb 2014 09:20:29 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Thu, 27 Feb 2014 09:20:29 -0800 (PST) In-Reply-To: <20140227081759.GA29517@x2.osted.lan> References: <20140227081759.GA29517@x2.osted.lan> Date: Thu, 27 Feb 2014 10:20:29 -0700 X-Google-Sender-Auth: pQjtxY1H7xqhbzBX6OcVvyVTIBk Message-ID: Subject: Re: 100% repeatability? From: Alan Somers To: Peter Holm Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Feb 2014 17:20:32 -0000 On Thu, Feb 27, 2014 at 1:17 AM, Peter Holm wrote: > I looked at creating a test scenario for the following piece of code > from src/sys/kern/kern_fork.c: > > 881 if ((nprocs >= maxproc - 10 && priv_check_cred(td->td_ucred, > 882 PRIV_MAXPROC, 0) != 0) || nprocs >= maxproc) { > 883 error = EAGAIN; > 884 goto fail; > 885 } > > I came up with a test scenario for this: > > http://people.freebsd.org/~pho/kern_fork_test.c > > But it is not 100% repeatable, as I would expect many other kernel > tests will be. > The test case runs as expected, until for example root decides to do a > parallel buildworld. > Is this a show stopper? First of all, let me say thank you for writing ATF tests. It's great that the community is starting to use it. As for the test: 1) nlistf and memf should be explicitly initialized, and may as well be private variables in getprocs(). 2) global variables are unfortunate. Would it be possible for getprocs() to return maxprocs and nprocs as return arguments instead of global variables? Alternatively, it could return (maxprocs - nprocs), since that's the only way they're used. 3) It's confusing to have a testcase with the same name as a global variable. It doesn't confuse the compiler, since ATF_TC() and friends mangle the name. But it does confuse cscope and humans. 4) Regarding repeatability: non-repeatability is very very bad. It leads to spurious failures appearing in kyua's results, which cause casual testers to ignore the entire suite. If they come to expect a few spurious failures, then they'll ignore real failures. Serious testers learn to ignore the intermittently failing tests, which isn't good either. It simply takes too much effort to debug each and every failure; people won't do it. I would not commit a test that isn't repeatable. If nprocs and maxproc were jail-specific variables, then you could simply run this test inside of a jail. However, they look global to me. I think that the only good way to test the feature would be to run the test inside of a private bhyve instance with an extremely minimal set of services. That way you could ensure that no unexpected processes would be started. It would be much more work, though. You would also need a way to communicate to the guest OS. Perhaps an emulated serial port? -Alan