From owner-freebsd-testing@FreeBSD.ORG Sat Mar 15 20:33:37 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7701F3F6; Sat, 15 Mar 2014 20:33:37 +0000 (UTC) Received: from mail-we0-x232.google.com (mail-we0-x232.google.com [IPv6:2a00:1450:400c:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8C7621D8; Sat, 15 Mar 2014 20:33:36 +0000 (UTC) Received: by mail-we0-f178.google.com with SMTP id u56so3254491wes.37 for ; Sat, 15 Mar 2014 13:33:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=PLEih5FhNMWiFasxt47/VMf68ZFC6mjnPCytZpzpzdQ=; b=Z+VF5DiKMcA25BTgp6CCquRhkvJCB9jmcdpPdPo/J1UKxZKUNtdvkHrZbeIjOK2pYs yMEvG4RHgOfBXteWx6urtmG+NTkZFzmu/JHkX7pvnyoZJ5Gap6mxZHpyK9xPaps/Quah j7C38h7JhJe9Ck+6ach+f0hPQ31lJZ9UYK3lSjlSG3u3hGpB4JBEz9z5UIbbbH+F5L2I naumGyoe+UsDNV3604Y8xDpa6wRnUM/8MD9bmguoNJk33C34wUSiuHrJIpr00E3KkWRV NJxnEkjgqbJxMj7uO1woRZxWieXUFfYt4ODVLl82mxxPP/OWm2LOQ+oxKpuFZ54cobxm Kn9g== MIME-Version: 1.0 X-Received: by 10.180.73.19 with SMTP id h19mr3474079wiv.40.1394915615059; Sat, 15 Mar 2014 13:33:35 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Sat, 15 Mar 2014 13:33:34 -0700 (PDT) In-Reply-To: References: Date: Sat, 15 Mar 2014 14:33:35 -0600 X-Google-Sender-Auth: NNmlc0xG_1BzXD2c__tdDz0fMUo Message-ID: Subject: Re: ATF config variables for FreeBSD tests From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" , "Kenneth D. Merry" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Mar 2014 20:33:37 -0000 On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino wrote: > On Sat, Mar 15, 2014 at 7:04 AM, Alan Somers wrote: >> Another thread recently raised the issue of tests that cause >> potentially harmful changes to sysctl variables. We all agreed to >> conditionalize that test based on an ATF config variable. Now Ken and >> I both want to add tests with different harmful side effects. I >> propose the following list of config variables. If they sound good to >> everyone, I'll add them to tests(7) and to an example kyua.conf file >> in devel/kyua. >> >> test_suites.FreeBSD.allow_sysctl_side_effects >> If defined, enables tests that cause potentially harmful changes to >> globally relevant sysctl variables. Requested by Peter for >> kern_descrip_test:maxfiles__increase >> >> test_suites.FreeBSD.fibs >> Defined to a space delimited list of fib numbers. Tests that change >> routing tables must use these fibs instead of the default. They will >> be skipped if the variable is undefined. I'm working on such tests >> right now. >> >> test_suites.FreeBSD.disks >> Defined to a space delimited list of disk device nodes. Tests that >> format, mount, write to, disable, or have any other harmful effect on >> disks must use these devices. >> >> test_suites.FreeBSD.allow_devfs_side_effects. >> If defined, enables tests that can create or destroy semipermanent >> device nodes, like disk devices. Without this variable, tests may >> still create and destroy device nodes that are normally transient, >> like /dev/tap* and /dev/pts/*. This variable requires further >> explanation: > [...] > > I think these all sound reasonable. > > Can these expected side-effects be reversed in the test cleanup? For the first two, yes. I doubt that Kyua could do it automatically, but each test case certainly can. Perhaps even for the fourth, though it would be tricky. But there's no way to reverse the effects for the third. The ZFS tests, for example, create and destroy zpools. How would you reverse that? You can't. Whatever was previously on the disk is gone. > > The descriptions above should clearly mention these details. For > example, for sysctl-related tests I'd expect them to restore the > original values just fine... but maybe that's not the case in the > other examples you mention? > >> test_suites.FreeBSD.unit_tests >> test_suites.FreeBSD.functional_tests >> test_suites.FreeBSD.stress_tests >> These three variables would address a different concern of Peter's. >> He was concerned that his test might require too much run time, >> annoying other users of the rest of the test suite. By using these >> variables, a sysadmin could easily disable stress tests, which tend to >> take a very long time to run. Conversely, unit tests are normally >> very fast. A kyua run that only ran unit tests could easily be done >> after every buildworld, for example, while one that ran functional >> tests as well could be done perhaps several times per day. This is >> just an idea I have. I haven't tried it in practice. >> >> What do you think? > > Doing this separation by sizes is important for the reasons you > outline, but I have a different proposal that I have had in mind for a > while and only recently started to work on. > > The idea is, simply, to introduce a "size" metadata variable to the > test cases and then make every test declare if they are "small", > "medium", "large", etc. With this, the user is allowed to customize, > in kyua.conf, the timeouts for every test case size -- with some > reasonable defaults for modern machines. (I've seen this implemented > in other test suites and is an approach that works pretty well.) > > The reason to do it this way is to kill the "timeout" property, which > is unreliable as it is not possible to make good predictions on run > time when you are going to run the same test on multiple > architectures. With this approach, we'd then say that qemu VMs have > much longer timeouts for every class, for example. This sounds good, as long as test authors have good guidelines on how to appraise their tests' sizes. > > And lastly, we'd just need a simple "filtering" feature in the kyua > cli to allow specifying which size of tests to run (or to filter by > any other metadata property, for that matter). I don't like this idea. We tried it at work, and it didn't work out very well. Basically, I automatically assigned sizes (short, medium, long) to all of our tests based on their runtimes, so users could select to only run the short or medium tests on the bench. The problem is that the classification didn't make sense. For an expedited test run, you don't want the shortest tests; what you want are the tests with the most value per unit time. Unit tests have a high value and a very short runtime. Stress tests have a very long runtime and arguably low value since they don't always have consistent results. At my previous job there was a large department whose duties included curating test suites and deciding which tests would be included in the short runs. FreeBSD isn't going to have that, but we could still ask test authors to classify tests along the lines that I suggested. -Alan