From owner-freebsd-testing@FreeBSD.ORG Thu Mar 20 17:16:21 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C304FA9; Thu, 20 Mar 2014 17:16:21 +0000 (UTC) Received: from mail-ve0-x232.google.com (mail-ve0-x232.google.com [IPv6:2607:f8b0:400c:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 91B6022E; Thu, 20 Mar 2014 17:16:20 +0000 (UTC) Received: by mail-ve0-f178.google.com with SMTP id jw12so1292803veb.37 for ; Thu, 20 Mar 2014 10:16:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=//kSXxTtoqSqclgvop1ADcZ8au5BgFRd2HbhiBf7aHk=; b=VDQGQnJnXg9bf2sgX4JraZV0cTBIEnxdRXXs0ce49I1z4nlst40i2rUYLm3VsrOOnr 0wA3TPDNGI6KihBzmJkpbnpG1Fsn5XiuZWpuB6yKdmjvVwwFA63jsP+Cp6KrgqjNypnL OdK3fZ5EBpRqNCzmGn20sjxk1G1VD8AsTgxoD2SI7VNScxZ+aHgpBUuPsdNBskoJ5xJh n/m8DqOAMkvNJlTUVzEShFe06KhFBHbRGfCZo31zaAxCaTKfNHiYQaLh8vg4H+MMSXuT maU2lmsWvtaDRkRSgVdI8LRqVp/HUsLIFAlgsF9SUND+1d9uv9fM20CFTXJMrHXNxlD9 xuSg== MIME-Version: 1.0 X-Received: by 10.221.29.196 with SMTP id rz4mr34665121vcb.8.1395335779688; Thu, 20 Mar 2014 10:16:19 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.58.231.105 with HTTP; Thu, 20 Mar 2014 10:16:19 -0700 (PDT) In-Reply-To: References: <6BADC2BD-3C99-44F7-B6FB-6139AB8752D9@gmail.com> Date: Thu, 20 Mar 2014 11:16:19 -0600 X-Google-Sender-Auth: 959S1sLS-u4gq87VvRV-GN9fv1c Message-ID: Subject: Re: ATF config variables for FreeBSD tests From: Alan Somers To: Alan Somers Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" , "Kenneth D. Merry" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 17:16:21 -0000 On Mon, Mar 17, 2014 at 9:14 AM, Alan Somers wrote: > On Sun, Mar 16, 2014 at 10:53 AM, Garrett Cooper wrote: >> >>> On Mar 16, 2014, at 8:56, Alan Somers wrote: >>> >>>> On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino wrote: >>>>> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers wrote: >>>>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino wrote: >>>> [...] >>>>>> I think these all sound reasonable. >>>>>> >>>>>> Can these expected side-effects be reversed in the test cleanup? >>>>> >>>>> For the first two, yes. I doubt that Kyua could do it automatically, >>>>> but each test case certainly can. >>>> >>>> Right. I do not think Kyua should get into these tricky details >>>> either, if only because they are too OS-specific. But we can certainly >>>> offer additional scripts/functions (where it makes sense) in the >>>> FreeBSD test suite to simplify the test cases that might need this. >>> >>> Definitely. This is a good case for a library of reusable test code. >>> For example, I have two separate test programs that would like to >>> share setup and cleanup code for FIBs. Should I put it in >>> /usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ? I don't like >>> those options, because it looks like fibs.sh is a test program >>> designed to test stuff in /include or /lib. Perhaps >>> /usr/tests/tests/sh/include/fibs.sh? I'm really not sure where the >>> best place would be. What do you think? >> >> /usr/tests/libexec seems logical.. > > But that's where tests for the stuff in /libexec go. If anything, I > think /usr/libexec/tests would be more appropriate. Unfortunately, > such files would share the same source location as the tests for the > stuff in /libexec. But I think it's preferable to have a confusing > source layout to a confusing installed layout. > >> >>>> >>>>> Perhaps even for the fourth, though >>>>> it would be tricky. But there's no way to reverse the effects for the >>>>> third. The ZFS tests, for example, create and destroy zpools. How >>>>> would you reverse that? You can't. Whatever was previously on the >>>>> disk is gone. >>>> >>>> Ah, I guess I missed that detail. >>>> >>>> Couldn't those tests run on top of vnd devices though? >>> >>> FreeBSD doesn't have vnd(4), though it has md(4), which is similar. >>> md(4) devices could be used for some ZFS tests. as it happens, ZFS >>> can also use file-backed vdevs. But these workarounds don't work in >>> all cases. Sometimes you need real disks: >>> 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've >>> one a lot of work. >>> 2) Neither file-backed vdevs nor md(4) devices have physical path >>> information, which is needed to test some hotspare functionality. >>> Only da(4) devices that are attached to ses(4) expanders have that. >>> 3) There is no way to remove an active file-backed vdev, and I don't >>> think that you can destroy an in-use md(4) device either. Therefore, >>> to test how ZFS handles drive removals requires real drives. >>> 4) gibbs has made many tweaks to ZFS to better support 4K and 8K >>> sector drives. There is no way to emulate those with file-backed >>> vdevs. Perhaps it could be done with md(4) and gnop(8), but I've >>> never tried. >>> 5) pjd and Steve Hartland have been working on TRIM support. That's >>> not supported by file-backed vdevs, and I doubt that md(4) supports it >>> either. >>> 6) Copy-on-write on top of copy-on-write is very slow. If your system >>> uses ZFS root, then both file-backed vdevs and file-backed md(4) >>> devices are doing COW-on-COW. Using physical disks for the tests >>> greatly speeds the tests' runtimes. >>> >>> Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is >>> defined, and use file-backed vdevs otherwise. Tests that absolutely >>> require physical disks would be skipped if test_suites.FreeBSD.disks >>> isn't defined. It would take some work, but it's doable. >>> >>>> >>>> [...] >>>>>> And lastly, we'd just need a simple "filtering" feature in the kyua >>>>>> cli to allow specifying which size of tests to run (or to filter by >>>>>> any other metadata property, for that matter). >>>>> >>>>> I don't like this idea. >>>> >>>> I hope you are objecting to the filtering by test sizes, not the >>>> filtering itself! See below. >>>> >>>>> We tried it at work, and it didn't work out >>>>> very well. Basically, I automatically assigned sizes (short, medium, >>>>> long) to all of our tests based on their runtimes, so users could >>>>> select to only run the short or medium tests on the bench. The >>>>> problem is that the classification didn't make sense. For an >>>>> expedited test run, you don't want the shortest tests; what you want >>>>> are the tests with the most value per unit time. Unit tests have a >>>>> high value and a very short runtime. Stress tests have a very long >>>>> runtime and arguably low value since they don't always have consistent >>>>> results. >>>>> >>>>> At my previous job there was a large department whose duties included >>>>> curating test suites and deciding which tests would be included in the >>>>> short runs. FreeBSD isn't going to have that, but we could still ask >>>>> test authors to classify tests along the lines that I suggested. >>>> >>>> That's a good point. If I understand you correctly, what is important >>>> is to manually curate the set of "smoke tests" that run very quickly >>>> and that provide a reasonably high assurance that major subsystems >>>> haven't broken. >>>> >>>> We can do this too pretty easily, but configuration variables are the >>>> wrong mechanism. We can already define this information in metadata >>>> properties as long as you prefix them with X-. So all we'd need is a >>>> way to tell Kyua to only run tests marked with such property (i.e. >>>> with the filtering feature described above). >>> >>> I was assuming that a test case could put in its header >>> 'require.config "stress_tests"', or something like that. Then runtime >>> filtering would work by which variables the user defines. That way, >>> no new work would be required in Kyua. But it sounds like you're >>> talking about something different. IIUC, you're suggesting that a >>> test case would say in its header 'atf_set "X-runtime" "stress_test"' >>> and then some TBD filtering mechanism in Kyua selects on it. That >>> would certainly be more intuitive. Is that what you had in mind? >>> >>> -Alan I committed the man page change for the four variables we agreed on. But I didn't add the runtime selection variables. It sounds like Julio has bigger plans for those, and they'll require upstream work on Kyua. -Alan