From owner-freebsd-testing@FreeBSD.ORG Sun Mar 16 16:53:44 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8DF4AE22; Sun, 16 Mar 2014 16:53:44 +0000 (UTC) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4BA3EC87; Sun, 16 Mar 2014 16:53:44 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id bj1so4736877pad.2 for ; Sun, 16 Mar 2014 09:53:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=sI9rv95pwKNcCdPNNJmDLmDI4KduVGQH/cWAuZPmwhw=; b=KPFA9rsTTHo2jhHHoRW885o8Io0DNySr9qcfsNySJtDKsrXrhcE8HvsZ1mpnQCwomN efIjAs4GQPBK6gdSQJna7QzF6YmViPs71GuROQMrz4GUWJvgn87PupzrrIIz088p9+Uq J52K3zRBtJBD9RposJtbJFlsEjshZP+hwnmxh33uX5Zxa0rP07F1fFluntM3rX7HCIMe fWIdKZpziVMtHW5IQh9zXFZmnSWmZXS0aHyrI7XkzGs+ds7JHgFPCFxLPnZ3tMr2Ksej 18fQudU4g1b/UjiFhHyhWNUwEKP5fyJYD+OL5HDj+6jfEszdz8rw4Yk8J4DslQOv3sg/ TZ+w== X-Received: by 10.68.201.67 with SMTP id jy3mr21134006pbc.20.1394988823938; Sun, 16 Mar 2014 09:53:43 -0700 (PDT) Received: from [192.168.20.11] (c-24-17-226-153.hsd1.wa.comcast.net. [24.17.226.153]) by mx.google.com with ESMTPSA id dk1sm35711721pbc.46.2014.03.16.09.53.42 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 16 Mar 2014 09:53:42 -0700 (PDT) References: Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <6BADC2BD-3C99-44F7-B6FB-6139AB8752D9@gmail.com> X-Mailer: iPhone Mail (11B651) From: Garrett Cooper Subject: Re: ATF config variables for FreeBSD tests Date: Sun, 16 Mar 2014 09:53:42 -0700 To: Alan Somers Cc: "freebsd-testing@freebsd.org" , "Kenneth D. Merry" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 16:53:44 -0000 > On Mar 16, 2014, at 8:56, Alan Somers wrote: > >> On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino wrote: >>> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers wrote: >>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino wrote: >> [...] >>>> I think these all sound reasonable. >>>> >>>> Can these expected side-effects be reversed in the test cleanup? >>> >>> For the first two, yes. I doubt that Kyua could do it automatically, >>> but each test case certainly can. >> >> Right. I do not think Kyua should get into these tricky details >> either, if only because they are too OS-specific. But we can certainly >> offer additional scripts/functions (where it makes sense) in the >> FreeBSD test suite to simplify the test cases that might need this. > > Definitely. This is a good case for a library of reusable test code. > For example, I have two separate test programs that would like to > share setup and cleanup code for FIBs. Should I put it in > /usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ? I don't like > those options, because it looks like fibs.sh is a test program > designed to test stuff in /include or /lib. Perhaps > /usr/tests/tests/sh/include/fibs.sh? I'm really not sure where the > best place would be. What do you think? /usr/tests/libexec seems logical.. >> >>> Perhaps even for the fourth, though >>> it would be tricky. But there's no way to reverse the effects for the >>> third. The ZFS tests, for example, create and destroy zpools. How >>> would you reverse that? You can't. Whatever was previously on the >>> disk is gone. >> >> Ah, I guess I missed that detail. >> >> Couldn't those tests run on top of vnd devices though? > > FreeBSD doesn't have vnd(4), though it has md(4), which is similar. > md(4) devices could be used for some ZFS tests. as it happens, ZFS > can also use file-backed vdevs. But these workarounds don't work in > all cases. Sometimes you need real disks: > 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've > one a lot of work. > 2) Neither file-backed vdevs nor md(4) devices have physical path > information, which is needed to test some hotspare functionality. > Only da(4) devices that are attached to ses(4) expanders have that. > 3) There is no way to remove an active file-backed vdev, and I don't > think that you can destroy an in-use md(4) device either. Therefore, > to test how ZFS handles drive removals requires real drives. > 4) gibbs has made many tweaks to ZFS to better support 4K and 8K > sector drives. There is no way to emulate those with file-backed > vdevs. Perhaps it could be done with md(4) and gnop(8), but I've > never tried. > 5) pjd and Steve Hartland have been working on TRIM support. That's > not supported by file-backed vdevs, and I doubt that md(4) supports it > either. > 6) Copy-on-write on top of copy-on-write is very slow. If your system > uses ZFS root, then both file-backed vdevs and file-backed md(4) > devices are doing COW-on-COW. Using physical disks for the tests > greatly speeds the tests' runtimes. > > Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is > defined, and use file-backed vdevs otherwise. Tests that absolutely > require physical disks would be skipped if test_suites.FreeBSD.disks > isn't defined. It would take some work, but it's doable. > >> >> [...] >>>> And lastly, we'd just need a simple "filtering" feature in the kyua >>>> cli to allow specifying which size of tests to run (or to filter by >>>> any other metadata property, for that matter). >>> >>> I don't like this idea. >> >> I hope you are objecting to the filtering by test sizes, not the >> filtering itself! See below. >> >>> We tried it at work, and it didn't work out >>> very well. Basically, I automatically assigned sizes (short, medium, >>> long) to all of our tests based on their runtimes, so users could >>> select to only run the short or medium tests on the bench. The >>> problem is that the classification didn't make sense. For an >>> expedited test run, you don't want the shortest tests; what you want >>> are the tests with the most value per unit time. Unit tests have a >>> high value and a very short runtime. Stress tests have a very long >>> runtime and arguably low value since they don't always have consistent >>> results. >>> >>> At my previous job there was a large department whose duties included >>> curating test suites and deciding which tests would be included in the >>> short runs. FreeBSD isn't going to have that, but we could still ask >>> test authors to classify tests along the lines that I suggested. >> >> That's a good point. If I understand you correctly, what is important >> is to manually curate the set of "smoke tests" that run very quickly >> and that provide a reasonably high assurance that major subsystems >> haven't broken. >> >> We can do this too pretty easily, but configuration variables are the >> wrong mechanism. We can already define this information in metadata >> properties as long as you prefix them with X-. So all we'd need is a >> way to tell Kyua to only run tests marked with such property (i.e. >> with the filtering feature described above). > > I was assuming that a test case could put in its header > 'require.config "stress_tests"', or something like that. Then runtime > filtering would work by which variables the user defines. That way, > no new work would be required in Kyua. But it sounds like you're > talking about something different. IIUC, you're suggesting that a > test case would say in its header 'atf_set "X-runtime" "stress_test"' > and then some TBD filtering mechanism in Kyua selects on it. That > would certainly be more intuitive. Is that what you had in mind? > > -Alan > _______________________________________________ > freebsd-testing@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-testing > To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.org"