From owner-freebsd-testing@FreeBSD.ORG Sun Mar 16 15:56:57 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1221DEE6; Sun, 16 Mar 2014 15:56:57 +0000 (UTC) Received: from mail-wg0-x22c.google.com (mail-wg0-x22c.google.com [IPv6:2a00:1450:400c:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 29BCF807; Sun, 16 Mar 2014 15:56:56 +0000 (UTC) Received: by mail-wg0-f44.google.com with SMTP id m15so3792641wgh.27 for ; Sun, 16 Mar 2014 08:56:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ytTKLYPhs8HOtoVzN6ggFErkPQrZbV0AN+/IY94BrhE=; b=rbEb0tGqZe1X/oUKgxSWByRWyl2/Q2J8qRd6VHJyEhgi9l6ACtiyYTzfaxO3+sTSz4 dnPNOqvT4gWFT7TlLHLnrAvc2vJzkkt2vIJWbwyNPTcBRpr4YO7zXTtVaxLEvHy3hcsH sJxDeMd2VwON+Z101jZctYbdrlsvex0FW1C37Llr61i+G0LMq17pF1Hsm2w7fNJfWcTo HM5eBSGx58Pj0WZbsZZ0QuHBJ7Crbag9V6Ij1I+7ul0IkU8CgUeBYGBeu2mO7q6BcDgh Msc2Sm03x1//HWxTH9gRhCx+pq2PxMBlfE58XLJUJbphFxyMBGgch1uU35X4YuYNZhuE yF1w== MIME-Version: 1.0 X-Received: by 10.180.106.167 with SMTP id gv7mr6255229wib.40.1394985414558; Sun, 16 Mar 2014 08:56:54 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.168.130 with HTTP; Sun, 16 Mar 2014 08:56:54 -0700 (PDT) In-Reply-To: References: Date: Sun, 16 Mar 2014 09:56:54 -0600 X-Google-Sender-Auth: dhHCmejZiFk5mFKPmpUC0zLz9X0 Message-ID: Subject: Re: ATF config variables for FreeBSD tests From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" , "Kenneth D. Merry" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 15:56:57 -0000 On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino wrote: > On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers wrote: >> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino wrote: > [...] >>> I think these all sound reasonable. >>> >>> Can these expected side-effects be reversed in the test cleanup? >> >> For the first two, yes. I doubt that Kyua could do it automatically, >> but each test case certainly can. > > Right. I do not think Kyua should get into these tricky details > either, if only because they are too OS-specific. But we can certainly > offer additional scripts/functions (where it makes sense) in the > FreeBSD test suite to simplify the test cases that might need this. Definitely. This is a good case for a library of reusable test code. For example, I have two separate test programs that would like to share setup and cleanup code for FIBs. Should I put it in /usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ? I don't like those options, because it looks like fibs.sh is a test program designed to test stuff in /include or /lib. Perhaps /usr/tests/tests/sh/include/fibs.sh? I'm really not sure where the best place would be. What do you think? > >> Perhaps even for the fourth, though >> it would be tricky. But there's no way to reverse the effects for the >> third. The ZFS tests, for example, create and destroy zpools. How >> would you reverse that? You can't. Whatever was previously on the >> disk is gone. > > Ah, I guess I missed that detail. > > Couldn't those tests run on top of vnd devices though? FreeBSD doesn't have vnd(4), though it has md(4), which is similar. md(4) devices could be used for some ZFS tests. as it happens, ZFS can also use file-backed vdevs. But these workarounds don't work in all cases. Sometimes you need real disks: 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've one a lot of work. 2) Neither file-backed vdevs nor md(4) devices have physical path information, which is needed to test some hotspare functionality. Only da(4) devices that are attached to ses(4) expanders have that. 3) There is no way to remove an active file-backed vdev, and I don't think that you can destroy an in-use md(4) device either. Therefore, to test how ZFS handles drive removals requires real drives. 4) gibbs has made many tweaks to ZFS to better support 4K and 8K sector drives. There is no way to emulate those with file-backed vdevs. Perhaps it could be done with md(4) and gnop(8), but I've never tried. 5) pjd and Steve Hartland have been working on TRIM support. That's not supported by file-backed vdevs, and I doubt that md(4) supports it either. 6) Copy-on-write on top of copy-on-write is very slow. If your system uses ZFS root, then both file-backed vdevs and file-backed md(4) devices are doing COW-on-COW. Using physical disks for the tests greatly speeds the tests' runtimes. Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is defined, and use file-backed vdevs otherwise. Tests that absolutely require physical disks would be skipped if test_suites.FreeBSD.disks isn't defined. It would take some work, but it's doable. > > [...] >>> And lastly, we'd just need a simple "filtering" feature in the kyua >>> cli to allow specifying which size of tests to run (or to filter by >>> any other metadata property, for that matter). >> >> I don't like this idea. > > I hope you are objecting to the filtering by test sizes, not the > filtering itself! See below. > >> We tried it at work, and it didn't work out >> very well. Basically, I automatically assigned sizes (short, medium, >> long) to all of our tests based on their runtimes, so users could >> select to only run the short or medium tests on the bench. The >> problem is that the classification didn't make sense. For an >> expedited test run, you don't want the shortest tests; what you want >> are the tests with the most value per unit time. Unit tests have a >> high value and a very short runtime. Stress tests have a very long >> runtime and arguably low value since they don't always have consistent >> results. >> >> At my previous job there was a large department whose duties included >> curating test suites and deciding which tests would be included in the >> short runs. FreeBSD isn't going to have that, but we could still ask >> test authors to classify tests along the lines that I suggested. > > That's a good point. If I understand you correctly, what is important > is to manually curate the set of "smoke tests" that run very quickly > and that provide a reasonably high assurance that major subsystems > haven't broken. > > We can do this too pretty easily, but configuration variables are the > wrong mechanism. We can already define this information in metadata > properties as long as you prefix them with X-. So all we'd need is a > way to tell Kyua to only run tests marked with such property (i.e. > with the filtering feature described above). I was assuming that a test case could put in its header 'require.config "stress_tests"', or something like that. Then runtime filtering would work by which variables the user defines. That way, no new work would be required in Kyua. But it sounds like you're talking about something different. IIUC, you're suggesting that a test case would say in its header 'atf_set "X-runtime" "stress_test"' and then some TBD filtering mechanism in Kyua selects on it. That would certainly be more intuitive. Is that what you had in mind? -Alan