Date: Sun, 16 Mar 2014 09:53:42 -0700 From: Garrett Cooper <yaneurabeya@gmail.com> To: Alan Somers <asomers@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, "Kenneth D. Merry" <ken@freebsd.org> Subject: Re: ATF config variables for FreeBSD tests Message-ID: <6BADC2BD-3C99-44F7-B6FB-6139AB8752D9@gmail.com> In-Reply-To: <CAOtMX2g2B2gbTwsuR-LGM-wi_fKzGfL%2B24yL4e0W1SEOi1gdxg@mail.gmail.com> References: <CAOtMX2j8d-AQXHbmmkOca9tV-5mfEjFvBnwZMPwXHcqmgGgtDw@mail.gmail.com> <CAFY7cWC3HR_1sxjrL9ZU=xZxtonGDz3K_QJx8LZA4G4%2BOXTVNQ@mail.gmail.com> <CAOtMX2iY=Xtck3UAco-U_1r55yA2NMX0q0zoMuhPMoFi1iwVCQ@mail.gmail.com> <CAFY7cWCZ-e=251PB%2BDbG-F6UEsBNbZLexdnqoLuonL7hVcjXsg@mail.gmail.com> <CAOtMX2g2B2gbTwsuR-LGM-wi_fKzGfL%2B24yL4e0W1SEOi1gdxg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Mar 16, 2014, at 8:56, Alan Somers <asomers@freebsd.org> wrote: > >> On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino <jmmv@freebsd.org> wrote: >>> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers <asomers@freebsd.org> wrote: >>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv@freebsd.org> wrote: >> [...] >>>> I think these all sound reasonable. >>>> >>>> Can these expected side-effects be reversed in the test cleanup? >>> >>> For the first two, yes. I doubt that Kyua could do it automatically, >>> but each test case certainly can. >> >> Right. I do not think Kyua should get into these tricky details >> either, if only because they are too OS-specific. But we can certainly >> offer additional scripts/functions (where it makes sense) in the >> FreeBSD test suite to simplify the test cases that might need this. > > Definitely. This is a good case for a library of reusable test code. > For example, I have two separate test programs that would like to > share setup and cleanup code for FIBs. Should I put it in > /usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ? I don't like > those options, because it looks like fibs.sh is a test program > designed to test stuff in /include or /lib. Perhaps > /usr/tests/tests/sh/include/fibs.sh? I'm really not sure where the > best place would be. What do you think? /usr/tests/libexec seems logical.. >> >>> Perhaps even for the fourth, though >>> it would be tricky. But there's no way to reverse the effects for the >>> third. The ZFS tests, for example, create and destroy zpools. How >>> would you reverse that? You can't. Whatever was previously on the >>> disk is gone. >> >> Ah, I guess I missed that detail. >> >> Couldn't those tests run on top of vnd devices though? > > FreeBSD doesn't have vnd(4), though it has md(4), which is similar. > md(4) devices could be used for some ZFS tests. as it happens, ZFS > can also use file-backed vdevs. But these workarounds don't work in > all cases. Sometimes you need real disks: > 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've > one a lot of work. > 2) Neither file-backed vdevs nor md(4) devices have physical path > information, which is needed to test some hotspare functionality. > Only da(4) devices that are attached to ses(4) expanders have that. > 3) There is no way to remove an active file-backed vdev, and I don't > think that you can destroy an in-use md(4) device either. Therefore, > to test how ZFS handles drive removals requires real drives. > 4) gibbs has made many tweaks to ZFS to better support 4K and 8K > sector drives. There is no way to emulate those with file-backed > vdevs. Perhaps it could be done with md(4) and gnop(8), but I've > never tried. > 5) pjd and Steve Hartland have been working on TRIM support. That's > not supported by file-backed vdevs, and I doubt that md(4) supports it > either. > 6) Copy-on-write on top of copy-on-write is very slow. If your system > uses ZFS root, then both file-backed vdevs and file-backed md(4) > devices are doing COW-on-COW. Using physical disks for the tests > greatly speeds the tests' runtimes. > > Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is > defined, and use file-backed vdevs otherwise. Tests that absolutely > require physical disks would be skipped if test_suites.FreeBSD.disks > isn't defined. It would take some work, but it's doable. > >> >> [...] >>>> And lastly, we'd just need a simple "filtering" feature in the kyua >>>> cli to allow specifying which size of tests to run (or to filter by >>>> any other metadata property, for that matter). >>> >>> I don't like this idea. >> >> I hope you are objecting to the filtering by test sizes, not the >> filtering itself! See below. >> >>> We tried it at work, and it didn't work out >>> very well. Basically, I automatically assigned sizes (short, medium, >>> long) to all of our tests based on their runtimes, so users could >>> select to only run the short or medium tests on the bench. The >>> problem is that the classification didn't make sense. For an >>> expedited test run, you don't want the shortest tests; what you want >>> are the tests with the most value per unit time. Unit tests have a >>> high value and a very short runtime. Stress tests have a very long >>> runtime and arguably low value since they don't always have consistent >>> results. >>> >>> At my previous job there was a large department whose duties included >>> curating test suites and deciding which tests would be included in the >>> short runs. FreeBSD isn't going to have that, but we could still ask >>> test authors to classify tests along the lines that I suggested. >> >> That's a good point. If I understand you correctly, what is important >> is to manually curate the set of "smoke tests" that run very quickly >> and that provide a reasonably high assurance that major subsystems >> haven't broken. >> >> We can do this too pretty easily, but configuration variables are the >> wrong mechanism. We can already define this information in metadata >> properties as long as you prefix them with X-. So all we'd need is a >> way to tell Kyua to only run tests marked with such property (i.e. >> with the filtering feature described above). > > I was assuming that a test case could put in its header > 'require.config "stress_tests"', or something like that. Then runtime > filtering would work by which variables the user defines. That way, > no new work would be required in Kyua. But it sounds like you're > talking about something different. IIUC, you're suggesting that a > test case would say in its header 'atf_set "X-runtime" "stress_test"' > and then some TBD filtering mechanism in Kyua selects on it. That > would certainly be more intuitive. Is that what you had in mind? > > -Alan > _______________________________________________ > freebsd-testing@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-testing > To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6BADC2BD-3C99-44F7-B6FB-6139AB8752D9>