Date: Mon, 17 Mar 2014 09:14:10 -0600 From: Alan Somers <asomers@freebsd.org> To: Garrett Cooper <yaneurabeya@gmail.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, "Kenneth D. Merry" <ken@freebsd.org> Subject: Re: ATF config variables for FreeBSD tests Message-ID: <CAOtMX2ji7q23fW1idcOzvZGc=Q3UBO=6bY8OYhtiSr2s7UN7Jw@mail.gmail.com> In-Reply-To: <6BADC2BD-3C99-44F7-B6FB-6139AB8752D9@gmail.com> References: <CAOtMX2j8d-AQXHbmmkOca9tV-5mfEjFvBnwZMPwXHcqmgGgtDw@mail.gmail.com> <CAFY7cWC3HR_1sxjrL9ZU=xZxtonGDz3K_QJx8LZA4G4%2BOXTVNQ@mail.gmail.com> <CAOtMX2iY=Xtck3UAco-U_1r55yA2NMX0q0zoMuhPMoFi1iwVCQ@mail.gmail.com> <CAFY7cWCZ-e=251PB%2BDbG-F6UEsBNbZLexdnqoLuonL7hVcjXsg@mail.gmail.com> <CAOtMX2g2B2gbTwsuR-LGM-wi_fKzGfL%2B24yL4e0W1SEOi1gdxg@mail.gmail.com> <6BADC2BD-3C99-44F7-B6FB-6139AB8752D9@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Mar 16, 2014 at 10:53 AM, Garrett Cooper <yaneurabeya@gmail.com> wrote: > >> On Mar 16, 2014, at 8:56, Alan Somers <asomers@freebsd.org> wrote: >> >>> On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino <jmmv@freebsd.org> wrote: >>>> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers <asomers@freebsd.org> wrote: >>>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv@freebsd.org> wrote: >>> [...] >>>>> I think these all sound reasonable. >>>>> >>>>> Can these expected side-effects be reversed in the test cleanup? >>>> >>>> For the first two, yes. I doubt that Kyua could do it automatically, >>>> but each test case certainly can. >>> >>> Right. I do not think Kyua should get into these tricky details >>> either, if only because they are too OS-specific. But we can certainly >>> offer additional scripts/functions (where it makes sense) in the >>> FreeBSD test suite to simplify the test cases that might need this. >> >> Definitely. This is a good case for a library of reusable test code. >> For example, I have two separate test programs that would like to >> share setup and cleanup code for FIBs. Should I put it in >> /usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ? I don't like >> those options, because it looks like fibs.sh is a test program >> designed to test stuff in /include or /lib. Perhaps >> /usr/tests/tests/sh/include/fibs.sh? I'm really not sure where the >> best place would be. What do you think? > > /usr/tests/libexec seems logical.. But that's where tests for the stuff in /libexec go. If anything, I think /usr/libexec/tests would be more appropriate. Unfortunately, such files would share the same source location as the tests for the stuff in /libexec. But I think it's preferable to have a confusing source layout to a confusing installed layout. > >>> >>>> Perhaps even for the fourth, though >>>> it would be tricky. But there's no way to reverse the effects for the >>>> third. The ZFS tests, for example, create and destroy zpools. How >>>> would you reverse that? You can't. Whatever was previously on the >>>> disk is gone. >>> >>> Ah, I guess I missed that detail. >>> >>> Couldn't those tests run on top of vnd devices though? >> >> FreeBSD doesn't have vnd(4), though it has md(4), which is similar. >> md(4) devices could be used for some ZFS tests. as it happens, ZFS >> can also use file-backed vdevs. But these workarounds don't work in >> all cases. Sometimes you need real disks: >> 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've >> one a lot of work. >> 2) Neither file-backed vdevs nor md(4) devices have physical path >> information, which is needed to test some hotspare functionality. >> Only da(4) devices that are attached to ses(4) expanders have that. >> 3) There is no way to remove an active file-backed vdev, and I don't >> think that you can destroy an in-use md(4) device either. Therefore, >> to test how ZFS handles drive removals requires real drives. >> 4) gibbs has made many tweaks to ZFS to better support 4K and 8K >> sector drives. There is no way to emulate those with file-backed >> vdevs. Perhaps it could be done with md(4) and gnop(8), but I've >> never tried. >> 5) pjd and Steve Hartland have been working on TRIM support. That's >> not supported by file-backed vdevs, and I doubt that md(4) supports it >> either. >> 6) Copy-on-write on top of copy-on-write is very slow. If your system >> uses ZFS root, then both file-backed vdevs and file-backed md(4) >> devices are doing COW-on-COW. Using physical disks for the tests >> greatly speeds the tests' runtimes. >> >> Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is >> defined, and use file-backed vdevs otherwise. Tests that absolutely >> require physical disks would be skipped if test_suites.FreeBSD.disks >> isn't defined. It would take some work, but it's doable. >> >>> >>> [...] >>>>> And lastly, we'd just need a simple "filtering" feature in the kyua >>>>> cli to allow specifying which size of tests to run (or to filter by >>>>> any other metadata property, for that matter). >>>> >>>> I don't like this idea. >>> >>> I hope you are objecting to the filtering by test sizes, not the >>> filtering itself! See below. >>> >>>> We tried it at work, and it didn't work out >>>> very well. Basically, I automatically assigned sizes (short, medium, >>>> long) to all of our tests based on their runtimes, so users could >>>> select to only run the short or medium tests on the bench. The >>>> problem is that the classification didn't make sense. For an >>>> expedited test run, you don't want the shortest tests; what you want >>>> are the tests with the most value per unit time. Unit tests have a >>>> high value and a very short runtime. Stress tests have a very long >>>> runtime and arguably low value since they don't always have consistent >>>> results. >>>> >>>> At my previous job there was a large department whose duties included >>>> curating test suites and deciding which tests would be included in the >>>> short runs. FreeBSD isn't going to have that, but we could still ask >>>> test authors to classify tests along the lines that I suggested. >>> >>> That's a good point. If I understand you correctly, what is important >>> is to manually curate the set of "smoke tests" that run very quickly >>> and that provide a reasonably high assurance that major subsystems >>> haven't broken. >>> >>> We can do this too pretty easily, but configuration variables are the >>> wrong mechanism. We can already define this information in metadata >>> properties as long as you prefix them with X-. So all we'd need is a >>> way to tell Kyua to only run tests marked with such property (i.e. >>> with the filtering feature described above). >> >> I was assuming that a test case could put in its header >> 'require.config "stress_tests"', or something like that. Then runtime >> filtering would work by which variables the user defines. That way, >> no new work would be required in Kyua. But it sounds like you're >> talking about something different. IIUC, you're suggesting that a >> test case would say in its header 'atf_set "X-runtime" "stress_test"' >> and then some TBD filtering mechanism in Kyua selects on it. That >> would certainly be more intuitive. Is that what you had in mind? >> >> -Alan >> _______________________________________________ >> freebsd-testing@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-testing >> To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2ji7q23fW1idcOzvZGc=Q3UBO=6bY8OYhtiSr2s7UN7Jw>