Date: Fri, 14 Mar 2014 16:04:29 -0600 From: Alan Somers <asomers@freebsd.org> To: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, "Kenneth D. Merry" <ken@freebsd.org> Subject: ATF config variables for FreeBSD tests Message-ID: <CAOtMX2j8d-AQXHbmmkOca9tV-5mfEjFvBnwZMPwXHcqmgGgtDw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Another thread recently raised the issue of tests that cause potentially harmful changes to sysctl variables. We all agreed to conditionalize that test based on an ATF config variable. Now Ken and I both want to add tests with different harmful side effects. I propose the following list of config variables. If they sound good to everyone, I'll add them to tests(7) and to an example kyua.conf file in devel/kyua. test_suites.FreeBSD.allow_sysctl_side_effects If defined, enables tests that cause potentially harmful changes to globally relevant sysctl variables. Requested by Peter for kern_descrip_test:maxfiles__increase test_suites.FreeBSD.fibs Defined to a space delimited list of fib numbers. Tests that change routing tables must use these fibs instead of the default. They will be skipped if the variable is undefined. I'm working on such tests right now. test_suites.FreeBSD.disks Defined to a space delimited list of disk device nodes. Tests that format, mount, write to, disable, or have any other harmful effect on disks must use these devices. test_suites.FreeBSD.allow_devfs_side_effects. If defined, enables tests that can create or destroy semipermanent device nodes, like disk devices. Without this variable, tests may still create and destroy device nodes that are normally transient, like /dev/tap* and /dev/pts/*. This variable requires further explanation: Naively, one would think that test_suites.FreeBSD.disks would be sufficient protection for tests that create and destroy disk device nodes. However, Ken is working on a test that will cause hard drives to disappear and reappear in a different order, possibly resulting in their devnames changing. It could even result in "holes" appearing in the list of da unit numbers. If the sysadmin defines test_suites.FreeBSD.disks to "/dev/da1 /dev/da2 /dev/da3", it would be possible for those same devices to be named "/dev/da2 /dev/da1 /dev/da4" by the time Ken's test finishes. The permanent disappearance of /dev/da3 could confuse future tests that rely on test_suites.FreeBSD.disks. They would try to open /dev/da3 and fail. Thus, it may not be safe to run Ken's test in the same run as other tests that access disks. test_suites.FreeBSD.unit_tests test_suites.FreeBSD.functional_tests test_suites.FreeBSD.stress_tests These three variables would address a different concern of Peter's. He was concerned that his test might require too much run time, annoying other users of the rest of the test suite. By using these variables, a sysadmin could easily disable stress tests, which tend to take a very long time to run. Conversely, unit tests are normally very fast. A kyua run that only ran unit tests could easily be done after every buildworld, for example, while one that ran functional tests as well could be done perhaps several times per day. This is just an idea I have. I haven't tried it in practice. What do you think? -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2j8d-AQXHbmmkOca9tV-5mfEjFvBnwZMPwXHcqmgGgtDw>