Date: Tue, 7 Jan 2014 18:07:48 +0000 From: Julio Merino <julio@meroh.net> To: Ivan Voras <ivoras@freebsd.org> Cc: freebsd-hackers <freebsd-hackers@freebsd.org> Subject: Re: Continual benchmarking / regression testing? Message-ID: <CADyfeQXnrUx4eKk0mp1o7NK24KyiBeCtM6T5JOwUEhPy7H_3Yg@mail.gmail.com> In-Reply-To: <CAF-QHFU2bSA0-3CW%2BVNvqTHgXX%2BeLagD2NpVAyERbQj3heFWMg@mail.gmail.com> References: <lah8s3$8ur$1@ger.gmane.org> <CADyfeQUwmGnNVjExJGTwzTaTh9VgDgzcX0JNUvOcnpkZ7RK5gg@mail.gmail.com> <CAF-QHFU2bSA0-3CW%2BVNvqTHgXX%2BeLagD2NpVAyERbQj3heFWMg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jan 7, 2014 at 4:17 PM, Ivan Voras <ivoras@freebsd.org> wrote: > On 7 January 2014 17:11, Julio Merino <julio@meroh.net> wrote: >> On Tue, Jan 7, 2014 at 4:09 PM, Ivan Voras <ivoras@freebsd.org> wrote: >>> Hello, >>> >>> Is someone working on a contitual benchmarking / regression testing >>> project for FreeBSD? I seem to recall there was a post several months >>> ago but I can't find it. >> >> See http://wiki.freebsd.org/TestSuite for the current efforts. > > Ok, by looking at the wiki page and http://kyua1.nyi.freebsd.org/ > (only stable/10 is available btw), it looks like this effort is > centered on correctness, not performance benchmarking? Correct. I have basically no thoughts at the moment on benchmarking, but that's certainly something worth tackling. My email crossed Alan's, but here go my thoughts anyway. I feel integrating some basic kind of performance testing into the test suite might be beneficial if only to catch regressions, but detailed performance testing may be difficult for all the reasons mentioned by Alan. A simple possibility could be to explicitly mark specific tests as "benchmark tests" so that Kyua could measure and record their run time. In fact, Kyua already records the run time of tests and maintains historical data. What would be missing is a way to graph the results and to alert when the measurements differ above some thresholds. But then, benchmarking tests will have special requirements -- particularly during the setup of dependencies, the setup of the machine (to ensure there is no background noise) and also due to the large amount of tunables that may be involved. Plugging such tests into a correctness test suite, except for simple tests, is hard and may be not such a great idea. > Is Kyua easily > adaptable to include graphs and other more visually attractive > presentation types? Not at the moment, but better reporting is the thing I want to tackle the soonest. See the planning details (http://julipedia.meroh.net/2014/01/freebsd-test-suite-goals-and-planning.html) for some more information. In particular, visit the "test matrix" sheet of the planning spreadsheet. But again, there is nothing there regarding performance testing. -- Julio Merino / @jmmv
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADyfeQXnrUx4eKk0mp1o7NK24KyiBeCtM6T5JOwUEhPy7H_3Yg>