Date: Tue, 7 Jan 2014 12:41:29 -0700 From: Alan Somers <asomers@freebsd.org> To: Julio Merino <julio@meroh.net> Cc: freebsd-hackers <freebsd-hackers@freebsd.org>, Ivan Voras <ivoras@freebsd.org> Subject: Re: Continual benchmarking / regression testing? Message-ID: <CAOtMX2gNF9%2BQDvLDYrtQg-_hpE2zCh5d6Q3b_U9cVTOg_rFRYw@mail.gmail.com> In-Reply-To: <CADyfeQXnrUx4eKk0mp1o7NK24KyiBeCtM6T5JOwUEhPy7H_3Yg@mail.gmail.com> References: <lah8s3$8ur$1@ger.gmane.org> <CADyfeQUwmGnNVjExJGTwzTaTh9VgDgzcX0JNUvOcnpkZ7RK5gg@mail.gmail.com> <CAF-QHFU2bSA0-3CW%2BVNvqTHgXX%2BeLagD2NpVAyERbQj3heFWMg@mail.gmail.com> <CADyfeQXnrUx4eKk0mp1o7NK24KyiBeCtM6T5JOwUEhPy7H_3Yg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jan 7, 2014 at 11:07 AM, Julio Merino <julio@meroh.net> wrote: > On Tue, Jan 7, 2014 at 4:17 PM, Ivan Voras <ivoras@freebsd.org> wrote: >> On 7 January 2014 17:11, Julio Merino <julio@meroh.net> wrote: >>> On Tue, Jan 7, 2014 at 4:09 PM, Ivan Voras <ivoras@freebsd.org> wrote: >>>> Hello, >>>> >>>> Is someone working on a contitual benchmarking / regression testing >>>> project for FreeBSD? I seem to recall there was a post several months >>>> ago but I can't find it. >>> >>> See http://wiki.freebsd.org/TestSuite for the current efforts. >> >> Ok, by looking at the wiki page and http://kyua1.nyi.freebsd.org/ >> (only stable/10 is available btw), it looks like this effort is >> centered on correctness, not performance benchmarking? > > Correct. I have basically no thoughts at the moment on benchmarking, > but that's certainly something worth tackling. > > My email crossed Alan's, but here go my thoughts anyway. I feel > integrating some basic kind of performance testing into the test suite > might be beneficial if only to catch regressions, but detailed > performance testing may be difficult for all the reasons mentioned by > Alan. For a good example of performance regression testing, see the Linux kernel tracker at http://www.phoromatic.com/kernel-tracker.php? . As you can see, the hardware and software profiling is weak, and regression detection is very inexact. It wouldn't be possible to distill a single benchmark's results into a "regressed"/"didn't regress" result. Accurately detecting regressions would probably take some moderately sophisticated statistics. > > A simple possibility could be to explicitly mark specific tests as > "benchmark tests" so that Kyua could measure and record their run > time. In fact, Kyua already records the run time of tests and > maintains historical data. What would be missing is a way to graph > the results and to alert when the measurements differ above some > thresholds. > > But then, benchmarking tests will have special requirements -- > particularly during the setup of dependencies, the setup of the > machine (to ensure there is no background noise) and also due to the > large amount of tunables that may be involved. Plugging such tests > into a correctness test suite, except for simple tests, is hard and > may be not such a great idea. Yeah, that's basically the same conclusion I came to. > >> Is Kyua easily >> adaptable to include graphs and other more visually attractive >> presentation types? > > Not at the moment, but better reporting is the thing I want to tackle > the soonest. See the planning details > (http://julipedia.meroh.net/2014/01/freebsd-test-suite-goals-and-planning.html) > for some more information. In particular, visit the "test matrix" > sheet of the planning spreadsheet. But again, there is nothing there > regarding performance testing. So is Julipedia now the best source for Kyua news instead of http://engineering-kyua.blogspot.com/ ? I have the latter in my RSS feed, but it hasn't updated for awhile. -Alan > > -- > Julio Merino / @jmmv > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2gNF9%2BQDvLDYrtQg-_hpE2zCh5d6Q3b_U9cVTOg_rFRYw>