From owner-freebsd-hackers@FreeBSD.ORG Tue Jan 7 19:41:31 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3EB2B777; Tue, 7 Jan 2014 19:41:31 +0000 (UTC) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A68B216F9; Tue, 7 Jan 2014 19:41:30 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id hn9so4616885wib.6 for ; Tue, 07 Jan 2014 11:41:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=nHpq0mTCEL+Mq4YHYVGtQR0MFTs8aXQsHexl777Ruu0=; b=bG3KWIPeQPZbey2x4c7EkW3LP2TwGOXBIvAf7PH+VQuNmtDYvl/QrlBkf+m1PYFR6z 1NFqJvc7GFVXEy9XLFJzg5AEKS1bOk5+0o+CCO26QUyDCOLpOFOXz5zqo6SBnik0NPQb okTJknCpLg1xjaqYGPKpupiKgkgK4VS76MofJLb6/Ubjc6bE/e+GZTkcEfmsMAkNIv+y gSbiP2tH08SOvTfVpzwDfTRAgacQy7RMwtFhhElLWMCmSQJA5tblqd/X8UeiRPErhwHC KH0EkjMwEgXWNiq5ahvrRCUBCS4In3Zzyrdcv6eEi3sfdaTsQs27q0dTKYC0AdHxRnNu +qnw== MIME-Version: 1.0 X-Received: by 10.180.37.69 with SMTP id w5mr18142482wij.53.1389123689184; Tue, 07 Jan 2014 11:41:29 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.22.35 with HTTP; Tue, 7 Jan 2014 11:41:29 -0800 (PST) In-Reply-To: References: Date: Tue, 7 Jan 2014 12:41:29 -0700 X-Google-Sender-Auth: Cn6Up_6e3aKYIjXPj5LCSB6421E Message-ID: Subject: Re: Continual benchmarking / regression testing? From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers , Ivan Voras X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 19:41:31 -0000 On Tue, Jan 7, 2014 at 11:07 AM, Julio Merino wrote: > On Tue, Jan 7, 2014 at 4:17 PM, Ivan Voras wrote: >> On 7 January 2014 17:11, Julio Merino wrote: >>> On Tue, Jan 7, 2014 at 4:09 PM, Ivan Voras wrote: >>>> Hello, >>>> >>>> Is someone working on a contitual benchmarking / regression testing >>>> project for FreeBSD? I seem to recall there was a post several months >>>> ago but I can't find it. >>> >>> See http://wiki.freebsd.org/TestSuite for the current efforts. >> >> Ok, by looking at the wiki page and http://kyua1.nyi.freebsd.org/ >> (only stable/10 is available btw), it looks like this effort is >> centered on correctness, not performance benchmarking? > > Correct. I have basically no thoughts at the moment on benchmarking, > but that's certainly something worth tackling. > > My email crossed Alan's, but here go my thoughts anyway. I feel > integrating some basic kind of performance testing into the test suite > might be beneficial if only to catch regressions, but detailed > performance testing may be difficult for all the reasons mentioned by > Alan. For a good example of performance regression testing, see the Linux kernel tracker at http://www.phoromatic.com/kernel-tracker.php? . As you can see, the hardware and software profiling is weak, and regression detection is very inexact. It wouldn't be possible to distill a single benchmark's results into a "regressed"/"didn't regress" result. Accurately detecting regressions would probably take some moderately sophisticated statistics. > > A simple possibility could be to explicitly mark specific tests as > "benchmark tests" so that Kyua could measure and record their run > time. In fact, Kyua already records the run time of tests and > maintains historical data. What would be missing is a way to graph > the results and to alert when the measurements differ above some > thresholds. > > But then, benchmarking tests will have special requirements -- > particularly during the setup of dependencies, the setup of the > machine (to ensure there is no background noise) and also due to the > large amount of tunables that may be involved. Plugging such tests > into a correctness test suite, except for simple tests, is hard and > may be not such a great idea. Yeah, that's basically the same conclusion I came to. > >> Is Kyua easily >> adaptable to include graphs and other more visually attractive >> presentation types? > > Not at the moment, but better reporting is the thing I want to tackle > the soonest. See the planning details > (http://julipedia.meroh.net/2014/01/freebsd-test-suite-goals-and-planning.html) > for some more information. In particular, visit the "test matrix" > sheet of the planning spreadsheet. But again, there is nothing there > regarding performance testing. So is Julipedia now the best source for Kyua news instead of http://engineering-kyua.blogspot.com/ ? I have the latter in my RSS feed, but it hasn't updated for awhile. -Alan > > -- > Julio Merino / @jmmv > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"