From owner-freebsd-hackers@FreeBSD.ORG Tue Jan 7 18:08:17 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B670FCE8 for ; Tue, 7 Jan 2014 18:08:17 +0000 (UTC) Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3D7591FDA for ; Tue, 7 Jan 2014 18:08:16 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id y6so540984lbh.19 for ; Tue, 07 Jan 2014 10:08:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=0VFf1abSTpCb30sqddgD6xifZdF2M+gOcmcUByMBYZQ=; b=BwaOvPymutLYokqyOJlnLNe02aO+33+OXL5YiXMVbuaeHpcs/VVDL3JwvzeWbLYGqn rnil4O9odGkujYYBRTCaqYP0ywOXbyaLDChmeYSf8QlMdHoZMwauYZSzqM7XtpEvuFB7 OfuBAPu4dNA8rBlB0bX0194argSafC66ajK6kOGII0KiXOvEaaDXiIKwGE7Dik8JxpML RHNjyonEP46Ki1YeBOHAqfRh4NRS6RVyiGk2LrP74vuubIw7UZ3uaYRkzZRxezN0BuIG yEQz5k8jjjUPq34gsZh4qlPIUQwKMRH4M/vm/m14JjUMFOrnaRekn1LhZ+nRdNbbbFYc /KZw== X-Gm-Message-State: ALoCoQk+ow3xWj2h+IuYn8E0DMuPET+530Mr/eSqDSkjNw/g3SlDF9CJ++udtW/0gFptiISZ+6zz X-Received: by 10.152.4.162 with SMTP id l2mr3264124lal.75.1389118088651; Tue, 07 Jan 2014 10:08:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.129.101 with HTTP; Tue, 7 Jan 2014 10:07:48 -0800 (PST) X-Originating-IP: [2620:0:1040:406:9098:ad1a:255:209e] In-Reply-To: References: From: Julio Merino Date: Tue, 7 Jan 2014 18:07:48 +0000 Message-ID: Subject: Re: Continual benchmarking / regression testing? To: Ivan Voras Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jan 2014 18:08:17 -0000 On Tue, Jan 7, 2014 at 4:17 PM, Ivan Voras wrote: > On 7 January 2014 17:11, Julio Merino wrote: >> On Tue, Jan 7, 2014 at 4:09 PM, Ivan Voras wrote: >>> Hello, >>> >>> Is someone working on a contitual benchmarking / regression testing >>> project for FreeBSD? I seem to recall there was a post several months >>> ago but I can't find it. >> >> See http://wiki.freebsd.org/TestSuite for the current efforts. > > Ok, by looking at the wiki page and http://kyua1.nyi.freebsd.org/ > (only stable/10 is available btw), it looks like this effort is > centered on correctness, not performance benchmarking? Correct. I have basically no thoughts at the moment on benchmarking, but that's certainly something worth tackling. My email crossed Alan's, but here go my thoughts anyway. I feel integrating some basic kind of performance testing into the test suite might be beneficial if only to catch regressions, but detailed performance testing may be difficult for all the reasons mentioned by Alan. A simple possibility could be to explicitly mark specific tests as "benchmark tests" so that Kyua could measure and record their run time. In fact, Kyua already records the run time of tests and maintains historical data. What would be missing is a way to graph the results and to alert when the measurements differ above some thresholds. But then, benchmarking tests will have special requirements -- particularly during the setup of dependencies, the setup of the machine (to ensure there is no background noise) and also due to the large amount of tunables that may be involved. Plugging such tests into a correctness test suite, except for simple tests, is hard and may be not such a great idea. > Is Kyua easily > adaptable to include graphs and other more visually attractive > presentation types? Not at the moment, but better reporting is the thing I want to tackle the soonest. See the planning details (http://julipedia.meroh.net/2014/01/freebsd-test-suite-goals-and-planning.html) for some more information. In particular, visit the "test matrix" sheet of the planning spreadsheet. But again, there is nothing there regarding performance testing. -- Julio Merino / @jmmv