From owner-freebsd-testing@FreeBSD.ORG Sun Mar 16 00:49:39 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C25977F7 for ; Sun, 16 Mar 2014 00:49:39 +0000 (UTC) Received: from mail-qg0-f46.google.com (mail-qg0-f46.google.com [209.85.192.46]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7A7858BF for ; Sun, 16 Mar 2014 00:49:39 +0000 (UTC) Received: by mail-qg0-f46.google.com with SMTP id e89so12259232qgf.5 for ; Sat, 15 Mar 2014 17:49:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=gQwCcxawyts8g61fj+/emWXwFqLvyfjdKYiHTBvt5H0=; b=DR2SOc9hKFmzlRUZN0V3Ab+1jSrkTFRTTd/8AUxZMVWWQ2ZecqW3WlxRH5n07lnMzb nYtL9UEG4robQO1N8vh3Cgg3fwG7kw0IXIAKK9tgQ+v+hOStHFQYm98Ns5TDSUGSQSa2 oEBh3BOE1SX9Q/elbygClKKs/CdEprz1dOzxWNr3we4DT8T9K3Q9ij2dJ0EbuA13ibhb 2ECSnh/KTT5E8G74fjLhbAZfXtnh5ECKs9XiP/yAKbs4TqPSveJ1hn96TlagunRNlETZ eiqte7zZ8yBm8Ri9OVfcXO+tGQuKpbGwjhC61iQZ5DA6xR5BxosRHcsf1NPZikyG8VQf 26jQ== X-Gm-Message-State: ALoCoQl9EdEHfbxHtTGvWg+Sey5Ikhr8HuFGfoiCYD7+gLyr/DFKmeYqPuMLD+Ae1nviQIImasQh X-Received: by 10.224.137.202 with SMTP id x10mr19224qat.80.1394930972238; Sat, 15 Mar 2014 17:49:32 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.102 with HTTP; Sat, 15 Mar 2014 17:49:11 -0700 (PDT) X-Originating-IP: [180.42.49.96] In-Reply-To: References: From: Julio Merino Date: Sun, 16 Mar 2014 09:49:11 +0900 X-Google-Sender-Auth: jnvFkdmNPRsLnkY65Uzw4s1pbdI Message-ID: Subject: Re: ATF config variables for FreeBSD tests To: Alan Somers Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" , "Kenneth D. Merry" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2014 00:49:39 -0000 On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers wrote: > On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino wrote: [...] >> I think these all sound reasonable. >> >> Can these expected side-effects be reversed in the test cleanup? > > For the first two, yes. I doubt that Kyua could do it automatically, > but each test case certainly can. Right. I do not think Kyua should get into these tricky details either, if only because they are too OS-specific. But we can certainly offer additional scripts/functions (where it makes sense) in the FreeBSD test suite to simplify the test cases that might need this. > Perhaps even for the fourth, though > it would be tricky. But there's no way to reverse the effects for the > third. The ZFS tests, for example, create and destroy zpools. How > would you reverse that? You can't. Whatever was previously on the > disk is gone. Ah, I guess I missed that detail. Couldn't those tests run on top of vnd devices though? [...] >> And lastly, we'd just need a simple "filtering" feature in the kyua >> cli to allow specifying which size of tests to run (or to filter by >> any other metadata property, for that matter). > > I don't like this idea. I hope you are objecting to the filtering by test sizes, not the filtering itself! See below. > We tried it at work, and it didn't work out > very well. Basically, I automatically assigned sizes (short, medium, > long) to all of our tests based on their runtimes, so users could > select to only run the short or medium tests on the bench. The > problem is that the classification didn't make sense. For an > expedited test run, you don't want the shortest tests; what you want > are the tests with the most value per unit time. Unit tests have a > high value and a very short runtime. Stress tests have a very long > runtime and arguably low value since they don't always have consistent > results. > > At my previous job there was a large department whose duties included > curating test suites and deciding which tests would be included in the > short runs. FreeBSD isn't going to have that, but we could still ask > test authors to classify tests along the lines that I suggested. That's a good point. If I understand you correctly, what is important is to manually curate the set of "smoke tests" that run very quickly and that provide a reasonably high assurance that major subsystems haven't broken. We can do this too pretty easily, but configuration variables are the wrong mechanism. We can already define this information in metadata properties as long as you prefix them with X-. So all we'd need is a way to tell Kyua to only run tests marked with such property (i.e. with the filtering feature described above).