From owner-freebsd-testing@FreeBSD.ORG Thu Mar 27 04:00:29 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CFE05BB4; Thu, 27 Mar 2014 04:00:29 +0000 (UTC) Received: from mail-ve0-x229.google.com (mail-ve0-x229.google.com [IPv6:2607:f8b0:400c:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78A21863; Thu, 27 Mar 2014 04:00:29 +0000 (UTC) Received: by mail-ve0-f169.google.com with SMTP id pa12so3601306veb.28 for ; Wed, 26 Mar 2014 21:00:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=JY+3eUxf5eL6gZWTYFKK0G8KuqDR2hpEa65iN7u7C2M=; b=kGbOA/Z9ADNqSxPNJdpfPbSF9mBe43ztxpsFMA1clPSius5cpBn2bKYtQikIH78V0H NCrJCNXrx8JeYf0paBkSx4fdJ/Rc7OWAEvpj7q5HeUdERaEZsorfXlTosOpKDKdu489v mBKMX39jJDKeuGy6QjbpRvHOSnZlwax0bwsQmdPbwcwAUJ6AL963b5nVrOya27Y8Tc0G z2l8FmMH703CSXxJYym8RYwOc6/EB/phBcW2fNBNRK7+ZPuSDMOftNoXjaK2hKY5Drku q8oRBZVGggzAsQa25QbwbKCwRFXejmzC+zt4buYvwFdtkqhUrVodT6nk6IE1a8muSi5y 9OuA== MIME-Version: 1.0 X-Received: by 10.52.26.161 with SMTP id m1mr42851756vdg.24.1395892828533; Wed, 26 Mar 2014 21:00:28 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.58.207.169 with HTTP; Wed, 26 Mar 2014 21:00:28 -0700 (PDT) In-Reply-To: References: Date: Wed, 26 Mar 2014 22:00:28 -0600 X-Google-Sender-Auth: P-T9SfDqYQy0ZwjqN_lChAedaJU Message-ID: Subject: Re: ATF Test Cases From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" , "Kilner, Peter" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 04:00:30 -0000 On Wed, Mar 26, 2014 at 7:47 PM, Julio Merino wrote: > On Thu, Mar 27, 2014 at 7:26 AM, Kilner, Peter wro= te: >> Hello, > > Hi Peter, > > Alan already answered this pretty well, but let me throw in my 2 cents: > >> I hope I am posting this question to the correct list. >> >> Is there a way to have a single ATF test case report multiple pass/fail = results? Currently I have seen that one can include a number of test condi= tions in a single test case (for example many aft_checks). However the tes= t case will only report one pass/fail result for that case. > > That's by design: one test case should represent one (and only one) > scenario. Therefore, there can only be one meaningful result. > >> I would like to build a test case that will iterate though many configur= ations and would like to see a pass/fail for each config. However this is = difficult to implement with multiply test cases because of the for loops th= at I am using. > > It sounds like you actually should split your test case into smaller > ones and name each test case accordingly. Your email doesn't provide > a lot of details, but I guess that if you are testing more than one > configuration file is because each configuration file is exposing a > specific behavior of the program under test. > > Consider this: "test_configuration" sounds like a really broad test > case name and provides no information about what is wrong when the > test fails. But if you consider > "test_configuration__missing_hostname", > "test_configuration__invalid_syntax", > "test_configuration__duplicate_host_stanzas" as separate test cases, > then when one fails it becomes obvious what specific case is broken. > > Doing this needn't be hard. > > If you are using atf-sh, my suggestion is to put all of the test case > body in a separate function. Then, with a loop, you dynamically > define small test case bodies (using eval) that call the auxiliary > function with the right parameters. I never thought about using eval to dynamically define testcases. Can you please give an example? > > If you are using atf-c, you can do something similar, maybe even abusing = macros. > > But basically: just move the code to an auxiliary function and have > all the individual test cases call it as a one-liner with the right > arguments. > > See the following for some more ideas: > > https://wiki.freebsd.org/TestSuite/Structure#Test_programs_vs._test_ca= ses > > Please let me know if anything is not clear or lacks detail. > _______________________________________________ > freebsd-testing@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-testing > To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.org= "