From owner-freebsd-testing@FreeBSD.ORG Thu Mar 27 01:48:24 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 456A9A99 for ; Thu, 27 Mar 2014 01:48:24 +0000 (UTC) Received: from mail-qa0-f43.google.com (mail-qa0-f43.google.com [209.85.216.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 03A9AABE for ; Thu, 27 Mar 2014 01:48:23 +0000 (UTC) Received: by mail-qa0-f43.google.com with SMTP id j15so3200665qaq.30 for ; Wed, 26 Mar 2014 18:48:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=TdjVgEwHK8Ghtx76vh6UzWj85Dg7fANsAe54TQAsM5Q=; b=hIs8G5yD6iD+Aqs6Y+XH6Y58BUwlPksZnilAe5PIq0Vhsk855i8ffLgt1hBnmFAw8t 1bb8Y6bSVKudM0CghLPqp+jBWh9MUzdnk1Eo4zYDbtCUi7ATeAWOvcm9BQA1fwxq90o/ GG5IjFkCc80QfzmbiZe9iOXYzQFG0uZzd9JhXw/ACLb7nLrrlyUWhbr2u7KeaAk9JNTC tcDwe5xunPOANJOswG2fn3M8LMZPtaBFkuE0KjMvA5RZ8ljAeLS5Gnb5E8tjpdwv5T8r RrKeUx5P0ufg+d4Jr+DxuhBRlEu0k77d1LVxYObTSxDazKQQJsF/QvVK3FxHXpZSHYfl Mxmw== X-Gm-Message-State: ALoCoQmIuJFsiMZYqymwsMxCBK0PwXn26lp4IYhgPse0KmjBu+L7elYt8PHeLKVf6Y8HoTPjrLiE X-Received: by 10.140.97.137 with SMTP id m9mr6207084qge.95.1395884897748; Wed, 26 Mar 2014 18:48:17 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.102 with HTTP; Wed, 26 Mar 2014 18:47:56 -0700 (PDT) X-Originating-IP: [2401:fa00:d:c:b03e:2e82:92b6:e21d] In-Reply-To: References: From: Julio Merino Date: Thu, 27 Mar 2014 10:47:56 +0900 X-Google-Sender-Auth: yDgbjXHPNWVSPYy6f73ouCQSMOk Message-ID: Subject: Re: ATF Test Cases To: "Kilner, Peter" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 01:48:24 -0000 On Thu, Mar 27, 2014 at 7:26 AM, Kilner, Peter wrote= : > Hello, Hi Peter, Alan already answered this pretty well, but let me throw in my 2 cents: > I hope I am posting this question to the correct list. > > Is there a way to have a single ATF test case report multiple pass/fail r= esults? Currently I have seen that one can include a number of test condit= ions in a single test case (for example many aft_checks). However the test= case will only report one pass/fail result for that case. That's by design: one test case should represent one (and only one) scenario. Therefore, there can only be one meaningful result. > I would like to build a test case that will iterate though many configura= tions and would like to see a pass/fail for each config. However this is d= ifficult to implement with multiply test cases because of the for loops tha= t I am using. It sounds like you actually should split your test case into smaller ones and name each test case accordingly. Your email doesn't provide a lot of details, but I guess that if you are testing more than one configuration file is because each configuration file is exposing a specific behavior of the program under test. Consider this: "test_configuration" sounds like a really broad test case name and provides no information about what is wrong when the test fails. But if you consider "test_configuration__missing_hostname", "test_configuration__invalid_syntax", "test_configuration__duplicate_host_stanzas" as separate test cases, then when one fails it becomes obvious what specific case is broken. Doing this needn't be hard. If you are using atf-sh, my suggestion is to put all of the test case body in a separate function. Then, with a loop, you dynamically define small test case bodies (using eval) that call the auxiliary function with the right parameters. If you are using atf-c, you can do something similar, maybe even abusing ma= cros. But basically: just move the code to an auxiliary function and have all the individual test cases call it as a one-liner with the right arguments. See the following for some more ideas: https://wiki.freebsd.org/TestSuite/Structure#Test_programs_vs._test_case= s Please let me know if anything is not clear or lacks detail.