From owner-freebsd-testing@FreeBSD.ORG Wed Apr 9 23:18:22 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4975A552 for ; Wed, 9 Apr 2014 23:18:22 +0000 (UTC) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 091DE117D for ; Wed, 9 Apr 2014 23:18:21 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id oz11so2696410veb.20 for ; Wed, 09 Apr 2014 16:18:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=L++sP2oZu3azhhoo9U8hZHLEVd2gaiFy8piWjS72kGs=; b=t3wgmhKvYEMRtRISpZVKKv8rvKU9n5rUpoRj27jh1x5UAfHJwsyFRpz2e8VagXqOxJ BFi9Lec48mSM8dZtkS9KnoRn+y6ibJ+zBJmUYsiUSujjmTK44SCMeemMcfw+q/PLkQSJ RcgtFvP1z7ihiVBW9+ITEyZM775auVaN283MRCEGtFXFYptjW2kfnjixeiirldb+A0qH PCW2o4lWL4tMobo4McTzBl8CVTZuiRvN4U8SNyvozT5R/y+UOrNVcsT1Suog7qyabFVF O2aa3X/gPMj9qplGZvL/pyYcl3c2FIR2yVnHAv3zI0EfKvgTEKOyqLMj5ZMg7zk/64IV To6A== MIME-Version: 1.0 X-Received: by 10.220.103.141 with SMTP id k13mr22153vco.25.1397085501160; Wed, 09 Apr 2014 16:18:21 -0700 (PDT) Received: by 10.221.67.136 with HTTP; Wed, 9 Apr 2014 16:18:21 -0700 (PDT) In-Reply-To: References: Date: Wed, 9 Apr 2014 16:18:21 -0700 Message-ID: Subject: Re: ATF Test Cases From: Garrett Cooper To: "Kilner, Peter" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Apr 2014 23:18:22 -0000 Hi again Peter! On Wed, Mar 26, 2014 at 3:26 PM, Kilner, Peter wrote= : > Hello, > > I hope I am posting this question to the correct list. > > Is there a way to have a single ATF test case report multiple pass/fail r= esults? Currently I have seen that one can include a number of test condit= ions in a single test case (for example many aft_checks). However the test= case will only report one pass/fail result for that case. > > I would like to build a test case that will iterate though many configura= tions and would like to see a pass/fail for each config. However this is d= ifficult to implement with multiply test cases because of the for loops tha= t I am using. > > Any suggestions would be appreciated. As others have suggested, your approach could be tweaked a bit to iterate over all of the different configurations you are trying to test. In general I would create either a generic library or test driver with common code and have a common validation method, then build off that with additional validation methods if needed. I do that all the time. I think that Julio's suggestion with dynamically eval'ing tests is spot on as well. There's an example here of how to do that (that we might want to extend to a more generic form and push into Kyua or FreeBSD's test infrastructure if it's ok): https://github.com/yaneurabeya/freebsd/blob/master/bin/sh/tests/functional_= test.sh . Important: this code is proof of concept -- please do _NOT_ copy-paste it if at all possible. Thanks, -Garrett