From owner-freebsd-testing@FreeBSD.ORG Wed Apr 9 23:30:37 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4FB126AB for ; Wed, 9 Apr 2014 23:30:37 +0000 (UTC) Received: from mailuogwdur.emc.com (mailuogwdur.emc.com [128.221.224.79]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mailuogwprd54.lss.emc.com", Issuer "RSA Corporate Server CA v2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF32412C5 for ; Wed, 9 Apr 2014 23:30:36 +0000 (UTC) Received: from maildlpprd55.lss.emc.com (maildlpprd55.lss.emc.com [10.106.48.159]) by mailuogwprd54.lss.emc.com (Sentrion-MTA-4.3.0/Sentrion-MTA-4.3.0) with ESMTP id s39NUYQY001942 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 9 Apr 2014 19:30:35 -0400 X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd54.lss.emc.com s39NUYQY001942 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=emc.com; s=jan2013; t=1397086235; bh=8/2CkPFVOrg0y8HI7SLYfEiO8Os=; h=From:To:CC:Date:Subject:Message-ID:References:In-Reply-To: Content-Type:MIME-Version; b=DhgMBukAodw14d43C8x0pzxTq1U2VOyma9/vBy4K16hTAYzj0HH5QlkQSeQqnKVJr QssQc4uEOTbOM9RFR61PuAErn4MZoc8dxn8yQYYfaoR/fJ/FpmwDS9Ec77hOQ8lm4l q08RpipS+TaEgh4GTLKPQFeeLGpkdmG3MtmfRrlM= X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd54.lss.emc.com s39NUYQY001942 Received: from mailusrhubprd03.lss.emc.com (mailusrhubprd03.lss.emc.com [10.253.24.21]) by maildlpprd55.lss.emc.com (RSA Interceptor); Wed, 9 Apr 2014 19:30:29 -0400 Received: from mxhub02.corp.emc.com (mxhub02.corp.emc.com [10.254.141.104]) by mailusrhubprd03.lss.emc.com (Sentrion-MTA-4.3.0/Sentrion-MTA-4.3.0) with ESMTP id s39NUTV3018853 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Wed, 9 Apr 2014 19:30:29 -0400 Received: from MX01A.corp.emc.com ([10.254.141.71]) by mxhub02.corp.emc.com ([10.254.141.104]) with mapi; Wed, 9 Apr 2014 19:30:28 -0400 From: "Kilner, Peter" To: Garrett Cooper Date: Wed, 9 Apr 2014 19:30:27 -0400 Subject: Re: ATF Test Cases Thread-Topic: ATF Test Cases Thread-Index: Ac9US7DY5B6QDP1DQs+JcnljnQYtJw== Message-ID: <1C62F69C-F225-42D8-B56D-A22E5019892E@isilon.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-Sentrion-Hostname: mailusrhubprd03.lss.emc.com X-RSA-Classifications: public Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: "freebsd-testing@freebsd.org" , "Kilner, Peter" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Apr 2014 23:30:37 -0000 Hi Garrett, Thanks for the example, this is great solution! I was starting to think ab= out creating a test case generator but this a much better way to go. Thank you, Peter K On Apr 9, 2014, at 4:18 PM, Garrett Cooper > wrote: Hi again Peter! On Wed, Mar 26, 2014 at 3:26 PM, Kilner, Peter > wrote: Hello, I hope I am posting this question to the correct list. Is there a way to have a single ATF test case report multiple pass/fail res= ults? Currently I have seen that one can include a number of test conditio= ns in a single test case (for example many aft_checks). However the test c= ase will only report one pass/fail result for that case. I would like to build a test case that will iterate though many configurati= ons and would like to see a pass/fail for each config. However this is dif= ficult to implement with multiply test cases because of the for loops that = I am using. Any suggestions would be appreciated. As others have suggested, your approach could be tweaked a bit to iterate over all of the different configurations you are trying to test. In general I would create either a generic library or test driver with common code and have a common validation method, then build off that with additional validation methods if needed. I do that all the time. I think that Julio's suggestion with dynamically eval'ing tests is spot on as well. There's an example here of how to do that (that we might want to extend to a more generic form and push into Kyua or FreeBSD's test infrastructure if it's ok): https://github.com/yaneurabeya/freebsd/blob/master/bin/sh/tests/functional_= test.sh . Important: this code is proof of concept -- please do _NOT_ copy-paste it if at all possible. Thanks, -Garrett