From owner-freebsd-testing@FreeBSD.ORG Thu Mar 6 19:55:22 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3E99CB1A for ; Thu, 6 Mar 2014 19:55:22 +0000 (UTC) Received: from mail-qc0-f181.google.com (mail-qc0-f181.google.com [209.85.216.181]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EE3C225D for ; Thu, 6 Mar 2014 19:55:21 +0000 (UTC) Received: by mail-qc0-f181.google.com with SMTP id e9so3477299qcy.26 for ; Thu, 06 Mar 2014 11:55:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=FHi3xQVSeUfJfphQBphcR9MzdLVex8PUDqfH5zvrmLY=; b=MnNBLE47/0A/jYZe0jBD6xSP0FTYCCNfT+WzO8xsWGBbW5RZmBToXtNgZ6uXR301xo F5EnCptfvbCkIPxGx2SxaJApR3InN2rmvw4UNVp/ZV+uWJSwhO4AMzsBYhlegFz8N9EA O2P1BVhRylvt4K1FA3Ld3Ei2vC7Fx9rRE/oTq70j2+qldvMhw2J7aBl7hlwDdRIHFjbH Q0BQwPu1PDx+StPyunIZhtxUHV6CcdYL+IpLXbomdmzGWNI5CW/Re703oxK5yCnPGgLG Xc8j9L7/JA8uZT/Cc/m9KNn8nPtzkqmoNW9mf3XHw70ip5swlKgCLGbH5b3GckA4Q7Ks qtig== X-Gm-Message-State: ALoCoQm2xcSsENWui1ZqGmXXGJYXsA79RaKDoZBlAKpuB7cYPHY5wNOIIRKpNHsoOeqNu8T+EcW6 X-Received: by 10.224.21.129 with SMTP id j1mr16552424qab.13.1394135715349; Thu, 06 Mar 2014 11:55:15 -0800 (PST) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.102 with HTTP; Thu, 6 Mar 2014 11:54:55 -0800 (PST) X-Originating-IP: [2620:0:1003:1021:5c49:13a2:f7c6:9920] In-Reply-To: References: From: Julio Merino Date: Thu, 6 Mar 2014 14:54:55 -0500 X-Google-Sender-Auth: eqBEKVqle0wI9WYfprL4vJSgFlI Message-ID: Subject: Re: Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles") To: Garrett Cooper Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" , kyua-discuss@googlegroups.com X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Mar 2014 19:55:22 -0000 On Thu, Mar 6, 2014 at 1:28 PM, Garrett Cooper wrote: > > Seems like another standalone tool should be developed to cleanup > variables after exit per test and per test suite. Which variables? I can see some benefit in providing helper functions to store and load arbitrary data structures to/from disk with one-liners, but is that what you mean? > Setup/cleanup in ATF doesn't seem to follow the model that many other > testing frameworks do when it comes to executing testcases and > managing setup/cleanup. I really wish things were like the > JUnit/unittest model, i.e. > > - Test suite setup > - For each testcase: > - Run the testcase setup > - If the setup passed, continue > - Run the testcase > - Run the testcase cleanup > - Test suite teardown Sure... but the systems that implement this setup assume that all the steps above run under one process. That's fine in the vast majority of unit tests but the model fails when tests modify system-wide state and crash before they get to the cleanup step. Keep in mind that one of the first design decisions for ATF was to support test cases like the above where system-wide state may be modified and you need to revert the changes. Because test cases can crash at any point, having an out-of-process cleanup routine was a must. This comes with its oddities though. All said, I'd also like a more JUnit-like model because it's easier to understand and satisfies 99% of the cases (assuming you have rump).