From owner-freebsd-testing@FreeBSD.ORG Thu Mar 6 20:02:59 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF3ABDD9; Thu, 6 Mar 2014 20:02:58 +0000 (UTC) Received: from mail-vc0-x230.google.com (mail-vc0-x230.google.com [IPv6:2607:f8b0:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8A15B35B; Thu, 6 Mar 2014 20:02:58 +0000 (UTC) Received: by mail-vc0-f176.google.com with SMTP id lc6so2650560vcb.35 for ; Thu, 06 Mar 2014 12:02:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZC8jRYVrhwbgWowkfMc9Y1vNiVPrXjTeZw0MBznsT04=; b=CC3aE//5hP0qxBP1fdPK/ztVOW19as5DgmIY9DQZTYv8/6vEKKnUAQFAmAcKIk5TCX 2VZgUQ1i80MmQKgRlShlYT5iyCASTfkQjG96Zij61dkLl6Ods90qcUgEbQsPw4ePBPzc pWT9KAFUXLhOFusP+i5YV7XxUsjZMzo8BDz3FATunOZWbwTVHsnLavrooxWSfd6HanWZ AEAm2fVBzWhp937wJAa6znLqoIslMpkmYJLUqZyNY3qpqaeaNOswrDVExP8/4C/w4naP 3OtbwxrknpURUadslQDsn736SSMbFsvX6KUtgvDy1zD8XgStuWENrLguCBmq3sE3g5jM GL4A== MIME-Version: 1.0 X-Received: by 10.52.248.138 with SMTP id ym10mr9213531vdc.18.1394136177716; Thu, 06 Mar 2014 12:02:57 -0800 (PST) Received: by 10.221.67.136 with HTTP; Thu, 6 Mar 2014 12:02:57 -0800 (PST) In-Reply-To: References: Date: Thu, 6 Mar 2014 12:02:57 -0800 Message-ID: Subject: Re: Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles") From: Garrett Cooper To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" , kyua-discuss@googlegroups.com X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Mar 2014 20:02:59 -0000 On Thu, Mar 6, 2014 at 11:54 AM, Julio Merino wrote: > On Thu, Mar 6, 2014 at 1:28 PM, Garrett Cooper wrote: >> >> Seems like another standalone tool should be developed to cleanup >> variables after exit per test and per test suite. > > Which variables? sysctls in this case. Isilon would probably love this and other folks would too no doubt. > I can see some benefit in providing helper functions to store and load > arbitrary data structures to/from disk with one-liners, but is that > what you mean? Sorta kinda. >> Setup/cleanup in ATF doesn't seem to follow the model that many other >> testing frameworks do when it comes to executing testcases and >> managing setup/cleanup. I really wish things were like the >> JUnit/unittest model, i.e. >> >> - Test suite setup >> - For each testcase: >> - Run the testcase setup >> - If the setup passed, continue >> - Run the testcase >> - Run the testcase cleanup >> - Test suite teardown > > Sure... but the systems that implement this setup assume that all the > steps above run under one process. That's fine in the vast majority of > unit tests but the model fails when tests modify system-wide state and > crash before they get to the cleanup step. Agreed. Hence the need for tools that restore system state and careful code reviews and code design. > Keep in mind that one of the first design decisions for ATF was to > support test cases like the above where system-wide state may be > modified and you need to revert the changes. Because test cases can > crash at any point, having an out-of-process cleanup routine was a > must. This comes with its oddities though. Yes, but it falls on the developer to handle global state outside of the atf test directory sandbox. ATF shouldn't worry about protecting someone from doing something like fiddling around with sysctls and files in /etc (for example) -- it's the test developer's job to not do things like this. > All said, I'd also like a more JUnit-like model because it's easier to > understand and satisfies 99% of the cases (assuming you have rump). Yup yup! -Garrett