From owner-freebsd-testing@FreeBSD.ORG Fri Mar 14 22:04:32 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BA21EA2B; Fri, 14 Mar 2014 22:04:32 +0000 (UTC) Received: from mail-we0-x236.google.com (mail-we0-x236.google.com [IPv6:2a00:1450:400c:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 320A6381; Fri, 14 Mar 2014 22:04:32 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id p61so2598257wes.41 for ; Fri, 14 Mar 2014 15:04:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=TJ7a2iTw/lIYsJfEVtth1815nHgtkZIs/vu89by6sgA=; b=b5KhTo8DesHmFen1gwywwuOncAOHP+nahEyvLSxB+FMLk7qzObmRyTEx6buEIN4pKc BqZjuU++R6GX+n7T+76WZMT1odecVWznrLJHS6ozqM8cGK5SjKJOt414rQ7blIdo9UDd A2BZLs4jg6epIg1+x6F4uefNzewtxKnxAowspLUaWGyi7Qlz1Iayd3hdF11JZmGl6KJO OS2Vw5dr92QS5BIMnEywxa6bwzIvY2CpTEYNTAOdVMHgCeRCrltrdoOnE1I7Xp7IhMJ1 HjERcRJYWIe0tu9RiwoY2NtJfZTBHbwNLxu3KSHbfZmFYlDYXwojNs2FGOaqGL7EyM5J hLZw== MIME-Version: 1.0 X-Received: by 10.194.63.236 with SMTP id j12mr8600875wjs.5.1394834669793; Fri, 14 Mar 2014 15:04:29 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Fri, 14 Mar 2014 15:04:29 -0700 (PDT) Date: Fri, 14 Mar 2014 16:04:29 -0600 X-Google-Sender-Auth: nbHoFTf03rWC7IIocXuXK5eEc6I Message-ID: Subject: ATF config variables for FreeBSD tests From: Alan Somers To: "freebsd-testing@freebsd.org" , "Kenneth D. Merry" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2014 22:04:32 -0000 Another thread recently raised the issue of tests that cause potentially harmful changes to sysctl variables. We all agreed to conditionalize that test based on an ATF config variable. Now Ken and I both want to add tests with different harmful side effects. I propose the following list of config variables. If they sound good to everyone, I'll add them to tests(7) and to an example kyua.conf file in devel/kyua. test_suites.FreeBSD.allow_sysctl_side_effects If defined, enables tests that cause potentially harmful changes to globally relevant sysctl variables. Requested by Peter for kern_descrip_test:maxfiles__increase test_suites.FreeBSD.fibs Defined to a space delimited list of fib numbers. Tests that change routing tables must use these fibs instead of the default. They will be skipped if the variable is undefined. I'm working on such tests right now. test_suites.FreeBSD.disks Defined to a space delimited list of disk device nodes. Tests that format, mount, write to, disable, or have any other harmful effect on disks must use these devices. test_suites.FreeBSD.allow_devfs_side_effects. If defined, enables tests that can create or destroy semipermanent device nodes, like disk devices. Without this variable, tests may still create and destroy device nodes that are normally transient, like /dev/tap* and /dev/pts/*. This variable requires further explanation: Naively, one would think that test_suites.FreeBSD.disks would be sufficient protection for tests that create and destroy disk device nodes. However, Ken is working on a test that will cause hard drives to disappear and reappear in a different order, possibly resulting in their devnames changing. It could even result in "holes" appearing in the list of da unit numbers. If the sysadmin defines test_suites.FreeBSD.disks to "/dev/da1 /dev/da2 /dev/da3", it would be possible for those same devices to be named "/dev/da2 /dev/da1 /dev/da4" by the time Ken's test finishes. The permanent disappearance of /dev/da3 could confuse future tests that rely on test_suites.FreeBSD.disks. They would try to open /dev/da3 and fail. Thus, it may not be safe to run Ken's test in the same run as other tests that access disks. test_suites.FreeBSD.unit_tests test_suites.FreeBSD.functional_tests test_suites.FreeBSD.stress_tests These three variables would address a different concern of Peter's. He was concerned that his test might require too much run time, annoying other users of the rest of the test suite. By using these variables, a sysadmin could easily disable stress tests, which tend to take a very long time to run. Conversely, unit tests are normally very fast. A kyua run that only ran unit tests could easily be done after every buildworld, for example, while one that ran functional tests as well could be done perhaps several times per day. This is just an idea I have. I haven't tried it in practice. What do you think? -Alan