From owner-freebsd-testing@FreeBSD.ORG Thu Jan 29 19:14:42 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50587165; Thu, 29 Jan 2015 19:14:42 +0000 (UTC) Received: from mail-ie0-x234.google.com (mail-ie0-x234.google.com [IPv6:2607:f8b0:4001:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10DFD9F8; Thu, 29 Jan 2015 19:14:42 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id rl12so37732605iec.11; Thu, 29 Jan 2015 11:14:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=RItBiq88UsfK/fMUSdygwHVN+XVZkPm88gr9WGb5aRA=; b=fGaskR60Hlu3hBCmej3CFDStLWt8FIHMLBBfyW3MAf1jUUW8rMXgSJXJSVIvopOnm0 45gxGAUjNlpexM5ndL7Ekr5XVQ1wTBGKujdVli9N+6GdRf6xkO3gtsfRemh3+5LJLDBx XYiMPizH1xH8wo59ZBAR+pqniwSVnTlNHe6FgR6QfFkIyNIeX5S0VmtrOkI2HfzmpPyV cDvtn/ca5nJPM3utFLzarNL48CSbhY7VXelaF0cEZVPcOtrZGtkvsOmylUwQJ3sjQ+JZ 8fa28dwFki6iKfx447EDRybrSivmddpij0/IhxyG5RoC3VEAPiU0TSIfaYtTWNonX647 9aKA== MIME-Version: 1.0 X-Received: by 10.50.66.131 with SMTP id f3mr2132506igt.7.1422558881459; Thu, 29 Jan 2015 11:14:41 -0800 (PST) Received: by 10.50.182.233 with HTTP; Thu, 29 Jan 2015 11:14:41 -0800 (PST) In-Reply-To: <20150129171841.GA23484@sol.firepipe.net> References: <201501291423.t0TENWOO048247@svn.freebsd.org> <2519DB92-D2AD-48BD-96AE-101168E9C163@gmail.com> <20150129171841.GA23484@sol.firepipe.net> Date: Thu, 29 Jan 2015 11:14:41 -0800 Message-ID: Subject: Re: svn commit: r277876 - in projects/zfsd/head/tests/sys/cddl/zfs/tests: cli_root/zpool_upgrade zil From: NGie Cooper To: Will Andrews Content-Type: text/plain; charset=UTF-8 Cc: "svn-src-projects@freebsd.org" , "freebsd-testing@freebsd.org" , "src-committers@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2015 19:14:42 -0000 On Thu, Jan 29, 2015 at 9:18 AM, Will Andrews wrote: > On Thu, Jan 29, 2015 at 08:58:57AM -0800, Garrett Cooper wrote: >> Please use atf_skip instead of atf_fail so the issues don't count >> as failures. If you have bug numbers for the issues that would be nice >> too in the error messages. > > To me, skipping tests is only done when the environment is not setup > correctly for the test. > > Panics are a special case in that they cause an irreversible abort of the > test run itself, and they *are* failures. > > We've adopted a practice of marking such tests as 'expected failure' and > then preemptively failing them. Does that work for you? I agree in concept. The only way to really do this in ATF is a two-step: atf_expect_fail "This test panics my box" atf_fail "Panic!" This will print out a message like "expected failure: This test panics my box: Panic!" though in the results, which is incredibly confusing... That's why Julio, NetBSD, me, and others prefer: atf_skip "This test panics my box" A "filtering" mechanism for dealing with broken/failing tests on FreeBSD hasn't been developed (and it kinds of invites maintenance pain as it numbs people to issues in tests). So while the atf_fail is known and expected for SpectraLogic, it's not for the Jenkins folks, Isilon, Microsoft, Netflix, Panzura, etc. > Bugs will come later if I don't have fixes I can commit. Bugging the symptoms sooner rather than later is better so others know what commands to avoid entering in (or situations to avoid in general) when dealing with code. Please, please bug them so others don't accidentally trip themselves up on the issues you've seen. PS Thank you again for the work that you, Alan, and the other folks at SpectraLogic have done! It'll be nice to add some intelligence to FreeBSD/ZFS with zfsd :).