From owner-svn-src-projects@FreeBSD.ORG Thu Jan 29 19:41:21 2015 Return-Path: Delivered-To: svn-src-projects@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BC5AAE09 for ; Thu, 29 Jan 2015 19:41:21 +0000 (UTC) Received: from mail-ie0-f172.google.com (mail-ie0-f172.google.com [209.85.223.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5000FE50 for ; Thu, 29 Jan 2015 19:41:21 +0000 (UTC) Received: by mail-ie0-f172.google.com with SMTP id rd18so37641137iec.3 for ; Thu, 29 Jan 2015 11:41:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=QlNMwXN5q4Rn04F7Aqij02WFui8IPJadZjyqg+6BJWI=; b=WrTt6IqbznSjjpD4MAP8Pu9hU2ngW2Qw7xlAnz7n43kOGay5MiYQFbANDMuFGeSW8/ 8Q/BmstTa1TzGwlnibJylzvJSAFpLdZhWbt3yBuJh3sJmmXJ8w3vnRxTn21zAKBIzGba 0xn741CHp51su0xQROosUuNcNDDb81O2gjX5X0Kbte97ZVK1grA1X6cRqgumt554h0SW ZaEOOiUfLE/hsj4Kyuiv4W2vS8H1GQnnHMyIB6Bj8z4xpB3RRcKuteTyIk47cAxrfBEM QRttE/TaPngO1cKz5C/WpU2Ms57seIWEmFoamfqIX1lc6npaxj5J1Fju0zHOXT5bzKMq wy+w== X-Gm-Message-State: ALoCoQkpQWzEUTWfHi6VF6Oz4SfNjpkiSapKYI/OwVcxRr0XG0FKEb6e097brunC5hXSCBm0bx8H X-Received: by 10.42.121.84 with SMTP id i20mr2477619icr.20.1422560479935; Thu, 29 Jan 2015 11:41:19 -0800 (PST) Received: from sol.firepipe.net ([2601:1:ad80:db8:7271:bcff:fe83:9550]) by mx.google.com with ESMTPSA id c70sm2913717ioc.3.2015.01.29.11.41.18 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Jan 2015 11:41:19 -0800 (PST) Sender: Will Andrews Date: Thu, 29 Jan 2015 12:41:17 -0700 From: Will Andrews To: NGie Cooper Subject: Re: svn commit: r277876 - in projects/zfsd/head/tests/sys/cddl/zfs/tests: cli_root/zpool_upgrade zil Message-ID: <20150129194111.GD23484@sol.firepipe.net> References: <201501291423.t0TENWOO048247@svn.freebsd.org> <2519DB92-D2AD-48BD-96AE-101168E9C163@gmail.com> <20150129171841.GA23484@sol.firepipe.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="osDK9TLjxFScVI/L" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "svn-src-projects@freebsd.org" , "freebsd-testing@freebsd.org" , "src-committers@freebsd.org" X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2015 19:41:21 -0000 --osDK9TLjxFScVI/L Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 29, 2015 at 11:14:41AM -0800, NGie Cooper wrote: > I agree in concept. The only way to really do this in ATF is a two-step: >=20 > atf_expect_fail "This test panics my box" > atf_fail "Panic!" >=20 > This will print out a message like "expected failure: This test panics > my box: Panic!" though in the results, which is incredibly > confusing... That's why Julio, NetBSD, me, and others prefer: >=20 > atf_skip "This test panics my box" >=20 > A "filtering" mechanism for dealing with broken/failing tests on > FreeBSD hasn't been developed (and it kinds of invites maintenance > pain as it numbs people to issues in tests). So while the atf_fail is > known and expected for SpectraLogic, it's not for the Jenkins folks, > Isilon, Microsoft, Netflix, Panzura, etc. I suspect that marking the tests skipped means they will be ignored more so than marking them as expected failures. It is inefficient to manually filter a (potentially very long) skipped list to determine which ones are actually failures and which ones are simply because the environment isn't set up for them or because they require functions not yet ported to FreeBSD. In the ZFS test suite, ~150 are skipped out of ~800 total. So, how do you (quickly) tell which ones are actually failures? > Bugging the symptoms sooner rather than later is better so others know > what commands to avoid entering in (or situations to avoid in general) > when dealing with code. Please, please bug them so others don't > accidentally trip themselves up on the issues you've seen. I understand that, but first I'm trying to get the test suite to run. As I said in reply to Steven, all of these failures will be investigated once I have an easily repeatable mechanism, and before merging to head. > PS Thank you again for the work that you, Alan, and the other folks at > SpectraLogic have done! It'll be nice to add some intelligence to > FreeBSD/ZFS with zfsd :). Appreciated. zfsd won't be included in the initial test suite import, unfortunately. There are unfinished KPI changes that zfsd depends on in order to do its job. However, the vast majority of the ZFS test suite does not depend on zfsd, and we've seen many regressions over the last few years that would have been caught if this suite were being run. --=20 wca --osDK9TLjxFScVI/L Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlTKjNYACgkQF47idPgWcsX+CACfaNzcAQ1rL1z56fg8BUPLKfcM RYoAoI3yf8jZOBR4C1Ouv+LBu5AcQHeZ =7IAm -----END PGP SIGNATURE----- --osDK9TLjxFScVI/L--