From owner-freebsd-testing@FreeBSD.ORG Thu Jan 29 19:41:21 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4D8F7E07 for ; Thu, 29 Jan 2015 19:41:21 +0000 (UTC) Received: from mail-ie0-f171.google.com (mail-ie0-f171.google.com [209.85.223.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0723DE4E for ; Thu, 29 Jan 2015 19:41:20 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id tr6so38109546ieb.2 for ; Thu, 29 Jan 2015 11:41:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=QlNMwXN5q4Rn04F7Aqij02WFui8IPJadZjyqg+6BJWI=; b=Li2ma8sv9Q8wGZeNOyid1lpOr6dR2One3sVf6Z309s2Dtx5ai18FAGTnVPFHuMlLpG 71LyXVsJOz/ZgxPp7s05fWTXODweY5lS1kPAG935jYxRbzmER5JOShDhjx3tyC139/Ad Qksu30nj2+2XMAuS3FcggY4JsPD1QmJDXJvMme8eV12s4jOOWPhy+behH8uQf6al0EOP W9tRDleGl60PLaghURV2OpFDR6TzZeE/bTYkQ//d3sCjDlvcpARis2RHDS77vCGQ779w 9L8iZ+jzNDvBoBhgMsXk/C2cwoHXkmVk7t59DlsMDJTRAv0+dokOVdamMNxN9z2FoKlF mrJQ== X-Gm-Message-State: ALoCoQlZbupp9lvYi4W9aS0iqURiUYN3ZQxX7TF5egDX3ZwSReMtlGl9pap5H6K4t1fBONaND9oI X-Received: by 10.42.121.84 with SMTP id i20mr2477619icr.20.1422560479935; Thu, 29 Jan 2015 11:41:19 -0800 (PST) Received: from sol.firepipe.net ([2601:1:ad80:db8:7271:bcff:fe83:9550]) by mx.google.com with ESMTPSA id c70sm2913717ioc.3.2015.01.29.11.41.18 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Jan 2015 11:41:19 -0800 (PST) Sender: Will Andrews Date: Thu, 29 Jan 2015 12:41:17 -0700 From: Will Andrews To: NGie Cooper Subject: Re: svn commit: r277876 - in projects/zfsd/head/tests/sys/cddl/zfs/tests: cli_root/zpool_upgrade zil Message-ID: <20150129194111.GD23484@sol.firepipe.net> References: <201501291423.t0TENWOO048247@svn.freebsd.org> <2519DB92-D2AD-48BD-96AE-101168E9C163@gmail.com> <20150129171841.GA23484@sol.firepipe.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="osDK9TLjxFScVI/L" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "svn-src-projects@freebsd.org" , "freebsd-testing@freebsd.org" , "src-committers@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2015 19:41:21 -0000 --osDK9TLjxFScVI/L Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 29, 2015 at 11:14:41AM -0800, NGie Cooper wrote: > I agree in concept. The only way to really do this in ATF is a two-step: >=20 > atf_expect_fail "This test panics my box" > atf_fail "Panic!" >=20 > This will print out a message like "expected failure: This test panics > my box: Panic!" though in the results, which is incredibly > confusing... That's why Julio, NetBSD, me, and others prefer: >=20 > atf_skip "This test panics my box" >=20 > A "filtering" mechanism for dealing with broken/failing tests on > FreeBSD hasn't been developed (and it kinds of invites maintenance > pain as it numbs people to issues in tests). So while the atf_fail is > known and expected for SpectraLogic, it's not for the Jenkins folks, > Isilon, Microsoft, Netflix, Panzura, etc. I suspect that marking the tests skipped means they will be ignored more so than marking them as expected failures. It is inefficient to manually filter a (potentially very long) skipped list to determine which ones are actually failures and which ones are simply because the environment isn't set up for them or because they require functions not yet ported to FreeBSD. In the ZFS test suite, ~150 are skipped out of ~800 total. So, how do you (quickly) tell which ones are actually failures? > Bugging the symptoms sooner rather than later is better so others know > what commands to avoid entering in (or situations to avoid in general) > when dealing with code. Please, please bug them so others don't > accidentally trip themselves up on the issues you've seen. I understand that, but first I'm trying to get the test suite to run. As I said in reply to Steven, all of these failures will be investigated once I have an easily repeatable mechanism, and before merging to head. > PS Thank you again for the work that you, Alan, and the other folks at > SpectraLogic have done! It'll be nice to add some intelligence to > FreeBSD/ZFS with zfsd :). Appreciated. zfsd won't be included in the initial test suite import, unfortunately. There are unfinished KPI changes that zfsd depends on in order to do its job. However, the vast majority of the ZFS test suite does not depend on zfsd, and we've seen many regressions over the last few years that would have been caught if this suite were being run. --=20 wca --osDK9TLjxFScVI/L Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlTKjNYACgkQF47idPgWcsX+CACfaNzcAQ1rL1z56fg8BUPLKfcM RYoAoI3yf8jZOBR4C1Ouv+LBu5AcQHeZ =7IAm -----END PGP SIGNATURE----- --osDK9TLjxFScVI/L--