Date: Thu, 29 Jan 2015 12:41:17 -0700 From: Will Andrews <will@freebsd.org> To: NGie Cooper <yaneurabeya@gmail.com> Cc: "svn-src-projects@freebsd.org" <svn-src-projects@freebsd.org>, "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, "src-committers@freebsd.org" <src-committers@freebsd.org> Subject: Re: svn commit: r277876 - in projects/zfsd/head/tests/sys/cddl/zfs/tests: cli_root/zpool_upgrade zil Message-ID: <20150129194111.GD23484@sol.firepipe.net> In-Reply-To: <CAGHfRMD8WxyVAm4iGbOkPT8xo0JjRGwKMaR1bLA_ePVVW4GUsQ@mail.gmail.com> References: <201501291423.t0TENWOO048247@svn.freebsd.org> <2519DB92-D2AD-48BD-96AE-101168E9C163@gmail.com> <20150129171841.GA23484@sol.firepipe.net> <CAGHfRMD8WxyVAm4iGbOkPT8xo0JjRGwKMaR1bLA_ePVVW4GUsQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--osDK9TLjxFScVI/L Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 29, 2015 at 11:14:41AM -0800, NGie Cooper wrote: > I agree in concept. The only way to really do this in ATF is a two-step: >=20 > atf_expect_fail "This test panics my box" > atf_fail "Panic!" >=20 > This will print out a message like "expected failure: This test panics > my box: Panic!" though in the results, which is incredibly > confusing... That's why Julio, NetBSD, me, and others prefer: >=20 > atf_skip "This test panics my box" >=20 > A "filtering" mechanism for dealing with broken/failing tests on > FreeBSD hasn't been developed (and it kinds of invites maintenance > pain as it numbs people to issues in tests). So while the atf_fail is > known and expected for SpectraLogic, it's not for the Jenkins folks, > Isilon, Microsoft, Netflix, Panzura, etc. I suspect that marking the tests skipped means they will be ignored more so than marking them as expected failures. It is inefficient to manually filter a (potentially very long) skipped list to determine which ones are actually failures and which ones are simply because the environment isn't set up for them or because they require functions not yet ported to FreeBSD. In the ZFS test suite, ~150 are skipped out of ~800 total. So, how do you (quickly) tell which ones are actually failures? > Bugging the symptoms sooner rather than later is better so others know > what commands to avoid entering in (or situations to avoid in general) > when dealing with code. Please, please bug them so others don't > accidentally trip themselves up on the issues you've seen. I understand that, but first I'm trying to get the test suite to run. As I said in reply to Steven, all of these failures will be investigated once I have an easily repeatable mechanism, and before merging to head. > PS Thank you again for the work that you, Alan, and the other folks at > SpectraLogic have done! It'll be nice to add some intelligence to > FreeBSD/ZFS with zfsd :). Appreciated. zfsd won't be included in the initial test suite import, unfortunately. There are unfinished KPI changes that zfsd depends on in order to do its job. However, the vast majority of the ZFS test suite does not depend on zfsd, and we've seen many regressions over the last few years that would have been caught if this suite were being run. --=20 wca --osDK9TLjxFScVI/L Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlTKjNYACgkQF47idPgWcsX+CACfaNzcAQ1rL1z56fg8BUPLKfcM RYoAoI3yf8jZOBR4C1Ouv+LBu5AcQHeZ =7IAm -----END PGP SIGNATURE----- --osDK9TLjxFScVI/L--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150129194111.GD23484>