From owner-freebsd-virtualization@FreeBSD.ORG Sat Oct 25 21:55:30 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 49A4D678; Sat, 25 Oct 2014 21:55:30 +0000 (UTC) Received: from mail-yh0-x231.google.com (mail-yh0-x231.google.com [IPv6:2607:f8b0:4002:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC162EB6; Sat, 25 Oct 2014 21:55:29 +0000 (UTC) Received: by mail-yh0-f49.google.com with SMTP id a41so2850484yho.22 for ; Sat, 25 Oct 2014 14:55:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=kKt6BmrW6I+rLbcDwNkEKwLjVTlpuwf/2Jn9S9xWo14=; b=gvhr0GrDcrOeHTC5nx8pgldKNvSbME/5jkI0kLJkw+pgl/cRnB0k84cK9wV+q56YbJ VG+woGlm/AS5K7No/+0DJPXINLGMUjgDc/4gJ6LaELUcskKepR5yvzetI1yGFu0F4+uO 3jezqTfumx1vwcoS0N/zhpI4fGNsHoBkXXMXigKHe1ZIb2TPyABVzf2nOlpUmMn5Zckn Kj/JRjkXKj182CHGbei9aYR3665sQdve/f4MPMbf5/E2REb+efnTPrGA3+D73rIUZImO RIjCnr39Dex8uCuCmWLyxTR3p30aBEmWJ18u4dlGZQXtp9rIrwShqqj2n3RMEP5uf50j gvfQ== MIME-Version: 1.0 X-Received: by 10.236.209.101 with SMTP id r65mr3468882yho.140.1414274129002; Sat, 25 Oct 2014 14:55:29 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Sat, 25 Oct 2014 14:55:28 -0700 (PDT) In-Reply-To: <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com> Date: Sat, 25 Oct 2014 14:55:28 -0700 X-Google-Sender-Auth: t0VcMeHAoagDRAA2pnmRsZr5KUA Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: "K. Macy" To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 21:55:30 -0000 On Sat, Oct 25, 2014 at 2:46 PM, Garrett Cooper wro= te: > On Oct 25, 2014, at 13:20, K. Macy wrote: > >>>>> Alan also suggested against integrating the test suite as-is, because= as he said, "Remember, don't run these tests on a production system. They= WILL cause panics and deadlocks, and they may cause data loss too.=E2=80= =9D >>>>> >>>>> Cheers, >>>>> -Garrett >>>> >>>> Wait, we want to sweep those bugs under the rug? What exactly is wron= g with making a test harness that can very easily reproduce a known problem= ? The chances are that anyone will dive into it once the bug is easily rep= roducible. >>> >>> Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. >> >> Yes, disabling tests that fail leads to an ineffectual test suite. A >> test suite that never has any failures is not very useful. However, >> there are two factors to take in to account in this context: >> a) frequent failures can lead users to stop running a test suite >> leading to further regressions >> b) long-term repeated failures can desensitize users leading them to >> ignore *new* failures facilitating further regressions >> >> Thus it's really a question of what context you're talking about >> running the test suite in. For purposes of Jenkins we want full >> visibility in to what is passing and what is failing and how long this >> has been going on for. > > (seeing as how my other post isn=E2=80=99t in the -testing archives yet..= ) > Panicking a node (what the tests are doing before last night) and= exiting with a non-zero exit code (what I=E2=80=99m making them do with th= e bail outs in tools/regression/zfs) are both considered test failures. The= difference being that I can safely run all of the tests on a production or= a test machine without having to panic/reboot the box and I get greater co= verage in one fell swoop. If a developer wants they can always delete the l= ines that bail out of the tests to get the desired panic. I don't think there is a RIGHT answer. I just ask that it be noisy about the choice that's being made. In other words, it should be very explicit that it's running a "safe" subset of the tests because FreeBSD's ZFS can't actually pass all of them, and that there be a flag to enable those of us who want to see the failure to see it without modifying any scripts or config files. A test framework that requires me to muck with settings to run failing tests isn't living up to its full potential. -K