From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 20:46:08 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 256C38D1; Sat, 25 Oct 2014 20:46:08 +0000 (UTC) Received: from mail-wg0-x22c.google.com (mail-wg0-x22c.google.com [IPv6:2a00:1450:400c:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1AE708A5; Sat, 25 Oct 2014 20:46:06 +0000 (UTC) Received: by mail-wg0-f44.google.com with SMTP id y10so3165082wgg.15 for ; Sat, 25 Oct 2014 13:46:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=t//pXgEd3Q3UH0iDU41IKFjnnABq5Sm5aPP3WwoOZYY=; b=SVH1blnEsqAYqr3PugLeWMACKsu9FoSXunMO38A0GhtfKZv0/sOTh3ATCkCS9jCQlD DAtFkj8tZBZefRlYGdMmO8HEeASGKxmgLOICXsaUtSVHbLYTBFdei7R3gKKoeKDHu6nt Ml6Jk9Nu7ZACfRC+Pe9L7Rbgb6wIDxIAk0fn91G//Plugu5QCeTEQldd3dXidRp6wOzd 4FQuRRsjKOgvCGhHeLqBnOJ6HQTRSOwbWRaxPWQ2YjRSVIYZ+EJ7FUlLQk5xooiMVr0V 8a0eM26oTPzJVM/gFgFcJPOJvV/GizrxUBE5Q6qiPdBpFQ/ErX4JxJQJREl+y+8+Tygp 2gOw== MIME-Version: 1.0 X-Received: by 10.194.91.176 with SMTP id cf16mr13105133wjb.60.1414269965374; Sat, 25 Oct 2014 13:46:05 -0700 (PDT) Reply-To: matt@ixsystems.com Sender: mattjeet@gmail.com Received: by 10.194.14.40 with HTTP; Sat, 25 Oct 2014 13:46:05 -0700 (PDT) In-Reply-To: References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> Date: Sat, 25 Oct 2014 13:46:05 -0700 X-Google-Sender-Auth: d4LEDwRd3jOMoF28Wxp0hRpOoQk Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Matt Olander To: "K. Macy" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" , Alfred Perlstein , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 20:46:08 -0000 On Sat, Oct 25, 2014 at 1:20 PM, K. Macy wrote: >>>> Alan also suggested against integrating the test suite as-is, because = as he said, "Remember, don't run these tests on a production system. They = WILL cause panics and deadlocks, and they may cause data loss too.=E2=80=9D >>>> >>>> Cheers, >>>> -Garrett >>> >>> Wait, we want to sweep those bugs under the rug? What exactly is wrong= with making a test harness that can very easily reproduce a known problem?= The chances are that anyone will dive into it once the bug is easily repr= oducible. >> >> Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. >> Cheers, >> -Garrett > > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions > > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for. Agreed. We're talking about doing an OpenZFS bug-tracker and maybe we could have tests post there automatically, once the bug is opened. I am going to put this on the board at MeetBSD Ca. next week, so we can have a discussion about it in real-time with some of the people present. Cheers, -matt