From owner-freebsd-testing@FreeBSD.ORG Mon Oct 27 15:10:40 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 61386E86 for ; Mon, 27 Oct 2014 15:10:40 +0000 (UTC) Received: from mail-qa0-f53.google.com (mail-qa0-f53.google.com [209.85.216.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1ADA2289 for ; Mon, 27 Oct 2014 15:10:39 +0000 (UTC) Received: by mail-qa0-f53.google.com with SMTP id n8so1571961qaq.40 for ; Mon, 27 Oct 2014 08:10:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=TBl6wMCDbY3weub38DtEmRd0Rf1UxjK1xrapWDvlc9M=; b=KPyIZYeCEwh5jXhyDAwV0JSPdY+LvJfqDy2aWQk0Rk+k+Imc2UyUNNXK3VNwYvX0Eo M9F6YWxQVAxCiaU2kMBNJtWQc4KsG9/EFce8n/Vot+4JoZixqglsd0El3NIt0Rm8WHD8 zyqHXTAaI1rnp/52ywUFMWQchDo0pFKWfRpdIKd16EFVcPGYDZn9foOmRKWe4M2MPUPC aN2NnTConNYfqQLU1DQD5Ld1Fe9WuEhQ5jx5KxTg785esOxaZmshU9N++I9oHfKtF55V OJn7LFcNzfbuHc7D/l8Pgo+GKd3of2Uiht54f71vsZKcBP+Oz5J6HBpN+YTp8GqPXWjR T0JQ== X-Gm-Message-State: ALoCoQnJmdq1HwuOjOjd+562FXYmXRa9qKwsJECh489DI17yJrSJnrG5ptePCfznCu8Dv/Tny5Zo X-Received: by 10.224.61.7 with SMTP id r7mr34364552qah.9.1414422298155; Mon, 27 Oct 2014 08:04:58 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.66.69 with HTTP; Mon, 27 Oct 2014 08:04:37 -0700 (PDT) X-Originating-IP: [81.184.188.119] In-Reply-To: <86E89F83-179D-4301-9EC0-0ECC3909B924@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <86E89F83-179D-4301-9EC0-0ECC3909B924@gmail.com> From: Julio Merino Date: Mon, 27 Oct 2014 16:04:37 +0100 X-Google-Sender-Auth: q2XxFKlkoMlXxB2tFCBTHuVOrFg Message-ID: Subject: Re: Panicking UUTs and integrating the ZFS test suite into the default run To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" , Alfred Perlstein X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Oct 2014 15:10:40 -0000 On Sat, Oct 25, 2014 at 8:59 PM, Garrett Cooper wrote: > (moving to -testing and renaming because this is turning into a bikeshed discussion) > > On Oct 25, 2014, at 9:49, Craig Rodrigues wrote: > >> On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein wrote: > > 6 > 7 [ "${os}" = "FreeBSD" ] && die "panics FreeBSD; see bug # 194589" > 8 > > 1. The issue is documented, so developers and testers know where to get the bug information > 2. The test is marked as a failure. > > Both of these items encourage testers and other developers to go prod developers with knowledge and cycles to go fix the bugs so things eventually turn green. That's bad as well. Marking tests as broken trains developers to ignore failures and to be more lax in tolerating breakage, and thus makes the signal from the test suite useless. Developers must be able to trust the results of the test suite, and to understand that a flip from green to red is really bad. Thinking "Hey, the build has been red like, forever, so one more broken test from me is irrelevant!" is quite common. These tests must be kept green if they are for known problems. A way to do so is with "expected failures", which I think you can denote in TAP with the "TODO" keyword and with expect_fail in ATF. Regarding the panics: making the box panic every time the test runs is probably not a great idea... but this is a shortcoming caused by our inability to properly test the kernel in an isolated environment. (In NetBSD, we'd use rump for this and thus the machine would never panic.)