From owner-freebsd-virtualization@FreeBSD.ORG Sat Oct 25 21:46:48 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62229484; Sat, 25 Oct 2014 21:46:48 +0000 (UTC) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1629EDD4; Sat, 25 Oct 2014 21:46:48 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id eu11so3107879pac.30 for ; Sat, 25 Oct 2014 14:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=ROBklQZYzRnOWEg4u6+mXXSEkyoA39Nb6IiVCDo1Xxw=; b=TxhmpWmG3d4QN+IylTmBpQ4iKoZbx8OXhNghtQONHkI+RGzBN2aOzpdlTaPeIua/zr rCd1a80riRlgYFrpHlJ5FuvMCZbPrL6oWUmajDRwVFMUWEILl01baRhHrwaFj6KljQmQ iq5i6PImVvu/vaOv7JJo5bgxDUqKGipAU9HAC/poO7y0DL5yt4Bbs1wvk7gkvLm1Hm0J S/4c5derxn850sFBlqrbnTE42zGlBGDkmbDmZEP43OBvkVdoHFNHBAb219BzbA7TaMP4 ofxFP139tjrcShC8epy6Vrn43rKU1K8lw66K+m9wp9uxbK7ePFV2Wfxd/4UmqlvwTYvW sZtw== X-Received: by 10.70.19.101 with SMTP id d5mr13514906pde.79.1414273607611; Sat, 25 Oct 2014 14:46:47 -0700 (PDT) Received: from [192.168.20.5] (c-98-247-240-204.hsd1.wa.comcast.net. [98.247.240.204]) by mx.google.com with ESMTPSA id jq5sm6960728pbc.32.2014.10.25.14.46.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 25 Oct 2014 14:46:46 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Garrett Cooper In-Reply-To: Date: Sat, 25 Oct 2014 14:46:46 -0700 Message-Id: <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> To: "K. Macy" X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 21:46:48 -0000 --Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 25, 2014, at 13:20, K. Macy wrote: >>>> Alan also suggested against integrating the test suite as-is, = because as he said, "Remember, don't run these tests on a production = system. They WILL cause panics and deadlocks, and they may cause data = loss too.=94 >>>>=20 >>>> Cheers, >>>> -Garrett >>>=20 >>> Wait, we want to sweep those bugs under the rug? What exactly is = wrong with making a test harness that can very easily reproduce a known = problem? The chances are that anyone will dive into it once the bug is = easily reproducible. >>=20 >> Sweeping bugs under the rug is not what I plan on doing; I=92m = marking these as expected failures, as opposed to having them = continually panic a machine. Once a ZFS dev takes a look at the issue = and resolves them, then the ZFS dev can remove the =93bail=94 calls I=92m = adding to the testcases. >=20 > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions >=20 > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for. (seeing as how my other post isn=92t in the -testing archives yet..) Panicking a node (what the tests are doing before last night) = and exiting with a non-zero exit code (what I=92m making them do with = the bail outs in tools/regression/zfs) are both considered test = failures. The difference being that I can safely run all of the tests on = a production or a test machine without having to panic/reboot the box = and I get greater coverage in one fell swoop. If a developer wants they = can always delete the lines that bail out of the tests to get the = desired panic. Cheers, -Garrett --Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUTBpGAAoJEMZr5QU6S73eroQH/0EUjiiTQEnkaDgG29GxsSP4 pmerYl9PXlSJiN1t0ApDz7HxxWh0qTsV8+5OIdrWOU4gvmK3DEaq+HOMqVboVtS5 WLFy+/bljFhnEbF5b3UqPUyNFNaCKV7Rdr96oLnA9NVJRvEo6aNHF0+rZKyTDeMd Qb7YDscluRqyqFCugb3rcatFWVHLycQKZdiUSx5Mdc7PnXiJ4nK9nK9nq6gU3lAA u9F3wKFZLHuzI9Ko2MEUp9jqwoK8kL2wx3Q5YBvyCUUI7OmoIf2GNhhxOZwa4Tga lfrTGH4gOcRLzjGmOGCOqXLAG3j1JpPetzJywdYjfZIVu+iEp+FmX4vH6WjF8nw= =a/Nc -----END PGP SIGNATURE----- --Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC--