From owner-freebsd-current@FreeBSD.ORG Sun Jun 8 16:39:08 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8852DB57 for ; Sun, 8 Jun 2014 16:39:08 +0000 (UTC) Received: from mail-qg0-f42.google.com (mail-qg0-f42.google.com [209.85.192.42]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 466A32484 for ; Sun, 8 Jun 2014 16:39:07 +0000 (UTC) Received: by mail-qg0-f42.google.com with SMTP id q107so7914208qgd.15 for ; Sun, 08 Jun 2014 09:39:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=APTrWVbNkjTw46AARtjn7pvNi8KVkQ061sggV/SPsq8=; b=AKjttTFMd3RaMuE7aMuG4/rz5XrDUzXyzlSIJqojCxrvrQ8zZqT+Ioe8uPt745q6hf G/O6K0+9+09sfnX2+S0jpc7mNEmW7mrGvKFWxinAZXlSuzIZFphl9MDTfFmIK04dJQDE 88iZ4zTRBvLZmQfj9uuosnHZIzPBgLp983klYiX0e8Waq7Uq2njswfh01BGsDtikwn2m 4VmmmzVl/oe3YqpH0UHmnWjUxIxZp96wKQTQgCAuHj4KjapLDe8G1akVyFDGJg3olAXz osl9+KXHM/SwX1tn/EiFGAbqHW95+NcCkDUiJhjnNNpkcIRVcJtsu08V9XLAFSqolaZD q4mQ== X-Gm-Message-State: ALoCoQk13h3F9kEkcRUfVm1J3v6n7kxb0oprLKBqc6mhUYdpabk0XUf98T3+aMTvB2HXTiTM1AI3 X-Received: by 10.224.130.196 with SMTP id u4mr26894086qas.13.1402245541547; Sun, 08 Jun 2014 09:39:01 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Sun, 8 Jun 2014 09:38:40 -0700 (PDT) X-Originating-IP: [89.101.236.178] In-Reply-To: <20140607195350.GU3991@kib.kiev.ua> References: <20140607033334.GO3991@kib.kiev.ua> <20140607195350.GU3991@kib.kiev.ua> From: Julio Merino Date: Sun, 8 Jun 2014 12:38:40 -0400 X-Google-Sender-Auth: Z9Ex5U7kVnWjJ0cWym7EGMjXiXk Message-ID: Subject: Re: Turning TESTS on by default To: Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD CURRENT , Alan Somers X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jun 2014 16:39:08 -0000 On Sat, Jun 7, 2014 at 3:53 PM, Konstantin Belousov wrote: > Could somebody point out a popular software system that spills the > tests or other developer-only[*] stuff into the production install ? I > immediately remember the perl and its modules which have very extensive > test suite, but the test suite is not installed. > > [*] As is, developers of the system, not developers utilizing the product > as the base. That's the key you are missing: the FreeBSD test suite, just as the NetBSD test suite, have been designed from the very beginning as a tool for *both* developers and end users. Most test suites out there, if not all, do not have such a goal and thus it is uninteresting, and likely harmful, to install them (if only because it is non-trivial to execute them in a controlled manner). The FreeBSD test suite is intended to let users (especially sysadmins) routinely assess the functionality of their system, especially right after deployment and during system upgrades. This approach is partly intended to cover the fact that the FreeBSD project cannot possibly have[*] a large-enough testing farm to validate most hardware/software combinations run by our users. It is also partly intended to inspire confidence in the system by our users by being very transparent about what works and what may possibly not work. NetBSD encourages you to do test runs (even from cron) in its afterboot(8) manual page. * Mind you: companies such as Microsoft have extremely large test environments and, yet, they continuously fail to catch specific failures in some weird system combination. It is just a fact of the complexity of OSes. The difference between "them and us" is that they cannot afford to ship their test suite because it's a very important asset of theirs and because of the potential PR implications of users uncovering failures here and there (my guesses though). I recommend reading "The Old New Thing" blog for insightful details on all the testing that Windows goes thorough, for example.