From owner-freebsd-arch@FreeBSD.ORG Mon Nov 19 16:16:53 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4E985572; Mon, 19 Nov 2012 16:16:53 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from exprod7og101.obsmtp.com (exprod7og101.obsmtp.com [64.18.2.155]) by mx1.freebsd.org (Postfix) with ESMTP id 15B848FC12; Mon, 19 Nov 2012 16:16:48 +0000 (UTC) Received: from P-EMHUB01-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob101.postini.com ([64.18.6.12]) with SMTP ID DSNKUKpbaU8BAqCB3zW+QQ0+qyrFb39g7s+2@postini.com; Mon, 19 Nov 2012 08:16:52 PST Received: from magenta.juniper.net (172.17.27.123) by P-EMHUB01-HQ.jnpr.net (172.24.192.33) with Microsoft SMTP Server (TLS) id 8.3.213.0; Mon, 19 Nov 2012 08:13:34 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.24.29.229]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id qAJGDX328225; Mon, 19 Nov 2012 08:13:33 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos.jnpr.net (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 6A37C58094; Mon, 19 Nov 2012 08:13:33 -0800 (PST) To: Poul-Henning Kamp Subject: Re: [RFC] test layout/standardization for FreeBSD In-Reply-To: <31881.1353312581@critter.freebsd.dk> References: <7099.1352886181@critter.freebsd.dk> <31881.1353312581@critter.freebsd.dk> Comments: In-reply-to: "Poul-Henning Kamp" message dated "Mon, 19 Nov 2012 08:09:41 +0000." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Mon, 19 Nov 2012 08:13:33 -0800 Message-ID: <20121119161333.6A37C58094@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain Cc: Garrett Cooper , George Neville-Neil , "freebsd-arch@FreeBSD.org Arch" , Matthew Fleming , Marcel Moolenaar X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 16:16:53 -0000 On Mon, 19 Nov 2012 08:09:41 +0000, "Poul-Henning Kamp" writes: >Some tests are not 100% deterministic no matter what. That is >an important distinction. Yes. It would probably be worth defining some meta tags to identify things like that. Even if the outcome were nothing more than a warning to the user "hey the results here can vary - don't panic" Still, to me, tests of that nature start to sound more like system tests than unit-tests. >>Also, the estimated duration for tests is very platform specific. > >Only if CPU or I/O bound. Tests for things like dynamic route-expiry, >and other timeout driven parts of the system will have long realtime >clock, but short cpu-clock. Again, that sounds more like a system test than unit-test. For a unit-test you would want to mock the environment so that your timer events or whatever are fully controlled (and compressed). >I fully agree with your "crawl before we run" approach, but testing >is a quite mature, if underappreciated, discipline, and we would do >well to learn from others mistakes. Agreed. One of the common misstakes it seems is bluring the boundaries between unit, function and system tests. System tests are needed for sure, but simple unit-tests (of the bits that can be simply unit-tested) are far more valuable for establishing good overal test coverage. >All I'm asking at this point, is that we make a space for storing >this metadata in a machine-readable format. Sounds reasonable, though exactly what and how, is to be determined, and my hope is that such info should not be needed until we start pushing unit-tests into system test territory. Thanks --sjg