From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 31 17:10:49 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BF76B5C; Tue, 31 Dec 2013 17:10:49 +0000 (UTC) Received: from mail-pa0-x22e.google.com (mail-pa0-x22e.google.com [IPv6:2607:f8b0:400e:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 08EBA113D; Tue, 31 Dec 2013 17:10:49 +0000 (UTC) Received: by mail-pa0-f46.google.com with SMTP id kp14so10939775pab.19 for ; Tue, 31 Dec 2013 09:10:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=WRV8WPk+1+qVdPQ7+4DQqkrNo8jpJAeRNryDHuPAhfs=; b=0VsJ0BE2axYyH4O1PMU+FmRe/KC198beMFLXd9pRUuc30/vBglbt9+fD6Iztl9csn1 g6Rg7vkMYsq19Q/C+H0CxTTE/vHn+cUJ4K1b9XYqZvoVvH9GMtamctPrUbNPdm4eZdna BAxc40Q8cnjDRpoxqgV8SyXfX5uYFfWOeqDUePfvgtkEqW7uLW7nXbTa/LLeh3Af+nzc xrd+opUpmzgkGF98rMNc9m3ZYfEY2YgE3fuSJdmCvNN6w4KslCeWLrwdIAK9UxEDwK7u lZFScHj07vuvzsQBiXtlas1neZ7NEOqjGayBMps1e4CFoHaO81TnUZhgIjIs6SiljZIQ 9WXg== X-Received: by 10.66.146.199 with SMTP id te7mr77152250pab.106.1388509848663; Tue, 31 Dec 2013 09:10:48 -0800 (PST) Received: from [10.214.56.107] (mobile-166-147-083-140.mycingular.net. [166.147.83.140]) by mx.google.com with ESMTPSA id un5sm25555594pab.3.2013.12.31.09.10.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 31 Dec 2013 09:10:46 -0800 (PST) References: Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <7B0C24C8-8F93-4658-8D77-09B7C4F26805@gmail.com> X-Mailer: iPhone Mail (11B554a) From: Garrett Cooper Subject: Re: First cut of the continuous testing infrastructure Date: Tue, 31 Dec 2013 09:10:42 -0800 To: Julio Merino X-Mailman-Approved-At: Tue, 31 Dec 2013 17:15:16 +0000 Cc: "freebsd-testing@freebsd.org" , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Dec 2013 17:10:49 -0000 > On Dec 31, 2013, at 5:22, Julio Merino wrote: >=20 > Dear all, >=20 > With all the work that has gone into the tree to add the barebones of > a test suite in FreeBSD, and with the generous availability of 3 > machines for testing purposes, I am pleased to announce the very first > cut of the testing infrastructure! >=20 > Before spoiling the fun, let me make clear that these pages are /not/ > final by any means. There is still a ton of work to be done to, at > least: 1) make the reports more amenable and usable; 2) extend the > infrastructure to cover more releases and architectures; 3) grow the > test suite to make it a useful tool; 4) fix existing breakage; and 5) > notify developers when things break (possibly even pinpointing at > faulty commits!). >=20 > Having said that, point your browser at: >=20 > http://kyua1.nyi.freebsd.org/ >=20 > and have fun! >=20 > You may read some more on this work on the project page: >=20 > http://wiki.freebsd.org/TestSuite >=20 > and on my blog post introducing the FreeBSD test suite: >=20 > http://julipedia.meroh.net/2013/12/introducing-freebsd-test-suite.html >=20 > Hoping you find this useful in its current form... let the feedback flow! >=20 > Enjoy your entry into 2014 :-) Well done! The only thing I see that could be improved in the HTML format is= : 1. The total number of tests should be the last row. 2. Non-engineers (and some engineers like me) like percentages, in addition t= o raw numbers. Helps reduce panic when someone goes "oh my gosh! We have 18 t= ests failing!!!" Cheers and have a happy new year :)! -Garrett PS some of the ATF tests (for 0.17 at least) won't work unless you have the r= ight configuration file [from share/?], but if you have that configuration f= ile, it will cause regular tests to bomb because of recursion with /bin/sh .= Granted, that was with ATF, not Kyua driving the tests...=