From owner-freebsd-testing@FreeBSD.ORG Tue Dec 31 13:23:03 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD1245E7 for ; Tue, 31 Dec 2013 13:23:03 +0000 (UTC) Received: from mail-lb0-f172.google.com (mail-lb0-f172.google.com [209.85.217.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 46D7B116B for ; Tue, 31 Dec 2013 13:23:02 +0000 (UTC) Received: by mail-lb0-f172.google.com with SMTP id x18so6271770lbi.3 for ; Tue, 31 Dec 2013 05:23:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=fEY6mFN8gHQJT0Xq6nCkceGzranI9V3cv8r0pU8DH9E=; b=YnF7g9bWu6f2B6ipZdyoNvGtiy2JEd+uJT9oy00LJe27jAYx52EJgyfh4YrHBNywGE BglGVaTL1crxyoHUfwFnNhOmIHdm9eXCp53Y84lKC0MNqp0P0yzYGtycxgOwKSx2Bp3X 4HkKBjRkpkFVyV7eO8wszX0VTz5T3MrlU5uCTBpi3ggPmcAC3oLtAOdCPy3Rj3+AwOwi s78IeD9SZNpj8shu3rUMnS1p1aX3fnqx+sPHrds0ehmr7fhmRtdh7dYdow9PV2mj62ak WF8hPvfAjwl/y7JrIBinXSw+XtNgAxaUoAEM7xJEWRorFdFxhGRyXkQ0LS0q7Jp1vZaX 1nww== X-Gm-Message-State: ALoCoQk8sCfdQFIoututIA1YnF6E4gI4nsiYq/ICzqm/zPdTn7Ec76mp1w2E6NANvWe615S6ouec X-Received: by 10.152.225.129 with SMTP id rk1mr18265lac.74.1388496180919; Tue, 31 Dec 2013 05:23:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.129.101 with HTTP; Tue, 31 Dec 2013 05:22:40 -0800 (PST) X-Originating-IP: [108.176.158.82] From: Julio Merino Date: Tue, 31 Dec 2013 08:22:40 -0500 Message-ID: Subject: First cut of the continuous testing infrastructure To: freebsd-testing@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Dec 2013 13:23:03 -0000 Dear all, With all the work that has gone into the tree to add the barebones of a test suite in FreeBSD, and with the generous availability of 3 machines for testing purposes, I am pleased to announce the very first cut of the testing infrastructure! Before spoiling the fun, let me make clear that these pages are /not/ final by any means. There is still a ton of work to be done to, at least: 1) make the reports more amenable and usable; 2) extend the infrastructure to cover more releases and architectures; 3) grow the test suite to make it a useful tool; 4) fix existing breakage; and 5) notify developers when things break (possibly even pinpointing at faulty commits!). Having said that, point your browser at: http://kyua1.nyi.freebsd.org/ and have fun! You may read some more on this work on the project page: http://wiki.freebsd.org/TestSuite and on my blog post introducing the FreeBSD test suite: http://julipedia.meroh.net/2013/12/introducing-freebsd-test-suite.html Hoping you find this useful in its current form... let the feedback flow! Enjoy your entry into 2014 :-) -- Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Tue Dec 31 16:08:00 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C2C843E7; Tue, 31 Dec 2013 16:08:00 +0000 (UTC) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 93A4E1D33; Tue, 31 Dec 2013 16:08:00 +0000 (UTC) Received: from [209.249.190.124] (port=55200 helo=gnnmac.hudson-trading.com) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1) (envelope-from ) id 1Vy1rK-0004Tl-FN; Tue, 31 Dec 2013 11:07:58 -0500 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: First cut of the continuous testing infrastructure From: George Neville-Neil In-Reply-To: Date: Tue, 31 Dec 2013 11:07:56 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Julio Merino X-Mailer: Apple Mail (2.1827) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: freebsd-testing@freebsd.org, freebsd-hackers@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Dec 2013 16:08:00 -0000 On Dec 31, 2013, at 8:22 , Julio Merino wrote: > Dear all, >=20 > With all the work that has gone into the tree to add the barebones of > a test suite in FreeBSD, and with the generous availability of 3 > machines for testing purposes, I am pleased to announce the very first > cut of the testing infrastructure! >=20 > Before spoiling the fun, let me make clear that these pages are /not/ > final by any means. There is still a ton of work to be done to, at > least: 1) make the reports more amenable and usable; 2) extend the > infrastructure to cover more releases and architectures; 3) grow the > test suite to make it a useful tool; 4) fix existing breakage; and 5) > notify developers when things break (possibly even pinpointing at > faulty commits!). >=20 > Having said that, point your browser at: >=20 > http://kyua1.nyi.freebsd.org/ >=20 > and have fun! >=20 > You may read some more on this work on the project page: >=20 > http://wiki.freebsd.org/TestSuite >=20 > and on my blog post introducing the FreeBSD test suite: >=20 > = http://julipedia.meroh.net/2013/12/introducing-freebsd-test-suite.html >=20 > Hoping you find this useful in its current form... let the feedback = flow! >=20 > Enjoy your entry into 2014 :-) >=20 This looks great! We need to get people to start writing tests as part = of their day to day coding now. Best, George From owner-freebsd-testing@FreeBSD.ORG Tue Dec 31 17:10:49 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BF76B5C; Tue, 31 Dec 2013 17:10:49 +0000 (UTC) Received: from mail-pa0-x22e.google.com (mail-pa0-x22e.google.com [IPv6:2607:f8b0:400e:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 08EBA113D; Tue, 31 Dec 2013 17:10:49 +0000 (UTC) Received: by mail-pa0-f46.google.com with SMTP id kp14so10939775pab.19 for ; Tue, 31 Dec 2013 09:10:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=WRV8WPk+1+qVdPQ7+4DQqkrNo8jpJAeRNryDHuPAhfs=; b=0VsJ0BE2axYyH4O1PMU+FmRe/KC198beMFLXd9pRUuc30/vBglbt9+fD6Iztl9csn1 g6Rg7vkMYsq19Q/C+H0CxTTE/vHn+cUJ4K1b9XYqZvoVvH9GMtamctPrUbNPdm4eZdna BAxc40Q8cnjDRpoxqgV8SyXfX5uYFfWOeqDUePfvgtkEqW7uLW7nXbTa/LLeh3Af+nzc xrd+opUpmzgkGF98rMNc9m3ZYfEY2YgE3fuSJdmCvNN6w4KslCeWLrwdIAK9UxEDwK7u lZFScHj07vuvzsQBiXtlas1neZ7NEOqjGayBMps1e4CFoHaO81TnUZhgIjIs6SiljZIQ 9WXg== X-Received: by 10.66.146.199 with SMTP id te7mr77152250pab.106.1388509848663; Tue, 31 Dec 2013 09:10:48 -0800 (PST) Received: from [10.214.56.107] (mobile-166-147-083-140.mycingular.net. [166.147.83.140]) by mx.google.com with ESMTPSA id un5sm25555594pab.3.2013.12.31.09.10.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 31 Dec 2013 09:10:46 -0800 (PST) References: Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <7B0C24C8-8F93-4658-8D77-09B7C4F26805@gmail.com> X-Mailer: iPhone Mail (11B554a) From: Garrett Cooper Subject: Re: First cut of the continuous testing infrastructure Date: Tue, 31 Dec 2013 09:10:42 -0800 To: Julio Merino Cc: "freebsd-testing@freebsd.org" , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Dec 2013 17:10:49 -0000 > On Dec 31, 2013, at 5:22, Julio Merino wrote: >=20 > Dear all, >=20 > With all the work that has gone into the tree to add the barebones of > a test suite in FreeBSD, and with the generous availability of 3 > machines for testing purposes, I am pleased to announce the very first > cut of the testing infrastructure! >=20 > Before spoiling the fun, let me make clear that these pages are /not/ > final by any means. There is still a ton of work to be done to, at > least: 1) make the reports more amenable and usable; 2) extend the > infrastructure to cover more releases and architectures; 3) grow the > test suite to make it a useful tool; 4) fix existing breakage; and 5) > notify developers when things break (possibly even pinpointing at > faulty commits!). >=20 > Having said that, point your browser at: >=20 > http://kyua1.nyi.freebsd.org/ >=20 > and have fun! >=20 > You may read some more on this work on the project page: >=20 > http://wiki.freebsd.org/TestSuite >=20 > and on my blog post introducing the FreeBSD test suite: >=20 > http://julipedia.meroh.net/2013/12/introducing-freebsd-test-suite.html >=20 > Hoping you find this useful in its current form... let the feedback flow! >=20 > Enjoy your entry into 2014 :-) Well done! The only thing I see that could be improved in the HTML format is= : 1. The total number of tests should be the last row. 2. Non-engineers (and some engineers like me) like percentages, in addition t= o raw numbers. Helps reduce panic when someone goes "oh my gosh! We have 18 t= ests failing!!!" Cheers and have a happy new year :)! -Garrett PS some of the ATF tests (for 0.17 at least) won't work unless you have the r= ight configuration file [from share/?], but if you have that configuration f= ile, it will cause regular tests to bomb because of recursion with /bin/sh .= Granted, that was with ATF, not Kyua driving the tests...=