From owner-freebsd-testing@FreeBSD.ORG Fri Feb 27 03:17:09 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E5EA8521; Fri, 27 Feb 2015 03:17:09 +0000 (UTC) Received: from mail-la0-x22a.google.com (mail-la0-x22a.google.com [IPv6:2a00:1450:4010:c03::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 62C38623; Fri, 27 Feb 2015 03:17:09 +0000 (UTC) Received: by labgq15 with SMTP id gq15so15004532lab.6; Thu, 26 Feb 2015 19:17:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=XEVw3sxCJdRdVmJzk8w7Pm/StExB5E6m2VOLuQPUk8Q=; b=SFvEMBX8P/nBLVeA3xnE+/x+oi5sm3HFoct6X5UD7yBZX0A5dO7pCdpNuGI2+y34DG 1fwttH0fssrwjLUgdAJYf6MMbwCRkD68Uqy2BDP5BWiWRDD4f1LfxsIu9b3Lga1ITbHz MiYJFr+EJhKD5cEU0DMYHQGQusZvjAfCWv5XpCBZwUez8eF2bopjHw7cRQyISOQXEoTG XLn0Fj2zShoOtEXQihW2ezTmVC9u0fNj8Y9OYWmNIdYGM1Cv0lLKo0V22VxqQUo4Zhdx eiP5xwmiHb4NbBAfsLDShEjCejaKIRSIgtT88WNhrZ6KivjAw5/gxFTp+Ua8cDcbnWqL 3ugw== MIME-Version: 1.0 X-Received: by 10.112.223.7 with SMTP id qq7mr10514614lbc.81.1425007027338; Thu, 26 Feb 2015 19:17:07 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.82.164 with HTTP; Thu, 26 Feb 2015 19:17:07 -0800 (PST) In-Reply-To: References: Date: Thu, 26 Feb 2015 19:17:07 -0800 X-Google-Sender-Auth: QpQStFOWg9ycH8CbtyczJ_7f4ow Message-ID: Subject: Re: Running tests as a developer prior to commit From: Craig Rodrigues To: Ed Maste Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Feb 2015 03:17:10 -0000 On Mon, Feb 23, 2015 at 7:20 PM, Ed Maste wrote: > > This is precisely why I'd like to find a way to make it easy to run a > (rather small) subset of tests, if a developer desires to do so. The > alternative is that they decide it's too awkward to run the tests, and > just don't bother. Even in that case we're still better off today > (since the Jenkins task will catch it) than we were some time ago > before consistent test infrastructure existed. But it seems we ought > to avoid the latency, mailing list churn, and masking effect of a > broken test run if we can facilitate the developer running a set of > tests that they've identified. > If I could do: cd /usr/src/bin/ls/ make test and have it run the unit tests associated with /bin/ls, that would be a good start. It wouldn't be perfect, because if a change to /bin/ls, affects some other test in /usr/src/usr.bin/foobar/ , this wouldn't catch it. You would have to do something more complicated by building an expert system which can compute the dependencies in the tree and run the appropriate tests. But having that is better than nothing. As far as I understand things, the way the unit tests are integrated in the Makefile infrastructure, the tests need to be staged to /usr/tests/ in order to be run. As part of the staging process, Kyuafiles are generated which are used as input to the "kyua" binary. I don't have any make-fu to know how to simplify all of this stuff. Well, on a positive note, at least we *are* running the unit tests every day as people commit code, and reporting the results. That's a huge deal. I've seen large companies who don't do as good a job with this stuff, so we are not doing too badly in FreeBSD. Over time, I am sure we will improve things with better automation. Julio Merino accomplished a huge milestone by getting *any* type of unit tests into the FreeBSD source tree, and integrated with the build. Regularly running the tests under automation was the next milestone. The next milestone will be making it easier for developers to use. I think that the solution which will be realistically achievable, which people will actually use and not rebel over is some type of test automation hooked into Github pull requests against the FreeBSD src tree. -- Craig -- Craig