From owner-freebsd-testing@FreeBSD.ORG Tue Feb 24 03:20:32 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2A372AC5; Tue, 24 Feb 2015 03:20:32 +0000 (UTC) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E2B88E81; Tue, 24 Feb 2015 03:20:31 +0000 (UTC) Received: by mail-ig0-f174.google.com with SMTP id b16so23653181igk.1; Mon, 23 Feb 2015 19:20:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=6xaMIBDpKr+EXXlYXEOQtdFGC5dtmrvOIHK4wPPDZws=; b=YOorTY5OxDeNMNhyMXGk8Um8xkMVXot2kR1UX2L/uzKkFYqMR/Vkze79NvtBjeTz89 FKaf5gH54dY+RkilrTyyJzOv14Yn3B4tLw3bRv9EGX94pOv+FXhZ5I7/GJMti5PnccaK jr1g4YdOxC+3tL4TIikGcDiU/E3Hgx+TVQe7Tmddqqdl6cZ0ZBpsKVQF8F7dqCji0PM0 aKsGyj/QISHFUlslyYTd+FdOOu3xWH4EmEeyAXmDyFQLgPwLTIg4FkzdDdA9P9iffLd1 I785jTljNua6GHXZX0IhrNfiwiKSwKEC4qN/b/3FxhNFclTNjXQoZbIxjvOSHj9HGEM7 AkHA== X-Received: by 10.50.111.168 with SMTP id ij8mr17453900igb.43.1424748031285; Mon, 23 Feb 2015 19:20:31 -0800 (PST) MIME-Version: 1.0 Sender: carpeddiem@gmail.com Received: by 10.107.158.19 with HTTP; Mon, 23 Feb 2015 19:20:11 -0800 (PST) In-Reply-To: References: From: Ed Maste Date: Mon, 23 Feb 2015 22:20:11 -0500 X-Google-Sender-Auth: hYXM7QweYfZ9kVo_nm6oqDgx2Qw Message-ID: Subject: Re: Running tests as a developer prior to commit To: Craig Rodrigues Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Feb 2015 03:20:32 -0000 On 23 February 2015 at 21:37, Craig Rodrigues wrote: > > If someone touches an arbitrary piece of code in FreeBSD, > what are the subset of tests that need to be run to validate the change? > Some changes are very simple, and it is quite easy to extrapolate which > tests need to be run. > > However, some changes are more subtle, and it is not so obvious which > tests are affected. Absolutely, but I'm not suggesting that we change the current setup, or avoid proceeding with other ideas you raise below. We'll still have the full test run from Jenkins if someone commits after running a chosen subset of tests. The use case I'm considering is someone making a change to a single binary under one of the bin directories, which has Kyua tests in its own subdirectory. My goal is to be able to: - run the test against the not-yet-installed tool built in $OBJDIR - report overall pass/failure and 0/non-0 return code (in addition to other output) - easily obtain stdout / stderr for failing tests to track down issues > The other suggestion I have seen in other companies is to require developers > to run all unit tests for every change before they commit. This works to > some > extent, because in a company you can force this behavior. > > For FreeBSD, I think this will fail. We already have over 3000 tests under > /usr/tests. > It's not that hard to run them with "cd /usr/tests && kyua test". > However, requiring FreeBSD developers to run these tests > for every change they do is a big change in how they submit code to FreeBSD. > This will probably result in rebellion. This is precisely why I'd like to find a way to make it easy to run a (rather small) subset of tests, if a developer desires to do so. The alternative is that they decide it's too awkward to run the tests, and just don't bother. Even in that case we're still better off today (since the Jenkins task will catch it) than we were some time ago before consistent test infrastructure existed. But it seems we ought to avoid the latency, mailing list churn, and masking effect of a broken test run if we can facilitate the developer running a set of tests that they've identified. > I would like to see a compromise solution. I would like an optional > workflow, > where someone does a pull request on the FreeBSD repository in GitHub, which > then > kicks off automation which builds an image, runs the kyua tests, and reports > the results. I think a project like this would be excellent to have, and I would personally make use of it. As you say it will be a lot of work, and it still introduces a great deal of latency over "make test" for a simple userland binary change.