From owner-freebsd-testing@FreeBSD.ORG Tue Feb 24 02:37:30 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3796AE29; Tue, 24 Feb 2015 02:37:30 +0000 (UTC) Received: from mail-la0-x22b.google.com (mail-la0-x22b.google.com [IPv6:2a00:1450:4010:c03::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A94808C1; Tue, 24 Feb 2015 02:37:29 +0000 (UTC) Received: by labgq15 with SMTP id gq15so22978707lab.6; Mon, 23 Feb 2015 18:37:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Sdse0+oqETE0LhSsYK+zUg0kd3ufyScfpZtUrk2V9Ak=; b=1AwV9/nNpc23boE0Ib1MH68dCqz/DhSGFFfV+5/fsZjGE7u3DHyAhFJqs4NpMY5Dlu B7vsJTc9O/KfbqgxFsfUmlbldOD9b+fG14L0YSvZ+pt6QjQlDcMldL1mElDdEgMIJMOZ W5J9LxkbhwpGILQGtvVEFKF58kp3BjnxTPjTXTMEPCsgqeuaxuPbfCs1iFHU5j/Bj/gg qc/Ff91t27qaAvaNYFTu0raDdN+TchdDGr4qhHkLk2AURuleomwqQgCd9rLebjVHB8GT eVsfo8vYTtnUnIZw5Q40YakmVeaogPrxcfMaRjOyZ+OblQVHSZ9EIlrGnl8eFKIqhZAG smpw== MIME-Version: 1.0 X-Received: by 10.112.204.197 with SMTP id la5mr6549019lbc.29.1424745447692; Mon, 23 Feb 2015 18:37:27 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.82.164 with HTTP; Mon, 23 Feb 2015 18:37:27 -0800 (PST) In-Reply-To: References: Date: Mon, 23 Feb 2015 18:37:27 -0800 X-Google-Sender-Auth: KqupghcdnycrVZ0ftGGV_4rdpx0 Message-ID: Subject: Re: Running tests as a developer prior to commit From: Craig Rodrigues To: Ed Maste Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Feb 2015 02:37:30 -0000 On Mon, Feb 23, 2015 at 2:19 PM, Ed Maste wrote: > > It'd be even better if we can facilitate simple runs of a > suitable subset of tests prior to a commit. It looks like "make test" > is close, despite the warnings it emits about being experimental. Is > there a path we can take to support it, for at least a common case of > developing and testing on a FreeBSD-CURRENT install? > On the surface, your request seems simple. However, I have seen the same request asked in multiple companies I have worked at, and it is not so simple. If someone touches an arbitrary piece of code in FreeBSD, what are the subset of tests that need to be run to validate the change? Some changes are very simple, and it is quite easy to extrapolate which tests need to be run. However, some changes are more subtle, and it is not so obvious which tests are affected. Having knowledge about what changes affect what unit tests requires almost an expert system to be built. The other suggestion I have seen in other companies is to require developers to run all unit tests for every change before they commit. This works to some extent, because in a company you can force this behavior. For FreeBSD, I think this will fail. We already have over 3000 tests under /usr/tests. It's not that hard to run them with "cd /usr/tests && kyua test". However, requiring FreeBSD developers to run these tests for every change they do is a big change in how they submit code to FreeBSD. This will probably result in rebellion. I would like to see a compromise solution. I would like an optional workflow, where someone does a pull request on the FreeBSD repository in GitHub, which then kicks off automation which builds an image, runs the kyua tests, and reports the results. After seeing these test results, the developer would still be responsible for manually committing the change to SVN. People who are OK with using GitHub can follow this workflow, and opt-in to the tests. People who hate git and GitHub can ignore it, and continue to commit directly to SVN as they do today. There are a whole lot of automation libraries built on top of Jenkins and GitHub which can facilitate this type of workflow. Projects like Mac Homebrew have very sophisticated workflows which kick off tests in response to pull requests. Someone in FreeBSD would need to build out this test infrastructure. It's doable, but a lot of work. It might be nice to make a Google Summer of Code project out of this. -- Craig