Date: Mon, 23 Feb 2015 18:37:27 -0800 From: Craig Rodrigues <rodrigc@FreeBSD.org> To: Ed Maste <emaste@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Running tests as a developer prior to commit Message-ID: <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com> In-Reply-To: <CAPyFy2DKMktGV2yp=u2aesoiGH=s_VMksO_7HkPwmwi6MU3S4Q@mail.gmail.com> References: <CAPyFy2DKMktGV2yp=u2aesoiGH=s_VMksO_7HkPwmwi6MU3S4Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 23, 2015 at 2:19 PM, Ed Maste <emaste@freebsd.org> wrote: > > It'd be even better if we can facilitate simple runs of a > suitable subset of tests prior to a commit. It looks like "make test" > is close, despite the warnings it emits about being experimental. Is > there a path we can take to support it, for at least a common case of > developing and testing on a FreeBSD-CURRENT install? > On the surface, your request seems simple. However, I have seen the same request asked in multiple companies I have worked at, and it is not so simple. If someone touches an arbitrary piece of code in FreeBSD, what are the subset of tests that need to be run to validate the change? Some changes are very simple, and it is quite easy to extrapolate which tests need to be run. However, some changes are more subtle, and it is not so obvious which tests are affected. Having knowledge about what changes affect what unit tests requires almost an expert system to be built. The other suggestion I have seen in other companies is to require developers to run all unit tests for every change before they commit. This works to some extent, because in a company you can force this behavior. For FreeBSD, I think this will fail. We already have over 3000 tests under /usr/tests. It's not that hard to run them with "cd /usr/tests && kyua test". However, requiring FreeBSD developers to run these tests for every change they do is a big change in how they submit code to FreeBSD. This will probably result in rebellion. I would like to see a compromise solution. I would like an optional workflow, where someone does a pull request on the FreeBSD repository in GitHub, which then kicks off automation which builds an image, runs the kyua tests, and reports the results. After seeing these test results, the developer would still be responsible for manually committing the change to SVN. People who are OK with using GitHub can follow this workflow, and opt-in to the tests. People who hate git and GitHub can ignore it, and continue to commit directly to SVN as they do today. There are a whole lot of automation libraries built on top of Jenkins and GitHub which can facilitate this type of workflow. Projects like Mac Homebrew have very sophisticated workflows which kick off tests in response to pull requests. Someone in FreeBSD would need to build out this test infrastructure. It's doable, but a lot of work. It might be nice to make a Google Summer of Code project out of this. -- Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA>