Date: Mon, 23 Feb 2015 22:20:11 -0500 From: Ed Maste <emaste@freebsd.org> To: Craig Rodrigues <rodrigc@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Running tests as a developer prior to commit Message-ID: <CAPyFy2AF%2BN=Ju0KmbLhheF94M7X8EOFAOPQN1FH6N213f4MKuQ@mail.gmail.com> In-Reply-To: <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com> References: <CAPyFy2DKMktGV2yp=u2aesoiGH=s_VMksO_7HkPwmwi6MU3S4Q@mail.gmail.com> <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23 February 2015 at 21:37, Craig Rodrigues <rodrigc@freebsd.org> wrote: > > If someone touches an arbitrary piece of code in FreeBSD, > what are the subset of tests that need to be run to validate the change? > Some changes are very simple, and it is quite easy to extrapolate which > tests need to be run. > > However, some changes are more subtle, and it is not so obvious which > tests are affected. Absolutely, but I'm not suggesting that we change the current setup, or avoid proceeding with other ideas you raise below. We'll still have the full test run from Jenkins if someone commits after running a chosen subset of tests. The use case I'm considering is someone making a change to a single binary under one of the bin directories, which has Kyua tests in its own subdirectory. My goal is to be able to: - run the test against the not-yet-installed tool built in $OBJDIR - report overall pass/failure and 0/non-0 return code (in addition to other output) - easily obtain stdout / stderr for failing tests to track down issues > The other suggestion I have seen in other companies is to require developers > to run all unit tests for every change before they commit. This works to > some > extent, because in a company you can force this behavior. > > For FreeBSD, I think this will fail. We already have over 3000 tests under > /usr/tests. > It's not that hard to run them with "cd /usr/tests && kyua test". > However, requiring FreeBSD developers to run these tests > for every change they do is a big change in how they submit code to FreeBSD. > This will probably result in rebellion. This is precisely why I'd like to find a way to make it easy to run a (rather small) subset of tests, if a developer desires to do so. The alternative is that they decide it's too awkward to run the tests, and just don't bother. Even in that case we're still better off today (since the Jenkins task will catch it) than we were some time ago before consistent test infrastructure existed. But it seems we ought to avoid the latency, mailing list churn, and masking effect of a broken test run if we can facilitate the developer running a set of tests that they've identified. > I would like to see a compromise solution. I would like an optional > workflow, > where someone does a pull request on the FreeBSD repository in GitHub, which > then > kicks off automation which builds an image, runs the kyua tests, and reports > the results. I think a project like this would be excellent to have, and I would personally make use of it. As you say it will be a lot of work, and it still introduces a great deal of latency over "make test" for a simple userland binary change.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPyFy2AF%2BN=Ju0KmbLhheF94M7X8EOFAOPQN1FH6N213f4MKuQ>