Date: Thu, 26 Feb 2015 19:17:07 -0800 From: Craig Rodrigues <rodrigc@FreeBSD.org> To: Ed Maste <emaste@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Running tests as a developer prior to commit Message-ID: <CAG=rPVecKLnU%2BPXRiftH7-0HWnoLnA%2B%2Bn0jhWNJdAaesOEqPng@mail.gmail.com> In-Reply-To: <CAPyFy2AF%2BN=Ju0KmbLhheF94M7X8EOFAOPQN1FH6N213f4MKuQ@mail.gmail.com> References: <CAPyFy2DKMktGV2yp=u2aesoiGH=s_VMksO_7HkPwmwi6MU3S4Q@mail.gmail.com> <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com> <CAPyFy2AF%2BN=Ju0KmbLhheF94M7X8EOFAOPQN1FH6N213f4MKuQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 23, 2015 at 7:20 PM, Ed Maste <emaste@freebsd.org> wrote: > > This is precisely why I'd like to find a way to make it easy to run a > (rather small) subset of tests, if a developer desires to do so. The > alternative is that they decide it's too awkward to run the tests, and > just don't bother. Even in that case we're still better off today > (since the Jenkins task will catch it) than we were some time ago > before consistent test infrastructure existed. But it seems we ought > to avoid the latency, mailing list churn, and masking effect of a > broken test run if we can facilitate the developer running a set of > tests that they've identified. > If I could do: cd /usr/src/bin/ls/ make test and have it run the unit tests associated with /bin/ls, that would be a good start. It wouldn't be perfect, because if a change to /bin/ls, affects some other test in /usr/src/usr.bin/foobar/ , this wouldn't catch it. You would have to do something more complicated by building an expert system which can compute the dependencies in the tree and run the appropriate tests. But having that is better than nothing. As far as I understand things, the way the unit tests are integrated in the Makefile infrastructure, the tests need to be staged to /usr/tests/ in order to be run. As part of the staging process, Kyuafiles are generated which are used as input to the "kyua" binary. I don't have any make-fu to know how to simplify all of this stuff. Well, on a positive note, at least we *are* running the unit tests every day as people commit code, and reporting the results. That's a huge deal. I've seen large companies who don't do as good a job with this stuff, so we are not doing too badly in FreeBSD. Over time, I am sure we will improve things with better automation. Julio Merino accomplished a huge milestone by getting *any* type of unit tests into the FreeBSD source tree, and integrated with the build. Regularly running the tests under automation was the next milestone. The next milestone will be making it easier for developers to use. I think that the solution which will be realistically achievable, which people will actually use and not rebel over is some type of test automation hooked into Github pull requests against the FreeBSD src tree. -- Craig -- Craig
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG=rPVecKLnU%2BPXRiftH7-0HWnoLnA%2B%2Bn0jhWNJdAaesOEqPng>