Date: Thu, 26 Feb 2015 19:24:53 -0800 From: Garrett Cooper <yaneurabeya@gmail.com> To: Craig Rodrigues <rodrigc@FreeBSD.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Running tests as a developer prior to commit Message-ID: <8E237BF1-E8DB-44D6-BA9F-89E386D0F3C0@gmail.com> In-Reply-To: <CAG=rPVecKLnU%2BPXRiftH7-0HWnoLnA%2B%2Bn0jhWNJdAaesOEqPng@mail.gmail.com> References: <CAPyFy2DKMktGV2yp=u2aesoiGH=s_VMksO_7HkPwmwi6MU3S4Q@mail.gmail.com> <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com> <CAPyFy2AF%2BN=Ju0KmbLhheF94M7X8EOFAOPQN1FH6N213f4MKuQ@mail.gmail.com> <CAG=rPVecKLnU%2BPXRiftH7-0HWnoLnA%2B%2Bn0jhWNJdAaesOEqPng@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
On Feb 26, 2015, at 19:17, Craig Rodrigues <rodrigc@FreeBSD.org> wrote:
> On Mon, Feb 23, 2015 at 7:20 PM, Ed Maste <emaste@freebsd.org> wrote:
>
>>
>> This is precisely why I'd like to find a way to make it easy to run a
>> (rather small) subset of tests, if a developer desires to do so. The
>> alternative is that they decide it's too awkward to run the tests, and
>> just don't bother. Even in that case we're still better off today
>> (since the Jenkins task will catch it) than we were some time ago
>> before consistent test infrastructure existed. But it seems we ought
>> to avoid the latency, mailing list churn, and masking effect of a
>> broken test run if we can facilitate the developer running a set of
>> tests that they've identified.
>>
>
> If I could do:
> cd /usr/src/bin/ls/
> make test
>
> and have it run the unit tests associated with /bin/ls, that would be a
> good start.
>
> It wouldn't be perfect, because if a change to /bin/ls, affects some other
> test
> in /usr/src/usr.bin/foobar/ , this wouldn't catch it. You would have to do
> something
> more complicated by building an expert system which can compute the
> dependencies
> in the tree and run the appropriate tests. But having that is better than
> nothing.
>
> As far as I understand things, the way the unit tests are integrated in the
> Makefile
> infrastructure, the tests need to be staged to /usr/tests/ in order to be
> run.
> As part of the staging process, Kyuafiles are generated which are used as
> input
> to the "kyua" binary. I don't have any make-fu to know how to simplify all
> of this stuff.
>
> Well, on a positive note, at least we *are* running the unit tests every
> day as people
> commit code, and reporting the results. That's a huge deal.
> I've seen large companies who don't do as good a job with this stuff, so we
> are not doing too badly in FreeBSD.
>
> Over time, I am sure we will improve things with better automation.
>
> Julio Merino accomplished a huge milestone by getting *any* type of unit
> tests into the FreeBSD source
> tree, and integrated with the build. Regularly running the tests under
> automation was the next milestone.
>
> The next milestone will be making it easier for developers to use. I think
> that the solution
> which will be realistically achievable, which people will actually use and
> not rebel over
> is some type of test automation hooked into Github pull requests against
> the FreeBSD src tree.
Hi,
This hook (running tests from a subdir) already exists, but is incomplete:
$ make test
*** Using this test does not preclude you from running the tests
*** installed in /usr/tests. This test run may raise false
*** positives and/or false negatives.
*** WARNING: make test is experimental
***
*** Using this test does not preclude you from running the tests
*** installed in /usr/tests. This test run may raise false
*** positives and/or false negatives.
kyua: E: Load of '/usr/tests/bin/pkill/Kyuafile' failed: File '/usr/tests/bin/pkill/Kyuafile' not found.
*** Error code 2
Stop.
make: stopped in /usr/src.svn/bin/pkill/tests
I have code on github which expands make test to install everything to ${.OBJDIR}, then run the tests.
make testworld doesn’t exist either, so I’ll work on plumbing that into the build system, because having to cd into every directory to run tests is silly.
Thanks for the reminder :).
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQEcBAEBCgAGBQJU7+OFAAoJEMZr5QU6S73e8S0H/3TbaLzJbQf7QrcnrFwW0aLR
h4RNxQgQQp2DO/OluAZikb6H1WfdxJ074AVRisA+4cFm8kgRKKQ/41jf04fciWXz
2zpMEMyPojD/n++aSiU1kRyaNNk2mAH/hS/6w/1Z9X/0C7ah9/yOWTlJqtATGKOT
yXdFchMjqTz1GFLGOJGtLPxPFuuLrnl1JXOFmYpsrZLpkcJhx8LokkPHgcLxrvXm
hGRbzOoKtI0dNPy3Z+bgbyCjc95zb/FFN2FBjDFVGtaHv6WthJYIe61LQ8jErWJq
wHQ2sEWWr2S4GBspi5eqyC8eq1O1xCs42+5xnWMtSU4x2bx3ZkaAWgaEhn9WV9A=
=Bhnx
-----END PGP SIGNATURE-----
help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8E237BF1-E8DB-44D6-BA9F-89E386D0F3C0>