Date: Thu, 26 Feb 2015 19:24:53 -0800 From: Garrett Cooper <yaneurabeya@gmail.com> To: Craig Rodrigues <rodrigc@FreeBSD.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Running tests as a developer prior to commit Message-ID: <8E237BF1-E8DB-44D6-BA9F-89E386D0F3C0@gmail.com> In-Reply-To: <CAG=rPVecKLnU%2BPXRiftH7-0HWnoLnA%2B%2Bn0jhWNJdAaesOEqPng@mail.gmail.com> References: <CAPyFy2DKMktGV2yp=u2aesoiGH=s_VMksO_7HkPwmwi6MU3S4Q@mail.gmail.com> <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com> <CAPyFy2AF%2BN=Ju0KmbLhheF94M7X8EOFAOPQN1FH6N213f4MKuQ@mail.gmail.com> <CAG=rPVecKLnU%2BPXRiftH7-0HWnoLnA%2B%2Bn0jhWNJdAaesOEqPng@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_B5E27DFE-3A9E-4B01-BB52-8A02313317B3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
On Feb 26, 2015, at 19:17, Craig Rodrigues <rodrigc@FreeBSD.org> wrote:
> On Mon, Feb 23, 2015 at 7:20 PM, Ed Maste <emaste@freebsd.org> wrote:
>=20
>>=20
>> This is precisely why I'd like to find a way to make it easy to run a
>> (rather small) subset of tests, if a developer desires to do so. The
>> alternative is that they decide it's too awkward to run the tests, =
and
>> just don't bother. Even in that case we're still better off today
>> (since the Jenkins task will catch it) than we were some time ago
>> before consistent test infrastructure existed. But it seems we ought
>> to avoid the latency, mailing list churn, and masking effect of a
>> broken test run if we can facilitate the developer running a set of
>> tests that they've identified.
>>=20
>=20
> If I could do:
> cd /usr/src/bin/ls/
> make test
>=20
> and have it run the unit tests associated with /bin/ls, that would be =
a
> good start.
>=20
> It wouldn't be perfect, because if a change to /bin/ls, affects some =
other
> test
> in /usr/src/usr.bin/foobar/ , this wouldn't catch it. You would have =
to do
> something
> more complicated by building an expert system which can compute the
> dependencies
> in the tree and run the appropriate tests. But having that is better =
than
> nothing.
>=20
> As far as I understand things, the way the unit tests are integrated =
in the
> Makefile
> infrastructure, the tests need to be staged to /usr/tests/ in order to =
be
> run.
> As part of the staging process, Kyuafiles are generated which are used =
as
> input
> to the "kyua" binary. I don't have any make-fu to know how to =
simplify all
> of this stuff.
>=20
> Well, on a positive note, at least we *are* running the unit tests =
every
> day as people
> commit code, and reporting the results. That's a huge deal.
> I've seen large companies who don't do as good a job with this stuff, =
so we
> are not doing too badly in FreeBSD.
>=20
> Over time, I am sure we will improve things with better automation.
>=20
> Julio Merino accomplished a huge milestone by getting *any* type of =
unit
> tests into the FreeBSD source
> tree, and integrated with the build. Regularly running the tests =
under
> automation was the next milestone.
>=20
> The next milestone will be making it easier for developers to use. I =
think
> that the solution
> which will be realistically achievable, which people will actually use =
and
> not rebel over
> is some type of test automation hooked into Github pull requests =
against
> the FreeBSD src tree.
Hi,
This hook (running tests from a subdir) already exists, but is =
incomplete:
$ make test
*** Using this test does not preclude you from running the tests
*** installed in /usr/tests. This test run may raise false
*** positives and/or false negatives.
*** WARNING: make test is experimental
***
*** Using this test does not preclude you from running the tests
*** installed in /usr/tests. This test run may raise false
*** positives and/or false negatives.
kyua: E: Load of '/usr/tests/bin/pkill/Kyuafile' failed: File =
'/usr/tests/bin/pkill/Kyuafile' not found.
*** Error code 2
Stop.
make: stopped in /usr/src.svn/bin/pkill/tests
I have code on github which expands make test to install everything to =
${.OBJDIR}, then run the tests.
make testworld doesn=92t exist either, so I=92ll work on plumbing that =
into the build system, because having to cd into every directory to run =
tests is silly.
Thanks for the reminder :).
--Apple-Mail=_B5E27DFE-3A9E-4B01-BB52-8A02313317B3
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=signature.asc
Content-Type: application/pgp-signature;
name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQEcBAEBCgAGBQJU7+OFAAoJEMZr5QU6S73e8S0H/3TbaLzJbQf7QrcnrFwW0aLR
h4RNxQgQQp2DO/OluAZikb6H1WfdxJ074AVRisA+4cFm8kgRKKQ/41jf04fciWXz
2zpMEMyPojD/n++aSiU1kRyaNNk2mAH/hS/6w/1Z9X/0C7ah9/yOWTlJqtATGKOT
yXdFchMjqTz1GFLGOJGtLPxPFuuLrnl1JXOFmYpsrZLpkcJhx8LokkPHgcLxrvXm
hGRbzOoKtI0dNPy3Z+bgbyCjc95zb/FFN2FBjDFVGtaHv6WthJYIe61LQ8jErWJq
wHQ2sEWWr2S4GBspi5eqyC8eq1O1xCs42+5xnWMtSU4x2bx3ZkaAWgaEhn9WV9A=
=Bhnx
-----END PGP SIGNATURE-----
--Apple-Mail=_B5E27DFE-3A9E-4B01-BB52-8A02313317B3--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8E237BF1-E8DB-44D6-BA9F-89E386D0F3C0>