Date: Tue, 2 Oct 2012 09:17:39 -0700 From: "Simon J. Gerraty" <sjg@juniper.net> To: Garrett Cooper <yanegomi@gmail.com> Cc: freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org, sjg@juniper.net Subject: Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program Message-ID: <20121002161739.EFE0E58093@chaos.jnpr.net> In-Reply-To: <CAGH67wSaFeVyr_uNfigNQ%2BOBy4JmiLMa_u7c7=aqgB7wDKzsLg@mail.gmail.com> References: <CAGH67wRkOmy7rWLkxXnT2155PuSQpwOMyu7dTAKeO1WW2dju7g@mail.gmail.com> <CDA41F27-73C1-47CF-B84D-2627B1F7E7D8@xcllnt.net> <20121001223100.E7D0D58093@chaos.jnpr.net> <FDE5A28C-297A-4476-B06E-638049E20C6C@gmail.com> <20121002000030.54CEE58093@chaos.jnpr.net> <CAGH67wSaFeVyr_uNfigNQ%2BOBy4JmiLMa_u7c7=aqgB7wDKzsLg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 2 Oct 2012 07:19:55 -0700, Garrett Cooper writes:
>> We put the test cases in a subdir of the lib/prog
>> This has multiple benefits, and eliminates any impact on the normal
>> build of said libs/progs.
>
>Hmmm... that's one of the 3 approaches I provided, but it turned out
>to be annoying to make this work at first inspection because of how
>things were done in our build system. Just for review (even though
>it's OT), the three approaches I presented were as follows...
>
>Note: in all 3 examples, clock.c is the source code and t_clock.c is
>the relevant unittest code.
There is actually another:
lib/libc/Makefile
lib/libc/gen/...
.../clock.c
.../t_clock.c
lib/libc/tests/Makefile
that is the makefile for building/running the tests lives in the subdir,
there are advantages to this, but the test code itself can be
anywhere you like. Either next to the code that it tests, or in the
tests/ subdir, another subdir, or whatever combination makes the most sense.
Most of our ATF users put their test code in the subdir fwiw.
>> Also a number of our teams find it necessary to create mock classes etc to
>> adequately test their libs. Keeping all this in a tests/ subdir makes
>> it easy, to keep things neat & up to date, and ensure that the tests
>> pass. Trying to do all that in the same dir as the lib would be ugly.
>
>This (consolidating everything under a single directory) is the way
>that was requested by the beforementioned parties.
Sorry for not participating in that discussion ;-)
John - perhaps we do need that ATF list?
For example building a library and test apps in the one directory
either requires turning bsd.lib.mk into an even more complex thing that
a multi-prog bsd.pog.mk, or having the makefile behave completely
differently depending on the target requested.
That later is a bad idea. It precludes being able to seamlessly
integrate unit-tests into the build, and would need to be fixed if/when
attempting to introduce meta mode. Typically the unit-tests depend on
the library they are testing, a separate subdir for the tests/Makefile
makes capturing that dependency easy - otherwise you are back to the
unspeakably complex bsd.lib.mk
Far better to get these things right first up.
>I wish I had known that. I pinged marcel/obrien about this some weeks
>ago and didn't receive a response.
Many appologies.
>What are the plans for importing bmake into FreeBSD (this should
>probably be a different thread)?
The import has been done - just needs to be merged to contrib,
and there's a small patch to be committed.
We hope to get that done this week.
>I agree that it's a good thing (in theory) -- it'd just help to
>discuss this with Julio first because I know of a couple cases where
>this would be unusable given how bsd.test.mk is coded:
I do talk to Julio about ATF (I'm sjg@netbsd.org ;-)
though not specifically this - beyond (I think) mentioning that I did it
differently.
>1. The "it works for me on my machine!" certification:
>
>It encourages environment tainting, which could be a really, really
>bad thing; I've seen developers take interesting shortcuts when
True. In our (Junos) build we cleanse the environment rather carefully
and have a very clear distinction between building for the "host" (which
may just happend to be i386 based) and the i386 "target" for example.
I can provide build smarts to do all that - but didn't want to shove it
down anyone's throat ;-)
What "works for us" doesn't necessarily work for everyone.
>2. Failure in a subdirectory results in lost coverage:
>
>a/...
> .../b/... <- tests live here
> .../c/d/... <- tests live here
>
>If I execute make test from a/ (one of my goals), then it should
>iterate down a/b, then a/c/d and run the tests in a DFS style (BFS if
This gets back to the bmake topic. We actually avoid "walking" the
tree at all, eg. in the Junos build I have a target "atf-tests" which
builds all ATF tests dirs regardless of where they are (and any
pre-requisits they have). This is handy to include as a dependency of
the build target we use for branch syncs etc, to ensure no unit-tests
atrophy.
But the most common case - and the one to optimize for, is the person
making changes to libfoo, testing that he hasn't broken anything.
>could disguise other bugs. This would be fixed by changing bsd.test.mk
>to use atf-run properly, but that's work (not hard) that still needs
>to be done.
Yes, atf.test.mk already does that.
>Hmmmm... My goal was to make ATF a "one ring to rule them all"
>infrastructure so that way everything could be unified under ATF in
ATF is a pretty good framework - which is why we (Juniper) use it.
We want the flexibility to have more than one framework, but the number
should be very small. So far I have persuaded teams that wanted to
adopt alternate frameworks, that ATF could easily meet their needs.
>one way, shape or form (even if it's just reporting), because lord
>knows we don't need a lua, unittest/nose, etc wrapper for reporting
>results. All of the jobs I've worked at write custom wrappers around
Yes, having all "frameworks" able to output results in a consistent
format is definitely a good thing.
Thanks
--sjg
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121002161739.EFE0E58093>