Date: Tue, 2 Oct 2012 07:19:55 -0700 From: Garrett Cooper <yanegomi@gmail.com> To: "Simon J. Gerraty" <sjg@juniper.net> Cc: freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program Message-ID: <CAGH67wSaFeVyr_uNfigNQ%2BOBy4JmiLMa_u7c7=aqgB7wDKzsLg@mail.gmail.com> In-Reply-To: <20121002000030.54CEE58093@chaos.jnpr.net> References: <CAGH67wRkOmy7rWLkxXnT2155PuSQpwOMyu7dTAKeO1WW2dju7g@mail.gmail.com> <CDA41F27-73C1-47CF-B84D-2627B1F7E7D8@xcllnt.net> <20121001223100.E7D0D58093@chaos.jnpr.net> <FDE5A28C-297A-4476-B06E-638049E20C6C@gmail.com> <20121002000030.54CEE58093@chaos.jnpr.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Oct 1, 2012 at 5:00 PM, Simon J. Gerraty <sjg@juniper.net> wrote:
>>> Not to mention the fact that bsd.prog.mk goes from being relatively
>>> simple, to unspeakably hard to read, and all for rather limited =
>>return.
>
> This btw I think is the more important issue.
> I was looking at bsd.prog.mk in netbsd the other day.
> It has no business being that complex.
Ok. Even though I originally thought that my changes were a bit hacky
for not modifying bsd.man.mk, bsd.nls.mk, etc, looking back the way
they were resolved was a little elegant in its simplicity (albeit not
optimized).
...
>>It order to do this, I need to be able to build multiple programs so =
>>things are as transparent. On the flipside, bsd.prog[s].mk can live on =
>
> We put the test cases in a subdir of the lib/prog
> This has multiple benefits, and eliminates any impact on the normal
> build of said libs/progs.
Hmmm... that's one of the 3 approaches I provided, but it turned out
to be annoying to make this work at first inspection because of how
things were done in our build system. Just for review (even though
it's OT), the three approaches I presented were as follows...
Note: in all 3 examples, clock.c is the source code and t_clock.c is
the relevant unittest code.
1. Test programs live with the sources (this was the requested approach), e.g.
lib/libc/gen/...
.../clock.c
.../t_clock.c
2. Test programs live in subdirs:
lib/libc/gen/...
.../clock.c
.../tests/...
.../t_clock.c
3. Test programs completely decoupled from the source tree:
lib/libc/gen/...
.../clock.c
tests/lib/libc/gen/...
.../t_clock.c
A hybrid approach (1.+3. / 2.+3.) should probably be used anyhow for
stress tests and the like that really have no business living in one
particular directory in the source tree (sort of achieving what
tools/regression does today, but hopefully in a less messy manner as
tools/regression appears to have grown organically minus any single
sane order).
> Also a number of our teams find it necessary to create mock classes etc to
> adequately test their libs. Keeping all this in a tests/ subdir makes
> it easy, to keep things neat & up to date, and ensure that the tests
> pass. Trying to do all that in the same dir as the lib would be ugly.
This (consolidating everything under a single directory) is the way
that was requested by the beforementioned parties.
>>> FWIW we use progs.mk (as bsd.progs.mk) from
>>> ftp://ftp.netbsd.org/pub/NetBSD/misc/sjg/mk-*.tar.gz=20
>>> It isn't ideal, but it certainly avoids a lot of churn and complexity
>>> for what is essentially a corner case.
>>
>>This requires pulling bmake into FreeBSD proper in order for things to =
>>function the last I checked as bsd.prog.mk depends upon bmake directives =
>
> This is already happening ;-)
I wish I had known that. I pinged marcel/obrien about this some weeks
ago and didn't receive a response.
Originally bsd.progs.mk was going to be a home grown effort that was
going to be killed off and replaced with NetBSD's build
infrastructure, but I didn't get a reply and had to put this (my work
to replace bsd.prog.mk) plan in motion given requests I was provided
above for resolving the source/unittest code consolidation effort.
>>Ideally however, I would like doing this compared to running a custom =
>>build infrastructure, but (as you probably know) this requires =
>>rototilling the current FreeBSD build system to a large degree =
>
> Actually building FreeBSD-current using bmake requires only modest changes.
Yes for the most part, and I agree that bmake is definitely more
advanced than FreeBSD pmake so I consider it a very welcome change.
What are the plans for importing bmake into FreeBSD (this should
probably be a different thread)?
...
> I know, but it is a very useful thing to be able to do.
> You can add knobs to control such things of course.
I agree that it's a good thing (in theory) -- it'd just help to
discuss this with Julio first because I know of a couple cases where
this would be unusable given how bsd.test.mk is coded:
1. The "it works for me on my machine!" certification:
It encourages environment tainting, which could be a really, really
bad thing; I've seen developers take interesting shortcuts when
testing their code (me included sometimes :)..); I've seen hardcoded
paths, harcoded "paths" for named semaphores, things that "just work"
because of someone's host environment, feature specific assumptions
(developer X was doing testing on feature Y and things broke when
he/she committed the testcase to head), etc where the "it works for me
on my machine!" certification would be true more often than not. These
same individuals would more likely than not execute things taking
shortcuts, but I want to avoid creating a system where developers cut
corners and commit too early because working within the confines of
the system is not conducive to getting work done.
2. Failure in a subdirectory results in lost coverage:
a/...
.../b/... <- tests live here
.../c/d/... <- tests live here
If I execute make test from a/ (one of my goals), then it should
iterate down a/b, then a/c/d and run the tests in a DFS style (BFS if
invoked with -j > 1 -- AIIIEEEE). If a/b fails to execute the make
process will bomb out because of how it executes shell statements.
However, if I let a/b pass, then developers/testers will accidentally
overlook failures, unless output is captured from the run and errors
are searched for in the end.
This isn't a super big problem if the tests are deterministic enough
that fixing one test permits the others to run, but the lost test
coverage in that period of time (or others if it transiently fails)
could disguise other bugs. This would be fixed by changing bsd.test.mk
to use atf-run properly, but that's work (not hard) that still needs
to be done.
(there are some other cases, but they're eluding me now)
> Yes, but making one test.mk handle potentially lots of frameworks
> will get rather ugly quite quickly.
Understood and I agree. I wasn't advocating having a single,
monolithic bsd.test.mk with all the test frameworks integrated into it
(that would be silly and unmaintainable).
> Better to add per-framework .mk files, and perhaps have them include a
> bsd.test.mk which only handles high level logic rather than the details
> of the frameworks.
>
> That's laregly why we didn't use bsd.test.mk
Hmmmm... My goal was to make ATF a "one ring to rule them all"
infrastructure so that way everything could be unified under ATF in
one way, shape or form (even if it's just reporting), because lord
knows we don't need a lua, unittest/nose, etc wrapper for reporting
results. All of the jobs I've worked at write custom wrappers around
testing infrastructures in order to present the data in a pretty
format (HTML), but that's a largely wasted effort (unless someone
REALLY wants to learn CSS/HTML/JS, which is an ongoing pain thanks to
browser incompatibilities), but I realize that this is also more
effort than I could get to right away, and it would have to be done as
needed instead.
Thanks!
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGH67wSaFeVyr_uNfigNQ%2BOBy4JmiLMa_u7c7=aqgB7wDKzsLg>