From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 2 16:19:03 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7714D106566B; Tue, 2 Oct 2012 16:19:03 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from exprod7og124.obsmtp.com (exprod7og124.obsmtp.com [64.18.2.26]) by mx1.freebsd.org (Postfix) with ESMTP id 0CC958FC15; Tue, 2 Oct 2012 16:19:01 +0000 (UTC) Received: from P-EMHUB03-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob124.postini.com ([64.18.6.12]) with SMTP ID DSNKUGsT71yeK0mQ0bP/uNjtntajTyvZOkML@postini.com; Tue, 02 Oct 2012 09:19:03 PDT Received: from magenta.juniper.net (172.17.27.123) by P-EMHUB03-HQ.jnpr.net (172.24.192.33) with Microsoft SMTP Server (TLS) id 8.3.213.0; Tue, 2 Oct 2012 09:17:40 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.24.29.229]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id q92GHeh43108; Tue, 2 Oct 2012 09:17:40 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos.jnpr.net (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id EFE0E58093; Tue, 2 Oct 2012 09:17:39 -0700 (PDT) To: Garrett Cooper In-Reply-To: References: <20121001223100.E7D0D58093@chaos.jnpr.net> <20121002000030.54CEE58093@chaos.jnpr.net> Comments: In-reply-to: Garrett Cooper message dated "Tue, 02 Oct 2012 07:19:55 -0700." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Tue, 2 Oct 2012 09:17:39 -0700 Message-ID: <20121002161739.EFE0E58093@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-Mailman-Approved-At: Tue, 02 Oct 2012 16:36:28 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org, sjg@juniper.net Subject: Re: [CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Oct 2012 16:19:03 -0000 On Tue, 2 Oct 2012 07:19:55 -0700, Garrett Cooper writes: >> We put the test cases in a subdir of the lib/prog >> This has multiple benefits, and eliminates any impact on the normal >> build of said libs/progs. > >Hmmm... that's one of the 3 approaches I provided, but it turned out >to be annoying to make this work at first inspection because of how >things were done in our build system. Just for review (even though >it's OT), the three approaches I presented were as follows... > >Note: in all 3 examples, clock.c is the source code and t_clock.c is >the relevant unittest code. There is actually another: lib/libc/Makefile lib/libc/gen/... .../clock.c .../t_clock.c lib/libc/tests/Makefile that is the makefile for building/running the tests lives in the subdir, there are advantages to this, but the test code itself can be anywhere you like. Either next to the code that it tests, or in the tests/ subdir, another subdir, or whatever combination makes the most sense. Most of our ATF users put their test code in the subdir fwiw. >> Also a number of our teams find it necessary to create mock classes etc to >> adequately test their libs. Keeping all this in a tests/ subdir makes >> it easy, to keep things neat & up to date, and ensure that the tests >> pass. Trying to do all that in the same dir as the lib would be ugly. > >This (consolidating everything under a single directory) is the way >that was requested by the beforementioned parties. Sorry for not participating in that discussion ;-) John - perhaps we do need that ATF list? For example building a library and test apps in the one directory either requires turning bsd.lib.mk into an even more complex thing that a multi-prog bsd.pog.mk, or having the makefile behave completely differently depending on the target requested. That later is a bad idea. It precludes being able to seamlessly integrate unit-tests into the build, and would need to be fixed if/when attempting to introduce meta mode. Typically the unit-tests depend on the library they are testing, a separate subdir for the tests/Makefile makes capturing that dependency easy - otherwise you are back to the unspeakably complex bsd.lib.mk Far better to get these things right first up. >I wish I had known that. I pinged marcel/obrien about this some weeks >ago and didn't receive a response. Many appologies. >What are the plans for importing bmake into FreeBSD (this should >probably be a different thread)? The import has been done - just needs to be merged to contrib, and there's a small patch to be committed. We hope to get that done this week. >I agree that it's a good thing (in theory) -- it'd just help to >discuss this with Julio first because I know of a couple cases where >this would be unusable given how bsd.test.mk is coded: I do talk to Julio about ATF (I'm sjg@netbsd.org ;-) though not specifically this - beyond (I think) mentioning that I did it differently. >1. The "it works for me on my machine!" certification: > >It encourages environment tainting, which could be a really, really >bad thing; I've seen developers take interesting shortcuts when True. In our (Junos) build we cleanse the environment rather carefully and have a very clear distinction between building for the "host" (which may just happend to be i386 based) and the i386 "target" for example. I can provide build smarts to do all that - but didn't want to shove it down anyone's throat ;-) What "works for us" doesn't necessarily work for everyone. >2. Failure in a subdirectory results in lost coverage: > >a/... > .../b/... <- tests live here > .../c/d/... <- tests live here > >If I execute make test from a/ (one of my goals), then it should >iterate down a/b, then a/c/d and run the tests in a DFS style (BFS if This gets back to the bmake topic. We actually avoid "walking" the tree at all, eg. in the Junos build I have a target "atf-tests" which builds all ATF tests dirs regardless of where they are (and any pre-requisits they have). This is handy to include as a dependency of the build target we use for branch syncs etc, to ensure no unit-tests atrophy. But the most common case - and the one to optimize for, is the person making changes to libfoo, testing that he hasn't broken anything. >could disguise other bugs. This would be fixed by changing bsd.test.mk >to use atf-run properly, but that's work (not hard) that still needs >to be done. Yes, atf.test.mk already does that. >Hmmmm... My goal was to make ATF a "one ring to rule them all" >infrastructure so that way everything could be unified under ATF in ATF is a pretty good framework - which is why we (Juniper) use it. We want the flexibility to have more than one framework, but the number should be very small. So far I have persuaded teams that wanted to adopt alternate frameworks, that ATF could easily meet their needs. >one way, shape or form (even if it's just reporting), because lord >knows we don't need a lua, unittest/nose, etc wrapper for reporting >results. All of the jobs I've worked at write custom wrappers around Yes, having all "frameworks" able to output results in a consistent format is definitely a good thing. Thanks --sjg