From owner-freebsd-testing@FreeBSD.ORG Wed Oct 30 22:26:31 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3B81EB8 for ; Wed, 30 Oct 2013 22:26:31 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from db8outboundpool.messaging.microsoft.com (mail-db8lp0186.outbound.messaging.microsoft.com [213.199.154.186]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9925B28B3 for ; Wed, 30 Oct 2013 22:26:30 +0000 (UTC) Received: from mail5-db8-R.bigfish.com (10.174.8.225) by DB8EHSOBE011.bigfish.com (10.174.4.74) with Microsoft SMTP Server id 14.1.225.22; Wed, 30 Oct 2013 21:25:58 +0000 Received: from mail5-db8 (localhost [127.0.0.1]) by mail5-db8-R.bigfish.com (Postfix) with ESMTP id 2D7DBC001CB; Wed, 30 Oct 2013 21:25:58 +0000 (UTC) X-Forefront-Antispam-Report: CIP:66.129.224.52; KIP:(null); UIP:(null); IPV:NLI; H:P-EMF01-SAC.jnpr.net; RD:none; EFVD:NLI X-SpamScore: 2 X-BigFish: VPS2(zz1432Izz1f42h208ch1ee6h1de0h1fdah2073h1202h1e76h1d1ah1d2ah1fc6h1082kzz1de097hz2fh2a8h839hd25hf0ah1288h12a5h12a9h12bdh12e5h137ah139eh13b6h1441h14ddh1504h1537h162dh1631h1758h1898h18e1h1946h19b5h1ad9h1b0ah1b2fh1b88h1fb3h1d0ch1d2eh1d3fh1de2h1dfeh1dffh1e23h1fe8h1ff5h2218h2216h1155h) Received-SPF: pass (mail5-db8: domain of juniper.net designates 66.129.224.52 as permitted sender) client-ip=66.129.224.52; envelope-from=sjg@juniper.net; helo=P-EMF01-SAC.jnpr.net ; SAC.jnpr.net ; Received: from mail5-db8 (localhost.localdomain [127.0.0.1]) by mail5-db8 (MessageSwitch) id 1383168356853304_25803; Wed, 30 Oct 2013 21:25:56 +0000 (UTC) Received: from DB8EHSMHS015.bigfish.com (unknown [10.174.8.243]) by mail5-db8.bigfish.com (Postfix) with ESMTP id C1BE128004B; Wed, 30 Oct 2013 21:25:56 +0000 (UTC) Received: from P-EMF01-SAC.jnpr.net (66.129.224.52) by DB8EHSMHS015.bigfish.com (10.174.4.25) with Microsoft SMTP Server (TLS) id 14.16.227.3; Wed, 30 Oct 2013 21:25:53 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF01-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 30 Oct 2013 14:25:49 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.24.29.229]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id r9UIF8L72377; Wed, 30 Oct 2013 11:15:15 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos.jnpr.net (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 84F085807E; Wed, 30 Oct 2013 11:15:08 -0700 (PDT) To: Julio Merino Subject: Re: Plugging ATF tests into the build and other cleanups In-Reply-To: References: <20131029203658.75BF35807E@chaos.jnpr.net> Comments: In-reply-to: Julio Merino message dated "Tue, 29 Oct 2013 17:46:19 -0400." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 30 Oct 2013 11:15:08 -0700 Message-ID: <20131030181508.84F085807E@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-OriginatorOrg: juniper.net X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% Cc: freebsd-testing@freebsd.org, Rui Paulo , sjg@juniper.net X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Oct 2013 22:26:31 -0000 On Tue, 29 Oct 2013 17:46:19 -0400, Julio Merino writes: >> You can't even do OLD_FILES+= ${KYUAFILES:H}/* > >I'm confused... Is that "can't" supposed to be "can"? Or in other >words, do you mean that this would or would not work if we assumed >that ATF/Kyua are the only thing populating /usr/tests ? No it wouldn't work - the existing delete-old-* targets are very careful about only deleting what was explicitly listed. That's probably a good thing. >That's part of the rationale for the next patch: to make /usr/tests/ >conditional on a single knob (WITH_TESTS) so that such assumptions can >be made: either you install the full test suite or you don't. I don't You could make that argument yes, and further, if folk want their own test suites they could be told to put them somewhere else. Since this isn't critical one way or the other yet, I'd leave some time for folk to ponder/discuss - ie. just ignore cleaning /usr/tests for now. >> freebsd-testing/recurse-subdir.diff >> >> I'm not so crazy about. >> What's wrong with requiring folk to set TESTS_SUBDIR[S] ? > >Because SUBDIR and TESTS_SUBDIRS serve different purposes. With This patch seems to blur that distinction no? >In the current patchset, this is useful in, for example, >src/lib/atf/Makefile (see last patch) : this file needs to generate a >Kyuafile that registers its 3 subdirectories (libatf-c, libatf-c++ and >test-programs) so that the recursion can be performed at runtime. >There is no need to list those as TESTS_SUBDIRS because they are >already in SUBDIR. But for this corner case, listing those dirs in TESTS_SUBDIRS (despite also being in SUBDIR) would do no great harm, and would avoid the need to blur the distiction noted above. >There are some alternatives to this behavior: > >1. Move the generation of the Kyuafile to src/lib/atf/tests/Makefile. >But if you do that, you cannot get the automatic generation to work >because that tests/ directory does not have any children (and thus the >automatic generation code has no idea about what to do). I'm not sure I fully understand the issue (given what we are doing - see below). >2. Manually set TESTS_SUBDIR in src/lib/atf/Makefile to SUBDIR... >which would be suboptimal because this is something that could be done >automatically (this patch). Optimizing for corner cases isn't necessarily a good idea. If it turns out that this is a common issue, we can always revisit. So this (or the next) would be my suggestion - for now. >3. Make src/lib/atf/tests/Makefile install a generic Kyuafile that >discovers its directories, just like /usr/tests/Kyuafile and >/usr/tests/lib/Kyuafile do. This is a reasonable optional and would >follow your comments below of keeping stuff in tests/ This is what we actually do. Before packaging tests, a generic Atffile is dropped into each dir under ${STAGE_OBJTOP}/usr/tests/ that does not already have one, this seems to work ok. >> freebsd-testing/move-kyuafiles.diff >> >> I'm also not crazy about this one (the change to lib/Makefile) >> What is the expected use? > >To populate /usr/tests/lib accordingly. You need: > >/usr/tests/Kyuafile - auto-recurse into directories >/usr/tests/lib/Kyuafile - auto-recurse into directories >/usr/tests/lib/libc/Kyuafile - list of tests to run and possibly other >subdirectories >and that /usr/tests/lib/Kyuafile has to come from somewhere. I think Sure, but if Kyuafile works anything like Atffile, a generic boilerplate one would do? and this could be handled from almost anywhere. Since we (junos - and same in projects/bmake) ignore intermediate makefiles like lib/ etc, that wouldn't be my choice. Doing it in a leaf dir, works just as well and is then independent of the orchestration of the build. >We could do this from src/lib/tests/ as easily; didn't think about it. > Gave this a try and updated the patches accordingly; check them out. Will do. >Still open is the question if we should do the same for >src/lib/atf/Makefile et. al. I suspect the answer is yes, but then >there is the issue mentioned above of the Kyuafile auto-generation. >(Haven't done it yet as I don't have the time now and would appreciate >your input.) All of the Atffiles in our stage tree are autogenerated. The leaf dirs usr/tests/bin/cat/Atffile are done by atf.test.mk: $ cat ../obj/stage/i386/usr/tests/bin/cat/Atffile Content-Type: application/X-atf-atffile; version="1" # Automatically generated by atf-test.mk. prop: test-suite = "FreeBSD" tp: t_cat $ and the intermediate dirs by the method mentioned above $ cat ../obj/stage/i386/usr/tests/bin/Atffile Content-Type: application/X-atf-atffile; version="1" prop: test-suite = "FreeBSD" tp-glob: * $ ls -li ../obj/stage/i386/usr/tests/bin/Atffile 93925694 -rw-r--r-- 26 sjg wheel 94 Sep 17 14:27 ../obj/stage/i386/usr/tests/bin/Atffile $ cat ../obj/stage/i386/usr/tests/bin/Atffile.dirdep juniper/pkgs/os-tests/finish.i386 $ that last let's you know who put it there. >Indeed, having a SRCTOP would have made things easier... but I didn't >find that available. No, not yet. It is needed for dirdeps.mk though. >Note: I have applied some of your suggestions to the existing patches >but some things still remain open so don't bother doing an in-depth >review again because they are broken. However, I gotta go now and I'd >prefer to send this out for further comments while I have your >attention ;) Sure - thanks very much for your help. --sjg