Date: Thu, 16 Feb 2017 05:14:07 +0000 (UTC) From: Ngie Cooper <ngie@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r313790 - in stable/10: . share/man/man7 share/mk Message-ID: <201702160514.v1G5E7ux034321@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: ngie Date: Thu Feb 16 05:14:07 2017 New Revision: 313790 URL: https://svnweb.freebsd.org/changeset/base/313790 Log: MFC r285119,r292502,r295380: r285119 (by jmmv): Add support for TEST_METADATA Allow Makefiles to define generic metadata settings that apply to all test programs defined by a Makefile. The generic TEST_METADATA variable extends the per-test program settings already supported via TEST_METADATA.<program>. This feature will be useful to easily apply some settings to all programs in a directory. In particular, Kyua 0.12 will support parallel execution of test programs and a bunch of them will need to be tagged as is_exclusive to indicate that they cannot be run in parallel with anything else due to their side-effects. It will be reasonable to set this setting on whole directories. r292502: Always expose LOCALBASE, not just when CROSS_TOOLCHAIN is defined Instead of using which(1) to look for doxygen, look for it in <LOCALBASE>/bin . $PATH gets mangled by make buildenv, etc so it's better to just be explicit about the path if someone uses that for instance. r295380: Simplify running the FreeBSD test suite Replace `make regress` (legacy test make target) and `make test` (incomplete test make target added with the FreeBSD test suite) with make check as it's consistent with other open source projects. `make check` defaults to running tests from `.OBJDIR`, but can be overridden with the `CHECKDIR` variable. Add `make checkworld` target to simplify running the FreeBSD test suite from `TESTSBASE` (i.e. the top-level tests directory), similar to buildworld. Document `make check` and `make checkworld` in build(7). Other minor changes: - Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify `make check`. - Remove terse warnings attached to `beforetest`/`aftertest`. - Add kyua binary check to check target in suite.test.mk; error out if it's not found The MFC is [partly] contingent on other build related changes being MFCed. X-MFC to: stable/10 Relnotes: yes Modified: stable/10/Makefile stable/10/Makefile.inc1 stable/10/share/man/man7/build.7 stable/10/share/mk/bsd.README stable/10/share/mk/bsd.subdir.mk stable/10/share/mk/bsd.sys.mk stable/10/share/mk/bsd.test.mk stable/10/share/mk/suite.test.mk Directory Properties: stable/10/ (props changed) Modified: stable/10/Makefile ============================================================================== --- stable/10/Makefile Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/Makefile Thu Feb 16 05:14:07 2017 (r313790) @@ -21,6 +21,7 @@ # kernel-toolchains - Build kernel-toolchain for all universe targets. # doxygen - Build API documentation of the kernel, needs doxygen. # update - Convenient way to update your source tree(s). +# checkworld - Run test suite on installed world. # check-old - List obsolete directories/files/libraries. # check-old-dirs - List obsolete directories. # check-old-files - List obsolete files. @@ -99,8 +100,8 @@ # For more information, see the build(7) manual page. # TGTS= all all-man buildenv buildenvvars buildkernel buildworld \ - check-old check-old-dirs check-old-files check-old-libs \ - checkdpadd clean cleandepend cleandir \ + check check-old check-old-dirs check-old-files check-old-libs \ + checkdpadd checkworld clean cleandepend cleandir \ delete-old delete-old-dirs delete-old-files delete-old-libs \ depend distribute distributekernel distributekernel.debug \ distributeworld distrib-dirs distribution doxygen \ @@ -108,7 +109,7 @@ TGTS= all all-man buildenv buildenvvars installkernel.debug packagekernel packageworld \ reinstallkernel reinstallkernel.debug \ installworld kernel-toolchain libraries lint maninstall \ - obj objlink regress rerelease showconfig tags toolchain update \ + obj objlink rerelease showconfig tags toolchain update \ _worldtmp _legacy _bootstrap-tools _cleanobj _obj \ _build-tools _cross-tools _includes _libraries _depend \ build32 builddtb distribute32 install32 xdev xdev-build xdev-install \ @@ -374,6 +375,10 @@ make bmake: .PHONY ${MMAKE} all; \ ${MMAKE} install DESTDIR=${MYMAKE:H} BINDIR= +regress: .PHONY + @echo "'make regress' has been renamed 'make check'" | /usr/bin/fmt + @false + tinderbox toolchains kernel-toolchains: upgrade_checks tinderbox: Modified: stable/10/Makefile.inc1 ============================================================================== --- stable/10/Makefile.inc1 Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/Makefile.inc1 Thu Feb 16 05:14:07 2017 (r313790) @@ -35,6 +35,7 @@ # The intended user-driven targets are: # buildworld - rebuild *everything*, including glue to help do upgrades # installworld- install everything built by "buildworld" +# checkworld - run test suite on installed world # doxygen - build API documentation of the kernel # update - convenient way to update your source tree (eg: svn/svnup) # @@ -50,6 +51,8 @@ .include <bsd.arch.inc.mk> .include <bsd.compiler.mk> +LOCALBASE?= /usr/local + # We must do share/info early so that installation of info `dir' # entries works correctly. Do it first since it is less likely to # grow dependencies on include and lib than vice versa. @@ -1190,12 +1193,26 @@ packagekernel: .endif # +# +# checkworld +# +# Run test suite on installed world. +# +checkworld: .PHONY + @if [ ! -x ${LOCALBASE}/bin/kyua ]; then \ + echo "You need kyua (devel/kyua) to run the test suite." | /usr/bin/fmt; \ + exit 1; \ + fi + ${_+_}${LOCALBASE}/bin/kyua test -k ${TESTSBASE}/Kyuafile + +# +# # doxygen # # Build the API documentation with doxygen # doxygen: .PHONY - @if [ ! -x `/usr/bin/which doxygen` ]; then \ + @if [ ! -x ${LOCALBASE}/bin/doxygen ]; then \ echo "You need doxygen (devel/doxygen) to generate the API documentation of the kernel." | /usr/bin/fmt; \ exit 1; \ fi Modified: stable/10/share/man/man7/build.7 ============================================================================== --- stable/10/share/man/man7/build.7 Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/share/man/man7/build.7 Thu Feb 16 05:14:07 2017 (r313790) @@ -107,6 +107,16 @@ section below, and by the variables docu The following list provides the names and actions for the targets supported by the build system: .Bl -tag -width ".Cm cleandepend" +.It Cm check +Run tests for a given subdirectory. +The default directory used is +.Pa ${.OBJDIR} , +but the check directory can be changed with +.Pa ${CHECKDIR} . +.It Cm checkworld +Run the +.Fx +test suite on installed world. .It Cm clean Remove any files created during the build process. .It Cm cleandepend @@ -653,6 +663,7 @@ make TARGET=sparc64 DESTDIR=/clients/spa .Xr mergemaster 8 , .Xr portsnap 8 , .Xr reboot 8 , -.Xr shutdown 8 +.Xr shutdown 8 , +.Xr tests 7 .Sh AUTHORS .An Mike W. Meyer Aq mwm@mired.org . Modified: stable/10/share/mk/bsd.README ============================================================================== --- stable/10/share/mk/bsd.README Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/share/mk/bsd.README Thu Feb 16 05:14:07 2017 (r313790) @@ -492,6 +492,17 @@ It has seven targets: all: build the test programs. + check: + runs the test programs from CHECKDIR with kyua test. + + The beforecheck and aftercheck targets will be invoked, if + defined, to execute commands before and after the realcheck + target has been executed, respectively. + + The devel/kyua package must be installed before invoking this + target. + + See CHECKDIR for more details. clean: remove the test programs and any object files. cleandir: @@ -510,12 +521,6 @@ It has seven targets: run lint on the source files. tags: create a tags file for the source files. - test: - runs the test programs from the object directory; if the - Makefile does not itself define the target test, the - targets beforetest and aftertest may also be used to - cause actions immediately before and after the test - target is executed. It sets/uses the following variables, among many others: @@ -529,6 +534,10 @@ TESTSDIR Path to the installed tests. M ${TESTSBASE}/${RELDIR:H} , e.g. /usr/tests/bin/ls when included from bin/ls/tests . +CHECKDIR The directory that 'make check' executes tests from. + + The value of CHECKDIR defaults to .OBJDIR. + KYUAFILE If 'auto' (the default), generate a Kyuafile out of the test programs defined in the Makefile. If 'yes', then a manually-crafted Kyuafile must be supplied with the Modified: stable/10/share/mk/bsd.subdir.mk ============================================================================== --- stable/10/share/mk/bsd.subdir.mk Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/share/mk/bsd.subdir.mk Thu Feb 16 05:14:07 2017 (r313790) @@ -24,9 +24,9 @@ # This is a variant of install, which will # put the stuff into the right "distribution". # -# afterinstall, all, all-man, beforeinstall, checkdpadd, clean, +# afterinstall, all, all-man, beforeinstall, check, checkdpadd, clean, # cleandepend, cleandir, cleanilinks depend, install, lint, -# maninstall, manlint, obj, objlink, realinstall, regress, tags +# maninstall, manlint, obj, objlink, realinstall, tags # .if !target(__<bsd.subdir.mk>__) @@ -70,9 +70,9 @@ ${SUBDIR:N.WAIT}: .PHONY .MAKE # Work around parsing of .if nested in .for by putting .WAIT string into a var. __wait= .WAIT -.for __target in all all-man checkdpadd clean cleandepend cleandir \ +.for __target in all all-man check checkdpadd clean cleandepend cleandir \ cleanilinks depend distribute lint maninstall manlint obj objlink \ - realinstall regress tags ${SUBDIR_TARGETS} + realinstall tags ${SUBDIR_TARGETS} .ifdef SUBDIR_PARALLEL __subdir_targets= .for __dir in ${SUBDIR} Modified: stable/10/share/mk/bsd.sys.mk ============================================================================== --- stable/10/share/mk/bsd.sys.mk Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/share/mk/bsd.sys.mk Thu Feb 16 05:14:07 2017 (r313790) @@ -158,11 +158,11 @@ CFLAGS+= ${CWARNFLAGS} ${CWARNFLAGS.${.I # or expect to ever be up-to-date. PHONY_NOTMAIN = afterdepend afterinstall all beforedepend beforeinstall \ beforelinking build build-tools buildfiles buildincludes \ - checkdpadd clean cleandepend cleandir cleanobj configure \ + check checkdpadd clean cleandepend cleandir cleanobj configure \ depend dependall distclean distribute exe extract \ html includes install installfiles installincludes lint \ obj objlink objs objwarn realall realdepend \ - realinstall regress subdir-all subdir-depend subdir-install \ + realinstall subdir-all subdir-depend subdir-install \ tags whereobj .if defined(PORTNAME) Modified: stable/10/share/mk/bsd.test.mk ============================================================================== --- stable/10/share/mk/bsd.test.mk Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/share/mk/bsd.test.mk Thu Feb 16 05:14:07 2017 (r313790) @@ -68,11 +68,15 @@ _TESTS= .include <plain.test.mk> .include <tap.test.mk> +# kyua automatically descends directories; only run make check on the +# top-level directory +.if !make(check) .for ts in ${TESTS_SUBDIRS} .if empty(SUBDIR:M${ts}) SUBDIR+= ${ts} .endif .endfor +.endif # it is rare for test cases to have man pages .if !defined(MAN) @@ -83,18 +87,13 @@ MAN= .include <suite.test.mk> .endif -.if !target(realtest) -realtest: .PHONY +.if !target(realcheck) +realcheck: .PHONY @echo "$@ not defined; skipping" .endif -test: .PHONY -.ORDER: beforetest realtest -test: beforetest realtest - -.if target(aftertest) -.ORDER: realtest aftertest -test: aftertest -.endif +beforecheck realcheck aftercheck check: .PHONY +.ORDER: beforecheck realcheck aftercheck +check: beforecheck realcheck aftercheck .include <bsd.progs.mk> Modified: stable/10/share/mk/suite.test.mk ============================================================================== --- stable/10/share/mk/suite.test.mk Thu Feb 16 04:23:45 2017 (r313789) +++ stable/10/share/mk/suite.test.mk Thu Feb 16 05:14:07 2017 (r313790) @@ -30,12 +30,19 @@ KYUAFILE?= auto # Kyua as this is later encoded in the Kyuafile test program definitions. #TEST_INTERFACE.<test-program>= interface-name +# Metadata properties applicable to all test programs. +# +# All the variables for a test program defined in the Makefile are appended +# to the test program's definition in the Kyuafile. This feature can be +# used to avoid having to explicitly supply a Kyuafile in the source +# directory, allowing the caller Makefile to rely on the KYUAFILE=auto +# behavior defined here. +#TEST_METADATA+= key="value" + # Per-test program metadata properties as a list of key/value pairs. # -# All the variables for a particular program are appended to the program's -# definition in the Kyuafile. This feature can be used to avoid having to -# explicitly supply a Kyuafile in the source directory, allowing the caller -# Makefile to rely on the KYUAFILE=auto behavior defined here. +# These per-test program settings _extend_ the values provided in the +# unqualified TEST_METADATA variable. #TEST_METADATA.<test-program>+= key="value" .if ${KYUAFILE:tl} != "no" @@ -43,11 +50,12 @@ FILES+= Kyuafile FILESDIR_Kyuafile= ${TESTSDIR} .endif -.if ${KYUAFILE:tl} == "auto" -CLEANFILES+= Kyuafile Kyuafile.tmp -.endif +.for _T in ${_TESTS} +_TEST_METADATA.${_T}= ${TEST_METADATA} ${TEST_METADATA.${_T}} +.endfor .if ${KYUAFILE:tl} == "auto" +CLEANFILES+= Kyuafile Kyuafile.tmp Kyuafile: Makefile @{ \ echo '-- Automatically generated by bsd.test.mk.'; \ @@ -59,10 +67,10 @@ Kyuafile: Makefile } > ${.TARGET}.tmp .for _T in ${_TESTS} .if defined(.PARSEDIR) - @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${TEST_METADATA.${_T}:C/$/,/:tW:C/^/, /W:C/,$//W}}' \ + @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${_TEST_METADATA.${_T}:C/$/,/:tW:C/^/, /W:C/,$//W}}' \ >>${.TARGET}.tmp .else - @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${TEST_METADATA.${_T}:C/^/, /:Q:S/\\ ,/,/g:S,\\,,g}}' \ + @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${_TEST_METADATA.${_T}:C/^/, /:Q:S/\\ ,/,/g:S,\\,,g}}' \ >>Kyuafile.auto.tmp .endif .endfor @@ -72,9 +80,11 @@ Kyuafile: Makefile @mv ${.TARGET}.tmp ${.TARGET} .endif +CHECKDIR?= ${DESTDIR}${TESTSDIR} + KYUA= ${LOCALBASE}/bin/kyua -.if exists(${KYUA}) -# Definition of the "make test" target and supporting variables. + +# Definition of the "make check" target and supporting variables. # # This target, by necessity, can only work for native builds (i.e. a FreeBSD # host building a release for the same system). The target runs Kyua, which is @@ -83,35 +93,15 @@ KYUA= ${LOCALBASE}/bin/kyua # Due to the dependencies of the binaries built by the source tree and how they # are used by tests, it is highly possible for a execution of "make test" to # report bogus results unless the new binaries are put in place. -realtest: .PHONY - @echo "*** WARNING: make test is experimental" - @echo "***" - @echo "*** Using this test does not preclude you from running the tests" - @echo "*** installed in ${TESTSBASE}. This test run may raise false" - @echo "*** positives and/or false negatives." - @echo - @${KYUA} test -k ${DESTDIR}${TESTSDIR}/Kyuafile; \ - result=0; \ - echo; \ - echo "*** Once again, note that "make test" is unsupported."; \ - test $${result} -eq 0 -.endif -beforetest: .PHONY -.if defined(TESTSDIR) -.if ${TESTSDIR} == ${TESTSBASE} -# Forbid running from ${TESTSBASE}. It can cause false positives/negatives and -# it does not cover all the tests (e.g. it misses testing software in external). - @echo "*** Sorry, you cannot use make test from src/tests. Install the" - @echo "*** tests into their final location and run them from ${TESTSBASE}" - @false -.else - @echo "*** Using this test does not preclude you from running the tests" - @echo "*** installed in ${TESTSBASE}. This test run may raise false" - @echo "*** positives and/or false negatives." -.endif -.else - @echo "*** No TESTSDIR defined; nothing to do." - @false -.endif - @echo +realcheck: .PHONY + @if [ ! -x ${KYUA} ]; then \ + echo; \ + echo "kyua binary not installed at expected location (${.TARGET})"; \ + echo; \ + echo "Please install via pkg install, or specify the path to the kyua"; \ + echo "package via the \$${LOCALBASE} variable, e.g. "; \ + echo "LOCALBASE=\"${LOCALBASE}\""; \ + false; \ + fi + @${KYUA} test -k ${CHECKDIR}/Kyuafile
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201702160514.v1G5E7ux034321>