Date: Tue, 7 Dec 2004 22:18:10 +0000 From: Nik Clayton <nik@freebsd.org> To: doc@freebsd.org Subject: Testing the build infrastructure Message-ID: <20041207221810.GC79404@clan.nothing-going-on.org>
next in thread | raw e-mail | index | archive | help
--RIYY1s2vRbPFwWeW
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Guys,
Given the comments that have been bandied about recently relating to
breakage of various parts of the build infrastructure, and functionality
that used to work no longer working, I'd like to make some concrete
progress towards fixing things.
To my mind, step 1 is describing, given a particular set of inputs
(probably arguments given to make(1)), what we expect the output to be.
This means we need to write some tests. Automated tests. The kinds of
tests that can get run automatically every 24 hours or so, to rapidly
pin-point any commits that have caused a regressions in build
functionality.
Thinking out loud for the moment, I think we need=20
doc/regression/{articles, books}/
Below each one of those directories we have one or more subdirectories,
one for each type of document that we're testing. Each document has all
the usual content that we would normally provide. DocBook markup,
images, use of marked sections, a Makefile, and so on.
In addition, we create multiple test files. By convention, these will
have a .t extension, and follow the protocol described in
src/tools/regression/README. Each .t file implements one or more
logically related tests. Each test can be quite simple (there's no such
thing as a useless test) or complex as necessary.
For example, we might have a test file that makes sure that the HTML
format works.
#!/bin/sh
# Test: Make sure that FORMATS=3Dhtml-split is understood
test=3D"\"html-split\" is a valid format"
make -V ALL_FORMATS | xargs -n 1 echo | grep '^html-split$'
if [ $? -eq 0 ]; then
echo "ok 1 - $test"
else
echo "not ok 1 - $test"
fi
# Test: Does 'make FORMATS=3Dhtml-split' complete, and produce=20
# sensible output
test=3D"make FORMATS=3Dhtml-split"
make FORMATS=3Dhtml-split
if [ $? -eq 0 ]; then
echo "ok 2 - $test"
else
echo "not ok 2 - $test"
fi
test=3D"Generated .html file exists"
if [ -e index.html ]; then
echo "ok 3 - $test"
else
echo "not ok 3 - $test"
fi
test=3D"and it's a sensible size"
if [ -s index.html ]; then
echo "ok 4 - $test"
else
echo "not ok 4 - $test"
fi
=2E.. and so on -- additional tests to verify the DOCTYPE, that all the
internal links work, that sort of thing. There should be tests to make
sure that things like building with a r/o doc/ tree works, and so forth.
Thoughts?
N
--=20
FreeBSD: The Power to Serve http://www.freebsd.org/ (__)
FreeBSD Documentation Project http://www.freebsd.org/docproj/ \\\'',)
\/ \=
^
--- 15B8 3FFC DDB4 34B0 AA5F 94B7 93A8 0764 2C37 E375 --- .\._/=
_)
--RIYY1s2vRbPFwWeW
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (FreeBSD)
iD8DBQFBtiwhk6gHZCw343URArzxAJ4vt94kh+JPM8MOlieLaKE0WBkxgACfV4q4
husspZUos6taJVb86V7AvJ4=
=T7Ry
-----END PGP SIGNATURE-----
--RIYY1s2vRbPFwWeW--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20041207221810.GC79404>