From owner-freebsd-doc@FreeBSD.ORG Tue Dec 7 22:33:59 2004 Return-Path: Delivered-To: freebsd-doc@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 23ECC16A4CE for ; Tue, 7 Dec 2004 22:33:59 +0000 (GMT) Received: from jc.ngo.org.uk (jc.ngo.org.uk [69.55.225.33]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5168943D1D for ; Tue, 7 Dec 2004 22:33:58 +0000 (GMT) (envelope-from nik@crf-consulting.co.uk) Received: from crf-consulting.co.uk (82-44-220-218.cable.ubr10.haye.blueyonder.co.uk [82.44.220.218]) by jc.ngo.org.uk (8.12.11/8.13.1) with ESMTP id iB7MXufO047013 for ; Tue, 7 Dec 2004 14:33:57 -0800 (PST) (envelope-from nik@crf-consulting.co.uk) Received: from clan.nothing-going-on.org (clan.nothing-going-on.org [192.168.1.20]) by crf-consulting.co.uk (8.13.1/8.12.9) with ESMTP id iB7MIBx7050474 for ; Tue, 7 Dec 2004 22:18:11 GMT (envelope-from nik@catkin) Received: from clan.nothing-going-on.org (localhost [127.0.0.1]) iB7MIA8o031896 for ; Tue, 7 Dec 2004 22:18:11 GMT (envelope-from nik@clan.nothing-going-on.org) Received: (from nik@localhost) by clan.nothing-going-on.org (8.13.1/8.13.1/Submit) id iB7MIAnc031895 for doc@freebsd.org; Tue, 7 Dec 2004 22:18:10 GMT (envelope-from nik) Date: Tue, 7 Dec 2004 22:18:10 +0000 From: Nik Clayton To: doc@freebsd.org Message-ID: <20041207221810.GC79404@clan.nothing-going-on.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="RIYY1s2vRbPFwWeW" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Organization: FreeBSD Project Subject: Testing the build infrastructure X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Dec 2004 22:33:59 -0000 --RIYY1s2vRbPFwWeW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Guys, Given the comments that have been bandied about recently relating to breakage of various parts of the build infrastructure, and functionality that used to work no longer working, I'd like to make some concrete progress towards fixing things. To my mind, step 1 is describing, given a particular set of inputs (probably arguments given to make(1)), what we expect the output to be. This means we need to write some tests. Automated tests. The kinds of tests that can get run automatically every 24 hours or so, to rapidly pin-point any commits that have caused a regressions in build functionality. Thinking out loud for the moment, I think we need=20 doc/regression/{articles, books}/ Below each one of those directories we have one or more subdirectories, one for each type of document that we're testing. Each document has all the usual content that we would normally provide. DocBook markup, images, use of marked sections, a Makefile, and so on. In addition, we create multiple test files. By convention, these will have a .t extension, and follow the protocol described in src/tools/regression/README. Each .t file implements one or more logically related tests. Each test can be quite simple (there's no such thing as a useless test) or complex as necessary. For example, we might have a test file that makes sure that the HTML format works. #!/bin/sh # Test: Make sure that FORMATS=3Dhtml-split is understood test=3D"\"html-split\" is a valid format" make -V ALL_FORMATS | xargs -n 1 echo | grep '^html-split$' if [ $? -eq 0 ]; then echo "ok 1 - $test" else echo "not ok 1 - $test" fi # Test: Does 'make FORMATS=3Dhtml-split' complete, and produce=20 # sensible output test=3D"make FORMATS=3Dhtml-split" make FORMATS=3Dhtml-split if [ $? -eq 0 ]; then echo "ok 2 - $test" else echo "not ok 2 - $test" fi test=3D"Generated .html file exists" if [ -e index.html ]; then echo "ok 3 - $test" else echo "not ok 3 - $test" fi test=3D"and it's a sensible size" if [ -s index.html ]; then echo "ok 4 - $test" else echo "not ok 4 - $test" fi =2E.. and so on -- additional tests to verify the DOCTYPE, that all the internal links work, that sort of thing. There should be tests to make sure that things like building with a r/o doc/ tree works, and so forth. Thoughts? N --=20 FreeBSD: The Power to Serve http://www.freebsd.org/ (__) FreeBSD Documentation Project http://www.freebsd.org/docproj/ \\\'',) \/ \= ^ --- 15B8 3FFC DDB4 34B0 AA5F 94B7 93A8 0764 2C37 E375 --- .\._/= _) --RIYY1s2vRbPFwWeW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFBtiwhk6gHZCw343URArzxAJ4vt94kh+JPM8MOlieLaKE0WBkxgACfV4q4 husspZUos6taJVb86V7AvJ4= =T7Ry -----END PGP SIGNATURE----- --RIYY1s2vRbPFwWeW--