From owner-dev-commits-doc-all@freebsd.org Thu Dec 31 12:29:18 2020 Return-Path: Delivered-To: dev-commits-doc-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BA7EC4C04FC for ; Thu, 31 Dec 2020 12:29:18 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D66sV4lw9z4jCL; Thu, 31 Dec 2020 12:29:18 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-lf1-f54.google.com with SMTP id h205so43758270lfd.5; Thu, 31 Dec 2020 04:29:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=BmvbiDFAloVSxvdU3zMpwhs2xU3IG6nazGNDPIJaLfU=; b=KvryrnqTZMDIuNfx/Wynu/B5Z4JU1CQ283m+xSYk9V2K+bCPXtxinY22Q+ndBEHt5d CBzYYEvFL3XhiVhW4uoZ8OxaF262MgFyw+zaAxSibzG2viCSAM9MqZAyjzzoMfZW6nSd 4P0os8rBgCVD0IUo7VR2u0ht58rJ2CsPip1VO2uAU6DEGmWtPuLdwlgpPEV/3yRFj7ci uy1tO5pDW5DeOLW71S5CFvjAiaRwQBwARZGHd/1pOP5caYI55qpE7UwTXyWmNFKoByBp qMyw+jmhGulU9clhusiCAgiqEi1TVYjGETKppo93rLj8ZDGBC5NM/dka6Ej7wjngN/IM 6XNw== X-Gm-Message-State: AOAM531LV4mtbg2+qGTo0VJRPQj9yYqmDSJP004Mn1jAuQYUlw+TkTOU 1nr1ThA8vMHoKHMwmwpsTDDIrlUGF1s= X-Google-Smtp-Source: ABdhPJzRm3XJKTMNHE2He68s4Fvmbhpjl++Wp3ik4Qc2bExY9zeY9viNBqBmA848V/1Cuj+zMwbV9w== X-Received: by 2002:a2e:6109:: with SMTP id v9mr2022058ljb.431.1609417756500; Thu, 31 Dec 2020 04:29:16 -0800 (PST) Received: from [192.168.0.128] (89-77-97-14.dynamic.chello.pl. [89.77.97.14]) by smtp.gmail.com with ESMTPSA id t30sm6190628lft.266.2020.12.31.04.29.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 31 Dec 2020 04:29:15 -0800 (PST) Subject: Re: git: cb1d034ab1 - main - docbuild: add scripts used to build the PDF versions of the Handbook To: Glen Barber , doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org References: <202012301653.0BUGrUPd003615@gitrepo.freebsd.org> From: Mateusz Piotrowski <0mp@FreeBSD.org> Message-ID: Date: Thu, 31 Dec 2020 13:29:38 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <202012301653.0BUGrUPd003615@gitrepo.freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4D66sV4lw9z4jCL X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: dev-commits-doc-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Commit messages for all branches of the doc repository." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2020 12:29:18 -0000 Hey, I've got a couple of minor suggestions and questions regarding this patch. On 12/30/20 5:53 PM, Glen Barber wrote: > The branch main has been updated by gjb: > > URL: https://cgit.FreeBSD.org/doc/commit/?id=cb1d034ab1e773758d20238be196aeeb1be68b18 > > commit cb1d034ab1e773758d20238be196aeeb1be68b18 > Author: Glen Barber > AuthorDate: 2020-12-30 16:40:07 +0000 > Commit: Glen Barber > CommitDate: 2020-12-30 16:40:07 +0000 > > docbuild: add scripts used to build the PDF versions of the Handbook > diff --git a/share/tools/dobuild.sh b/share/tools/dobuild.sh > new file mode 100755 > index 0000000000..a2ae5c092d > --- /dev/null > +++ b/share/tools/dobuild.sh > @@ -0,0 +1,117 @@ > +#!/bin/sh > +# Copyright (c) 2001 Wolfram Schneider > +# Copyright (c) 2001 Dima Dorfman > +# Copyright (c) 2004 Simon L. Nielsen > +# > +# Build the FreeBSD docs from the CVS^WSVN repository. > +# > +# > +# Major variables: > +# > +# PATH - The search path as interpreted by the shell. > +# CVSROOT - Path to the FreeBSD CVS repository (obsolete) > +# DOCSVN - Url to the FreeBSD doc SVN repository. > +# BUILDDIR - Where the checked out copies of the files are stored. > +# DESTDIR - Where the rendered copies should wind up. > +# BUILDARGS - Arguments to pass to make(1) when {build,install}ing. > +# INSTARGS - Arguments to pass to make(1) when installing. > +# WEBMAILTO - Address to send mail to if the build fails. > +# > +# subtrees - List of directores in $BUILDDIR which are from SCM. > +# > +# Variables which are in uppercase are derived from the environment > +# unless they don't exist, in which case a value suitable for > +# www.FreeBSD.org is used. Variables in lowercase can't be safely > +# changed without editing other parts of the script; thus, their value > +# in the environment is ignored. > +# > +# Exit codes: > +# > +# 0 - success > +# 1 - unknown failure > +# 2 - failure in CVS^SVN operations > +# 3 - failure in make operations > +# > +# $FreeBSD: www/tools/webupdate,v 1.10 2003/11/18 11:00:54 kuriyama Exp $ > +# > + > +# > +# WARNING! This script depend on dobuild_wrap.sh setting the > +# configuration variables in the enviroment. > +# > +if [ -z "${DOBUILDWRAP}" ]; then > + echo "Error: This script should only be called from dobuild_wrap.sh!" > + exit 1 > +fi > + > +# Only install some compression types > +INSTALL_COMPRESSED=${COMPTYPES}; export INSTALL_COMPRESSED > + > +# Abort on all errors > +set -e > + > +umask 002 > +cd $BUILDDIR || exit 1 > + > +# Remove the old copies. > +if [ -z "${NOCLEAN}" ]; then > + echo "===> Removing old stuff" > + rm -Rf doc out 2>/dev/null || true > + chflags -R noschg doc out 2>/dev/null || true > + rm -Rf doc out 2>/dev/null > + > + echo "===> Check out the new doc" > + #cvs -qR checkout -P doc || exit 2 > + svn checkout ${DOCSVN} doc || exit 2 > +fi > + > +cd $BUILDDIR/doc || exit 1 > +mkdir -p $DOCDIR > +mkdir -p $DOCDIR/packages > +rm -f ${BUILDDIR}/packages # TMP > +ln -s ${DOCDIR}/packages ${BUILDDIR}/doc/packages > + > +if [ -d ${PATCHDIR} -a -n "$(find ${PATCHDIR} -name \*.patch)" ]; then > + echo "===> Patching bugs in doc tree" > + for p in `echo ${PATCHDIR}/*.patch`; do Could this be just ``` for p in ${PATCHDIR}/*.patch; do ``` instead? > + patch < ${p} > + done > +fi > + > +echo "===> Building doc" > +time make ${BUILDARGS} all || exit 3 > + > +echo "===> Installing doc package to temp roots" > +time make ${INSTARGS} install || exit 3 > + > +#echo "===> Building doc packages" > +#time make ${BUILDARGS} package || exit 3 > + > +# XXX TODO, check for correct string when tex run out of resources > +#echo "===> Testing for broken tex output" > +#grep 'hash' $LOGFILE > /dev/null && \ > +# (echo "Error: Possible TeX out of resources."; false) > + > +#OKFILENAME > + > +# build a string to use with find(1) to identify > +# uncompressed files to be copied > +UNCOMP_FIND_STR="" > +for fmt in ${UNCOMPRESSED_FORMATS}; do > + UNCOMP_FIND_STR="${UNCOMP_FIND_STR}( -name article.${fmt} -or -name book.${fmt} ) -or " > +done > +# add outer parens, remove the final "-or " > +UNCOMP_FIND_STR="( ${UNCOMP_FIND_STR%-or } )" > + > +if [ -n "${UNCOMP_FIND_STR}" ]; then > + echo "copying uncompressed formats: ${UNCOMPRESSED_FORMATS}" > + cd ${BUILDDIR}/doc > + find -X ??_* ${UNCOMP_FIND_STR} -print0 \ > + | xargs -0 -I % install -v -C -o docbuild -g docbuild -m 444 % ${DOCDIR}/% > +fi > + > +rm -rf ${OUTDIR}/* > +mkdir -p ${OUTDIR} > +cp -Rp ${DOCDIR}/* ${OUTDIR} > + > +exit 0 > diff --git a/share/tools/dobuild_wrap.sh b/share/tools/dobuild_wrap.sh > new file mode 100755 > index 0000000000..d0bdf3e246 > --- /dev/null > +++ b/share/tools/dobuild_wrap.sh > @@ -0,0 +1,135 @@ > +#!/bin/sh > +# > +# Copyright (c) 2004 Simon L. Nielsen > +# All rights reserved. > +# > +# Redistribution and use in source and binary forms, with or without > +# modification, are permitted provided that the following conditions > +# are met: > +# 1. Redistributions of source code must retain the above copyright > +# notice, this list of conditions and the following disclaimer. > +# 2. Redistributions in binary form must reproduce the above copyright > +# notice, this list of conditions and the following disclaimer in the > +# documentation and/or other materials provided with the distribution. > +# > +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > +# SUCH DAMAGE. > +# > +# $FreeBSD$ > +# > + > +# Wrapper around doc build. Simplifies main build script. > + > +# > +# Default configuration. > +# > +DEFAULT_PATH=/bin:/usr/bin:/usr/local/bin > +DEFAULT_BUILDROOT=/local0/docbuild > +DEFAULT_CVSROOT=/home/dcvs > +DEFAULT_DOCSVN="svn://svn.freebsd.org/doc/head" > +DEFAULT_BUILDARGS="NO_JPMAN=yes GEN_INDEX=yes" > +DEFAULT_INSTARGS="NO_JPMAN=yes GEN_INDEX=yes INSTALL_ONLY_COMPRESSED=yes" > +# Must keep space between addresses. > +DEFAULT_WEBMAILTO="doceng@FreeBSD.org, wblock@wonkity.com" > +#DEFAULT_FORMATS="html html.tar html-split html-split.tar txt ps pdf rtf pdb" > +#DEFAULT_FORMATS="html html.tar html-split html-split.tar txt ps pdf epub" > +DEFAULT_FORMATS="html html.tar html-split html-split.tar txt pdf epub" > +DEFAULT_UNCOMPRESSED_FORMATS="pdf epub" > +DEFAULT_COMPTYPES="bz2 zip" > + > +# > +# Variable setup. > +# > +BUILDROOT=${BUILDROOT:-${DEFAULT_BUILDROOT}}; export BUILDROOT > +PATH="${PATH}:${DEFAULT_PATH}"; export PATH > +CVSROOT=${CVSROOT:-${DEFAULT_CVSROOT}}; export CVSROOT > +DOCSVN=${DOCSVN:-${DEFAULT_DOCSVN}}; export DOCSVN > +BUILDDIR=${BUILDDIR:-${BUILDROOT}/build}; export BUILDDIR > +DOCDIR=${DOCDIR:-${BUILDROOT}/build/out}; export DOCDIR > +FORMATS=${FORMATS:-${DEFAULT_FORMATS}}; export FORMATS > +UNCOMPRESSED_FORMATS=${UNCOMPRESSED_FORMATS:-${DEFAULT_UNCOMPRESSED_FORMATS}}; export UNCOMPRESSED_FORMATS > +COMPTYPES=${COMPTYPES:-${DEFAULT_COMPTYPES}}; export COMPTYPES > +BUILDARGS=${BUILDARGS:-${DEFAULT_BUILDARGS}}; export BUILDARGS > +INSTARGS=${INSTARGS:-${DEFAULT_INSTARGS}}; export INSTARGS > +WEBMAILTO=${WEBMAILTO:-${DEFAULT_WEBMAILTO}}; export WEBMAILTO > +PATCHDIR=${PATCHDIR:-${BUILDROOT}/patches}; export PATCHDIR > +OUTDIR=${OUTDIR:-${BUILDROOT}/docout}; export OUTDIR > +LOGROOT=${LOGBUILDROOT:-${BUILDROOT}/logs}; export LOGROOT > +LOGDIR=${LOGROOT}/$(date +%Y%m%d%H%M%S); export LOGDIR > +LOGFILE=${LOGDIR}/buildlog; export LOGFILE > +LOCKFILE=${LOCKFILE:-${BUILDROOT}/lock}; export LOCKFILE > +STATUSFILE=${LOGDIR}/status.xml; export STATUSFILE > +BUILDSFILE=${LOGROOT}/builds.xml; export BUILDSFILE > + > +DOBUILDWRAP="yes"; export DOBUILDWRAP > + > +# Run the actual script inside a lock so only one instance can run at > +# the time. > +lockf -t 0 ${LOCKFILE} /bin/sh << 'E*O*F' > + if [ ! -d ${LOGDIR} ]; then > + mkdir -p ${LOGDIR} > + fi > + LOGFILE_REL=${LOGFILE##${LOGROOT}/} > + echo "" > ${STATUSFILE} > + echo " $(hostname)" >> ${STATUSFILE} > + echo " $(date +%Y%m%d%H%M%S)" >> ${STATUSFILE} > + echo " ${LOGFILE_REL}" >> ${STATUSFILE} > + echo " running" >> ${STATUSFILE} > + echo "" >> ${STATUSFILE} > + > + echo '' > ${BUILDSFILE} > + find ${LOGROOT} -name status.xml | xargs cat >> ${BUILDSFILE} > + echo '' >> ${BUILDSFILE} > + > + # Do the actual build. > + starttime=$(date +%s) > + sh -x ${BUILDROOT}/dobuild.sh > ${LOGFILE} 2>&1 > + ret=$? > + touch ${LOGFILE} > + endtime=$(date +%s) > + runtime=$((${endtime} - ${starttime})) > + > + # Remove the status and "footer" of ${STATUSFILE}. > + keeplines=$(($(wc -l < ${STATUSFILE}) - 2)) > + mv ${STATUSFILE} ${STATUSFILE}.tmp > + head -n ${keeplines} ${STATUSFILE}.tmp > ${STATUSFILE} > + rm -f ${STATUSFILE}.tmp > + > + echo " $(date +%Y%m%d%H%M%S)" >> ${STATUSFILE} > + echo " ${runtime}" >> ${STATUSFILE} > + echo " ${ret}" >> ${STATUSFILE} > + if [ ${ret} -eq 0 ]; then > + echo " completed" >> ${STATUSFILE} > + else > + echo " failed" >> ${STATUSFILE} > + fi > + > + if [ ${ret} -ne 0 ]; then > + tail -100 ${LOGFILE} | \ > + mail -s "FreeBSD FTP doc build failed" ${WEBMAILTO} > + fi > + > + if egrep '^! TeX capacity exceeded' ${LOGFILE} > /dev/null; then > + egrep '^! TeX capacity exceeded' ${LOGFILE} | \ > + mail -s "FreeBSD FTP doc build warning" ${WEBMAILTO} > + echo " Possibly out of TeX resources." \ > + >> ${STATUSFILE} > + fi > + echo "" >> ${STATUSFILE} > + > + echo '' > ${BUILDSFILE} Is this ">" instead of ">>" on purpose? > + find ${LOGROOT} -name status.xml | xargs cat >> ${BUILDSFILE} "-exec cat +" might do as well. > + echo '' >> ${BUILDSFILE} > +E*O*F > + > + > +exit 0