From owner-freebsd-doc@FreeBSD.ORG Sun Nov 16 20:35:21 2003 Return-Path: Delivered-To: freebsd-doc@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E101616A4D1; Sun, 16 Nov 2003 20:35:21 -0800 (PST) Received: from electra.cse.Buffalo.EDU (electra.cse.Buffalo.EDU [128.205.32.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id CB4AC43FF5; Sun, 16 Nov 2003 20:35:15 -0800 (PST) (envelope-from kensmith@cse.Buffalo.EDU) Received: from electra.cse.Buffalo.EDU (kensmith@localhost [127.0.0.1]) hAH4ZFus016761; Sun, 16 Nov 2003 23:35:15 -0500 (EST) Received: (from kensmith@localhost) by electra.cse.Buffalo.EDU (8.12.10/8.12.9/Submit) id hAH4ZFVn016760; Sun, 16 Nov 2003 23:35:15 -0500 (EST) Date: Sun, 16 Nov 2003 23:35:15 -0500 From: Ken Smith To: Joe Marcus Clarke Message-ID: <20031117043515.GA16285@electra.cse.Buffalo.EDU> References: <1068365672.64416.2.camel@shumai.marcuscom.com> <20031109031918.40437858.trhodes@FreeBSD.org> <1068366438.64416.8.camel@shumai.marcuscom.com> <20031116173729.GC5895@electra.cse.Buffalo.EDU> <1069007319.3752.11.camel@shumai.marcuscom.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1069007319.3752.11.camel@shumai.marcuscom.com> User-Agent: Mutt/1.4.1i cc: doc@FreeBSD.org Subject: Re: DocBook help X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Nov 2003 04:35:22 -0000 On Sun, Nov 16, 2003 at 01:28:39PM -0500, Joe Marcus Clarke wrote: > Yeah, that's fine. Thanks! How's this? It's viewable in a very stripped-down web version at: http://www.cse.buffalo.edu/~kensmith/FreeBSD/cluster_article.html if you want to see it. Lost of the support files are missing so it will look much better once integrated on the main web site. %man; %freebsd; %authors; %teams; %mailing-lists; %trademarks; ]>
Package Building Procedures The &os; Ports Management Team $FreeBSD$ 2003 The &os; Ports Management Team &tm-attrib.freebsd; &tm-attrib.general; Introduction and Conventions In order to provide pre-compiled binaries of third-party applications for &os;, the ports collection is regularly built on one of the Package Building Clusters. Currently, there are two such clusters: bento.FreeBSD.org and dosirak.kr.freebsd.org. Most of the package building magic occurs under the /var/portbuild directory. Unless otherwise specified, all paths will be relative to this location. ${arch} will be used to specify one of the package architectures (&i386;, alpha, &sparc64;, ia64, and amd64), and ${branch} will be used to specify the build branch (4, 5, 4-exp). Build Client Management The &i386; and alpha clients currently netboot from bento; the others are self-hosted. In all cases they set themselves up at boot-time to prepare to build packages. The ports-${arch} user can &man.ssh.1; as root onto each of the ${arch} nodes. The scripts/allgohans script can be used to run a command on all of the ${arch} clients [XXX Currently slightly broken - it suspends itself after dispatching the command to each client]. The scripts/checkmachines script is used to monitor the load on all the nodes of the build cluster, and schedule which nodes build which ports. This script is not very robust, and has a tendency to die. It is best to start up this script on the build master (either bento or dosirak) after boot time using a &man.while.1; loop. Chroot Build Environment Setup Package builds are performed in a chroot populated by the portbuild script using the ${arch}/${branch}/tarballs/bindist.tar file. This tarball is created by the mkbindist script which reads the ${arch}/${branch}/mkbindist.conf file to decide how to create the tarball. The script should be run as root with the following command: /var/portbuild&prompt.root; scripts/mkbindist ${arch} ${branch} If ftp=1 in mkbindist.conf then a prebuilt release will be downloaded via FTP from the location specified by ftp://${ftpserver}/${ftpurl}/${rel}. If ftp=0 and buildworld=1 then mkbindist will call makeworld to build a new world [XXX This is currently broken]. If both ftp=0 and buildworld=0 then mkbindist will use the pre-existing contents of ${worlddir} to create bindist.tar. In practice this means that you must have already installed a world in ${worlddir}, which is typically installed with the makeworld script: /var/portbuild&prompt.root; scripts/makeworld ${arch} ${branch} [-nocvs] This command builds a world from the ${arch}/${branch}/src tree and installs it into ${worlddir}. The tree will be updated first unless -nocvs is specified. The bindist.tar file is extracted onto each client at client boot time, and at the start of each pass of the dopackages script. Starting the Build The scripts/dopackages* scripts are used to perform the builds. Most useful are: dopackages.steve3 - Perform a 5.x build dopackages.steve4 - Perform a 4.x build dopackages.steve4-exp - Perform a 4.x build with experimental patches (4-exp branch) These are wrappers around dopackages, and take a number of arguments. For example: dopackages.steve3 ${arch} [-options] [-options] may be zero or more of the following: -nofinish - Do not perform post-processing once the build is complete. Useful if you expect that the build will need to be restarted once it finishes. This option should always be used for normal build operations. -finish - Perform post-processing only. -restart - Restart an interrupted (or non-finished) build from the beginning. Ports that failed on the previous build will be rebuilt. -continue - Restart an interrupted (or non-finished) build. Will not rebuild ports that failed on the previous build. -cdrom - This package build is intended to end up on a CD-ROM, so NO_CDROM packages and distfiles should be deleted in post-processing. -nobuild - Perform all the preprocessing steps, but do not actually do the package build. -noindex - Do not rebuild INDEX during preprocessing. -noduds - Do not rebuild the duds file (ports that are never built, e.g. those marked IGNORE, NO_PACKAGE, etc.) during preprocessing. -nocvs - Do not cvs update the src tree during preprocessing. -noportscvs - Do not cvs update the ports tree during preprocessing. -norestr - Do not attempt to build RESTRICTED ports. -plistcheck - Make it fatal for ports to leave behind files after deinstallation. -distfiles - Collect distfiles that pass make checksum for later uploading to ftp-master. Use this sparingly because it takes up a lot of disk space. You should remove the distfiles once they have been transfered to ftp-master. -fetch-original - Fetch the distfile from the original MASTER_SITES rather than ftp-master. Make sure the ${arch} build is run as the ports-${arch} user or it will complain loudly. The actual package build itself occurs in two identical phases. The reason for this is that sometimes transient NFS failures may halt the build. Doing things in two phases is a workaround for these types of problems. Be careful that ports/Makefile does not specify any empty subdirectories. This is especially important if you are doing a 4-exp build. If the build process encounters an empty subdirectory, both package build phases will stop short, and an error similar to the following will be written to ${arch}/${branch}/make.[0|1]: don't know how to make dns-all(continuing) To correct this problem, simply comment out or remove the SUBDIR entries that point to empty subdirectories. After doing this, you can restart the build by running the proper dopackages command with the -restart option. Anatomy of a Build A full build without any -no options performs the following operations in the specified order: A CVS checkout of the current ports tree [*] A CVS checkout of the running branch's doc tree [*] A CVS checkout of the running branch's src tree [*] Checks which ports do not have a SUBDIR entry in their respective category's Makefile [*] Creates the duds file, which is a list of ports not to build [*] [+] Generates a fresh INDEX file [*] [+] Sets up the nodes that will be used in the build [*] [+] Builds a list of restricted ports [*] [+] Builds packages (phase 1) [++] Performs another node setup [+] Builds packages (phase 2) [++] [*] Status of these steps can be found in ${arch}/${branch}/build.log as well as on stderr of the tty running the dopackages command. [+] If any of these steps fail, the build will stop cold in its tracks. [++] Status of these steps can be found in ${arch}/${branch}/make.[0|1], where make.0 is the log file used by phase 1 of the package build and make.1 is the log file used by phase 2. Individual ports will write their build logs to ${arch}/${branch}/logs and their error logs to ${arch}/${branch}/errors. Interrupting a Build Sending a HUP signal to the dopackages* shell processes or to any make process invoked by those scripts is usually sufficient to interrupt the build. The package builds dispatched by make to the client machines will clean themselves up after a few minutes (check with ps x until they all go away). Remove the ${arch}/lock file before trying to restart the build. Monitoring the Build The scripts/stats${branch} command counts the number of packages currently built. Running cat /var/portbuild/*/loads/* shows the client loads and number of concurrent builds in progress. Running tail -f ${arch}/${branch}/build.log shows the overall build progress. If a build is failing, and it is immediately obvious from the port build log as to why, you can preserve the WRKDIR for further analysis. To do this, touch a file called .keep in the port's directory. The next time the cluster tries to build this port, it will tar, compress, and copy the WRKDIR to ${arch}/${branch}/wrkdirs. Keep an eye on &man.df.1; output. If the /var/portbuild file system becomes full then Bad Things happen. Release Builds When building packages for a release, it may be necessary to manually update the ports and src trees to the release tag and use -nocvs and -noportscvs. To build package sets intended for use on a CD-ROM, use the -cdrom option to dopackages. Assuming disk space is available on the cluster, use -distfiles to collect distfiles. You must run the initial build with -distfiles to collect all the fetched distfiles. After the initial build completes, restart the build with -restart -distfiles -fetch-original to collect updated distfiles as well. Then, once the build is post-processed, take an inventory of the list of files fetched: &prompt.user; cd ${arch}/${branch} &prompt.user; find distfiles > ${release}-distfiles This inventory file typically lives in i386/${branch} on the cluster master. This is useful to aid in periodically cleaning out the distfiles from ftp-master. When space gets tight, distfiles from recent releases can be kept while others can be thrown away. Once the distfiles have been uploaded (see below), the final release package set must be created. Just to be on the safe side, run the ${arch}/${branch}/cdrom.sh script by hand to make sure all the CD-ROM restricted packages and distfiles have been pruned. Then, copy the ${arch}/${branch}/packages directory to ${arch}/${branch}/packages-${release}. Once the packages are safely moved off, contact the &a.re; and inform them of the release package location. Remember to coordinate with the &a.re; about the timing and status of the release builds. Uploading Packages Once a build has completed, packages and/or distfiles can be transferred to ftp-master for propagation to the FTP mirror network. If the build was run with -nofinish, then make sure to follow up with dopackages -finish to post-process the packages (removes RESTRICTED and NO_CDROM packages where appropriate, prunes packages not listed in INDEX, removes from INDEX references to packages not built, and generates a CHECKSUM.MD5 summary); and distfiles (moves them from the temporary distfiles/.pbtmp directory into distfiles/ and removes RESTRICTED and NO_CDROM distfiles). It is usually a good idea to run the restricted.sh and/or cdrom.sh scripts by hand after dopackages finishes just to be safe. Run the restricted.sh script before uploading to ftp-master, then run cdrom.sh before preparing the final package set for a release. Packages can be copied to the staging area on ftp-master with something like the following: &prompt.root; cd /var/portbuild/${arch}/${branch} &prompt.root; tar cfv - packages/ | ssh portmgr@ftp-master tar xfC - w/ports/${arch}/tmp/${branch} Then log into ftp-master, verify that the package set was transferred successfully, remove the package set that the new package set is to replace (in ~/w/ports/${arch}), and move the new set into place. Some of the directories on ftp-master are, in fact, symlinks. Be sure you move the new packages directory over the real destination directory, and not one of the symlinks that points to it. Distfiles can be transferred via rsync: &prompt.root; cd /var/portbuild/${arch}/${branch} &prompt.root; rsync -r -v -l -p -c -n distfiles/ portmgr@ftp-master:w/ports/distfiles/ | tee log ALWAYS use -n first with rsync and check the output to make sure it is sane. If it looks good, re-run the rsync without the -n option.
-- Ken Smith - From there to here, from here to | kensmith@cse.buffalo.edu there, funny things are everywhere. | - Theodore Geisel |