From owner-svn-doc-head@FreeBSD.ORG Mon Dec 24 09:41:46 2012 Return-Path: Delivered-To: svn-doc-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9ABDE581; Mon, 24 Dec 2012 09:41:46 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) by mx1.freebsd.org (Postfix) with ESMTP id 7D8B48FC17; Mon, 24 Dec 2012 09:41:46 +0000 (UTC) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.5/8.14.5) with ESMTP id qBO9fkdf079002; Mon, 24 Dec 2012 09:41:46 GMT (envelope-from hrs@svn.freebsd.org) Received: (from hrs@localhost) by svn.freebsd.org (8.14.5/8.14.5/Submit) id qBO9fk9Z079001; Mon, 24 Dec 2012 09:41:46 GMT (envelope-from hrs@svn.freebsd.org) Message-Id: <201212240941.qBO9fk9Z079001@svn.freebsd.org> From: Hiroki Sato Date: Mon, 24 Dec 2012 09:41:46 +0000 (UTC) To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r40467 - head/en_US.ISO8859-1/articles/portbuild X-SVN-Group: doc-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-doc-head@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SVN commit messages for the doc tree for head List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 09:41:46 -0000 Author: hrs Date: Mon Dec 24 09:41:46 2012 New Revision: 40467 URL: http://svnweb.freebsd.org/changeset/doc/40467 Log: Markup style/whitespace cleanup. No content change. Modified: head/en_US.ISO8859-1/articles/portbuild/article.xml Modified: head/en_US.ISO8859-1/articles/portbuild/article.xml ============================================================================== --- head/en_US.ISO8859-1/articles/portbuild/article.xml Mon Dec 24 09:14:24 2012 (r40466) +++ head/en_US.ISO8859-1/articles/portbuild/article.xml Mon Dec 24 09:41:46 2012 (r40467) @@ -50,61 +50,64 @@ . This article documents the internal workings of the - cluster. + cluster. Many of the details in this article will be of interest only to - those on the Ports Management - team. + those on the Ports Management + team. The codebase - Most of the package building magic occurs under the - /var/portbuild directory. Unless - otherwise specified, all paths will be relative to - this location. ${arch} will - be used to specify one of the package architectures - (e.g., amd64, arm, &i386;, ia64, powerpc, &sparc64;), and - ${branch} will be used - to specify the build branch (e.g., 7, 7-exp, 8, 8-exp, 9, 9-exp, 10, 10-exp). - The set of branches that portmgr currently - supports is the same as those that the &os; - security team - supports. - - - - Packages are no longer built for branches 4, 5, or 6, nor - for the alpha architecture. - - - The scripts that control all of this live in - /var/portbuild/scripts/. - These are the checked-out copies from the Subversion repository at - - base/projects/portbuild/scripts/ - . - - Typically, incremental builds are done that use previous - packages as dependencies; this takes less time, and puts less - load on the mirrors. Full builds are usually only done: + Most of the package building magic occurs under the + /var/portbuild directory. Unless + otherwise specified, all paths will be relative to + this location. ${arch} will + be used to specify one of the package architectures + (e.g., amd64, arm, &i386;, ia64, powerpc, &sparc64;), and + ${branch} will be used + to specify the build branch (e.g., 7, 7-exp, 8, 8-exp, 9, 9-exp, 10, 10-exp). + The set of branches that portmgr currently + supports is the same as those that the &os; + security team + supports. + - - right after release time, for the - -STABLE branches + + Packages are no longer built for branches 4, 5, or 6, nor + for the alpha architecture. + - periodically to test changes to - -CURRENT - + The scripts that control all of this live in + /var/portbuild/scripts/. + These are the checked-out copies from the Subversion repository at + + base/projects/portbuild/scripts/ + . + + Typically, incremental builds are done that use previous + packages as dependencies; this takes less time, and puts less + load on the mirrors. Full builds are usually only done: - for experimental ("exp-") builds + + + right after release time, for the + -STABLE branches + - + + periodically to test changes to + -CURRENT + - Packages from experimental builds are not uploaded. + + for experimental ("exp-") builds + + + Packages from experimental builds are not uploaded. @@ -116,25 +119,32 @@ for other hosts to be head nodes. Among the changes were: - removal of the hard-coding of the string - pointyhat + + removal of the hard-coding of the string + pointyhat + - factoring out all configuration constants (which - were previously scattered throughout the code) into configuration - files (see below) - - - appending the hostname to the directories - specified by buildid (this will allow - directories to be unambigious when copied between machines.) - + + factoring out all configuration constants (which + were previously scattered throughout the code) into configuration + files (see below) + - making the scripts more robust in terms of setting - up directories and symlinks + + appending the hostname to the directories + specified by buildid (this will allow + directories to be unambigious when copied between machines.) + - where necessary, changing certain script invocations - to make all the above easier + + making the scripts more robust in terms of setting + up directories and symlinks + + + where necessary, changing certain script invocations + to make all the above easier + This document was originally written before these changes @@ -220,13 +230,11 @@ The bindist.tar file is extracted onto each client at client boot time, and at the start of each pass of the dopackages - script. - + script. - For both commands above, if - ${buildid} is - latest, it may be omitted. - + For both commands above, if + ${buildid} is + latest, it may be omitted. @@ -339,281 +347,260 @@ PKG_BIN=/usr/local/sbin/pkg <command>dopackages</command> scripts - The scripts/dopackages.wrapper script - is used to perform the builds. - - &prompt.root; dopackages.wrapper ${arch} ${branch} ${buildid} - - Most often, you will be using latest for - the value of buildid. + The scripts/dopackages.wrapper script + is used to perform the builds. - [-options] may be zero or more of the - following: + &prompt.root; dopackages.wrapper ${arch} ${branch} ${buildid} - - - - Do not delete this build in the - future, when it would be normally deleted as part of the - latest - previous cycle. - Do not forget to clean it up manually when you no longer need it. - - + Most often, you will be using latest for + the value of buildid. - - - Do not perform - post-processing once the build is complete. Useful - if you expect that the build will need to be restarted - once it finishes. If you use this option, do not forget to cleanup - the clients when you do not need the build any more. - - + [-options] may be zero or more of the + following: - - - Perform - post-processing only. - - + + + - Do not delete this build in the + future, when it would be normally deleted as part of the + latest - previous cycle. + Do not forget to clean it up manually when you no longer need it. + - - - By default, when the - stage of the build is complete, the build - data will be deleted from the clients. This option will prevent - that. - + + - Do not perform + post-processing once the build is complete. Useful + if you expect that the build will need to be restarted + once it finishes. If you use this option, do not forget to cleanup + the clients when you do not need the build any more. + - - - Restart an interrupted - (or non-finished) build from the - beginning. Ports that failed on the previous build will - be rebuilt. - - + + - Perform + post-processing only. + - - - Restart an interrupted - (or non-finished) build. Will not - rebuild ports that failed on the previous build. - - + + - By default, when the + stage of the build is complete, the build + data will be deleted from the clients. This option will prevent + that. + - - - Compare the - interesting fields of the new - INDEX with the previous one, - remove packages and log files for the old ports that - have changed, and rebuild the rest. This - cuts down on build times substantially since - unchanged ports do not get rebuilt every time. - - + + - Restart an interrupted + (or non-finished) build from the + beginning. Ports that failed on the previous build will + be rebuilt. + - - - This package build is - intended to end up on a CD-ROM, so - NO_CDROM packages and distfiles - should be deleted in post-processing. - - + + - Restart an interrupted + (or non-finished) build. Will not + rebuild ports that failed on the previous build. + - - - Perform all - the preprocessing steps, but do not actually do - the package build. - - + + - Compare the + interesting fields of the new + INDEX with the previous one, + remove packages and log files for the old ports that + have changed, and rebuild the rest. This + cuts down on build times substantially since + unchanged ports do not get rebuilt every time. + - - - Do not rebuild - INDEX during preprocessing. - - + + - This package build is + intended to end up on a CD-ROM, so + NO_CDROM packages and distfiles + should be deleted in post-processing. + - - - Do not rebuild the - duds file (ports that are never - built, e.g., those marked IGNORE, - NO_PACKAGE, etc.) during - preprocessing. - - + + - Perform all + the preprocessing steps, but do not actually do + the package build. + - - - Do not check the - SUBDIRS for ports that are not connected - to the build. - - + + - Do not rebuild + INDEX during preprocessing. + - - - Try to build - BROKEN ports (off by default - because the amd64/&i386; clusters are fast enough now - that when doing incremental builds, more time - was spent rebuilding things that were going to - fail anyway. Conversely, the other clusters - are slow enough that it would be a waste of time - to try and build BROKEN ports). - - - With , you probably - also want to use - and - . - - + + - Do not rebuild the + duds file (ports that are never + built, e.g., those marked IGNORE, + NO_PACKAGE, etc.) during + preprocessing. + - - - Do not update the - src tree from the ZFS snapshot, keep the tree from - previous build instead. - - + + - Do not check the + SUBDIRS for ports that are not connected + to the build. + - - - Do not update the - src tree from the ZFS snapshot, update it with - a fresh checkout instead. - - + + - Try to build + BROKEN ports (off by default + because the amd64/&i386; clusters are fast enough now + that when doing incremental builds, more time + was spent rebuilding things that were going to + fail anyway. Conversely, the other clusters + are slow enough that it would be a waste of time + to try and build BROKEN ports). - - - Do not update the - ports tree from the ZFS snapshot, keep the tree from - previous build instead. - - + + With , you probably + also want to use + and + . + + - - - Do not update the - ports tree from the ZFS snapshot, update it with - a fresh checkout instead. - - + + - Do not update the + src tree from the ZFS snapshot, keep the tree from + previous build instead. + - - - Do not attempt to build - RESTRICTED ports. - - + + - Do not update the + src tree from the ZFS snapshot, update it with + a fresh checkout instead. + - - - Do not make it fatal for - ports to leave behind files after deinstallation. - - + + - Do not update the + ports tree from the ZFS snapshot, keep the tree from + previous build instead. + - - - Do not collect distfiles - that pass make checksum for later - uploading to ftp-master. - - + + - Do not update the + ports tree from the ZFS snapshot, update it with + a fresh checkout instead. + - - - Fetch the - distfile from the original MASTER_SITES - rather than any cache such as on ftp-master. - - + + - Do not attempt to build + RESTRICTED ports. + - - - - defeat the "qmanager threshhold" check for runaway - builds. You want this primarily when doing a - of a build that you expect to mostly - fail, or perhaps a run. By default, - the threshhold check is done. - - + + - Do not make it fatal for + ports to leave behind files after deinstallation. + - Unless you specify , - , or , - the symlinks for the existing builds will be rotated. i.e, - the existing symlink for previous will - be deleted; the most recent build will have its symlink changed - to previous/; and a new build will be - created and symlinked into latest/. - + + - Do not collect distfiles + that pass make checksum for later + uploading to ftp-master. + - If the last build finished cleanly you do not need to delete - anything. If it was interrupted, or you selected - , you need to clean up clients by running - + + - Fetch the + distfile from the original MASTER_SITES + rather than any cache such as on ftp-master. + - &prompt.user; build cleanup ${arch} ${branch} ${buildid} -full + + + - defeat the "qmanager threshhold" check for runaway + builds. You want this primarily when doing a + of a build that you expect to mostly + fail, or perhaps a run. By default, + the threshhold check is done. + + - When a new build is created, the directories errors/, - logs/, packages/, and so - forth, are cleaned by the scripts. If you are short of space, - you can also clean out ports/distfiles/. - Leave the latest/ directory alone; it is - a symlink for the webserver. + Unless you specify , + , or , + the symlinks for the existing builds will be rotated. i.e, + the existing symlink for previous will + be deleted; the most recent build will have its symlink changed + to previous/; and a new build will be + created and symlinked into latest/. + + If the last build finished cleanly you do not need to delete + anything. If it was interrupted, or you selected + , you need to clean up clients by running + + &prompt.user; build cleanup ${arch} ${branch} ${buildid} -full + + When a new build is created, the directories errors/, + logs/, packages/, and so + forth, are cleaned by the scripts. If you are short of space, + you can also clean out ports/distfiles/. + Leave the latest/ directory alone; it is + a symlink for the webserver. - - dosetupnodes is supposed to be run from - the dopackages script in the - case, but it can be a good idea to - run it by hand and then verify that the clients all have the - expected job load. Sometimes, - dosetupnode cannot clean up a build and you - need to do it by hand. (This is a bug.) - + + dosetupnodes is supposed to be run from + the dopackages script in the + case, but it can be a good idea to + run it by hand and then verify that the clients all have the + expected job load. Sometimes, + dosetupnode cannot clean up a build and you + need to do it by hand. (This is a bug.) + - Make sure the ${arch} build - is run as the ports-${arch} user - or it will complain loudly. - - The actual package build itself occurs in two - identical phases. The reason for this is that sometimes - transient problems (e.g., NFS failures, FTP sites being - unreachable, etc.) may halt a build. Doing things - in two phases is a workaround for these types of - problems. - - Be careful that ports/Makefile - does not specify any empty subdirectories. This is especially - important if you are doing an -exp build. If the build - process encounters an empty subdirectory, both package build - phases will stop short, and an error similar to the following - will be written to - ${arch}/${branch}/journal: - + Make sure the ${arch} build + is run as the ports-${arch} user + or it will complain loudly. - don't know how to make dns-all(continuing) + + The actual package build itself occurs in two + identical phases. The reason for this is that sometimes + transient problems (e.g., NFS failures, FTP sites being + unreachable, etc.) may halt a build. Doing things + in two phases is a workaround for these types of + problems. + - To correct this problem, simply comment out or remove - the SUBDIR entries that point to empty - subdirectories. After doing this, you can restart the build - by running the proper dopackages command - with the option. - + Be careful that ports/Makefile + does not specify any empty subdirectories. This is especially + important if you are doing an -exp build. If the build + process encounters an empty subdirectory, both package build + phases will stop short, and an error similar to the following + will be written to + ${arch}/${branch}/journal: + + don't know how to make dns-all(continuing) + + To correct this problem, simply comment out or remove + the SUBDIR entries that point to empty + subdirectories. After doing this, you can restart the build + by running the proper dopackages command + with the option. - - This problem also appears if you create a new category - Makefile with no SUBDIRs - in it. This is probably a bug. - + + This problem also appears if you create a new category + Makefile with no SUBDIRs + in it. This is probably a bug. + - - Update the i386-7 tree and do a complete build + + Update the i386-7 tree and do a complete build - &prompt.user; dopackages.wrapper i386 7 -nosrc -norestr -nofinish - + &prompt.user; dopackages.wrapper i386 7 -nosrc -norestr -nofinish + - - Restart an interrupted amd64-8 build without updating + + Restart an interrupted amd64-8 build without updating - &prompt.user; dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish - + &prompt.user; dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish + - - Post-process a completed sparc64-7 tree + + Post-process a completed sparc64-7 tree - &prompt.user; dopackages.wrapper sparc64 7 -finish - + &prompt.user; dopackages.wrapper sparc64 7 -finish + - Hint: it is usually best to run the dopackages - command inside of screen(1). + Hint: it is usually best to run the dopackages + command inside of screen(1). @@ -630,8 +617,7 @@ PKG_BIN=/usr/local/sbin/pkgbranch [newid] - Creates newid (or a datestamp if not specified). - Only needed when bringing up a new branch or a new architecture. - + Only needed when bringing up a new branch or a new architecture. @@ -639,8 +625,7 @@ PKG_BIN=/usr/local/sbin/pkgbranch oldid [newid] - Clones oldid to - newid (or a datestamp if not specified). - + newid (or a datestamp if not specified). @@ -649,8 +634,7 @@ PKG_BIN=/usr/local/sbin/pkgbuildid - Replaces the src tree with a new ZFS snapshot. Do not forget to use -nosrc flag to dopackages - later! - + later! @@ -659,10 +643,8 @@ PKG_BIN=/usr/local/sbin/pkgbuildid - Replaces the ports tree with a new ZFS snapshot. Do not forget to use -noports flag to dopackages - later! - + later! - @@ -673,7 +655,7 @@ PKG_BIN=/usr/local/sbin/pkg - &prompt.root; path/qmanager/packagebuild amd64 7-exp 20080904212103 aclock-0.2.3_2.tbz + &prompt.root; path/qmanager/packagebuild amd64 7-exp 20080904212103 aclock-0.2.3_2.tbz @@ -696,15 +678,13 @@ PKG_BIN=/usr/local/sbin/pkg An update of the running branch's - src tree from the ZFS snapshot - + src tree from the ZFS snapshot Checks which ports do not have a SUBDIR entry in their respective - category's Makefile - + category's Makefile @@ -716,14 +696,12 @@ PKG_BIN=/usr/local/sbin/pkg Generates a fresh INDEX - file - + file Sets up the nodes that will be used in the - build - + build @@ -755,8 +733,7 @@ PKG_BIN=/usr/local/sbin/pkgBuild Maintenance There are several cases where you will need to manually clean - up a build: - + up a build: @@ -772,101 +749,98 @@ PKG_BIN=/usr/local/sbin/pkgqmanager has crashed and has been restarted. - - - - Interrupting a Build - - Manually interrupting a build is a bit messy. First you need to - identify the tty in which it's running (either record the output - of &man.tty.1; when you start the build, or use ps x - to identify it. You need to make sure that nothing else important - is running in this tty, e.g., ps -t p1 or whatever. - If there is not, you can just kill off the whole term easily with - pkill -t pts/1; otherwise issue a - kill -HUP in there by, for example, -ps -t pts/1 -o pid= | xargs kill -HUP. Replace - p1 by whatever the tty is, of course. + - The - package builds dispatched by make to - the client machines will clean themselves up after a - few minutes (check with ps x until they - all go away). - - If you do not kill &man.make.1;, then it will spawn more jobs. - If you do not kill dopackages, then it will restart - the entire build. If you do not kill the pdispatch - processes, they'll keep going (or respawn) until they've built their - package. + + Interrupting a Build - + Manually interrupting a build is a bit messy. First you need to + identify the tty in which it's running (either record the output + of &man.tty.1; when you start the build, or use ps x + to identify it. You need to make sure that nothing else important + is running in this tty, e.g., ps -t p1 or whatever. + If there is not, you can just kill off the whole term easily with + pkill -t pts/1; otherwise issue a + kill -HUP in there by, for example, + ps -t pts/1 -o pid= | xargs kill -HUP. Replace + p1 by whatever the tty is, of course. + + The + package builds dispatched by make to + the client machines will clean themselves up after a + few minutes (check with ps x until they + all go away). + + If you do not kill &man.make.1;, then it will spawn more jobs. + If you do not kill dopackages, then it will restart + the entire build. If you do not kill the pdispatch + processes, they'll keep going (or respawn) until they've built their + package. + - - Cleaning up a Build + + Cleaning up a Build - To free up resources, you will need to clean up client machines by - running build cleanup command. For example: + To free up resources, you will need to clean up client machines by + running build cleanup command. For example: &prompt.user; /var/portbuild/scripts/build cleanup i386 8-exp 20080714120411 -full - If you forget to do this, then the old build - jails will not be cleaned up for 24 hours, and no - new jobs will be dispatched in their place since - pointyhat thinks the job slot is still occupied. - - To check, cat ~/loads/* to display the - status of client machines; the first column is the number of jobs - it thinks is running, and this should be roughly concordant - with the load average. loads is refreshed - every 2 minutes. If you do ps x | grep pdispatch - and it is less than the number of jobs that loads - thinks are in use, you are in trouble. - - You may have problem with the umount - commands hanging. If so, you are going to have to use the - allgohans script to run an &man.ssh.1; - command across all clients for that buildenv. For example: + If you forget to do this, then the old build + jails will not be cleaned up for 24 hours, and no + new jobs will be dispatched in their place since + pointyhat thinks the job slot is still occupied. + + To check, cat ~/loads/* to display the + status of client machines; the first column is the number of jobs + it thinks is running, and this should be roughly concordant + with the load average. loads is refreshed + every 2 minutes. If you do ps x | grep pdispatch + and it is less than the number of jobs that loads + thinks are in use, you are in trouble. + + You may have problem with the umount + commands hanging. If so, you are going to have to use the + allgohans script to run an &man.ssh.1; + command across all clients for that buildenv. For example: &prompt.user; ssh gohan24 df - will get you a df, and + will get you a df, and &prompt.user; allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports" &prompt.user; allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/src" - are supposed to get rid of the hanging mounts. You will have to - keep doing them since there can be multiple mounts. + are supposed to get rid of the hanging mounts. You will have to + keep doing them since there can be multiple mounts. - - Ignore the following: + + Ignore the following: - umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory + umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: unknown file system umount: Cleanup of /x/tmp/8-exp/chroot/53837/compat/linux/proc failed! /x/tmp/8-exp/chroot/53837/compat/linux/proc: not a file system root directory - The former two mean that the client did not have those mounted; - the latter two are a bug. - - You may also see messages about procfs. - - - After you have done all the above, remove the - ${arch}/lock - file before trying to restart the build. If you do not, - dopackages will simply exit. - + The former two mean that the client did not have those mounted; + the latter two are a bug. - If you have to do a ports tree update before - restarting, you may have to rebuild either duds, - INDEX, or both. + You may also see messages about procfs. + + After you have done all the above, remove the + ${arch}/lock + file before trying to restart the build. If you do not, + dopackages will simply exit. + + If you have to do a ports tree update before + restarting, you may have to rebuild either duds, + INDEX, or both. Maintaining builds with the <command>build</command> - command + command Here are the rest of the options for the build command: @@ -875,21 +849,16 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 build destroy arch branch - Destroy the - build id. - + build id. build list arch branch - Shows the current set - of build ids. - + of build ids. - - - @@ -930,8 +899,7 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 directory. The next time the cluster tries to build this port, it will tar, compress, and copy the WRKDIR to - ${arch}/${branch}/wrkdirs/. - + ${arch}/${branch}/wrkdirs/. If you find that the system is looping trying to build the same package over and over again, you may be able to fix the @@ -944,8 +912,7 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 Keep an eye on &man.df.1; output. If the /var/portbuild file system becomes full - then Bad Things happen. - + then Bad Things happen. The status of all current builds is generated periodically into the packagestats.html file, e.g., @@ -1072,8 +1039,7 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 and inform them of the release package location. Remember to coordinate with the &a.re; about the timing - and status of the release builds. - + and status of the release builds. @@ -1132,8 +1098,9 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 - Some of the directories on - ftp-master are, in fact, symlinks. Examples: + + Some of the directories on + ftp-master are, in fact, symlinks. Examples: *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***