Date: Mon, 24 Dec 2012 09:41:46 +0000 (UTC) From: Hiroki Sato <hrs@FreeBSD.org> To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r40467 - head/en_US.ISO8859-1/articles/portbuild Message-ID: <201212240941.qBO9fk9Z079001@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: hrs Date: Mon Dec 24 09:41:46 2012 New Revision: 40467 URL: http://svnweb.freebsd.org/changeset/doc/40467 Log: Markup style/whitespace cleanup. No content change. Modified: head/en_US.ISO8859-1/articles/portbuild/article.xml Modified: head/en_US.ISO8859-1/articles/portbuild/article.xml ============================================================================== --- head/en_US.ISO8859-1/articles/portbuild/article.xml Mon Dec 24 09:14:24 2012 (r40466) +++ head/en_US.ISO8859-1/articles/portbuild/article.xml Mon Dec 24 09:41:46 2012 (r40467) @@ -50,61 +50,64 @@ <ulink url="http://pointyhat.FreeBSD.org"></ulink>.</para>; <para>This article documents the internal workings of the - cluster.</para> + cluster.</para> <note> <para>Many of the details in this article will be of interest only to - those on the <ulink url="&url.base;/portmgr/">Ports Management</ulink> - team.</para> + those on the <ulink url="&url.base;/portmgr/">Ports Management</ulink> + team.</para> </note> <sect2 id="codebase"> <title>The codebase</title> - <para>Most of the package building magic occurs under the - <filename>/var/portbuild</filename> directory. Unless - otherwise specified, all paths will be relative to - this location. <replaceable>${arch}</replaceable> will - be used to specify one of the package architectures - (e.g., amd64, arm, &i386;, ia64, powerpc, &sparc64;), and - <replaceable>${branch}</replaceable> will be used - to specify the build branch (e.g., 7, 7-exp, 8, 8-exp, 9, 9-exp, 10, 10-exp). - The set of branches that <username>portmgr</username> currently - supports is the same as those that the &os; - <ulink url="http://www.freebsd.org/security/index.html#supported-branches">security team</ulink> - supports. - </para> - - <note> - <para>Packages are no longer built for branches 4, 5, or 6, nor - for the alpha architecture.</para> - </note> - - <para>The scripts that control all of this live in - <filename role="directory">/var/portbuild/scripts/</filename>. - These are the checked-out copies from the Subversion repository at - <ulink url="http://svnweb.freebsd.org/base/projects/portbuild/scripts/">; - <filename role="directory">base/projects/portbuild/scripts/</filename> - </ulink>.</para> - - <para>Typically, incremental builds are done that use previous - packages as dependencies; this takes less time, and puts less - load on the mirrors. Full builds are usually only done:</para> + <para>Most of the package building magic occurs under the + <filename>/var/portbuild</filename> directory. Unless + otherwise specified, all paths will be relative to + this location. <replaceable>${arch}</replaceable> will + be used to specify one of the package architectures + (e.g., amd64, arm, &i386;, ia64, powerpc, &sparc64;), and + <replaceable>${branch}</replaceable> will be used + to specify the build branch (e.g., 7, 7-exp, 8, 8-exp, 9, 9-exp, 10, 10-exp). + The set of branches that <username>portmgr</username> currently + supports is the same as those that the &os; + <ulink url="http://www.freebsd.org/security/index.html#supported-branches">security team</ulink> + supports. + </para> - <itemizedlist> - <listitem><para>right after release time, for the - <literal>-STABLE</literal> branches</para></listitem> + <note> + <para>Packages are no longer built for branches 4, 5, or 6, nor + for the alpha architecture.</para> + </note> - <listitem><para>periodically to test changes to - <literal>-CURRENT</literal> - </para></listitem> + <para>The scripts that control all of this live in + <filename role="directory">/var/portbuild/scripts/</filename>. + These are the checked-out copies from the Subversion repository at + <ulink url="http://svnweb.freebsd.org/base/projects/portbuild/scripts/">; + <filename role="directory">base/projects/portbuild/scripts/</filename> + </ulink>.</para> + + <para>Typically, incremental builds are done that use previous + packages as dependencies; this takes less time, and puts less + load on the mirrors. Full builds are usually only done:</para> - <listitem><para>for experimental (<literal>"exp-"</literal>) builds</para></listitem> + <itemizedlist> + <listitem> + <para>right after release time, for the + <literal>-STABLE</literal> branches</para> + </listitem> - </itemizedlist> + <listitem> + <para>periodically to test changes to + <literal>-CURRENT</literal></para> + </listitem> - <para>Packages from experimental builds are not uploaded.</para> + <listitem> + <para>for experimental (<literal>"exp-"</literal>) builds</para> + </listitem> + </itemizedlist> + <para>Packages from experimental builds are not uploaded.</para> </sect2> <sect2 id="codebase-notes"> @@ -116,25 +119,32 @@ for other hosts to be head nodes. Among the changes were:</para> <itemizedlist> - <listitem><para>removal of the hard-coding of the string - <literal>pointyhat</literal></para></listitem> + <listitem> + <para>removal of the hard-coding of the string + <literal>pointyhat</literal></para> + </listitem> - <listitem><para>factoring out all configuration constants (which - were previously scattered throughout the code) into configuration - files (see <link linkend="new-head-node">below</link>) - </para></listitem> - - <listitem><para>appending the hostname to the directories - specified by <literal>buildid</literal> (this will allow - directories to be unambigious when copied between machines.) - </para></listitem> + <listitem> + <para>factoring out all configuration constants (which + were previously scattered throughout the code) into configuration + files (see <link linkend="new-head-node">below</link>)</para> + </listitem> - <listitem><para>making the scripts more robust in terms of setting - up directories and symlinks</para></listitem> + <listitem> + <para>appending the hostname to the directories + specified by <literal>buildid</literal> (this will allow + directories to be unambigious when copied between machines.)</para> + </listitem> - <listitem><para>where necessary, changing certain script invocations - to make all the above easier</para></listitem> + <listitem> + <para>making the scripts more robust in terms of setting + up directories and symlinks</para> + </listitem> + <listitem> + <para>where necessary, changing certain script invocations + to make all the above easier</para> + </listitem> </itemizedlist> <para>This document was originally written before these changes @@ -220,13 +230,11 @@ <para>The <filename>bindist.tar</filename> file is extracted onto each client at client boot time, and at the start of each pass of the <command>dopackages</command> - script. - </para> + script.</para> - <para>For both commands above, if - <replaceable>${buildid}</replaceable> is - <literal>latest</literal>, it may be omitted. - </para> + <para>For both commands above, if + <replaceable>${buildid}</replaceable> is + <literal>latest</literal>, it may be omitted.</para> </sect1> <sect1 id="customizing"> @@ -339,281 +347,260 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <sect2 id="build-dopackages"> <title><command>dopackages</command> scripts</title> - <para>The <filename>scripts/dopackages.wrapper</filename> script - is used to perform the builds.</para> - - <screen>&prompt.root; <userinput>dopackages.wrapper <replaceable>${arch}</replaceable> <replaceable>${branch}</replaceable> <replaceable>${buildid}</replaceable> <option>[-options]</option></userinput></screen> - - <para>Most often, you will be using <literal>latest</literal> for - the value of <replaceable>buildid</replaceable>.</para> + <para>The <filename>scripts/dopackages.wrapper</filename> script + is used to perform the builds.</para> - <para><literal>[-options]</literal> may be zero or more of the - following:</para> + <screen>&prompt.root; <userinput>dopackages.wrapper <replaceable>${arch}</replaceable> <replaceable>${branch}</replaceable> <replaceable>${buildid}</replaceable> <option>[-options]</option></userinput></screen> - <itemizedlist> - <listitem> - <para><option>-keep</option> - Do not delete this build in the - future, when it would be normally deleted as part of the - <literal>latest</literal> - <literal>previous</literal> cycle. - Do not forget to clean it up manually when you no longer need it. - </para> - </listitem> + <para>Most often, you will be using <literal>latest</literal> for + the value of <replaceable>buildid</replaceable>.</para> - <listitem> - <para><option>-nofinish</option> - Do not perform - post-processing once the build is complete. Useful - if you expect that the build will need to be restarted - once it finishes. If you use this option, do not forget to cleanup - the clients when you do not need the build any more. - </para> - </listitem> + <para><literal>[-options]</literal> may be zero or more of the + following:</para> - <listitem> - <para><option>-finish</option> - Perform - post-processing only. - </para> - </listitem> + <itemizedlist> + <listitem> + <para><option>-keep</option> - Do not delete this build in the + future, when it would be normally deleted as part of the + <literal>latest</literal> - <literal>previous</literal> cycle. + Do not forget to clean it up manually when you no longer need it.</para> + </listitem> - <listitem> - <para><option>-nocleanup</option> - By default, when the - <option>-finish</option> stage of the build is complete, the build - data will be deleted from the clients. This option will prevent - that.</para> - </listitem> + <listitem> + <para><option>-nofinish</option> - Do not perform + post-processing once the build is complete. Useful + if you expect that the build will need to be restarted + once it finishes. If you use this option, do not forget to cleanup + the clients when you do not need the build any more.</para> + </listitem> - <listitem> - <para><option>-restart</option> - Restart an interrupted - (or non-<literal>finish</literal>ed) build from the - beginning. Ports that failed on the previous build will - be rebuilt. - </para> - </listitem> + <listitem> + <para><option>-finish</option> - Perform + post-processing only.</para> + </listitem> - <listitem> - <para><option>-continue</option> - Restart an interrupted - (or non-<literal>finish</literal>ed) build. Will not - rebuild ports that failed on the previous build. - </para> - </listitem> + <listitem> + <para><option>-nocleanup</option> - By default, when the + <option>-finish</option> stage of the build is complete, the build + data will be deleted from the clients. This option will prevent + that.</para> + </listitem> - <listitem> - <para><option>-incremental</option> - Compare the - interesting fields of the new - <filename>INDEX</filename> with the previous one, - remove packages and log files for the old ports that - have changed, and rebuild the rest. This - cuts down on build times substantially since - unchanged ports do not get rebuilt every time. - </para> - </listitem> + <listitem> + <para><option>-restart</option> - Restart an interrupted + (or non-<literal>finish</literal>ed) build from the + beginning. Ports that failed on the previous build will + be rebuilt.</para> + </listitem> - <listitem> - <para><option>-cdrom</option> - This package build is - intended to end up on a CD-ROM, so - <makevar>NO_CDROM</makevar> packages and distfiles - should be deleted in post-processing. - </para> - </listitem> + <listitem> + <para><option>-continue</option> - Restart an interrupted + (or non-<literal>finish</literal>ed) build. Will not + rebuild ports that failed on the previous build.</para> + </listitem> - <listitem> - <para><option>-nobuild</option> - Perform all - the preprocessing steps, but do not actually do - the package build. - </para> - </listitem> + <listitem> + <para><option>-incremental</option> - Compare the + interesting fields of the new + <filename>INDEX</filename> with the previous one, + remove packages and log files for the old ports that + have changed, and rebuild the rest. This + cuts down on build times substantially since + unchanged ports do not get rebuilt every time.</para> + </listitem> - <listitem> - <para><option>-noindex</option> - Do not rebuild - <filename>INDEX</filename> during preprocessing. - </para> - </listitem> + <listitem> + <para><option>-cdrom</option> - This package build is + intended to end up on a CD-ROM, so + <makevar>NO_CDROM</makevar> packages and distfiles + should be deleted in post-processing.</para> + </listitem> - <listitem> - <para><option>-noduds</option> - Do not rebuild the - <filename>duds</filename> file (ports that are never - built, e.g., those marked <literal>IGNORE</literal>, - <makevar>NO_PACKAGE</makevar>, etc.) during - preprocessing. - </para> - </listitem> + <listitem> + <para><option>-nobuild</option> - Perform all + the preprocessing steps, but do not actually do + the package build.</para> + </listitem> - <listitem> - <para><option>-nochecksubdirs</option> - Do not check the - <makevar>SUBDIRS</makevar> for ports that are not connected - to the build. - </para> - </listitem> + <listitem> + <para><option>-noindex</option> - Do not rebuild + <filename>INDEX</filename> during preprocessing.</para> + </listitem> - <listitem> - <para><option>-trybroken</option> - Try to build - <makevar>BROKEN</makevar> ports (off by default - because the amd64/&i386; clusters are fast enough now - that when doing incremental builds, more time - was spent rebuilding things that were going to - fail anyway. Conversely, the other clusters - are slow enough that it would be a waste of time - to try and build <makevar>BROKEN</makevar> ports). - </para> - <note> - <para>With <option>-trybroken</option>, you probably - also want to use <option>-fetch-original</option> - and - <option>-unlimited-errors</option>.</para> - </note> - </listitem> + <listitem> + <para><option>-noduds</option> - Do not rebuild the + <filename>duds</filename> file (ports that are never + built, e.g., those marked <literal>IGNORE</literal>, + <makevar>NO_PACKAGE</makevar>, etc.) during + preprocessing.</para> + </listitem> - <listitem> - <para><option>-nosrc</option> - Do not update the - <filename>src</filename> tree from the ZFS snapshot, keep the tree from - previous build instead. - </para> - </listitem> + <listitem> + <para><option>-nochecksubdirs</option> - Do not check the + <makevar>SUBDIRS</makevar> for ports that are not connected + to the build.</para> + </listitem> - <listitem> - <para><option>-srcvcs</option> - Do not update the - <filename>src</filename> tree from the ZFS snapshot, update it with - a fresh checkout instead. - </para> - </listitem> + <listitem> + <para><option>-trybroken</option> - Try to build + <makevar>BROKEN</makevar> ports (off by default + because the amd64/&i386; clusters are fast enough now + that when doing incremental builds, more time + was spent rebuilding things that were going to + fail anyway. Conversely, the other clusters + are slow enough that it would be a waste of time + to try and build <makevar>BROKEN</makevar> ports).</para> - <listitem> - <para><option>-noports</option> - Do not update the - <filename>ports</filename> tree from the ZFS snapshot, keep the tree from - previous build instead. - </para> - </listitem> + <note> + <para>With <option>-trybroken</option>, you probably + also want to use <option>-fetch-original</option> + and + <option>-unlimited-errors</option>.</para> + </note> + </listitem> - <listitem> - <para><option>-portsvcs</option> - Do not update the - <filename>ports</filename> tree from the ZFS snapshot, update it with - a fresh checkout instead. - </para> - </listitem> + <listitem> + <para><option>-nosrc</option> - Do not update the + <filename>src</filename> tree from the ZFS snapshot, keep the tree from + previous build instead.</para> + </listitem> - <listitem> - <para><option>-norestr</option> - Do not attempt to build - <makevar>RESTRICTED</makevar> ports. - </para> - </listitem> + <listitem> + <para><option>-srcvcs</option> - Do not update the + <filename>src</filename> tree from the ZFS snapshot, update it with + a fresh checkout instead.</para> + </listitem> - <listitem> - <para><option>-noplistcheck</option> - Do not make it fatal for - ports to leave behind files after deinstallation. - </para> - </listitem> + <listitem> + <para><option>-noports</option> - Do not update the + <filename>ports</filename> tree from the ZFS snapshot, keep the tree from + previous build instead.</para> + </listitem> - <listitem> - <para><option>-nodistfiles</option> - Do not collect distfiles - that pass <command>make checksum</command> for later - uploading to <hostid>ftp-master</hostid>. - </para> - </listitem> + <listitem> + <para><option>-portsvcs</option> - Do not update the + <filename>ports</filename> tree from the ZFS snapshot, update it with + a fresh checkout instead.</para> + </listitem> - <listitem> - <para><option>-fetch-original</option> - Fetch the - distfile from the original <makevar>MASTER_SITES</makevar> - rather than any cache such as on <hostid>ftp-master</hostid>. - </para> - </listitem> + <listitem> + <para><option>-norestr</option> - Do not attempt to build + <makevar>RESTRICTED</makevar> ports.</para> + </listitem> - <listitem> - <para><option>-unlimited-errors</option> - - defeat the "qmanager threshhold" check for runaway - builds. You want this primarily when doing a - <option>-restart</option> of a build that you expect to mostly - fail, or perhaps a <option>-trybroken</option> run. By default, - the threshhold check is done.</para> - </listitem> - </itemizedlist> + <listitem> + <para><option>-noplistcheck</option> - Do not make it fatal for + ports to leave behind files after deinstallation.</para> + </listitem> - <para>Unless you specify <option>-restart</option>, - <option>-continue</option>, or <option>-finish</option>, - the symlinks for the existing builds will be rotated. i.e, - the existing symlink for <filename>previous</filename> will - be deleted; the most recent build will have its symlink changed - to <filename>previous/</filename>; and a new build will be - created and symlinked into <filename>latest/</filename>. - </para> + <listitem> + <para><option>-nodistfiles</option> - Do not collect distfiles + that pass <command>make checksum</command> for later + uploading to <hostid>ftp-master</hostid>.</para> + </listitem> - <para>If the last build finished cleanly you do not need to delete - anything. If it was interrupted, or you selected - <option>-nocleanup</option>, you need to clean up clients by running - </para> + <listitem> + <para><option>-fetch-original</option> - Fetch the + distfile from the original <makevar>MASTER_SITES</makevar> + rather than any cache such as on <hostid>ftp-master</hostid>.</para> + </listitem> - <para>&prompt.user; <userinput>build cleanup <replaceable>${arch}</replaceable> <replaceable>${branch}</replaceable> <replaceable>${buildid}</replaceable> -full</userinput></para> + <listitem> + <para><option>-unlimited-errors</option> + - defeat the "qmanager threshhold" check for runaway + builds. You want this primarily when doing a + <option>-restart</option> of a build that you expect to mostly + fail, or perhaps a <option>-trybroken</option> run. By default, + the threshhold check is done.</para> + </listitem> + </itemizedlist> - <para>When a new build is created, the directories <filename>errors/</filename>, - <filename>logs/</filename>, <filename>packages/</filename>, and so - forth, are cleaned by the scripts. If you are short of space, - you can also clean out <filename>ports/distfiles/</filename>. - Leave the <filename>latest/</filename> directory alone; it is - a symlink for the webserver.</para> + <para>Unless you specify <option>-restart</option>, + <option>-continue</option>, or <option>-finish</option>, + the symlinks for the existing builds will be rotated. i.e, + the existing symlink for <filename>previous</filename> will + be deleted; the most recent build will have its symlink changed + to <filename>previous/</filename>; and a new build will be + created and symlinked into <filename>latest/</filename>.</para> + + <para>If the last build finished cleanly you do not need to delete + anything. If it was interrupted, or you selected + <option>-nocleanup</option>, you need to clean up clients by running</para> + + <screen>&prompt.user; <userinput>build cleanup <replaceable>${arch}</replaceable> <replaceable>${branch}</replaceable> <replaceable>${buildid}</replaceable> -full</userinput></screen> + + <para>When a new build is created, the directories <filename>errors/</filename>, + <filename>logs/</filename>, <filename>packages/</filename>, and so + forth, are cleaned by the scripts. If you are short of space, + you can also clean out <filename>ports/distfiles/</filename>. + Leave the <filename>latest/</filename> directory alone; it is + a symlink for the webserver.</para> - <note> - <para><literal>dosetupnodes</literal> is supposed to be run from - the <literal>dopackages</literal> script in the - <option>-restart</option> case, but it can be a good idea to - run it by hand and then verify that the clients all have the - expected job load. Sometimes, - <filename>dosetupnode</filename> cannot clean up a build and you - need to do it by hand. (This is a bug.)</para> - </note> + <note> + <para><literal>dosetupnodes</literal> is supposed to be run from + the <literal>dopackages</literal> script in the + <option>-restart</option> case, but it can be a good idea to + run it by hand and then verify that the clients all have the + expected job load. Sometimes, + <filename>dosetupnode</filename> cannot clean up a build and you + need to do it by hand. (This is a bug.)</para> + </note> - <para>Make sure the <replaceable>${arch}</replaceable> build - is run as the ports-<replaceable>${arch}</replaceable> user - or it will complain loudly.</para> - - <note><para>The actual package build itself occurs in two - identical phases. The reason for this is that sometimes - transient problems (e.g., NFS failures, FTP sites being - unreachable, etc.) may halt a build. Doing things - in two phases is a workaround for these types of - problems.</para></note> - - <para>Be careful that <filename>ports/Makefile</filename> - does not specify any empty subdirectories. This is especially - important if you are doing an -exp build. If the build - process encounters an empty subdirectory, both package build - phases will stop short, and an error similar to the following - will be written to - <filename><replaceable>${arch}</replaceable>/<replaceable>${branch}</replaceable>/journal</filename>: - </para> + <para>Make sure the <replaceable>${arch}</replaceable> build + is run as the ports-<replaceable>${arch}</replaceable> user + or it will complain loudly.</para> - <screen>don't know how to make dns-all(continuing)</screen> + <note> + <para>The actual package build itself occurs in two + identical phases. The reason for this is that sometimes + transient problems (e.g., NFS failures, FTP sites being + unreachable, etc.) may halt a build. Doing things + in two phases is a workaround for these types of + problems.</para> + </note> - <para>To correct this problem, simply comment out or remove - the <makevar>SUBDIR</makevar> entries that point to empty - subdirectories. After doing this, you can restart the build - by running the proper <command>dopackages</command> command - with the <option>-restart</option> option. - </para> + <para>Be careful that <filename>ports/Makefile</filename> + does not specify any empty subdirectories. This is especially + important if you are doing an -exp build. If the build + process encounters an empty subdirectory, both package build + phases will stop short, and an error similar to the following + will be written to + <filename><replaceable>${arch}</replaceable>/<replaceable>${branch}</replaceable>/journal</filename>:</para> + + <screen>don't know how to make dns-all(continuing)</screen> + + <para>To correct this problem, simply comment out or remove + the <makevar>SUBDIR</makevar> entries that point to empty + subdirectories. After doing this, you can restart the build + by running the proper <command>dopackages</command> command + with the <option>-restart</option> option.</para> - <note> - <para>This problem also appears if you create a new category - <filename>Makefile</filename> with no <makevar>SUBDIR</makevar>s - in it. This is probably a bug.</para> - </note> + <note> + <para>This problem also appears if you create a new category + <filename>Makefile</filename> with no <makevar>SUBDIR</makevar>s + in it. This is probably a bug.</para> + </note> - <example> - <title>Update the i386-7 tree and do a complete build</title> + <example> + <title>Update the i386-7 tree and do a complete build</title> - <screen>&prompt.user; <userinput>dopackages.wrapper i386 7 -nosrc -norestr -nofinish</userinput></screen> - </example> + <screen>&prompt.user; <userinput>dopackages.wrapper i386 7 -nosrc -norestr -nofinish</userinput></screen> + </example> - <example> - <title>Restart an interrupted amd64-8 build without updating</title> + <example> + <title>Restart an interrupted amd64-8 build without updating</title> - <screen>&prompt.user; <userinput>dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish</userinput></screen> - </example> + <screen>&prompt.user; <userinput>dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish</userinput></screen> + </example> - <example> - <title>Post-process a completed sparc64-7 tree</title> + <example> + <title>Post-process a completed sparc64-7 tree</title> - <screen>&prompt.user; <userinput>dopackages.wrapper sparc64 7 -finish</userinput></screen> - </example> + <screen>&prompt.user; <userinput>dopackages.wrapper sparc64 7 -finish</userinput></screen> + </example> - <para>Hint: it is usually best to run the <command>dopackages</command> - command inside of <command>screen(1)</command>.</para> + <para>Hint: it is usually best to run the <command>dopackages</command> + command inside of <command>screen(1)</command>.</para> </sect2> <sect2 id="build-command"> @@ -630,8 +617,7 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <replaceable>branch</replaceable> [<replaceable>newid</replaceable>]</literal> - Creates <replaceable>newid</replaceable> (or a datestamp if not specified). - Only needed when bringing up a new branch or a new architecture. - </para> + Only needed when bringing up a new branch or a new architecture.</para> </listitem> <listitem> @@ -639,8 +625,7 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <replaceable>branch</replaceable> <replaceable>oldid</replaceable> [<replaceable>newid</replaceable>]</literal> - Clones <replaceable>oldid</replaceable> to - <replaceable>newid</replaceable> (or a datestamp if not specified). - </para> + <replaceable>newid</replaceable> (or a datestamp if not specified).</para> </listitem> <listitem> @@ -649,8 +634,7 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <replaceable>buildid</replaceable></literal> - Replaces the src tree with a new ZFS snapshot. Do not forget to use <literal>-nosrc</literal> flag to <command>dopackages</command> - later! - </para> + later!</para> </listitem> <listitem> @@ -659,10 +643,8 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <replaceable>buildid</replaceable></literal> - Replaces the ports tree with a new ZFS snapshot. Do not forget to use <literal>-noports</literal> flag to <command>dopackages</command> - later! - </para> + later!</para> </listitem> - </itemizedlist> </sect2> @@ -673,7 +655,7 @@ PKG_BIN=/usr/local/sbin/pkg</programlist package set. This can be accomplished with the following invocation:</para> - <para>&prompt.root; <command><replaceable>path</replaceable>/qmanager/packagebuild <replaceable>amd64</replaceable> <replaceable>7-exp</replaceable> <replaceable>20080904212103</replaceable> <replaceable>aclock-0.2.3_2.tbz</replaceable></command></para> + <screen>&prompt.root; <command><replaceable>path</replaceable>/qmanager/packagebuild <replaceable>amd64</replaceable> <replaceable>7-exp</replaceable> <replaceable>20080904212103</replaceable> <replaceable>aclock-0.2.3_2.tbz</replaceable></command></screen> </sect2> </sect1> @@ -696,15 +678,13 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <listitem> <para>An update of the running branch's - <filename>src</filename> tree from the ZFS snapshot<footnoteref linkend='footnote-status1'></footnoteref> - </para> + <filename>src</filename> tree from the ZFS snapshot<footnoteref linkend='footnote-status1'></footnoteref></para> </listitem> <listitem> <para>Checks which ports do not have a <makevar>SUBDIR</makevar> entry in their respective - category's <filename>Makefile</filename><footnoteref linkend='footnote-status1'></footnoteref> - </para> + category's <filename>Makefile</filename><footnoteref linkend='footnote-status1'></footnoteref></para> </listitem> <listitem> @@ -716,14 +696,12 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <listitem> <para>Generates a fresh <filename>INDEX</filename> - file<footnoteref linkend='footnote-status1'></footnoteref><footnoteref linkend='footnote-buildstop'></footnoteref> - </para> + file<footnoteref linkend='footnote-status1'></footnoteref><footnoteref linkend='footnote-buildstop'></footnoteref></para> </listitem> <listitem> <para>Sets up the nodes that will be used in the - build<footnoteref linkend='footnote-status1'></footnoteref><footnoteref linkend='footnote-buildstop'></footnoteref> - </para> + build<footnoteref linkend='footnote-status1'></footnoteref><footnoteref linkend='footnote-buildstop'></footnoteref></para> </listitem> <listitem> @@ -755,8 +733,7 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <title>Build Maintenance</title> <para>There are several cases where you will need to manually clean - up a build: - </para> + up a build:</para> <orderedlist> <listitem> @@ -772,101 +749,98 @@ PKG_BIN=/usr/local/sbin/pkg</programlist <para><filename>qmanager</filename> has crashed and has been restarted.</para> </listitem> - </orderedlist> - - <sect2 id="interrupting"> - <title>Interrupting a Build</title> - - <para>Manually interrupting a build is a bit messy. First you need to - identify the tty in which it's running (either record the output - of &man.tty.1; when you start the build, or use <command>ps x</command> - to identify it. You need to make sure that nothing else important - is running in this tty, e.g., <userinput>ps -t p1</userinput> or whatever. - If there is not, you can just kill off the whole term easily with - <userinput>pkill -t pts/1</userinput>; otherwise issue a - <userinput>kill -HUP</userinput> in there by, for example, -<userinput>ps -t pts/1 -o pid= | xargs kill -HUP</userinput>. Replace - <replaceable>p1</replaceable> by whatever the tty is, of course.</para> + </orderedlist> - <para>The - package builds dispatched by <command>make</command> to - the client machines will clean themselves up after a - few minutes (check with <command>ps x</command> until they - all go away).</para> - - <para>If you do not kill &man.make.1;, then it will spawn more jobs. - If you do not kill <command>dopackages</command>, then it will restart - the entire build. If you do not kill the <command>pdispatch</command> - processes, they'll keep going (or respawn) until they've built their - package.</para> + <sect2 id="interrupting"> + <title>Interrupting a Build</title> - </sect2> + <para>Manually interrupting a build is a bit messy. First you need to + identify the tty in which it's running (either record the output + of &man.tty.1; when you start the build, or use <command>ps x</command> + to identify it. You need to make sure that nothing else important + is running in this tty, e.g., <userinput>ps -t p1</userinput> or whatever. + If there is not, you can just kill off the whole term easily with + <userinput>pkill -t pts/1</userinput>; otherwise issue a + <userinput>kill -HUP</userinput> in there by, for example, + <userinput>ps -t pts/1 -o pid= | xargs kill -HUP</userinput>. Replace + <replaceable>p1</replaceable> by whatever the tty is, of course.</para> + + <para>The + package builds dispatched by <command>make</command> to + the client machines will clean themselves up after a + few minutes (check with <command>ps x</command> until they + all go away).</para> + + <para>If you do not kill &man.make.1;, then it will spawn more jobs. + If you do not kill <command>dopackages</command>, then it will restart + the entire build. If you do not kill the <command>pdispatch</command> + processes, they'll keep going (or respawn) until they've built their + package.</para> + </sect2> - <sect2 id="cleanup"> - <title>Cleaning up a Build</title> + <sect2 id="cleanup"> + <title>Cleaning up a Build</title> - <para>To free up resources, you will need to clean up client machines by - running <command>build cleanup</command> command. For example:</para> + <para>To free up resources, you will need to clean up client machines by + running <command>build cleanup</command> command. For example:</para> <screen>&prompt.user; <userinput>/var/portbuild/scripts/build cleanup i386 8-exp 20080714120411 -full</userinput></screen> - <para>If you forget to do this, then the old build - <literal>jail</literal>s will not be cleaned up for 24 hours, and no - new jobs will be dispatched in their place since - <hostid>pointyhat</hostid> thinks the job slot is still occupied.</para> - - <para>To check, <userinput>cat ~/loads/*</userinput> to display the - status of client machines; the first column is the number of jobs - it thinks is running, and this should be roughly concordant - with the load average. <literal>loads</literal> is refreshed - every 2 minutes. If you do <userinput>ps x | grep pdispatch</userinput> - and it is less than the number of jobs that <literal>loads</literal> - thinks are in use, you are in trouble.</para> - - <para>You may have problem with the <command>umount</command> - commands hanging. If so, you are going to have to use the - <command>allgohans</command> script to run an &man.ssh.1; - command across all clients for that buildenv. For example:</para> + <para>If you forget to do this, then the old build + <literal>jail</literal>s will not be cleaned up for 24 hours, and no + new jobs will be dispatched in their place since + <hostid>pointyhat</hostid> thinks the job slot is still occupied.</para> + + <para>To check, <userinput>cat ~/loads/*</userinput> to display the + status of client machines; the first column is the number of jobs + it thinks is running, and this should be roughly concordant + with the load average. <literal>loads</literal> is refreshed + every 2 minutes. If you do <userinput>ps x | grep pdispatch</userinput> + and it is less than the number of jobs that <literal>loads</literal> + thinks are in use, you are in trouble.</para> + + <para>You may have problem with the <command>umount</command> + commands hanging. If so, you are going to have to use the + <command>allgohans</command> script to run an &man.ssh.1; + command across all clients for that buildenv. For example:</para> <screen>&prompt.user; ssh gohan24 df</screen> - <para>will get you a df, and</para> + <para>will get you a df, and</para> <screen>&prompt.user; allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports" &prompt.user; allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/src"</screen> - <para>are supposed to get rid of the hanging mounts. You will have to - keep doing them since there can be multiple mounts.</para> + <para>are supposed to get rid of the hanging mounts. You will have to + keep doing them since there can be multiple mounts.</para> - <note> - <para>Ignore the following: + <note> + <para>Ignore the following:</para> - <screen>umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory + <screen>umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: unknown file system umount: Cleanup of /x/tmp/8-exp/chroot/53837/compat/linux/proc failed! /x/tmp/8-exp/chroot/53837/compat/linux/proc: not a file system root directory</screen> - The former two mean that the client did not have those mounted; - the latter two are a bug.</para> - - <para>You may also see messages about <literal>procfs</literal>.</para> - </note> - - <para>After you have done all the above, remove the - <filename><replaceable>${arch}</replaceable>/lock</filename> - file before trying to restart the build. If you do not, - <filename>dopackages</filename> will simply exit. - </para> + <para>The former two mean that the client did not have those mounted; + the latter two are a bug.</para> - <para>If you have to do a ports tree update before - restarting, you may have to rebuild either <filename>duds</filename>, - <filename>INDEX</filename>, or both.</para> + <para>You may also see messages about <literal>procfs</literal>.</para> + </note> + <para>After you have done all the above, remove the + <filename><replaceable>${arch}</replaceable>/lock</filename> + file before trying to restart the build. If you do not, + <filename>dopackages</filename> will simply exit.</para> + + <para>If you have to do a ports tree update before + restarting, you may have to rebuild either <filename>duds</filename>, + <filename>INDEX</filename>, or both.</para> </sect2> <sect2 id="build-command-2"> <title>Maintaining builds with the <command>build</command> - command</title> + command</title> <para>Here are the rest of the options for the <command>build</command> command:</para> @@ -875,21 +849,16 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 <listitem> <para><literal>build destroy <replaceable>arch</replaceable> <replaceable>branch</replaceable></literal> - Destroy the - build id. - </para> + build id.</para> </listitem> <listitem> <para><literal>build list <replaceable>arch</replaceable> <replaceable>branch</replaceable></literal> - Shows the current set - of build ids. - </para> + of build ids.</para> </listitem> - </itemizedlist> - </sect2> - </sect1> <sect1 id="monitoring"> @@ -930,8 +899,7 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 directory. The next time the cluster tries to build this port, it will tar, compress, and copy the <makevar>WRKDIR</makevar> to - <filename><replaceable>${arch}</replaceable>/<replaceable>${branch}</replaceable>/wrkdirs/</filename>. - </para> + <filename><replaceable>${arch}</replaceable>/<replaceable>${branch}</replaceable>/wrkdirs/</filename>.</para> <para>If you find that the system is looping trying to build the same package over and over again, you may be able to fix the @@ -944,8 +912,7 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 <para>Keep an eye on &man.df.1; output. If the <filename>/var/portbuild</filename> file system becomes full - then <trademark>Bad Things</trademark> happen. - </para> + then <trademark>Bad Things</trademark> happen.</para> <para>The status of all current builds is generated periodically into the <filename>packagestats.html</filename> file, e.g., @@ -1072,8 +1039,7 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 and inform them of the release package location.</para> <para>Remember to coordinate with the &a.re; about the timing - and status of the release builds. - </para> + and status of the release builds.</para> </sect1> <sect1 id="uploading"> @@ -1132,8 +1098,9 @@ umount: Cleanup of /x/tmp/8-exp/chroot/5 </listitem> </itemizedlist> - <note><para>Some of the directories on - <hostid>ftp-master</hostid> are, in fact, symlinks. Examples:</para> + <note> + <para>Some of the directories on + <hostid>ftp-master</hostid> are, in fact, symlinks. Examples:</para> <itemizedlist> *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201212240941.qBO9fk9Z079001>