From owner-svn-doc-all@FreeBSD.ORG Fri Nov 7 22:06:53 2014 Return-Path: Delivered-To: svn-doc-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3975FA2F; Fri, 7 Nov 2014 22:06:53 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 242C32F2; Fri, 7 Nov 2014 22:06:53 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.9/8.14.9) with ESMTP id sA7M6r1A082223; Fri, 7 Nov 2014 22:06:53 GMT (envelope-from mat@FreeBSD.org) Received: (from mat@localhost) by svn.freebsd.org (8.14.9/8.14.9/Submit) id sA7M6rbS082222; Fri, 7 Nov 2014 22:06:53 GMT (envelope-from mat@FreeBSD.org) Message-Id: <201411072206.sA7M6rbS082222@svn.freebsd.org> X-Authentication-Warning: svn.freebsd.org: mat set sender to mat@FreeBSD.org using -f From: Mathieu Arnold Date: Fri, 7 Nov 2014 22:06:53 +0000 (UTC) To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r45951 - head/en_US.ISO8859-1/books/porters-handbook/testing X-SVN-Group: doc-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-doc-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire doc trees \(except for " user" , " projects" , and " translations" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2014 22:06:53 -0000 Author: mat (ports committer) Date: Fri Nov 7 22:06:52 2014 New Revision: 45951 URL: https://svnweb.freebsd.org/changeset/doc/45951 Log: Enhance the Poudriere chapter a great bit. Differential Revision: https://reviews.freebsd.org/D770 Submitted by: riggs Reviewed by: wblock Sponsored by: Absolight Modified: head/en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml Modified: head/en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml ============================================================================== --- head/en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml Fri Nov 7 22:02:59 2014 (r45950) +++ head/en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml Fri Nov 7 22:06:52 2014 (r45951) @@ -187,6 +187,728 @@ + + <application>Poudriere</application> + + For a ports contributor, + Poudriere is one of the most + important and helpful testing and build tools. Its main + features include: + + + + Bulk building of the entire ports tree, specific subsets + of the ports tree, or a single port including its + dependencies + + + + Automatic packaging of build results + + + + Generation of build log files per port + + + + Providing a signed &man.pkg.8; repository + + + + Testing of port builds before submitting a patch to the + &os; bug tracker or committing to the ports tree + + + + Testing for successful ports builds using different + options + + + + Because Poudriere performs its + building in a clean &man.jail.8; environment and uses + &man.zfs.8; features, it has several advantages over traditional + testing on the host system: + + + + No pollution of the host environment: No leftover files, + no accidental removals, no changes of existing configuration + files. + + + + Verify pkg-plist for missing or + superfluous entries + + + + Ports committers sometimes ask for a + Poudriere log alongside a patch + submission to assess whether the patch is ready for + integration into the ports tree + + + + It is also quite straightforward to set up and use, has no + dependencies, and will run on any supported &os; release. This + section shows how to install, configure, and run + Poudriere as part of the normal + workflow of a ports contributor. + + The examples in this section show a default file layout, as + standard in &os;. Substitute any local changes accordingly. + The ports tree, represented by ${PORTSDIR}, + is located in /usr/ports. Both + ${LOCALBASE} and ${PREFIX} + are /usr/local by default. + + + Installing <application>Poudriere</application> + + Poudriere is available in the + ports tree in ports-mgmt/poudriere. It can be + installed using &man.pkg.8; or from ports: + + &prompt.root; pkg install poudriere + + or + + &prompt.root; make -C /usr/ports/ports-mgmt/poudriere install clean + + There is also a work-in-progress version of + Poudriere which will eventually + become the next release. It is available in ports-mgmt/poudriere-devel. This + development version is used for the official &os; package + builds, so it is well tested. It often has newer interesting + features. A ports committer will want to use the development + version because it is what is used in production, and has all + the new features that will make sure everything is exactly + right. A contributor will not necessarily need those as the + most important fixes are backported to released version. The + main reason for the use of the development version to build + the official package is because it is faster, in a way that + will shorten a full build from 18 hours to 17 hours when using + a high end 32 CPU server with 128GB of + RAM. Those optimizations will not matter a + lot when building ports on a desktop machine. + + + + + Setting Up <application>Poudriere</application> + + The port installs a default configuration file, + /usr/local/etc/poudriere.conf. Each + parameter is documented in the configuration file and in + &man.poudriere.8;. Here is a minimal example config + file: + + ZPOOL=tank +ZROOTFS=/poudriere +BASEFS=/poudriere +DISTFILES_CACHE=/usr/ports/distfiles +RESOLV_CONF=/etc/resolv.conf +FREEBSD_HOST=ftp://ftp.freebsd.org +SVN_HOST=svn0.eu.FreeBSD.org + + + + ZPOOL + + + The name of the ZFS storage pool + which Poudriere shall use. + Must be listed in the output of zpool + status. + + + + + ZROOTFS + + + The root of + Poudriere-managed file + systems. This entry will cause + Poudriere to create + &man.zfs.8; file systems under + tank/poudriere. + + + + + BASEFS + + + The root mount point for + Poudriere file systems. This + entry will cause Poudriere to + mount tank/poudriere to + /poudriere. + + + + + DISTFILES_CACHE + + + Defines where distfiles are stored. In this + example, Poudriere and the + host share the distfiles storage directory. This avoids + downloading tarballs which are already present on the + system. + + + + + RESOLV_CONF + + + Use the host /etc/resolv.conf + inside jails for DNS. This is needed + so jails can resolve the URLs of + distfiles when downloading. It is not needed when using + a proxy. Refer to the default configuration file for + proxy configuration. + + + + + FREEBSD_HOST + + + The FTP/HTTP + server to use when the jails are installed from &os; + releases and updated with &man.freebsd-update.8;. + Choose a server location which is close, for example if + the machine is located in Australia, use + ftp.au.freebsd.org. + + + + + SVN_HOST + + + The server from where jails are installed and + updated when using + Subversion. Also used for + ports tree when not using &man.portsnap.8;. Again, + choose a nearby location. A list of official + Subversion mirrors can be + found in the &os; + Handbook Subversion + section. + + + + + + + + Creating <application>Poudriere</application> + Jails + + Create the base jails which + Poudriere will use for + building: + + &prompt.root; poudriere jail -c -j 93Ramd64 -v 9.3-RELEASE -a amd64 + + Fetch a 9.3-RELEASE for + amd64 from the FTP + server given by FREEBSD_HOST in + poudriere.conf, create the zfs file + system tank/poudriere/jails/93Ramd64, mount + it on /poudriere/jails/93Ramd64 and + extract the 9.3-RELEASE tarballs into this + file system. + + &prompt.root; poudriere jail -c -j 10i386 -v stable/10 -a i386 -m svn+https + + Create tank/poudriere/jails/10i386, + mount it on /poudriere/jails/10i386, then + check out the tip of the Subversion + branch of &os;-10-STABLE from + SVN_HOST in + poudriere.conf into + /poudriere/jails/10i386/usr/src, then + complete a buildworld and install + it into /poudriere/jails/10i386. + + + If a specific Subversion + revision is needed, append it to the version string. For + example: + + &prompt.root; poudriere jail -c -j 10i386 -v stable/10@123456 -a i386 -m svn+https + + + + While it is possible to build a newer version of &os; on + an older version, most of the time it will not run. For + example, if a stable/10 jail is needed, + the host will have to run stable/10 too. + Running 10.0-RELEASE is not + enough. + + + + The default svn protocol works but is + not very secure. Using svn+https along + with verifying the remote server's SSL + fingerprint is advised. It will ensure that the files used + for building the jail are from a trusted source. + + + A list of jails currently known to + Poudriere can be shown with + poudriere jail -l: + + &prompt.root; poudriere jail -l +JAILNAME VERSION ARCH METHOD +93Ramd64 9.3-RELEASE amd64 ftp +10i386 10.0-STABLE i386 svn+https + + + + + Keeping <application>Poudriere</application> Jails + Updated + + Managing updates is very straightforward. The + command: + + &prompt.root; poudriere jail -u -j JAILNAME + + updates the specified jail to the latest version + available. For &os; releases, update to the latest patchlevel + with &man.freebsd-update.8;. For &os; versions built from + source, update to the latest + Subversion revision in the + branch. + + + For jails employing a + svn+* method, + it is helpful to add -J + NumberOfParallelBuildJobs + to speed up the build by increasing the number of parallel + compile jobs used. For example, if the building machine has + 6 CPUs, use: + + &prompt.root; poudriere jail -u -J 6 -j JAILNAME + + + + + Setting Up Ports Trees for Use with + <application>Poudriere</application> + + There are multiple ways to use ports trees in + Poudriere. The most + straightforward way is to have + Poudriere create a default ports + tree for itself: + + &prompt.root; poudriere ports -c + + This command creates + tank/poudriere/ports/default, mount it on + /poudriere/ports/default, and populate it + using &man.portsnap.8;. Afterward it is included in the list + of known ports trees: + + &prompt.root; poudriere ports -l +PORTSTREE METHOD PATH +default portsnap /poudriere/ports/default + + + Note that the default ports tree is + special. Each of the build commands explained later will + implicitly use this ports tree unless specifically specified + otherwise. To use another tree, add -p + treename to the + commands. + + + While useful for regular bulk builds, having this default + ports tree with the &man.portsnap.8; method may not be the + best way to deal with local modifications for a ports + contributor. As with the creation of jails, it is possible to + use a different method for creating the ports tree. To add an + additional ports tree for testing local modifications and + ports development, checking out the tree via + Subversion is possible: + + &prompt.root; poudriere ports -c -m svn+https -p subversive + + Creates tank/poudriere/ports/subversive + and mounts it on + /poudriere/ports/subversive. It is then + populated using Subversion. + Finally, it is added to the list of known ports trees: + + &prompt.root; poudriere ports -l +PORTSTREE METHOD PATH +default portsnap /poudriere/ports/default +subversive svn+https /poudriere/ports/subversive + + + The svn method allows extra + qualifiers to tell Subversion + exactly how to fetch data. This is explained in + &man.poudriere.8;. For instance, poudriere ports + -c -m svn+ssh -p subversive uses + SSH for the checkout. + + + + + + Using Manually Managed Ports Trees with Poudriere + + Depending on the workflow, it can be extremely helpful to + use ports trees which are maintained manually. For instance, + if there is a local copy of the ports tree in + /work/ports, point + Poudriere to the location: + + &prompt.root; poudriere ports -c -F -f none -M /work/ports -p development + + This will be listed in the table of known trees: + + &prompt.root; poudriere ports -l +PORTSTREE METHOD PATH +development - /work/ports + + + The dash in the METHOD column means + that Poudriere will not update or + change this ports tree, ever. It is completely up to the + user to maintain this tree, including all local + modifications that may be used for testing new ports and + submitting patches. + + + + + + Keeping Poudriere Ports Trees Updated + + As straightforward as with jails described earlier: + + &prompt.root; poudriere ports -u -p PORTSTREE + + Will update the given + PORTSTREE, one tree given by the + output of poudriere -l, to the latest + revision available on the official servers. + + + Ports trees without a method, see , cannot be + updated like this. They must be updated manually by the + porter. + + + + + + Testing Ports + + After jails and ports trees have been set up, the result + of a contributor's modifications to the ports tree can be + tested. + + For example, local modifications to the www/firefox port located in + /work/ports/www/firefox can be tested in + the previously created 9.3-RELEASE jail: + + &prompt.root; poudriere testport -j 93Ramd64 -p development -o www/firefox + + This will build all dependencies of + Firefox. If a dependency has been + built previously and is still up-to-date, the pre-built + package is installed. If a dependency has no up-to-date + package, one will be built with default options in a jail. + Then Firefox itself is + built. + + The complete build of every port is logged to + /poudriere/data/logs/bulk/93Ri386-development/build-time/logs. + + The directory name 93Ri386-development + is derived from the arguments to -j and + -p, respectively. For convenience, a + symbolic link + /poudriere/data/logs/bulk/93Ri386-development/latest + is also maintained. The link points to the latest + build-time directory. Also in this + directory is an index.html for observing + the build process with a web browser. + + By default, Poudriere cleans up + the jails and leaves log files in the directories mentioned + above. To ease investigation, jails can be kept running after + the build by adding to + testport: + + &prompt.root; poudriere testport -j 93Ramd64 -p development -i -o www/firefox + + After the build completes, and regardless of whether it + was successful, a shell is provided within the jail. The + shell is used to investigate further. + Poudriere can be told to leave the + jail running after the build finishes with + . Poudriere + will show the command to run when the jail is no longer + needed. It is then possible to &man.jexec.8; into it: + + &prompt.root; poudriere testport -j 93Ramd64 -p development -I -o www/firefox +[...] +====>> Installing local Pkg repository to /usr/local/etc/pkg/repos +====>> Leaving jail 93Ramd64-development-n running, mounted at /poudriere/data/.m/93Ramd64-development/ref for interactive run testing +====>> To enter jail: jexec 93Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root +====>> To stop jail: poudriere jail -k -j 93Ramd64 -p development +&prompt.root; jexec 93Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root +&prompt.root; [do some stuff in the jail] +&prompt.root; exit +&prompt.root; poudriere jail -k -j 93Ramd64 -p development +====>> Umounting file systems + + An integral part of the &os; ports build infrastructure is + the ability to tweak ports to personal preferences with + options. These can be tested with + Poudriere as well. Adding the + : + + &prompt.root; poudriere testport -c -o www/firefox + + Presents the port configuration dialog before the port is + built. The ports given after in the + format + category/portname + will use the specified options, all dependencies will use the + default options. Testing dependent ports with non-default + options can be accomplished using sets, see . + + + When testing ports where pkg-plist + is altered during build depending on the selected options, + it is recommended to perform a test run with all options + selected and one with all options + deselected. + + + + + Using Sets + + For all actions involving builds, a so-called + set can be specified using -z + setname. A set refers + to a fully independent build. This allows, for instance, + usage of testport with non-standard options + for the dependent ports. + + To use sets, Poudriere expects + an existing directory structure similar to + PORT_DBDIR, defaults to + /var/db/ports in its configuration + directory. This directory is then nullfs-mounted into the + jails where the ports and their dependencies are built. + Usually a suitable starting point can be obtained by + recursively copying the existing PORT_DBDIR + to + /usr/local/etc/poudriere.d/jailname-portname-setname-options. + This is described in detail in &man.poudriere.8;. For + instance, testing www/firefox + in a specific set named devset, add the + -z devset parameter to the testport + command: + + &prompt.root; poudriere testport -j 93Ramd64 -p development -z devset -o www/firefox + + This will look for the existence of these directories in + this order: + + + + /usr/local/etc/poudriere.d/93Ramd64-development-devset-options + + + + /usr/local/etc/poudriere.d/93Ramd64-devset-options + + + + /usr/local/etc/poudriere.d/93Ramd64-development-options + + + + /usr/local/etc/poudriere.d/devset-options + + + + /usr/local/etc/poudriere.d/development-options + + + + /usr/local/etc/poudriere.d/93Ramd64-options + + + + /usr/local/etc/poudriere.d/options + + + + From this list, Poudriere + nullfs-mounts the first existing + directory tree into the /var/db/ports + directory of the build jails. Hence, all custom options are + used for all the ports during this run of + testport. + + After the directory structure for a set is provided, the + options for a particular port can be altered. For + example: + + &prompt.root; poudriere options -c www/firefox -z devset + + The configuration dialog for www/firefox is shown, and options can + be edited. The selected options are saved to the + devset set. + + + Poudriere is very flexible in + the option configuration. They can be set for particular + jails, ports trees, and for multiple ports by one command. + Refer to &man.poudriere.8; for details. + + + + + + Providing a Custom <filename>make.conf</filename> + File + + Similar to using sets, + Poudriere will also use a custom + make.conf if it is provided. No special + command line argument is necessary. Instead, + Poudriere looks for existing files + matching a name scheme derived from the command line. For + instance: + + &prompt.root; poudriere testport -j 93Ramd64 -p development -z devset -o www/firefox + + causes Poudriere to check for + the existence of these files in this order: + + + + /usr/local/etc/poudriere.d/make.conf + + + + /usr/local/etc/poudriere.d/devset-make.conf + + + + /usr/local/etc/poudriere.d/development-make.conf + + + + /usr/local/etc/poudriere.d/93Ramd64-make.conf + + + + /usr/local/etc/poudriere.d/93Ramd64-development-make.conf + + + + /usr/local/etc/poudriere.d/93Ramd64-devset-make.conf + + + + /usr/local/etc/poudriere.d/93Ramd64-development-devset-make.conf + + + + Unlike with sets, all of the found files will be appended, + in that order, into one + make.conf inside the build jails. It is + hence possible to have general make variables, intended to + affect all builds in + /usr/local/etc/poudriere.d/make.conf. + Special variables, intended to affect only certain jails or + sets can be set in specialised make.conf + files, such as + /usr/local/etc/poudriere.d/93Ramd64-development-devset-make.conf. + + + Using <filename>make.conf</filename> to Change Default + <application>Perl</application> + + To build a set with a non default + Perl version, for example, + 5.20, using a set named + perl5-20, create a + perl5-20-make.conf with this + line: + + DEFAULT_VERSIONS+= perl=5.20 + + + Note the use of += so that if the + variable is already set in the default + make.conf its content will not be + overwritten. + + + + + + + Pruning no Longer Needed Distfiles + + Poudriere comes with a built-in + mechanism to remove outdated distfiles that are no longer used + by any port of a given tree. The command + + &prompt.root; poudriere distclean -p portstree + + will scan the distfiles folder, + DISTFILES_CACHE in + poudriere.conf, versus the ports tree + given by the -p + portstree argument and + prompt for removal of those distfiles. To skip the prompt and + remove all unused files unconditionally, the + -y argument can be added: + + &prompt.root; poudriere distclean -p portstree -y + + + + + + Tinderbox @@ -202,23 +924,4 @@ Tinderbox website for more details. - - - Poudriere - - As a ports contributor, consider installing - poudriere. It is a powerful - system for building and testing ports. - Poudriere can be installed with - ports-mgmt/poudriere. - - There is also a ports-mgmt/poudriere-devel that often - has newer features that are mostly helpful when testing - ports. - - Visit the Poudriere - website for more details. -