Date: Sun, 29 Jun 2014 19:22:50 +0000 (UTC) From: Nathan Whitehorn <nwhitehorn@FreeBSD.org> To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r268020 - user/nwhitehorn/condorports Message-ID: <201406291922.s5TJMoYB090798@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: nwhitehorn Date: Sun Jun 29 19:22:49 2014 New Revision: 268020 URL: http://svnweb.freebsd.org/changeset/base/268020 Log: The ports tree does a number of things to configure itself to the installed system, some of which (like testing the capabilities of /usr/bin/cc) don't work in the context of building a dependency graph on the job submit node. Instead, ship graph generation to the worker nodes and run it in the same chroot as everything else. This reduces the ports tree operations on the submit node to just distfile fetching. Added: user/nwhitehorn/condorports/build_dag_dep_list.py (contents, props changed) user/nwhitehorn/condorports/builddag.sub user/nwhitehorn/condorports/builddagwrapper.sh (contents, props changed) user/nwhitehorn/condorports/startbuild.sh (contents, props changed) Modified: user/nwhitehorn/condorports/README user/nwhitehorn/condorports/buildportsdag.sh user/nwhitehorn/condorports/stagebuildfiles.sh Modified: user/nwhitehorn/condorports/README ============================================================================== --- user/nwhitehorn/condorports/README Sun Jun 29 18:54:41 2014 (r268019) +++ user/nwhitehorn/condorports/README Sun Jun 29 19:22:49 2014 (r268020) @@ -24,6 +24,8 @@ Submit node prerequisites: - sysutils/condor, configured to allow job submission - Copies of the relevant ports trees, at the same paths as they are on the build nodes (in /buildshare/ports/XXX) +- python +- A lot of disk space (> 100 GB) Build node prerequisites: ------------------------- @@ -41,45 +43,17 @@ Using: The basic procedure is: cd condorports -./buildportsdag.sh portsdir -condor_submit_dag -maxpre 10 ports.dag +./startbuild.sh 355755 /path/to/output amd64 10.0-RELEASE </scratch/horde> <Eat one to several meals depending on cluster size> -This will configure a Condor DAG for the ports tree in -/buildshare/ports/portsdir, the current system's architecture (see options -below), and release (again), placing the packages in the current directory -(once more). Submitting the DAG to the scheduler will begin the build. - -NOTE: The -maxpre option is *very* important, as this controls the number of -simultaneous make fetch operations to run on the submit node. If you don't -set it, you will get thousands of fetches at once. - -As the build runs, log files will appear in logs and built packages in a -directory by default in condorports. The build queue can be inspected using -condor_q. - -Options to buildportsdag.sh are specified using environment variables: -- PORTSDIR: Name of ports directory to build in. This *MUST* occur on the same - path on both the build and submit nodes. This directory may be read only. - Default: NONE -- PKGSDIR: Name of directory on submit node in which to place built packages. - Need not exist on the build nodes and can be on a local disk. - Default: $PORTSDIR/packages -- DISTHORDE: Name of directory on submit node in which to store and find - distfiles. Can be shared across ports tree. Need not exist on the build nodes - and can be on a local disk. This directory must be writable by the user - running the job submission. - Default: $PORTSDIR/distfiles -- ARCH: Value of uname -p for which to build - Default: Value of uname -p on submit node -- RELEASE: Name of FreeBSD release to use. This directory must exist on the - build nodes (it need not be accessible from the submit node) in - /releases/$RELEASE - Default: $ARCH/`uname -r` -- STAGEDIR: Name of directory in which to stage files for transfer for this - particular build. Must be unique to this particular package build run and - will be deleted at completion. - Default: $(pwd)/stage-$ARCH/$PKGSDIR +This will configure and submit a Condor DAG for the ports tree in +/buildshare/ports/355755, the amd64 architecture, and the 10.0-RELEASE world, +using the common distfiles directory in /scratch/horde (on the submit node). +The last argument is optional. If unspecified, it will instead use a +build-private directory in /path/to/output. + +As the build runs, log files and built packages will appear under +/path/to/output. The build queue can be inspected using condor_q -wide. TODO: ---- Added: user/nwhitehorn/condorports/build_dag_dep_list.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/nwhitehorn/condorports/build_dag_dep_list.py Sun Jun 29 19:22:49 2014 (r268020) @@ -0,0 +1,50 @@ +#!/usr/bin/env python + +import sys, os + +dag = file(sys.argv[1], 'r') +dag = dag.readlines() + +jobs = {} +pkgs = {} +for line in dag: + line = line.split(' ') + if line[0] == 'JOB': + jobs[line[1].strip()] = [] + if line[0] == 'PARENT': + deps = [] + for dep in line[1:]: + if dep == 'CHILD': + break + deps.append(dep) + jobs[line[-1].strip()] += deps + if line[0] == 'VARS': + kvpair = line[2].split('=') + if kvpair[0] == 'pkg': + pkgs[line[1]] = kvpair[1].strip().strip('"') + if kvpair[0] == 'pkgdir': + pkgs[line[1]] = kvpair[1].strip().strip('"') + '/' + pkgs[line[1]] + +def deepdeps(job, deps=None): + if deps is None: + deps = [] + for dep in jobs[job]: + pkg = pkgs[dep] + if pkg in deps: + continue + deps.append(pkg) + deepdeps(dep, deps) + return deps + +outdag = [] +for line in dag: + sline = line.split(' ') + if sline[0] == 'SCRIPT' and sline[1] == 'PRE': + line = line.strip() + line += ' ' + line += ' '.join(deepdeps(sline[2])) + line += '\n' + outdag.append(line) + +dagout = file(sys.argv[1], 'w') +dagout.writelines(outdag) Added: user/nwhitehorn/condorports/builddag.sub ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/nwhitehorn/condorports/builddag.sub Sun Jun 29 19:22:49 2014 (r268020) @@ -0,0 +1,19 @@ +executable = builddagwrapper.sh +universe = vanilla +notification = never # failure? + +# Require that the job run on a machine that supports our build architecture +# The "potemkin" is to suppress Condor's implicit Arch requirement +requirements = Arch=="potemkin" || stringListMember("$(BuildArch)", TARGET.SupportedABIs) + +log=/tmp/condorlog +output=logs/dagbuild.out +error=logs/dagbuild.err + +should_transfer_files=YES +when_to_transfer_output=ON_EXIT +transfer_output_files=ports.dag +transfer_input_files=buildportsdag.sh + +arguments= $(release_tarball) $(ports) ports.dag $(distfiles) $(outpkgs) $(stage) $(release) +queue Added: user/nwhitehorn/condorports/builddagwrapper.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/nwhitehorn/condorports/builddagwrapper.sh Sun Jun 29 19:22:49 2014 (r268020) @@ -0,0 +1,17 @@ +#!/bin/sh + +export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin + +release_tarball=$1 +shift + +portsdir=$1 + +set -e +sudo /pkgscripts/prepbuildjail $_CONDOR_SLOT $release_tarball $portsdir +cat buildportsdag.sh | sudo jexec $_CONDOR_SLOT sh -c 'cat > /buildportsdag.sh' +sudo jexec $_CONDOR_SLOT sh /buildportsdag.sh $@ +sudo jexec $_CONDOR_SLOT cat /ports.dag > ports.dag +sudo /pkgscripts/reapbuildjail $_CONDOR_SLOT +set +e + Modified: user/nwhitehorn/condorports/buildportsdag.sh ============================================================================== --- user/nwhitehorn/condorports/buildportsdag.sh Sun Jun 29 18:54:41 2014 (r268019) +++ user/nwhitehorn/condorports/buildportsdag.sh Sun Jun 29 19:22:49 2014 (r268020) @@ -1,28 +1,22 @@ #!/bin/sh -# Example: -# DISTHORDE=/scratch/horde ARCH=powerpc RELEASE=10.0-RELEASE PKGSDIR=/scratch/output/powerpc ./buildportsdag.sh 355755 ports-ppc.dag - export PORTSDIR=/buildshare/ports/$1 -: ${PKGSDIR=$PORTSDIR/packages} -: ${DISTHORDE=$PORTSDIR/distfiles} -: ${ARCH=`uname -p`} -: ${RELEASE=`uname -r`} -: ${STAGEDIR=$(pwd)/stage-$ARCH/$PKGSDIR} -: ${__MAKE_CONF=/dev/null}; export __MAKE_CONF +ARCH=`uname -p` -export PORT_DBDIR=/var/empty # Avoid host contamination -export PACKAGE_BUILDING=1 # Some things have different dependencies OUTDAG=$2 PORTSET=$1 +DISTHORDE=$3 +PKGSDIR=$4 +STAGEDIR=$5 +RELEASE=$6 +OSVERSION=$(make -C $PORTSDIR -V OSVERSION) release_tarball=$ARCH-`echo $RELEASE | tr . -` +export PACKAGE_BUILDING=1 # Some things have different dependencies mkdir -p $STAGEDIR -export OSVERSION=$(tar xOf /buildshare/releases/$release_tarball.tar usr/include/sys/param.h | awk '/^\#define[[:blank:]]__FreeBSD_version/ {print $3}') - dagjobs=$(pwd)/ports.dagjobs dagdeps=$(pwd)/ports.dagdeps dag=$(pwd)/$OUTDAG @@ -34,9 +28,9 @@ for category in $(make -C $PORTSDIR -V S echo JOB $job package.sub >> $dagjobs # One fetch for all archs with dummy fetch job? - export UNAME_p=$ARCH PKGNAME=$(make package-name) TARBALL=$STAGEDIR/stage-$category-$port.tar + # Note: these are not the complete arguments to stagebuildfiles.sh. These will be filled in by build_full_deps.py echo SCRIPT PRE $job stagebuildfiles.sh $job $PKGSDIR $ARCH $OSVERSION $TARBALL $DISTHORDE $PORTSDIR >> $dagjobs echo SCRIPT POST $job postbuild.sh $TARBALL \$RETURN >> $dagjobs echo VARS $job BuildArch=\"$ARCH\" >> $dagjobs @@ -53,5 +47,6 @@ for category in $(make -C $PORTSDIR -V S done; done cat $dagjobs $dagdeps > $dag -echo JOBSTATE_LOG $dag.joblog >> $dag +echo JOBSTATE_LOG $(basename $dag).joblog >> $dag +echo NODE_STATUS_FILE $(basename $dag).status >> $dag rm -f $dagjobs $dagdeps Modified: user/nwhitehorn/condorports/stagebuildfiles.sh ============================================================================== --- user/nwhitehorn/condorports/stagebuildfiles.sh Sun Jun 29 18:54:41 2014 (r268019) +++ user/nwhitehorn/condorports/stagebuildfiles.sh Sun Jun 29 19:22:49 2014 (r268020) @@ -4,8 +4,10 @@ # dependent packages? mkdir -p $(dirname $5) +mkdir -p $2 +logfile=$5.log exec 2>&1 -exec 1>$5.log +exec 1>$logfile set -e export UNAME_p=$3 @@ -19,9 +21,10 @@ export PACKAGE_BUILDING=1 export LOCALBASE=/var/empty # Fetch distfiles if unfetched -cd $1 -make -d e -DBATCH checksum DISTDIR=$DISTHORDE CLEAN_FETCH_ENV=1 -make -d e -DBATCH package-links PACKAGES=$2 CLEAN_FETCH_ENV=1 +echo 'Fetching distfiles' +make -C $1 -d e -DBATCH checksum DISTDIR=$DISTHORDE CLEAN_FETCH_ENV=1 +echo 'Making links' +make -C $1 -d e -DBATCH package-links PACKAGES=$(pwd)/$2 CLEAN_FETCH_ENV=1 mkdir -p $2/All # Make a tarball containing all the bits we need to build: the distfiles, @@ -32,24 +35,25 @@ TARBALL=$5 mkdir -p $(dirname $TARBALL) # Find distfiles -DIST_SUBDIR=$DISTHORDE/$(make -V DIST_SUBDIR) -ALLFILES=$(for f in `make -V ALLFILES`; do echo $DIST_SUBDIR/$f; done) +dist_subdir=$(make -C $1 -V DIST_SUBDIR) +DIST_SUBDIR=$DISTHORDE/$dist_subdir +ALLFILES=$(for f in `make -C $1 -V ALLFILES`; do echo $DIST_SUBDIR/$f; done) # Find packages -DEPENDS=`make all-depends-list` -PKGLIST=$(for x in $DEPENDS; do make -C $x PACKAGES=$2 -V PKGFILE; done) +shift 7 +PKGLIST="$@" rm -rf $TARBALL-dir -mkdir -p $TARBALL-dir/packages $TARBALL-dir/distfiles/$(make -V DIST_SUBDIR) +mkdir -p $TARBALL-dir/packages $TARBALL-dir/distfiles/$dist_subdir if [ -n "$ALLFILES" ]; then - ln -s $ALLFILES $TARBALL-dir/distfiles/$(make -V DIST_SUBDIR) + ln -s $ALLFILES $TARBALL-dir/distfiles/$dist_subdir fi if [ -n "$PKGLIST" ]; then - ln -s $PKGLIST $TARBALL-dir/packages + ln $PKGLIST $TARBALL-dir/packages fi echo 'Building tarball' tar -c -C $TARBALL-dir -v -L -f $TARBALL distfiles packages rm -rf $TARBALL-dir set +e -rm $5.log +rm $logfile Added: user/nwhitehorn/condorports/startbuild.sh ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/nwhitehorn/condorports/startbuild.sh Sun Jun 29 19:22:49 2014 (r268020) @@ -0,0 +1,40 @@ +# Example: +# startbuild.sh 355755 /path/to/build/area amd64 buildrelease [/path/to/distfiles] + +ports=$1 +buildpath=$2 +buildarch=$3 +buildrelease=$4 +distfiles=$5 + +release_tarball=$buildarch-`echo $buildrelease | tr . -` + +if [ -z "$distfiles" ]; then + distfiles=$buildpath/distfiles +fi + +mkdir -p $buildpath +mkdir -p $buildpath/logs +mkdir -p $distfiles + +cp builddag.sub package.sub $buildpath +cp buildport.sh builddagwrapper.sh buildportsdag.sh stagebuildfiles.sh postbuild.sh $buildpath +cp build_dag_dep_list.py $buildpath + +# First, build the real DAG +echo JOB builddag builddag.sub > $buildpath/build.dag +echo VARS builddag BuildArch=\"$buildarch\" >> $buildpath/build.dag +echo VARS builddag ports=\"$ports\" >> $buildpath/build.dag +echo VARS builddag distfiles=\"$distfiles\" >> $buildpath/build.dag +echo VARS builddag outpkgs=\"packages\" >> $buildpath/build.dag +echo VARS builddag stage=\"stage\" >> $buildpath/build.dag +echo VARS builddag release_tarball=\"$release_tarball\" >> $buildpath/build.dag +echo VARS builddag release=\"$buildrelease\" >> $buildpath/build.dag +echo SCRIPT POST builddag build_dag_dep_list.py ports.dag >> $buildpath/build.dag + +# Then, run it +echo SUBDAG EXTERNAL ports ports.dag >> $buildpath/build.dag +echo PARENT builddag CHILD ports >> $buildpath/build.dag + +cd $buildpath +condor_submit_dag -maxpre 15 -allowversionmismatch build.dag
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201406291922.s5TJMoYB090798>