Skip site navigation (1)Skip section navigation (2) 
Date:      Sat, 23 Jan 2010 12:33:14 +0200
From:      David Naylor <naylor.b.david@gmail.com>
To:        freebsd-current@freebsd.org
Subject:   "tinderbox" using stacked unionfs
Message-ID:  <201001231233.18832.naylor.b.david@gmail.com>
next in thread | raw e-mail | index | archive | help 
--nextPart1831293.5iA22QSUzj
Content-Type: multipart/mixed;
  boundary="Boundary-01=_qBtWLCFxlRT0g7U"
Content-Transfer-Encoding: 7bit


--Boundary-01=_qBtWLCFxlRT0g7U
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hi,

I have been experimenting with using unionfs to build ports in a "tinderbox=
"=20
environment.  This avoids having to remove and extract files for the build =
of=20
each port and it allows the discovery of installed/modified files by the po=
rt. =20

Please see attached for a (updated) shell script that handles all the "heav=
y=20
lifting" of building ports.  Of importance is LOCALBASE and BUILDDIR.  If y=
ou=20
want to override LOCALBASE please use `env` as the script needs to know abo=
ut=20
it.  BUILDDIR (/usr/build by default) is where the script stores everything=
=20
(including PKG_DBDIR). =20

# env LOCALBASE=3D/tmp/local BUILDDIR=3D/tmp/build ./ports-union-builder.sh

Will install x11/xorg without affecting already installed systems. =20

CURRENT STATUS:
 - *** Currently kernel stack size is too small and the above will trigger =
a=20
stack overflow.  Recompiling a kernel with ``options KSTACK_PAGES=3D32'' wi=
ll=20
alleviate that problem. =20
 - Currently there is a build problem that affects eggdbus/polkit (possibly=
=20
others) thus preventing x11/xorg from being built.  I will investigate the=
=20
cause (help welcome). =20
 - I highly recommend running this in a chroot
 - NO WARRANTY, SLIPPERY WHEN WET, EATS CHILDREN. =20

Since the script doesn't complete a full build I am unable to compare the=20
build speeds (thus the overhead of unionfs) but here are some partial resul=
ts=20
(with FORCE_MAKE_JOBS and quad core):

# time make -C /usr/ports/x11/xorg install clean
2185.180u 1066.927s 38:12.27 141.8%     4123+1745k 5185+1890io 51638pf+0w

# time ./ports-union-builder.sh
Port /usr/ports/devel/eggdbus failed to build
Port /usr/ports/x11/xorg failed due to dependency /usr/ports/devel/eggdbus
1065.356u 862.073s 25:03.88 128.1%      4602+1840k 4582+57338io 18327pf+0w

Regards,

David

--Boundary-01=_qBtWLCFxlRT0g7U
Content-Type: text/plain; charset="ISO-8859-1"; name="ports-union-builder.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="ports-union-builder.txt"

#!/bin/sh

BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}

MAKE="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES make"

set -e

mkdir -p $BUILDDIR $LOCALBASE $PKG_DBDIR $PACKAGES

[ -n "$(kldstat -v | grep unionfs)" ] || kldload unionfs

port2name() {

  echo $1 | sed 's|[/.-]|_|g'

}

port2pkg() {

  local pkg_name=
  local port=

  port=$1; shift
  eval pkg_name=PKG$(port2name $port)
  eval pkg=\$$pkg_name
  if [ -z "$pkg" ]
  then
    pkg=$($MAKE -C $port -V PKGNAME)
    eval $pkg_name=$pkg
  fi

}

depends() {

  local depend=
  local depends_name=
  local _deps=
  local name=
  local port=
  local type

  type=$1
  port=$2

  eval depends_name=DEPEND_${type}_$(port2name $port)
  eval deps=\"\$$depends_name\"

  if [ -z "$deps" ]
  then
    echo "Getting dependancies for $port" > /dev/stderr

    if [ "$type" = "build" ]
    then
      depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS -V RUN_DEPENDS)"
    else
      depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)"
    fi
    for depend in $depend_list
    do
      name=$(echo $depend | cut -f 2 -d ':')
      depends run $name
      _deps="$_deps $deps $name"
    done

    deps=$(for depend in $_deps
    do
      echo $depend
    done | sort -u)

    depends_name=$depends_name
    eval $depends_name=\"$deps \"
  fi

}

build() {

  local _deps=
  local dep=
  local port=

  port=$1

  depends build $port
  _deps="$deps"
  for dep in $_deps
  do
    port2pkg $dep
    if [ ! -d $BUILDDIR/$pkg ]
    then
      if ! build $dep
      then
        echo "Port $port failed due to dependancy $dep" > /dev/stderr
        return 255
      fi
    fi
  done

  echo "Building port $port..."

  for pkg in $_deps
  do
    port2pkg $pkg
    mount -t unionfs -r -o noatime $BUILDDIR/$pkg $LOCALBASE
  done
  port2pkg $port
  mkdir -p $BUILDDIR/$pkg
  mount -t unionfs -o noatime $BUILDDIR/$pkg $LOCALBASE

  set +e
  trap "true" INT TERM EXIT
  $MAKE -C $port clean build install package -DNO_DEPENDS -DBATCH
  status=$?
  trap - INT TERM EXIT
  set -e

  [ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE
  for pkg in $(echo $_deps | sort -r)
  do
    [ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE
  done

  if [ $status -ne 0 ]
  then
    echo "Port $port failed to build" > /dev/stderr
    port2pkg $port
    rm -rf $BUILDDIR/$pkg || (chflags -R 0 $BUILDDIR/$pkg; rm -rf $BUILDDIR/$pkg)
  else
    $MAKE -C $port clean
  fi

  return $status

}

build /usr/ports/x11/xorg

--Boundary-01=_qBtWLCFxlRT0g7U--

--nextPart1831293.5iA22QSUzj
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.13 (FreeBSD)

iEYEABECAAYFAkta0G4ACgkQUaaFgP9pFrItTQCdGKDQlAJ09H9EoUzJXuvQd6Nt
wAUAnA2xKq05+IzRRQ+uYLD9LxJfKB9S
=NlYM
-----END PGP SIGNATURE-----

--nextPart1831293.5iA22QSUzj--

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201001231233.18832.naylor.b.david>