From owner-freebsd-current@FreeBSD.ORG Sat Jan 23 10:33:13 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B1DB1065696 for ; Sat, 23 Jan 2010 10:33:13 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: from mail-fx0-f218.google.com (mail-fx0-f218.google.com [209.85.220.218]) by mx1.freebsd.org (Postfix) with ESMTP id 0538D8FC1D for ; Sat, 23 Jan 2010 10:33:12 +0000 (UTC) Received: by fxm10 with SMTP id 10so247670fxm.14 for ; Sat, 23 Jan 2010 02:33:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:organization:to:subject :date:user-agent:mime-version:content-type:content-transfer-encoding :message-id; bh=H2530PhJPVjsq9ILSQ73SbOeb591vZ6wzXhDKPD+1a8=; b=BEe7HH31Dh+vG0ZRIE1/lS2BwONf6c3gndCDXiC/wN9GRzy8hqp9K+ZtFCDy9PHQe6 wR2xSWl48nZnyJGQZbBMnxubzzO/SgNBCpCH0KeQJVhfos/g5PZKxoeZekdaCwMVdh6E M0+64zTgF0A0faLZfa/e53l/ca7w809fut3h4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:organization:to:subject:date:user-agent:mime-version :content-type:content-transfer-encoding:message-id; b=elsGrSKjElfsBbAy/nr2q6G5YIBPkyOQeTXFrR8LLSQMMqNKOsTtlZDECLRDyKd0Jj ifv/pvCoKGjjLRpe2DOeRFWS+XMZa7jYhofpveykjMKMPm2CWg22N0UFNyCy/mBmAA68 g1LLRz6cCjKXYyjbUO3tx27hCOsGqnu73m4T0= Received: by 10.223.77.129 with SMTP id g1mr3881736fak.11.1264242791776; Sat, 23 Jan 2010 02:33:11 -0800 (PST) Received: from dragon.dg ([41.216.197.17]) by mx.google.com with ESMTPS id 15sm1744952fxm.6.2010.01.23.02.33.09 (version=SSLv3 cipher=RC4-MD5); Sat, 23 Jan 2010 02:33:11 -0800 (PST) From: David Naylor Organization: Private To: freebsd-current@freebsd.org Date: Sat, 23 Jan 2010 12:33:14 +0200 User-Agent: KMail/1.12.3 (FreeBSD/8.0-STABLE; KDE/4.3.3; amd64; ; ) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1831293.5iA22QSUzj"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <201001231233.18832.naylor.b.david@gmail.com> Subject: "tinderbox" using stacked unionfs X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2010 10:33:13 -0000 --nextPart1831293.5iA22QSUzj Content-Type: multipart/mixed; boundary="Boundary-01=_qBtWLCFxlRT0g7U" Content-Transfer-Encoding: 7bit --Boundary-01=_qBtWLCFxlRT0g7U Content-Type: Text/Plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi, I have been experimenting with using unionfs to build ports in a "tinderbox= "=20 environment. This avoids having to remove and extract files for the build = of=20 each port and it allows the discovery of installed/modified files by the po= rt. =20 Please see attached for a (updated) shell script that handles all the "heav= y=20 lifting" of building ports. Of importance is LOCALBASE and BUILDDIR. If y= ou=20 want to override LOCALBASE please use `env` as the script needs to know abo= ut=20 it. BUILDDIR (/usr/build by default) is where the script stores everything= =20 (including PKG_DBDIR). =20 # env LOCALBASE=3D/tmp/local BUILDDIR=3D/tmp/build ./ports-union-builder.sh Will install x11/xorg without affecting already installed systems. =20 CURRENT STATUS: - *** Currently kernel stack size is too small and the above will trigger = a=20 stack overflow. Recompiling a kernel with ``options KSTACK_PAGES=3D32'' wi= ll=20 alleviate that problem. =20 - Currently there is a build problem that affects eggdbus/polkit (possibly= =20 others) thus preventing x11/xorg from being built. I will investigate the= =20 cause (help welcome). =20 - I highly recommend running this in a chroot - NO WARRANTY, SLIPPERY WHEN WET, EATS CHILDREN. =20 Since the script doesn't complete a full build I am unable to compare the=20 build speeds (thus the overhead of unionfs) but here are some partial resul= ts=20 (with FORCE_MAKE_JOBS and quad core): # time make -C /usr/ports/x11/xorg install clean 2185.180u 1066.927s 38:12.27 141.8% 4123+1745k 5185+1890io 51638pf+0w # time ./ports-union-builder.sh Port /usr/ports/devel/eggdbus failed to build Port /usr/ports/x11/xorg failed due to dependency /usr/ports/devel/eggdbus 1065.356u 862.073s 25:03.88 128.1% 4602+1840k 4582+57338io 18327pf+0w Regards, David --Boundary-01=_qBtWLCFxlRT0g7U Content-Type: text/plain; charset="ISO-8859-1"; name="ports-union-builder.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="ports-union-builder.txt" #!/bin/sh BUILDDIR=${BUILDDIR:-/usr/build} LOCALBASE=${LOCALBASE:-/usr/local} PORTSDIR=${PORTSDIR:-/usr/ports} PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports} PKG_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_pkg} PACKAGES=${PACKAGES:-$BUILDDIR/packages} MAKE="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES make" set -e mkdir -p $BUILDDIR $LOCALBASE $PKG_DBDIR $PACKAGES [ -n "$(kldstat -v | grep unionfs)" ] || kldload unionfs port2name() { echo $1 | sed 's|[/.-]|_|g' } port2pkg() { local pkg_name= local port= port=$1; shift eval pkg_name=PKG$(port2name $port) eval pkg=\$$pkg_name if [ -z "$pkg" ] then pkg=$($MAKE -C $port -V PKGNAME) eval $pkg_name=$pkg fi } depends() { local depend= local depends_name= local _deps= local name= local port= local type type=$1 port=$2 eval depends_name=DEPEND_${type}_$(port2name $port) eval deps=\"\$$depends_name\" if [ -z "$deps" ] then echo "Getting dependancies for $port" > /dev/stderr if [ "$type" = "build" ] then depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS -V RUN_DEPENDS)" else depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)" fi for depend in $depend_list do name=$(echo $depend | cut -f 2 -d ':') depends run $name _deps="$_deps $deps $name" done deps=$(for depend in $_deps do echo $depend done | sort -u) depends_name=$depends_name eval $depends_name=\"$deps \" fi } build() { local _deps= local dep= local port= port=$1 depends build $port _deps="$deps" for dep in $_deps do port2pkg $dep if [ ! -d $BUILDDIR/$pkg ] then if ! build $dep then echo "Port $port failed due to dependancy $dep" > /dev/stderr return 255 fi fi done echo "Building port $port..." for pkg in $_deps do port2pkg $pkg mount -t unionfs -r -o noatime $BUILDDIR/$pkg $LOCALBASE done port2pkg $port mkdir -p $BUILDDIR/$pkg mount -t unionfs -o noatime $BUILDDIR/$pkg $LOCALBASE set +e trap "true" INT TERM EXIT $MAKE -C $port clean build install package -DNO_DEPENDS -DBATCH status=$? trap - INT TERM EXIT set -e [ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE for pkg in $(echo $_deps | sort -r) do [ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE done if [ $status -ne 0 ] then echo "Port $port failed to build" > /dev/stderr port2pkg $port rm -rf $BUILDDIR/$pkg || (chflags -R 0 $BUILDDIR/$pkg; rm -rf $BUILDDIR/$pkg) else $MAKE -C $port clean fi return $status } build /usr/ports/x11/xorg --Boundary-01=_qBtWLCFxlRT0g7U-- --nextPart1831293.5iA22QSUzj Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.13 (FreeBSD) iEYEABECAAYFAkta0G4ACgkQUaaFgP9pFrItTQCdGKDQlAJ09H9EoUzJXuvQd6Nt wAUAnA2xKq05+IzRRQ+uYLD9LxJfKB9S =NlYM -----END PGP SIGNATURE----- --nextPart1831293.5iA22QSUzj--