From owner-freebsd-hackers@FreeBSD.ORG Sun Feb 28 17:12:19 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8AB8E1065670 for ; Sun, 28 Feb 2010 17:12:19 +0000 (UTC) (envelope-from naylor.b.david@gmail.com) Received: from mail-ew0-f226.google.com (mail-ew0-f226.google.com [209.85.219.226]) by mx1.freebsd.org (Postfix) with ESMTP id 0EEAF8FC18 for ; Sun, 28 Feb 2010 17:12:18 +0000 (UTC) Received: by ewy26 with SMTP id 26so891846ewy.3 for ; Sun, 28 Feb 2010 09:12:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:organization:to:subject :date:user-agent:cc:references:in-reply-to:mime-version:content-type :content-transfer-encoding:message-id; bh=NmcPJ0ipZyu3EHE4Cm2DVsSSvi6dhBqk66DBysFNeKE=; b=N5bxf3UDGSQDfHY4dc+s7tKxgOZo1DTL8O26qUOm7RfLlCNQZOPKfbczLOF7Yi20Il JERt3JnP198+PNW8pscQqhmWnmXL2c9poM2KgdgX2HjgC1IuyMlEZ+Ni+x9P91V9MbDR AQS/666uflLFmX8mFVtzAy7kEa1S1WWDEMq94= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:organization:to:subject:date:user-agent:cc:references :in-reply-to:mime-version:content-type:content-transfer-encoding :message-id; b=MEHvnYbr9uPXi16n3VpWuUBCvRnsKzolKG8UYIC1GhjXhySyKcluAoGkXr1NGNtqZn BxsLlBGjr3Tk9lFHW333+R9LQYpxErdKC0kryhlr2u1OP9cfpFsBnOcrdztf1bTwOZ6E YooJhnTpVyZ6Ox4YnT1uKBp0s84dUtM6VF3+o= Received: by 10.213.66.1 with SMTP id l1mr1994817ebi.91.1267377130082; Sun, 28 Feb 2010 09:12:10 -0800 (PST) Received: from dragon.dg ([41.216.197.15]) by mx.google.com with ESMTPS id 7sm7794224eyg.24.2010.02.28.09.12.06 (version=SSLv3 cipher=RC4-MD5); Sun, 28 Feb 2010 09:12:08 -0800 (PST) From: David Naylor Organization: Private To: Ulrich =?iso-8859-1?q?Sp=F6rlein?= Date: Sun, 28 Feb 2010 19:12:15 +0200 User-Agent: KMail/1.13.0 (FreeBSD/8.0-STABLE; KDE/4.3.95; amd64; ; ) References: <20100225135303.GM57731@acme.spoerlein.net> In-Reply-To: <20100225135303.GM57731@acme.spoerlein.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart5443290.T9Utag5tMc"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <201002281912.19406.naylor.b.david@gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: [Proof of Concept] Stacked unionfs based 'tinderbox' X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 17:12:19 -0000 --nextPart5443290.T9Utag5tMc Content-Type: multipart/mixed; boundary="Boundary-01=_wPqiLJBMTL+8B5b" Content-Transfer-Encoding: 7bit --Boundary-01=_wPqiLJBMTL+8B5b Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi, Apologies for the delayed reply. =20 On Thursday 25 February 2010 15:53:03 Ulrich Sp=F6rlein wrote: > On Thu, 25.02.2010 at 10:08:15 +0200, David Naylor wrote: > > Hi, > >=20 > > As some may have noticed on -current I have been working on using > > stacked unionfs to implement a 'tinderbox' type build system. I have > > successfully used the scripts to build x11/xorg (and have compared the > > results to using the traditional approach using pkg_add). The build > > system is stable except for one nasty corner case: deadlocks. >=20 > When I did this a couple of years ago, the major problems were failing > chdir(2) calls during ports build, etc. I'm hoping time and -current will solve this problem as it is beyond my=20 ability to fix. =20 > > To setup a compatible test environment requires: > > - recompile the kernel with `options KSTACK_PAGES=3D32`, otherwise the > > kernel will panic with a double fault. WITNESS options results in > > substantial performance degradation. > > - patch mtree (see below) [optional] > > - create the appropriate chroot environment (and reboot) [see below > > for corner case] > >=20 > > A performance bottleneck in mtree was identified. This resulted in > > mtree (as run by port install) consuming ~20% of the build time. See > > bin/143732 for a patch and further details. >=20 > Good work! >=20 > > The normal tinderbox approach takes ~80% more time to install compared = to > > the quick and dirty approach. The stacked unionfs approach takes ~170% > > more time (an increase of ~50% over the tinderbox approach). Some > > performance gains can be had if one uses memory backed storage (vs HDD > > in this case). >=20 > Please explain: what is the quick and dirty approach and which one is > faster now? The quick and dirty is `make -C /usr/ports/x11/xorg install clean`. The=20 stacked unionfs is still the slowest (even with a 20% improvement from=20 patching mtree). =20 If one is interested in performance in building ports in a sandbox then the= =20 tinderbox route is the way to go. =20 > As your scripts did not make it through, perhaps you can upload them to > the wiki? What I did back then was using a clean base system as the > underlying unionfs store to avoid re-generating the clean base over and > over again. Nowadays, a ZFS clone would probably be the way to go. I've attached the scripts with a .txt suffix. This should hopefully make i= t=20 past mailman. To which wiki page do you refer? You are welcome to add the= =20 scripts there. =20 I skipped the base system and only unionfs /usr/local. I never cleaned the= =20 base system. I figured that was the easiest to implement when the script r= uns=20 in a chroot. An area for further improvements? =20 > I'm not sure if a recursive approach is feasible here, as you can have > only one underlying unionfs mount. But special casing, e.g., perl may > still give a massive speedup. So for each port that has perl as > dependancy, you would not pull in the clean base + pkg_add perl, but > instead grab the clean-base+perl directory as an underlying unionfs. Have you done any speed comparisons of running programs (such as perl) from= a=20 unionfs. If the disk cache is big enough I don't think there will be a big= =20 performance hit? It is certainly a viable approach. =20 Regards, David --Boundary-01=_wPqiLJBMTL+8B5b Content-Type: text/plain; charset="ISO-8859-1"; name="ports-tinder-builder.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="ports-tinder-builder.txt" #!/bin/sh BUILDDIR=${BUILDDIR:-/usr/build} LOCALBASE=${LOCALBASE:-/usr/local} PORTSDIR=${PORTSDIR:-/usr/ports} PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports} PKG_DBDIR=${PKG_DBDIR:-$LOCALBASE/db_pkg} PACKAGES=${PACKAGES:-$BUILDDIR/packages} ENV="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES" MAKE="$ENV make" PKG_ADD="$ENV pkg_add" PKG_DELETE="$ENV pkg_delete" set -e mkdir -p $BUILDDIR $PACKAGES port2name() { echo $1 | sed 's|[/.-]|_|g' } port2pkg() { local pkg_name= local port= port=$1; shift eval pkg_name=PKG$(port2name $port) eval pkg=\$$pkg_name if [ -z "$pkg" ] then pkg=$($MAKE -C $port -V PKGNAME) eval $pkg_name=$pkg fi } depends() { local depend= local depends_name= local _deps= local name= local port= local type type=$1 port=$2 eval depends_name=DEPEND_${type}_$(port2name $port) eval deps=\"\$$depends_name\" if [ -z "$deps" ] then echo "Getting $type dependancies for $port" > /dev/stderr if [ "$type" = "build" ] then depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS -V RUN_DEPENDS)" else depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)" fi for depend in $depend_list do name=$(echo $depend | cut -f 2 -d ':') depends runtime $name _deps="$_deps $deps $name" done deps=" " for depend in $_deps do if [ -z "`echo "$deps" | grep " $depend "`" ] then deps="$deps$depend " fi done [ "`echo $deps | tr ' ' '\n' | sort`" = "`echo $deps | tr ' ' '\n' | sort -u`" ] depends_name=$depends_name eval $depends_name=\"$deps \" fi } run() { set +e trap "echo Terminating..." INT TERM EXIT "$@" _status=$? [ $status -ne 0 ] || status=$_status trap - INT TERM EXIT set -e } build() { local _deps= local dep= local port= port=$1 depends build $port _deps="$deps" for dep in $_deps do port2pkg $dep if [ ! -f $PACKAGES/All/$pkg.tbz ] then if ! build $dep then echo "Port $port failed due to dependency $dep" > /dev/stderr return 255 fi fi done echo "Building port $port..." mkdir -p $LOCALBASE $PKG_DBDIR for pkg in $_deps do port2pkg $pkg run $PKG_ADD $PACKAGES/All/$pkg.tbz [ $status -eq 0 ] || break done [ $status -ne 0 ] || run $MAKE -C $port clean build -DNO_DEPENDS -DBATCH if [ $status -eq 0 ] then run $MAKE -C $port install package -DNO_DEPENDS -DBATCH fi if [ $status -eq 0 -o -z "$NO_CLEANUP" ] then port2pkg $port run $PKG_DELETE -f $pkg for pkg in $(echo $_deps | sort -r) do port2pkg $pkg run $PKG_DELETE -f $pkg done run rm -rf $LOCALBASE || (run chflags -R 0 $LOCALBASE; run rm -rf $LOCALBASE) fi if [ $status -ne 0 ] then echo "Port $port failed to build" > /dev/stderr else run $MAKE -C $port clean fi return $status } build /usr/ports/x11/xorg --Boundary-01=_wPqiLJBMTL+8B5b Content-Type: text/plain; charset="ISO-8859-1"; name="ports-union-builder.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="ports-union-builder.txt" #!/bin/sh BUILDDIR=${BUILDDIR:-/usr/build} LOCALBASE=${LOCALBASE:-/usr/local} PORTSDIR=${PORTSDIR:-/usr/ports} PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports} PKG_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_pkg} PACKAGES=${PACKAGES:-$BUILDDIR/packages} MAKE="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES make" set -e mkdir -p $BUILDDIR $LOCALBASE $PKG_DBDIR $PACKAGES [ -n "$(kldstat -v | grep unionfs)" ] || kldload unionfs [ ! -e $BUILDDIR/.installing_port ] || rm -r `cat $BUILDDIR/.installing_port` $BUILDDIR/.installing_port port2name() { echo $1 | sed 's|[/.-]|_|g' } port2pkg() { local pkg_name= local port= port=$1; shift eval pkg_name=PKG$(port2name $port) eval pkg=\$$pkg_name if [ -z "$pkg" ] then pkg=$($MAKE -C $port -V PKGNAME) eval $pkg_name=$pkg fi } depends() { local depend= local depends_name= local _deps= local name= local port= local type type=$1 port=$2 eval depends_name=DEPEND_${type}_$(port2name $port) eval deps=\"\$$depends_name\" if [ -z "$deps" ] then echo "Getting $type dependancies for $port" > /dev/stderr if [ "$type" = "build" ] then depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS -V RUN_DEPENDS)" else depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)" fi for depend in $depend_list do name=$(echo $depend | cut -f 2 -d ':') depends runtime $name _deps="$_deps $deps $name" done deps=" " for depend in $_deps do if [ -z "`echo "$deps" | grep " $depend "`" ] then deps="$deps$depend " fi done [ "`echo $deps | tr ' ' '\n' | sort`" = "`echo $deps | tr ' ' '\n' | sort -u`" ] depends_name=$depends_name eval $depends_name=\"$deps \" fi } run_make() { set +e trap "true" INT TERM EXIT $MAKE "$@" status=$? trap - INT TERM EXIT set -e } build() { local _deps= local dep= local port= port=$1 depends build $port _deps="$deps" for dep in $_deps do port2pkg $dep if [ ! -d $BUILDDIR/$pkg ] then if ! build $dep then echo "Port $port failed due to dependency $dep" > /dev/stderr return 255 fi fi done echo "Building port $port..." for pkg in $_deps do port2pkg $pkg mount -t unionfs -r -o noatime $BUILDDIR/$pkg $LOCALBASE done run_make -C $port clean build -DNO_DEPENDS -DBATCH if [ $status -eq 0 ] then port2pkg $port mkdir -p $BUILDDIR/$pkg mount -t unionfs -o noatime $BUILDDIR/$pkg $LOCALBASE echo $BUILDDIR/$pkg > $BUILDDIR/.installing_port run_make -C $port install package -DNO_DEPENDS -DBATCH rm $BUILDDIR/.installing_port [ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE fi if [ $status -eq 0 -o -z "$NO_CLEANUP" ] then for pkg in $(echo $_deps | sort -r) do umount $LOCALBASE done fi if [ $status -ne 0 ] then echo "Port $port failed to build" > /dev/stderr port2pkg $port rm -rf $BUILDDIR/$pkg || (chflags -R 0 $BUILDDIR/$pkg; rm -rf $BUILDDIR/$pkg) else $MAKE -C $port clean fi return $status } build /usr/ports/x11/xorg --Boundary-01=_wPqiLJBMTL+8B5b-- --nextPart5443290.T9Utag5tMc Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEABECAAYFAkuKo/MACgkQUaaFgP9pFrLNzgCfQHeChGtFdD5ABrtEEwg4+faa iMoAni07Iv4P0P1rKjBwXEUQA85amUO0 =MigZ -----END PGP SIGNATURE----- --nextPart5443290.T9Utag5tMc--