Date: Sun, 28 Feb 2010 19:12:15 +0200 From: David Naylor <naylor.b.david@gmail.com> To: Ulrich =?iso-8859-1?q?Sp=F6rlein?= <uqs@freebsd.org> Cc: freebsd-hackers@freebsd.org Subject: Re: [Proof of Concept] Stacked unionfs based 'tinderbox' Message-ID: <201002281912.19406.naylor.b.david@gmail.com> In-Reply-To: <20100225135303.GM57731@acme.spoerlein.net> References: <b53f6f941002250008mbe82d46m68ea304359d16203@mail.gmail.com> <20100225135303.GM57731@acme.spoerlein.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--nextPart5443290.T9Utag5tMc
Content-Type: multipart/mixed;
boundary="Boundary-01=_wPqiLJBMTL+8B5b"
Content-Transfer-Encoding: 7bit
--Boundary-01=_wPqiLJBMTL+8B5b
Content-Type: Text/Plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Hi,
Apologies for the delayed reply. =20
On Thursday 25 February 2010 15:53:03 Ulrich Sp=F6rlein wrote:
> On Thu, 25.02.2010 at 10:08:15 +0200, David Naylor wrote:
> > Hi,
> >=20
> > As some may have noticed on -current I have been working on using
> > stacked unionfs to implement a 'tinderbox' type build system. I have
> > successfully used the scripts to build x11/xorg (and have compared the
> > results to using the traditional approach using pkg_add). The build
> > system is stable except for one nasty corner case: deadlocks.
>=20
> When I did this a couple of years ago, the major problems were failing
> chdir(2) calls during ports build, etc.
I'm hoping time and -current will solve this problem as it is beyond my=20
ability to fix. =20
> > To setup a compatible test environment requires:
> > - recompile the kernel with `options KSTACK_PAGES=3D32`, otherwise the
> > kernel will panic with a double fault. WITNESS options results in
> > substantial performance degradation.
> > - patch mtree (see below) [optional]
> > - create the appropriate chroot environment (and reboot) [see below
> > for corner case]
> >=20
> > A performance bottleneck in mtree was identified. This resulted in
> > mtree (as run by port install) consuming ~20% of the build time. See
> > bin/143732 for a patch and further details.
>=20
> Good work!
>=20
> > The normal tinderbox approach takes ~80% more time to install compared =
to
> > the quick and dirty approach. The stacked unionfs approach takes ~170%
> > more time (an increase of ~50% over the tinderbox approach). Some
> > performance gains can be had if one uses memory backed storage (vs HDD
> > in this case).
>=20
> Please explain: what is the quick and dirty approach and which one is
> faster now?
The quick and dirty is `make -C /usr/ports/x11/xorg install clean`. The=20
stacked unionfs is still the slowest (even with a 20% improvement from=20
patching mtree). =20
If one is interested in performance in building ports in a sandbox then the=
=20
tinderbox route is the way to go. =20
> As your scripts did not make it through, perhaps you can upload them to
> the wiki? What I did back then was using a clean base system as the
> underlying unionfs store to avoid re-generating the clean base over and
> over again. Nowadays, a ZFS clone would probably be the way to go.
I've attached the scripts with a .txt suffix. This should hopefully make i=
t=20
past mailman. To which wiki page do you refer? You are welcome to add the=
=20
scripts there. =20
I skipped the base system and only unionfs /usr/local. I never cleaned the=
=20
base system. I figured that was the easiest to implement when the script r=
uns=20
in a chroot. An area for further improvements? =20
> I'm not sure if a recursive approach is feasible here, as you can have
> only one underlying unionfs mount. But special casing, e.g., perl may
> still give a massive speedup. So for each port that has perl as
> dependancy, you would not pull in the clean base + pkg_add perl, but
> instead grab the clean-base+perl directory as an underlying unionfs.
Have you done any speed comparisons of running programs (such as perl) from=
a=20
unionfs. If the disk cache is big enough I don't think there will be a big=
=20
performance hit? It is certainly a viable approach. =20
Regards,
David
--Boundary-01=_wPqiLJBMTL+8B5b
Content-Type: text/plain; charset="ISO-8859-1"; name="ports-tinder-builder.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="ports-tinder-builder.txt"
#!/bin/sh
BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$LOCALBASE/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}
ENV="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES"
MAKE="$ENV make"
PKG_ADD="$ENV pkg_add"
PKG_DELETE="$ENV pkg_delete"
set -e
mkdir -p $BUILDDIR $PACKAGES
port2name() {
echo $1 | sed 's|[/.-]|_|g'
}
port2pkg() {
local pkg_name=
local port=
port=$1; shift
eval pkg_name=PKG$(port2name $port)
eval pkg=\$$pkg_name
if [ -z "$pkg" ]
then
pkg=$($MAKE -C $port -V PKGNAME)
eval $pkg_name=$pkg
fi
}
depends() {
local depend=
local depends_name=
local _deps=
local name=
local port=
local type
type=$1
port=$2
eval depends_name=DEPEND_${type}_$(port2name $port)
eval deps=\"\$$depends_name\"
if [ -z "$deps" ]
then
echo "Getting $type dependancies for $port" > /dev/stderr
if [ "$type" = "build" ]
then
depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS -V RUN_DEPENDS)"
else
depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)"
fi
for depend in $depend_list
do
name=$(echo $depend | cut -f 2 -d ':')
depends runtime $name
_deps="$_deps $deps $name"
done
deps=" "
for depend in $_deps
do
if [ -z "`echo "$deps" | grep " $depend "`" ]
then
deps="$deps$depend "
fi
done
[ "`echo $deps | tr ' ' '\n' | sort`" = "`echo $deps | tr ' ' '\n' | sort -u`" ]
depends_name=$depends_name
eval $depends_name=\"$deps \"
fi
}
run() {
set +e
trap "echo Terminating..." INT TERM EXIT
"$@"
_status=$?
[ $status -ne 0 ] || status=$_status
trap - INT TERM EXIT
set -e
}
build() {
local _deps=
local dep=
local port=
port=$1
depends build $port
_deps="$deps"
for dep in $_deps
do
port2pkg $dep
if [ ! -f $PACKAGES/All/$pkg.tbz ]
then
if ! build $dep
then
echo "Port $port failed due to dependency $dep" > /dev/stderr
return 255
fi
fi
done
echo "Building port $port..."
mkdir -p $LOCALBASE $PKG_DBDIR
for pkg in $_deps
do
port2pkg $pkg
run $PKG_ADD $PACKAGES/All/$pkg.tbz
[ $status -eq 0 ] || break
done
[ $status -ne 0 ] || run $MAKE -C $port clean build -DNO_DEPENDS -DBATCH
if [ $status -eq 0 ]
then
run $MAKE -C $port install package -DNO_DEPENDS -DBATCH
fi
if [ $status -eq 0 -o -z "$NO_CLEANUP" ]
then
port2pkg $port
run $PKG_DELETE -f $pkg
for pkg in $(echo $_deps | sort -r)
do
port2pkg $pkg
run $PKG_DELETE -f $pkg
done
run rm -rf $LOCALBASE || (run chflags -R 0 $LOCALBASE; run rm -rf $LOCALBASE)
fi
if [ $status -ne 0 ]
then
echo "Port $port failed to build" > /dev/stderr
else
run $MAKE -C $port clean
fi
return $status
}
build /usr/ports/x11/xorg
--Boundary-01=_wPqiLJBMTL+8B5b
Content-Type: text/plain; charset="ISO-8859-1"; name="ports-union-builder.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="ports-union-builder.txt"
#!/bin/sh
BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}
MAKE="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES make"
set -e
mkdir -p $BUILDDIR $LOCALBASE $PKG_DBDIR $PACKAGES
[ -n "$(kldstat -v | grep unionfs)" ] || kldload unionfs
[ ! -e $BUILDDIR/.installing_port ] || rm -r `cat $BUILDDIR/.installing_port` $BUILDDIR/.installing_port
port2name() {
echo $1 | sed 's|[/.-]|_|g'
}
port2pkg() {
local pkg_name=
local port=
port=$1; shift
eval pkg_name=PKG$(port2name $port)
eval pkg=\$$pkg_name
if [ -z "$pkg" ]
then
pkg=$($MAKE -C $port -V PKGNAME)
eval $pkg_name=$pkg
fi
}
depends() {
local depend=
local depends_name=
local _deps=
local name=
local port=
local type
type=$1
port=$2
eval depends_name=DEPEND_${type}_$(port2name $port)
eval deps=\"\$$depends_name\"
if [ -z "$deps" ]
then
echo "Getting $type dependancies for $port" > /dev/stderr
if [ "$type" = "build" ]
then
depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS -V RUN_DEPENDS)"
else
depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)"
fi
for depend in $depend_list
do
name=$(echo $depend | cut -f 2 -d ':')
depends runtime $name
_deps="$_deps $deps $name"
done
deps=" "
for depend in $_deps
do
if [ -z "`echo "$deps" | grep " $depend "`" ]
then
deps="$deps$depend "
fi
done
[ "`echo $deps | tr ' ' '\n' | sort`" = "`echo $deps | tr ' ' '\n' | sort -u`" ]
depends_name=$depends_name
eval $depends_name=\"$deps \"
fi
}
run_make() {
set +e
trap "true" INT TERM EXIT
$MAKE "$@"
status=$?
trap - INT TERM EXIT
set -e
}
build() {
local _deps=
local dep=
local port=
port=$1
depends build $port
_deps="$deps"
for dep in $_deps
do
port2pkg $dep
if [ ! -d $BUILDDIR/$pkg ]
then
if ! build $dep
then
echo "Port $port failed due to dependency $dep" > /dev/stderr
return 255
fi
fi
done
echo "Building port $port..."
for pkg in $_deps
do
port2pkg $pkg
mount -t unionfs -r -o noatime $BUILDDIR/$pkg $LOCALBASE
done
run_make -C $port clean build -DNO_DEPENDS -DBATCH
if [ $status -eq 0 ]
then
port2pkg $port
mkdir -p $BUILDDIR/$pkg
mount -t unionfs -o noatime $BUILDDIR/$pkg $LOCALBASE
echo $BUILDDIR/$pkg > $BUILDDIR/.installing_port
run_make -C $port install package -DNO_DEPENDS -DBATCH
rm $BUILDDIR/.installing_port
[ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE
fi
if [ $status -eq 0 -o -z "$NO_CLEANUP" ]
then
for pkg in $(echo $_deps | sort -r)
do
umount $LOCALBASE
done
fi
if [ $status -ne 0 ]
then
echo "Port $port failed to build" > /dev/stderr
port2pkg $port
rm -rf $BUILDDIR/$pkg || (chflags -R 0 $BUILDDIR/$pkg; rm -rf $BUILDDIR/$pkg)
else
$MAKE -C $port clean
fi
return $status
}
build /usr/ports/x11/xorg
--Boundary-01=_wPqiLJBMTL+8B5b--
--nextPart5443290.T9Utag5tMc
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (FreeBSD)
iEYEABECAAYFAkuKo/MACgkQUaaFgP9pFrLNzgCfQHeChGtFdD5ABrtEEwg4+faa
iMoAni07Iv4P0P1rKjBwXEUQA85amUO0
=MigZ
-----END PGP SIGNATURE-----
--nextPart5443290.T9Utag5tMc--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201002281912.19406.naylor.b.david>