Date: Thu, 23 Aug 2012 11:23:03 -0700 From: David Wolfskill <david@catwhisker.org> To: Dennis Glatting <dg@pki2.com> Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 9.1-RC1 Available... Message-ID: <20120823182303.GK1537@albert.catwhisker.org> In-Reply-To: <alpine.BSF.2.00.1208230903210.92471@btw.pki2.com> References: <1345697446.84337.11.camel@neo.cse.buffalo.edu> <20120823225855.U33776@sola.nimnet.asn.au> <1345729674.52121.4.camel@bauer.cse.buffalo.edu> <5036497F.7020501@icarz.com> <alpine.BSF.2.00.1208230903210.92471@btw.pki2.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--I4VOKWutKNZEOIPu
Content-Type: multipart/mixed; boundary="bX/mw5riLlTkt+Gv"
Content-Disposition: inline
--bX/mw5riLlTkt+Gv
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Thu, Aug 23, 2012 at 09:13:33AM -0700, Dennis Glatting wrote:
> ...=20
> > The second primer in the committer handbook seems to indicate that it is
> > difficult to run an SVN mirror. This appears to me to be the biggest
> > drawback. I have been using CVS and perforce for years, but subversion
> > is new to me.
> >
>=20
> Thanks.
>=20
> I have several cases where I cannot have machines going to the Internet t=
o=20
> update for various reasons, including WAN loading and policies, but I can=
=20
> have them hit local mirrors. I have three sites:
> ...
I asked Dennis if a description of how I handle my private read-only SVN
mirrors might help, and his reaction was positive, so....
First: here's what I'm trying (and succeeding, AFAICT) to accomplish:
* I have a "build" machine at home, where I currently track stable/8,
stable/9, and head (on different slices) on a daily basis. I also
update the installed ports on that machine daily.
Periodically (by default, weekly), I also install the "Sunday"
snapshot of stable/8 on my 2 "production" machines (and update their
installed ports). [I expect to switch these machines to stable/9
soon. The build machine has already been building stable/9 kerenls
for them. Shortly after that switch, I expect to stop tracking
stable/8.]
Separate from the above, I also track stable/8, stable/9, and head on
a daily basis on my laptop; I also update its installed ports on a
daily basis. One of the reasons I do this is to gain actual
experience with the code I'm expecting to install on the "production"
machines: the build machine is used only for that, and when its job is
done for the day, I power it off.
Since the laptop spends some of the time it may be building FreeBSD
or ports disconnected from a network (e.g., while I'm commuting
to work), I prefer to just keep a local mirror of the SVN repo
on my laptop.
* So, I started with one of the recent "seed" repositories in each case,
and have my build machine (freebeast.catwhisker.org) use svnsync to
re-sync its local mirror with the repo on svn.freebsd.org.
* As somewhat of a relic of my experiences doing this with CVS mirrors,
I perform the re-sync in 2 stages overnight. After the 2nd one, I
update the local /usr/ports hierarchy (which used to be a CVS working
direwctory, and is now an SVN working copy).
The timing is intended so that during the daytime, my local mirror
is "stable": if I were to check out (say) head from that mirror
at 5AM, noon, 6PM, and 10PM, I would get the same result each
time (even if I didn't specify a GRN).
* As another relic of doing something similar with CVSup, I use the
login "cvsupin" to actually do this work.
* Somewhere, I acquired the perception that if, after setting up the
SVN mirror on freebeast, if I were to try to also use svnsync on my
laptop, that effort would end up talking back to svn.freebsd.org --
which is NOT what I want, for 2 reasons:
* I want to keep my mirrors in sync with one another, so I have
reasonable assurance that (kernel configs and other local
provisioning aside) I'm running the same code on my laptop that I
would be running on the others.
* I am also trying to avoid using more sv,freebsd.org resources tyhan
necessary.
Therefore, for re-syncing the SVN mirrors on my latop, I use rsync(1).
As a bit of an implementation detail, I consign my repositories to
separate file systems, and I have symlinks with names that depict the
function pointing to appropriate places in the file system -- e.g.:
d134(9.1-P)[13] df /repo
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/ada0s4f 40622796 22955728 14417246 61% /repo
d134(9.1-P)[14] ls -F /repo
cvs/ svn/
d134(9.1-P)[15] ls -F /repo/*
/repo/cvs:
local/
/repo/svn:
freebsd/ local/
d134(9.1-P)[16] ls -lT /{cvs,svn}
lrwxr-xr-x 1 root wheel 9 Sep 27 08:49:09 2010 /cvs -> /repo/cvs
lrwxr-xr-x 1 root wheel 9 Sep 27 08:49:09 2010 /svn -> /repo/svn
d134(9.1-P)[17] ls -F /svn/freebsd
doc/ ports/ src/
d134(9.1-P)[18]=20
I have attached a copy of the script I use to do this. It isn't really
set up for more general consumption -- it hard-codes the name of my
build machine, for example.
Here are some sample entries from /etc/crontab. First, for the build
machine:
# Local additions
30 01 * * * cvsupin /usr/local/etc/svn-repo -f
30 03 * * * cvsupin /usr/local/etc/svn-repo -f -w=
/usr/ports
And from my laptop:
# Local additions
40 01 * * * cvsupin /usr/local/etc/svn-repo -l
40 03 * * * cvsupin /usr/local/etc/svn-repo -l =
-w /usr/ports
# Copied from freebeast, for when I'm on the road....
# 30 01 * * * cvsupin /usr/local/etc/svn-repo -f
# 30 03 * * * cvsupin /usr/local/etc/svn-repo -f =
-w /usr/ports
One other thing: I rather like to have the logs from the svnsync
handy, so the script copies those, to. There's a bit of sleight-of-hand
involved, as the user cvsupin can't create files in /var/log, and
I still want the log files there.
PS: Yes, I also rotate the svnsync log files. Sample entry from
/etc/newsyslog.conf:
/var/log/svnsync.log cvsupin: 644 12 * $M1D0 JC
Please feel free to make use of the ideas -- or even the code.
Peace,
david
--=20
David H. Wolfskill david@catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.
See http://www.catwhisker.org/~david/publickey.gpg for my public key.
--bX/mw5riLlTkt+Gv
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="svn-repo.2jnuJ1"
#! /bin/sh
set -e
t_mode=""
mode=""
LOGFILE="svnsync.log"
RSYNC="/usr/local/bin/rsync"
RSYNC_SVN_ARGS="-aHAX"
RSYNC_SVN_SOURCE="rsync://freebeast/svn/freebsd"
RSYNC_SVN_TARGET="/svn/"
RSYNC_LOG_ARGS="-ptgoAX"
RSYNC_LOG_SOURCE="rsync://freebeast/svn_log/svnsync.log"
TMP_DIR="/tmp"
LOG_DIR="/var/log"
REPO_BASE="file:///svn/freebsd/"
REPO_LIST="src:src/base ports:ports doc:doc"
SVNSYNC="/usr/local/bin/svnsync"
SVN="/usr/local/bin/svn"
WORKING_COPIES=""
while getopts "flTw:" arg; do
case ${arg} in
f) mode="freebsd"; continue;;
l) mode="local"; continue;;
T) t_mode="echo "; continue;;
w) WORKING_COPIES="$OPTARG"; continue;;
?) echo "Usage: $0 -{f|l} [-T] [-w working_copy...]" >&2; exit 2;;
esac
done
if [ -n "$t_mode" ]; then
LOG_DIR="/dev"
LOGFILE="tty"
fi
if [ -z "$mode" ]; then
echo "$0: Need to specify -l (for rsync from local machine) or -f (for svnsync from svn.freebsd.org)" >&2
exit 1;
fi
if [ "$mode" = "local" ]; then
cmd="$t_mode $RSYNC $RSYNC_SVN_ARGS $RSYNC_SVN_SOURCE $RSYNC_SVN_TARGET"
logit='$t_mode $RSYNC $RSYNC_LOG_ARGS $RSYNC_LOG_SOURCE $TMP_DIR && \
$t_mode /bin/cp -p $TMP_DIR/$LOGFILE $LOG_DIR && \
$t_mode /bin/rm $TMP_DIR/$LOGFILE'
elif [ "$mode" = "freebsd" ]; then
cmd="$t_mode $SVNSYNC sync --non-interactive \$REPO >>$LOG_DIR/$LOGFILE 2>&1"
logit='$t_mode echo "svnsync for $NAME ended at `date` exit status $?" >>$LOG_DIR/$LOGFILE'
else
echo "$0: Logic error; mode => $mode; not recognized" >&2
exit 3
fi
log_init="$t_mode echo 'svnsync started at `date`' >>$LOG_DIR/$LOGFILE"
status=-1
eval "$log_init"
until [ $status -eq 0 ]; do
if [ "$mode" = "local" ]; then
eval "$cmd"
status="$?"
eval "$logit"
else
for ITEM in $REPO_LIST; do
NAME=$( echo $ITEM | sed -e 's/:.*$//' )
SUFFIX=$( echo $ITEM | sed -e 's/^.*://' )
REPO="${REPO_BASE}${SUFFIX}"
eval "$cmd"
status="$?"
eval "$logit"
done
fi
done
for dir in $WORKING_COPIES; do
cmd="$SVN update $dir >/var/tmp/"$(basename $dir)"_update 2>&1"
$t_mode eval "$cmd"
done
exit $?
--bX/mw5riLlTkt+Gv--
--I4VOKWutKNZEOIPu
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
iEYEARECAAYFAlA2dQcACgkQmprOCmdXAD3mlQCfXBvk/rQvTwntngI0whZKARkN
BQ8An1uHTxXtYQAWfJpTrgF/YYtYo/Vj
=5+yu
-----END PGP SIGNATURE-----
--I4VOKWutKNZEOIPu--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120823182303.GK1537>