Date: Mon, 26 Mar 2012 12:37:14 -0700 From: Devin Teske <devin.teske@fisglobal.com> To: "'Doug Hardie'" <bc979@lafn.org>, "'Martin McCormick'" <martin@dc.cis.okstate.edu> Cc: freebsd-questions@freebsd.org Subject: RE: Remote System Builds Message-ID: <014501cd0b87$d9452410$8bcf6c30$@fisglobal.com> In-Reply-To: <8CB1037A-9843-48E2-9F10-27FDC7721EE3@lafn.org> References: <201203261820.q2QIKaFt008532@dc.cis.okstate.edu> <8CB1037A-9843-48E2-9F10-27FDC7721EE3@lafn.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> -----Original Message-----
> From: owner-freebsd-questions@freebsd.org [mailto:owner-freebsd-
> questions@freebsd.org] On Behalf Of Doug Hardie
> Sent: Monday, March 26, 2012 11:54 AM
> To: Martin McCormick
> Cc: freebsd-questions@freebsd.org
> Subject: Re: Remote System Builds
>
>
> On 26 March 2012, at 11:20, Martin McCormick wrote:
>
> > Is there yet any way to remotely rebuild a FreeBSD
> > system? I have two FreeBSD systems on two remote campuses that
> > presently run FreeBSD6.3. They need to be running FreeBSD9.0 and
> > I don't really care how I get there as long as it can be done
> > over the network. If we were physically there, I would put a
> > CDROM in and blow them away since it is such a large jump.
> >
> > I can have staff members there install CDROM's that were
> > remastered to use the serial console, but I am hoping that maybe
> > we are moving past this sort of logistics.
> >
> > I just tried to unpack the 9.0 image using tar which has
> > worked in the past to let one modify loader.conf but I got a
> > bunch of errors this time about files that couldn't be created
> > so maybe this is not the recommended headless installation
> > technique any longer.
>
> I am going to be facing the same issue in a few months. My experiences with
the
> serial console are that it is great for correcting small issues, trying to use
it for
> initial configuration is not going to be real easy. I would like to be able
to build a
> custom CD for that specific machine that doesn't need any operator input.
They
> just install it and boot the machine. It would need to format the disk and do
the
> complete installation (base and uniquely configured ports etc.). Is that
> possible?
Our company is facing a jump from FreeBSD-4.11 to FreeBSD-8.1 later this year
(in a few months).
The challenge is recognized as:
1. Upgrade 3,000+ systems in a matter of 60-90 days.
2. Many of those systems need to be upgraded remotely where no technical staff
exists (e.g. India, Philippines, Canada, etc.)
3. The remote upgrade must be done via SSH and NFS
4. Machines being upgraded must reboot and come back onto the network without
local technical assistance
The technique that I developed to accomplish this is a shell script weighing in
at [currently] 1954 lines of code. In addition to the 1954 lines of code, there
are another 631 lines of code dedicated to specific migrations that have to be
performed specific to (say) jumping from 4.11 to 8.1 (e.g., you should remove
the "nodev" option from your NFS mounts in /etc/fstab, among other things). On
top of THAT, there's another 8778 lines of code dedicated to "cruft removal"
(lists of files/directories that are unique to either the source-binary distro
or the destination-binary distro; depending on the direction of migration).
I will be releasing the full script soon (in a couple weeks) and even though
it's not specifically targeted to your migration path (6.3 to 9.0), it can be
learning-tool to illustrate exactly HOW you can go about making that binary
migration a reality.
Fortunately, it's very easy for me to provide explicit instructions on how
exactly we jump from 4.11 to 8.1, or 8.1 back to 4.11, or even from 8.1 i386 to
8.1 amd64 (and back) ... the script that I've written takes a "-nv" syntax which
means "don't actually do anything, but show me the explicit commands that I can
execute by-hand to migrate from one OS to the next as an in-place migration"
which has the following output:
NOTE: The output below is a sample migration from 4.11 to 8.1-amd64
WARNING: This is not intended to be a full solution provided to the OP but
rather a conversation-starter that should illustrate not only how HARD and/or
DIFFICULT it is to do what the OP asked, but also to show how it IS POSSIBLE to
achieve.
WARNING: Also, it's worth noting that this procedure will NOT work to jump to
"9.x" because the binary distribution sets from 9.0 and higher are in a
different format. You can work around this by adjusting for this fact --
replacing syntax such as "cat base/base.?? | tar ..." to simply "cat base.txz |
tar ..." (accounting for the new monolithic distribution-sets in 9.0 and
higher).
=== BEGIN OUTPUT OF OUR BINARY MIGRATION SCRIPT BELOW ===
Rebuild / with repository 8.1-RELEASE-amd64
TEST-ONLY! No actions will be performed.
#################### Source 8.1-RELEASE-amd64 specific config file
. "./etc/8.1-RELEASE-amd64.conf"
#################### Run 8.1-RELEASE-amd64 specific pre_install
pre_install
# Preflight sanity check (check release)
# Migration from 4.11-STABLE to 8.1-RELEASE-amd64 allowed
# Prevent disappearance of tar(1)
rm -Rfv /var/db/pkg/tar-1.5
# Move `/usr/local/etc/rc.d' to `/usr/local/etc/rc.d.xbak-2012-03-26.12:32:08'
mv -v /usr/local/etc/rc.d /usr/local/etc/rc.d.xbak-2012-03-26.12:32:08
mkdir -pv /usr/local/etc/rc.d
chmod -v 0755 /usr/local/etc/rc.d
chown -v root:wheel /usr/local/etc/rc.d
#################### Find/Remove core-dumps to free-up disk space
find -x "/" -maxdepth 2 -name '*.core' -type f | xargs rm -fv
#################### Back up /etc to /etc.xbak-2012-03-26.12:32:08
cp -RPpfv /etc /etc.xbak-2012-03-26.12:32:08
#################### Uninstall 289 packages...
pkg_delete -fav
#################### Prepare /oldbase directory
mkdir -pv /oldbase
cp -pfv ./crunch/rebuild_crunch4 /oldbase/rebuild_crunch4
ln -sfv rebuild_crunch4 /oldbase/cat
ln -sfv rebuild_crunch4 /oldbase/chmod
ln -sfv rebuild_crunch4 /oldbase/cp
ln -sfv rebuild_crunch4 /oldbase/date
ln -sfv rebuild_crunch4 /oldbase/ln
ln -sfv rebuild_crunch4 /oldbase/ls
ln -sfv rebuild_crunch4 /oldbase/mkdir
ln -sfv rebuild_crunch4 /oldbase/mv
ln -sfv rebuild_crunch4 /oldbase/rm
ln -sfv rebuild_crunch4 /oldbase/sh
ln -sfv rebuild_crunch4 /oldbase/sleep
ln -sfv rebuild_crunch4 /oldbase/awk
ln -sfv rebuild_crunch4 /oldbase/chflags
ln -sfv rebuild_crunch4 /oldbase/chpass
ln -sfv rebuild_crunch4 /oldbase/chsh
ln -sfv rebuild_crunch4 /oldbase/env
ln -sfv rebuild_crunch4 /oldbase/find
ln -sfv rebuild_crunch4 /oldbase/sed
ln -sfv rebuild_crunch4 /oldbase/stat
ln -sfv rebuild_crunch4 /oldbase/xargs
ln -sfv rebuild_crunch4 /oldbase/dialog
ln -sfv rebuild_crunch4 /oldbase/gzip
ln -sfv rebuild_crunch4 /oldbase/tar
ln -sfv rebuild_crunch4 /oldbase/reboot
ln -sfv rebuild_crunch4 /oldbase/chown
ln -sfv rebuild_crunch4 /oldbase/chroot
ln -sfv rebuild_crunch4 /oldbase/mtree
ln -sfv rebuild_crunch4 /oldbase/pwd_mkdb
#################### Sandbox myself into /oldbase
# For csh(1) or tcsh(1):
# setenv SHELL "/oldbase/sh"
# setenv PATH "/oldbase"
# rehash
# For sh(1) or bash(1):
# export SHELL="/oldbase/sh"
# export PATH="/oldbase"
# hash -r
#################### Perform final sanity checks
# Make sure /oldbase exists
# Make sure /oldbase is a directory
# Make sure $SHELL exists within /oldbase
# Make sure awk exists within /oldbase
# Make sure cat exists within /oldbase
# Make sure chflags exists within /oldbase
# Make sure chmod exists within /oldbase
# Make sure chown exists within /oldbase
# Make sure chpass exists within /oldbase
# Make sure chroot exists within /oldbase
# Make sure chsh exists within /oldbase
# Make sure cp exists within /oldbase
# Make sure date exists within /oldbase
# Make sure dialog exists within /oldbase
# Make sure env exists within /oldbase
# Make sure find exists within /oldbase
# Make sure gzip exists within /oldbase
# Make sure ln exists within /oldbase
# Make sure ls exists within /oldbase
# Make sure mkdir exists within /oldbase
# Make sure mtree exists within /oldbase
# Make sure mv exists within /oldbase
# Make sure pwd_mkdb exists within /oldbase
# Make sure reboot exists within /oldbase
# Make sure rm exists within /oldbase
# Make sure sed exists within /oldbase
# Make sure sh exists within /oldbase
# Make sure sleep exists within /oldbase
# Make sure stat exists within /oldbase
# Make sure tar exists within /oldbase
# Make sure xargs exists within /oldbase
#################### Clear distribution schg flag: base/base
cat "./../repos/8.1-RELEASE-amd64"/base/base.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: base/base
cat "./../repos/8.1-RELEASE-amd64"/base/base.?? | tar --unlink -xzpvf - -C "/"
--exclude etc/crontab --exclude etc/group --exclude etc/sysctl.conf --exclude
etc/master.passwd --exclude etc/passwd --exclude etc/spwd.db --exclude
etc/pwd.db --exclude etc/ssh/sshd_config --exclude boot/loader.conf
#################### Employ distribution mtree(8): base/base
mtree -eU -f ./../repos/8.1-RELEASE-amd64/base/base.mtree -p /
#################### Clear distribution schg flag: kernels/fis
cat "./../repos/8.1-RELEASE-amd64"/kernels/fis.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: kernels/fis
cat "./../repos/8.1-RELEASE-amd64"/kernels/fis.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): kernels/fis
mtree -eU -f ./../repos/8.1-RELEASE-amd64/kernels/fis.mtree -p /
#################### Clear distribution schg flag: local/local
cat "./../repos/8.1-RELEASE-amd64"/local/local.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: local/local
cat "./../repos/8.1-RELEASE-amd64"/local/local.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): local/local
mtree -eU -f ./../repos/8.1-RELEASE-amd64/local/local.mtree -p /
#################### Clear distribution schg flag: perl/perl
cat "./../repos/8.1-RELEASE-amd64"/perl/perl.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: perl/perl
cat "./../repos/8.1-RELEASE-amd64"/perl/perl.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): perl/perl
mtree -eU -f ./../repos/8.1-RELEASE-amd64/perl/perl.mtree -p /
#################### Clear distribution schg flag: lib32/lib32
cat "./../repos/8.1-RELEASE-amd64"/lib32/lib32.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: lib32/lib32
cat "./../repos/8.1-RELEASE-amd64"/lib32/lib32.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): lib32/lib32
mtree -eU -f ./../repos/8.1-RELEASE-amd64/lib32/lib32.mtree -p /
#################### Clear distribution schg flag: doc/doc
cat "./../repos/8.1-RELEASE-amd64"/doc/doc.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: doc/doc
cat "./../repos/8.1-RELEASE-amd64"/doc/doc.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): doc/doc
mtree -eU -f ./../repos/8.1-RELEASE-amd64/doc/doc.mtree -p /
#################### Clear distribution schg flag: games/games
cat "./../repos/8.1-RELEASE-amd64"/games/games.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: games/games
cat "./../repos/8.1-RELEASE-amd64"/games/games.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): games/games
mtree -eU -f ./../repos/8.1-RELEASE-amd64/games/games.mtree -p /
#################### Clear distribution schg flag: manpages/manpages
cat "./../repos/8.1-RELEASE-amd64"/manpages/manpages.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: manpages/manpages
cat "./../repos/8.1-RELEASE-amd64"/manpages/manpages.?? | tar --unlink -xzpvf -
-C "/"
#################### Employ distribution mtree(8): manpages/manpages
mtree -eU -f ./../repos/8.1-RELEASE-amd64/manpages/manpages.mtree -p /
#################### Clear distribution schg flag: proflibs/proflibs
cat "./../repos/8.1-RELEASE-amd64"/proflibs/proflibs.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: proflibs/proflibs
cat "./../repos/8.1-RELEASE-amd64"/proflibs/proflibs.?? | tar --unlink -xzpvf -
-C "/"
#################### Employ distribution mtree(8): proflibs/proflibs
mtree -eU -f ./../repos/8.1-RELEASE-amd64/proflibs/proflibs.mtree -p /
#################### Clear distribution schg flag: dict/dict
cat "./../repos/8.1-RELEASE-amd64"/dict/dict.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: dict/dict
cat "./../repos/8.1-RELEASE-amd64"/dict/dict.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): dict/dict
mtree -eU -f ./../repos/8.1-RELEASE-amd64/dict/dict.mtree -p /
#################### Clear distribution schg flag: info/info
cat "./../repos/8.1-RELEASE-amd64"/info/info.?? | tar tzf - | sed -e
"s:^/::;s:^\(\./\)\{0,1\}:/:" | xargs chflags noschg
#################### Unpack distribution contents: info/info
cat "./../repos/8.1-RELEASE-amd64"/info/info.?? | tar --unlink -xzpvf - -C "/"
#################### Employ distribution mtree(8): info/info
mtree -eU -f ./../repos/8.1-RELEASE-amd64/info/info.mtree -p /
#################### Run final mtree commands...
mtree -eU -f /etc/mtree/BSD.root.dist -p /
mtree -eU -f /etc/mtree/BSD.var.dist -p /var
mtree -eU -f /etc/mtree/BSD.usr.dist -p /usr
#################### Copy post-installation scripts...
mkdir -p /dist
cp -RPpfv ./../repos/8.1-RELEASE-amd64/druid /dist/
cp -RPpfv ./../repos/8.1-RELEASE-amd64/run_once /dist/
cp -RPpfv ./../repos/8.1-RELEASE-amd64/sys_custom /dist/
find "/dist/sys_custom" -type d -name CVS -exec rm -Rfv '{}' ';'
find "/dist/sys_custom" -type f -name .keep -exec rm -fv '{}' ';'
#################### Make-way for sys_custom items...
# Pre-unlink destination files
rm -f "/usr/local/sbin/host-setup"
#################### Install sys_custom items...
env SHELL=/oldbase/sh PATH=/oldbase chroot "/" /oldbase/sh /dist/druid/syscustom
#################### Prepare legacy pwd_mkdb(8) for run_once...
cp -fv /usr/sbin/pwd_mkdb /usr/sbin/pwd_mkdb.xnew
ln -sfv /oldbase/pwd_mkdb /usr/sbin/pwd_mkdb
#################### Check dhclient.conf(5) configuration...
# dhclient.conf(5) not configured (skip)
#################### Execute post-installation scripts...
env SHELL=/oldbase/sh PATH=/oldbase chroot "/" /oldbase/sh /dist/druid/run_once
#################### Run 8.1-RELEASE-amd64 specific post_install
post_install
# Remove `nodev' option from fstab(5) NFS mounts
cp -pf /etc/fstab /etc/fstab.nodev.xbak-2012-03-26.12:32:08
awk '
{
if ( $0 !~ /^[[:space:]]*#/ && ( $3 == "nfs" ) )
{
sub(/^nodev$/, "none", $4)
sub(/^nodev,/, "", $4)
sub(/,nodev,/, ",", $4)
sub(/,nodev$/, "", $4)
}
print
}' "/etc/fstab.nodev.xbak-2012-03-26.12:32:08" > "/etc/fstab"
# Change `null' fstype in fstab(5) to `nullfs'
cp -pf /etc/fstab /etc/fstab.null.xbak-2012-03-26.12:32:08
awk '
{
if ( $0 !~ /^[[:space:]]*#/ && ( $3 == "null" ) )
{
$3 = "nullfs"
}
print
}' "/etc/fstab.null.xbak-2012-03-26.12:32:08" > "/etc/fstab"
# Remove old files/directories:
# See `./etc/REMOVE_LIST-8.1' for a full listing
# Move `/usr/local/etc/xdm' to `/usr/local/etc/xdm.xbak-2012-03-26.12:32:08'
mv -v /usr/local/etc/xdm /usr/local/etc/xdm.xbak-2012-03-26.12:32:08
#################### Clean up...
rm -Rfv /dist
mv -fv /usr/sbin/pwd_mkdb.xnew /usr/sbin/pwd_mkdb
#################### End
=== END OUTPUT OF OUR BINARY MIGRATION SCRIPT BELOW ===
As you'll notice from the above output, we also make use of a custom
"rebuild_crunch" binary which allows us to sandbox ourself into a directory
containing a 1.2MB executable of statically-compiled utilities (crunched
together with crunchgen(8)) during the replacement of the base (running) system.
NOTE: It's very important that -- unless you understand fully the concepts of
the above commands -- that you NOT attempt this procedure as there are MANY
gotchas in the twists-and-turns that we use to upgrade binary systems. For
example, it's not shown above all that the code does in order to prevent the box
from hanging during the next boot.
Cheers, and happy digesting. As always, feel free to ask any questions.
I look forward to releasing this feature in a future revision of the "FreeBSD
Druid" available on druidbsd.sf.net. It really is a gem. It provides no less
than 270 different ways to perform the binary migration (including booting from
the Druid and rebuilding a non-bootable system).
--
Devin
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?014501cd0b87$d9452410$8bcf6c30$>