Date: Tue, 18 Apr 1995 10:44:32 -0400 From: Travis L Priest <T.L.Priest@LaRC.NASA.GOV> To: "Jordan K. Hubbard" <jkh@time.cdrom.com> Cc: ports@freefall.cdrom.com Subject: Re: A warning on pkg_deletes. Message-ID: <199504181444.KAA07365@licorice.larc.nasa.gov> In-Reply-To: <199411131024.CAA25267@time.cdrom.com> References: <199411131024.CAA25267@time.cdrom.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Sun, Jordan K. Hubbard <jkh@time.cdrom.com> wrote:
>
> I'm sure Satoshi will say this a lot more times in the next few
> days, but ports authors - PLEASE check your packing lists! If you
> do a pkg_delete of something and it blows the user's whole
> /usr/local (or worse!) away, you can take my word for it that said
> user will NOT be happy and it will give the _entire_ packages
> collection a black eye! The other porters will not be happy with
> you either since that user will most likely be screaming "the
> packages suck!! they deleted my entire collection of software!!"
> in the newsgroups and making it looks like *all* the packages are
> bad, not just yours.
>
> In summary: pkg_deletes that do anything other than exactly what
> they're supposed to are very very evil and *everyone* should run a
> pkg_delete -n on their package to see exactly what it would do. If
> the output makes your hair stand on end, please fix it! :-)
I've been pounding on poor Satoshi to read a fairly long e-mail
message I sent him about configuring software to avoid some of these
problems and a whole host of others, but perhaps this is a better
forum. I'd really like to see the FreeBSD ports built this way or in
some very similar fashion. I, along with a fellow convert :-), have
started bulding software for our FreeBSD systems that conform to this
scheme, and we'd like to give it back to the ports folks...
I'd like to see some discussion on this, even if it's "go away, this
scheme stinks."
Travis
>From priest Tue Apr 11 17:50:25 -0400 1995
To: asami@cs.berkeley.edu (Satoshi Asami)
Cc: Travis L Priest <T.L.Priest@LaRC.NASA.GOV>, branson@dvals1.larc.nasa.gov,
Tad Guy <E.E.Guy@LaRC.NASA.GOV>
Subject: Re: Ports hackers wanted! (fwd)
[This reply will be long; read it when you can dedicate some time.
This will be good practice for me, since I have to co-present it to a
group of System Administrators next week. :-)]
I'd like to make a recommendation on configuration 'standards.'
Pretend that this is a presentation, where the following topics will
be covered:
1 The problem
2 How software should be configured
3 How software should be installed
4 What are the advantages of using these 'standards'
5 What are some of the disadvantages
6 How is it currently being used
7 Future directions
Read this message all of the way through; a question you have an in
earlier section will be answered (with luck) in a proceeding session.
1 THE PROBLEM
This scheme was originally worked out by a co-worker and I while we
were trying to separate the vendor supplied OS from the local
modifications. Whenever a vendor's system was upgraded, local
software usually stood a good chance of getting lost or replaced by
vendor supplied versions. Additionally, it was difficult to find all
of the pieces of a package when it was time to install an updated
copy. Although it is possible to keep a record of where each file was
installed onto the system, this method is clumsy and error-prone. A
better, but compatible, method was clearly needed.
2 HOW SOFTWARE SHOULD BE CONFIGURED
The use of /usr/local for locally installed software is fairly
common practice on the Internet, and we wanted to stick with that.
When you get a software package (let's use emacs as the cannonical
example), it should be configured thusly:
PKGDIR = /usr/local/emacs-19.28
BINDIR = ${PKGDIR}/bin
LIBDIR = ${PKGDIR}/lib
APPRESDIR = ${PKGDIR}/lib/app-defaults
MANDIR = ${PKGDIR}/man
INFODIR = ${PKGDIR}/info
This scheme is made painfully easy with GNU's configure by simply
invoking the command "./configure --prefix=/usr/local/pkg-rev." When
the package is built, it will reference itself through the path
/usr/local/pkg-rev. For instance, emacs would find it's lisp files in
/usr/local/emacs-19.28/lib/lisp.
3 HOW SOFTWARE SHOULD BE INSTALLED
Once the package is built, it should be installed into /usr/local
in a subdirectory called pkg-rev. For the most part, this is simply a
"make install." You will now have a self-contained package directory
that looks something like this:
%ls /usr/local/emacs-19.25
bin info lib man
You'll find the emacs and and emacsclient (and other) binaries in the
bin directory, manual pages in man/man?, info files in the info
subdirectory, etc. You now have a nice little "package" contained in
a clean unit, but in order for users to get to it, they would have to
add '/usr/local/emacs-19.25/bin' to their paths, which is clearly
unacceptable. This is where the little bit of software steps in:
makelinks. This perl script (can be done in sh, C, or foo, if you
like, but I really like perl) scans a package directory for well known
subdirs (bin, lib, man, info, etc...) and makes links into the system
directories (/usr/local/bin, /usr/local/lib, etc..). Now, the new
software is in the user's paths for them to use. An example session
now goes like this:
% gnutar zxfp emacs-19.28.tar.gz
% cd emacs-19.28
% ./configure --prefix=/usr/local/emacs-19.28
% make
# make install
# makelinks -d /usr/local/emacs-19.28
I will append a copy of the makelinks perl script to the bottom of
this letter. There are a number of options, such as changing the base
directory (say to /local), making a shell script of the command that
would have been run for later execution, and running a
post-installation script (say to restart a daemon). [Makelinks was
inspired by something out of the Cygnus gcc installation procedure,
but it's over a year old so I forget what it was.]
4 WHAT ARE THE ADVANTAGES OF USING THESE 'STANDARDS'
This has turned out to have more advantages than we originally
anticipated. I'll enumerate:
* A package is now a well-formed collection of files under a common
directory (/usr/local/pkg-rev).
* You no longer have files belonging to one package in directories
where the vendor OS is installed, so it is less likely to be deleted
or modified when you perform an OS upgrade.
* You can instantly determine what version of a package you are using
simply by using 'ls -l /usr/local/bin/binary,' which will be a link
into the pkg-rev directory.
* These packages are painless to distribute. Use tar on the package
directory, ship it to the new system, untar it, and run makelinks.
* You can install two or more revisions of the same package
simultaneously, and test one revision (the SA would use
/usr/local/pkg-rev/bin to access it) while the other is in active
use by users. Once the SA is satisfied that the new package
functions, a "makelinks -r -d /usr/local/pkg-oldrev" followed by
"makelinks -d /usr/local/pkg-newrev" makes it immediately available
to users with an very (< 2 seconds) brief period of unavailability.
* If you are running low on space in /usr/local, it is now very easy
to make a symlink from /usr/local/pkg-rev to /bigone/pkg-rev and
have the files live on the bigone partition. It's easy to automount
packages, now, too, because you can link /usr/local/pkg-rev to the
automount point.
* From one major release to another, users will not have to change
their access habits. I did not mention earlier that the final link
made by makelinks is /usr/local/pkg -> /usr/local/pkg-rev. This
allows users to add /usr/local/emacs/lib/lisp/... paths to their
.emacs files and not have to change that when emacs is upgraded, or
to access irc scripts through /usr/local/ircii/scripts. The link
/usr/local/pkg always points to the 'supported' version.
* Since libraries like xpm, pvm, jpeg, tiff, etc., will exist in
/usr/local/lib (as links), users simply have to add one -L option to
their command line to link against them. The same as for include.
5 WHAT ARE SOME OF THE DISADVANTAGES
Well, I don't want to be accused of being one-sided, so here are some
of the disadvantages we've found so far:
* X does not work well if you use makelinks on the X distribution. It
is best to consider X to be an exception and globally add its bin
directory to the user's path. What we generally do is configure the
ProjectRoot to be /usr/local/X11R? and "make World ; make install"
which deposits X into /usr/local/X11R?. Then we move it to a
revision directory with "mv /usr/local/X11R? /usr/local/X11R?-??"
and make the link "ln /usr/local/X11R?-?? /usr/local/X11R?."
* Some speicial items need their own directory, like local lisp files
(vm, gnus) and perl contributions (locally written things). These
can go in /usr/local/lisp-rev or perl-rev so that they don't
disappear when you upgrade emacs or perl.
That's all I can think of at the moment. Please feel free to add your
concerns here.
6 HOW IS IT CURRENTLY BEING USED
We have this scheme successfully running on the following
architectures for about a year now:
Machine OS
------- --
Sun SunOS / Solaris
SGI IRIX (4 through 6)
HP HP/UX 9
DEC OSF/1
Cray Unicos 8
I'm working on setting it up on my FreeBSD box now so that I can
distribute binaries to some of the folks here that will be running
FreeBSD soon. All I do is build the stuff into /usr/local/pkg-rev,
tar it up and put it in our Mass Storage System, and everyone else
grabs the tarball, untars it into /usr/local, and runs makelinks on
it.
7 FUTURE DIRECTIONS
There are some topics that I did not cover in this message that are
worth mentioning. I like to consider /usr/local to be very static (no
log files are written here) after the initial software population, and
sharable among many machines of the same architecture. This means
that things like tcp_wrapper, which has local configuration files,
should be built to reference itself through /usr/local/pkg-rev like
everything else, but should reference it's config files in
/local/pkg-rev. The paradigm is that /usr/local is sharable among
machines, and that /local is local to one machine. It is *very*
desirable to separate vendor supplied files from what is locally
installed, which means configuration files don't go in /etc. Separate
partitions are nice so that you can just newfs the vendor OS
partitions when you upgrade, then mount /local and /usr/local when you
are done, and assuming binary compatibility between releases (very
common), you're good to go.
I've written a paper on this scheme, and it's available via the WWW.
Point your browser at:
http://ice-www.larc.nasa.gov/ICE/papers/system-software.html
if you'd like to see the finer points.
You'll find a copy of the makelinks perl script attached. I'd
appreciate good and bad comments about the scheme, especially if they
expose any weaknesses we haven't seen. I'd really like to see FreeBSD
go with this scheme (for personal and practical reasons :-). Feel
free to redistribute if the mood strikes you.
Travis
#!/usr/local/bin/perl -- # -*- Perl -*-
#
# $Id: makelinks,v 2.6 1994/08/05 13:24:05 priest Exp $
#
#
# Travis L Priest <T.L.Priest@LaRC.NASA.GOV>
# NASA Langley Research Center
# Information Systems Division
#
#
# -- Configuration Options --
#
$BASE = '/usr/local'; # local install tree base
# Rather this than extra code to deal with
# app-defaults and manual pages
@SUBDIRS = ('bin', 'info', 'lib', 'lib/app-defaults', 'include',
'man/man1', 'man/man2', 'man/man3', 'man/man4', 'man/man5',
'man/man6', 'man/man7', 'man/man8', 'man/mann', 'man/manl',
'man/cat1', 'man/cat2', 'man/cat3', 'man/cat4', 'man/cat5',
'man/cat6', 'man/cat7', 'man/cat8', 'man/catn', 'man/catl');
#
# -- Standard Definitions --
#
$FALSE = 0;
$TRUE = 1;
@ME = split(/\//, $0); $ME = pop(@ME); undef(@ME); # My name
($REVISION) = q$Revision: 2.6 $ =~ /^Revision:\s+([\d\.]+)/; # RCS version
#
# -- Process Options --
#
require("getopts.pl");
# -b base Change the base installed base to something else
# -d dir Make links from the specified directory
# -h Print help
# -l lib Create links in a system library
# -n Simply print what would be done without doing it
# -r Remove links (un-install the package)
# -s file Create a /bin/sh script named file to do the work instead
# -v Print version
# -x file Execute script at completion
$USAGE = "Usage: $ME -[hnrv] [-b base] [-d dir] [-l lib] [-[sx] script]\n";
do Getopts('b:d:hl:nrrs:vx:') || die $USAGE;
$base = $opt_b ? $opt_b : $BASE;
$path = $opt_d ? $opt_d : `pwd`;
$help = $opt_h ? &printhelp : $FALSE;
$lib = $opt_l ? $opt_l : $FALSE;
$FAKE = $opt_n ? $TRUE : $FALSE;
$remove = $opt_r ? $TRUE : $FALSE;
$script = $opt_s ? $opt_s : $FALSE;
$revision = $opt_v ? &printrev : $FALSE;
$execute = $opt_x ? $opt_x : $FALSE;
exit if ($help || $revision ); # already done
select(STDERR); $| = $TRUE; # unbuffered I/O
select(STDOUT); $| = $TRUE; # unbuffered I/O
&startscript if $script; # create a MakeLinks script instead of doing it
#
# -- Main --
#
#
# $pkg is the name of the package
# $rev is the revision number
# $location is the path to the install dir
# (ie. /usr/local/$pkg-$rev)
# $path is the standard access path
# (ie. /usr/local/$pkg)
#
if ( $lib ) {
($pkg, $location, $path) = &parselib($lib);
&linkents("$location", "$path") if -d "$location";
} else {
($pkg, $rev, $location, $path) = &parsedir($path);
# link the access dir to the installation dir
&dolink($location, "$base/$pkg") unless $lib;
while ( $dir = shift @SUBDIRS ) {
&linkents("$location/$dir", "$base/$dir") if -d "$location/$dir";
}
}
&runscript($execute) if $execute; # should we run a closing config program?
close(SCRIPT) if $script;
exit(0); # end main
#
# -- Subroutines --
#
#
# Attempt to determine the package name and revision level based on the
# directory name.
#
sub parsedir {
local($path) = @_;
local(@path, $pkg, @rev, $rev, $loc);
# check the path for validity
$path =~ s/\n$//;
chdir($path) || die $!;
$loc = $path; # save location
@path = split(/\//, $path);
($pkg, @rev) = split('-', pop(@path));
do {
$rev = shift @rev; # save revision
} while ( ($rev !~ /\d/) && ($pkg .= "-$rev") && @rev );
$path = join('/', @path, $pkg); # save path
# sanity check
die "Can't determine package name" if $pkg =~ /^$/;
die "Can't determine revision" if $rev =~ /^$/;
return($pkg, $rev, $loc, $path);
}
#
# Attempt to determine similar information as above, but for a system
# library
#
sub parselib {
local($path) = @_;
local(@path, $pkg, $loc, $lib);
# check the path for validity
$path =~ s/\n$//;
chdir($path) || die "$ME: $!";
$loc = $path; # save location
@path = split(/\//, $path);
$pkg = pop(@path);
$path = join('/', @path, 'lib'); # save path
return($pkg, $loc, $path);
}
#
# Create a links for all files in the src directory to the target directory.
#
sub linkents {
local($src, $target) = @_;
opendir(DIR, "$src") || die "Can't read $src: $!";
while ( $entry = readdir DIR ) {
next if $entry =~ /^\./; # ignore dot files
next if $entry =~ /~$/; # ignore ~ (emacs backup) files
next if $entry =~ /\.p[hl]$/; # special -- ignore perl .pl and .ph
next if $entry =~ /\.(bak)|(orig)|(old)$/; # ignore human backup files
next if -d "$src/$entry";
return unless &mkpath($target);
&dolink("$src/$entry", "$target/$entry");
}
closedir DIR;
return;
}
#
# Perform the unlink() and optinally symlink() to a file.
#
sub dolink {
local($old, $new) = @_;
local($dir);
if ( $script ) {
$new =~ s/^$base/\$BASE/;
print SCRIPT "\$RM $new\n";
print SCRIPT "\$LN $old $new\n" if ! $remove;
} else {
if ( -e $new || -l $new ) {
if ( $FAKE ) {
print "remove $new\n";
} else {
unlink $new || die "Can't unlink $new: $!";
}
}
if ( ! $remove ) { # global var!
if ( $FAKE ) {
print "link $old -> $new\n";
} else {
symlink($old, $new) || warn "linking $old -> $new failed: $!";
}
}
}
return;
}
sub startscript {
if ( $FAKE ) {
open(SCRIPT, ">&STDOUT") || die "$ME: $!";
} else {
open(SCRIPT, ">$script") || die "$ME: $!";
}
print SCRIPT "#!/bin/sh -x\n\n";
print SCRIPT "# Created by $ME-$REVISION\n\n";
print SCRIPT "BASE='$base'\n";
print SCRIPT "RM='rm -f'\n";
print SCRIPT "LN='ln -s'\n";
print SCRIPT "\n";
return;
}
#
# Print out a helpful usage message
#
sub printhelp {
print "$USAGE\n";
while ( <DATA> ) {
print;
}
return(1);
}
#
# Print out an informative revision message
#
sub printrev {
print "$ME-$REVISION\n";
return(1);
}
#
# Some packages require special things to be run
#
sub runscript {
local($name) = @_;
local(@file, $file);
@file = split(/\//, $name); $file = pop(@file); undef(@file);
unless ( -x $name ) {
print STDERR "'$name' not executable by you -- not executed\n";
return($FALSE);
}
# This is hard to determine. I have chosen
# to force the name to be in the physical
# directory, so lump it.
if ( $script ) {
print SCRIPT "\n# Run the configuration script '$name'\n";
print SCRIPT "$location/$file\n";
} else {
print `$location/$file`;
}
return($TRUE);
}
sub mkpath { # make needed subdirs
local(@parts) = split(/\//,@_[0]);
local($path) = '';
$! = ''; # reset and check later
foreach $part ( @parts ) {
next if $part eq '';
$path .= "/$part";
unless ( -d $path ) {
unless ( mkdir($path,0777) ) {
print STDERR "Failed to make path: $!\n";
return($FALSE);
}
}
}
if ( $! ) {
}
return($TRUE);
}
#
# -- DATA --
#
__END__
-b base Change the base installed base to something else
-d dir Make links from the specified directory
-h Print help
-l lib Create links in a system library
-n Simply print what would be done without doing it
-r Remove links (un-install the package)
-s file Create a /bin/sh script named file to do the work instead
-v Print version
-x file Execute command file after all other work is complete
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199504181444.KAA07365>