From owner-freebsd-ports Tue Apr 18 07:44:25 1995 Return-Path: ports-owner Received: (from majordom@localhost) by freefall.cdrom.com (8.6.10/8.6.6) id HAA08977 for ports-outgoing; Tue, 18 Apr 1995 07:44:25 -0700 Received: from licorice.larc.nasa.gov (licorice.larc.nasa.gov [128.155.2.31]) by freefall.cdrom.com (8.6.10/8.6.6) with ESMTP id HAA08970 for ; Tue, 18 Apr 1995 07:44:23 -0700 Received: by licorice.larc.nasa.gov (8.6.11/server2.4) id KAA07365; Tue, 18 Apr 1995 10:44:32 -0400 Message-Id: <199504181444.KAA07365@licorice.larc.nasa.gov> Date: Tue, 18 Apr 1995 10:44:32 -0400 From: Travis L Priest To: "Jordan K. Hubbard" Cc: ports@freefall.cdrom.com Subject: Re: A warning on pkg_deletes. In-Reply-To: <199411131024.CAA25267@time.cdrom.com> References: <199411131024.CAA25267@time.cdrom.com> Sender: ports-owner@FreeBSD.org Precedence: bulk Sun, Jordan K. Hubbard wrote: > > I'm sure Satoshi will say this a lot more times in the next few > days, but ports authors - PLEASE check your packing lists! If you > do a pkg_delete of something and it blows the user's whole > /usr/local (or worse!) away, you can take my word for it that said > user will NOT be happy and it will give the _entire_ packages > collection a black eye! The other porters will not be happy with > you either since that user will most likely be screaming "the > packages suck!! they deleted my entire collection of software!!" > in the newsgroups and making it looks like *all* the packages are > bad, not just yours. > > In summary: pkg_deletes that do anything other than exactly what > they're supposed to are very very evil and *everyone* should run a > pkg_delete -n on their package to see exactly what it would do. If > the output makes your hair stand on end, please fix it! :-) I've been pounding on poor Satoshi to read a fairly long e-mail message I sent him about configuring software to avoid some of these problems and a whole host of others, but perhaps this is a better forum. I'd really like to see the FreeBSD ports built this way or in some very similar fashion. I, along with a fellow convert :-), have started bulding software for our FreeBSD systems that conform to this scheme, and we'd like to give it back to the ports folks... I'd like to see some discussion on this, even if it's "go away, this scheme stinks." Travis >From priest Tue Apr 11 17:50:25 -0400 1995 To: asami@cs.berkeley.edu (Satoshi Asami) Cc: Travis L Priest , branson@dvals1.larc.nasa.gov, Tad Guy Subject: Re: Ports hackers wanted! (fwd) [This reply will be long; read it when you can dedicate some time. This will be good practice for me, since I have to co-present it to a group of System Administrators next week. :-)] I'd like to make a recommendation on configuration 'standards.' Pretend that this is a presentation, where the following topics will be covered: 1 The problem 2 How software should be configured 3 How software should be installed 4 What are the advantages of using these 'standards' 5 What are some of the disadvantages 6 How is it currently being used 7 Future directions Read this message all of the way through; a question you have an in earlier section will be answered (with luck) in a proceeding session. 1 THE PROBLEM This scheme was originally worked out by a co-worker and I while we were trying to separate the vendor supplied OS from the local modifications. Whenever a vendor's system was upgraded, local software usually stood a good chance of getting lost or replaced by vendor supplied versions. Additionally, it was difficult to find all of the pieces of a package when it was time to install an updated copy. Although it is possible to keep a record of where each file was installed onto the system, this method is clumsy and error-prone. A better, but compatible, method was clearly needed. 2 HOW SOFTWARE SHOULD BE CONFIGURED The use of /usr/local for locally installed software is fairly common practice on the Internet, and we wanted to stick with that. When you get a software package (let's use emacs as the cannonical example), it should be configured thusly: PKGDIR = /usr/local/emacs-19.28 BINDIR = ${PKGDIR}/bin LIBDIR = ${PKGDIR}/lib APPRESDIR = ${PKGDIR}/lib/app-defaults MANDIR = ${PKGDIR}/man INFODIR = ${PKGDIR}/info This scheme is made painfully easy with GNU's configure by simply invoking the command "./configure --prefix=/usr/local/pkg-rev." When the package is built, it will reference itself through the path /usr/local/pkg-rev. For instance, emacs would find it's lisp files in /usr/local/emacs-19.28/lib/lisp. 3 HOW SOFTWARE SHOULD BE INSTALLED Once the package is built, it should be installed into /usr/local in a subdirectory called pkg-rev. For the most part, this is simply a "make install." You will now have a self-contained package directory that looks something like this: %ls /usr/local/emacs-19.25 bin info lib man You'll find the emacs and and emacsclient (and other) binaries in the bin directory, manual pages in man/man?, info files in the info subdirectory, etc. You now have a nice little "package" contained in a clean unit, but in order for users to get to it, they would have to add '/usr/local/emacs-19.25/bin' to their paths, which is clearly unacceptable. This is where the little bit of software steps in: makelinks. This perl script (can be done in sh, C, or foo, if you like, but I really like perl) scans a package directory for well known subdirs (bin, lib, man, info, etc...) and makes links into the system directories (/usr/local/bin, /usr/local/lib, etc..). Now, the new software is in the user's paths for them to use. An example session now goes like this: % gnutar zxfp emacs-19.28.tar.gz % cd emacs-19.28 % ./configure --prefix=/usr/local/emacs-19.28 % make # make install # makelinks -d /usr/local/emacs-19.28 I will append a copy of the makelinks perl script to the bottom of this letter. There are a number of options, such as changing the base directory (say to /local), making a shell script of the command that would have been run for later execution, and running a post-installation script (say to restart a daemon). [Makelinks was inspired by something out of the Cygnus gcc installation procedure, but it's over a year old so I forget what it was.] 4 WHAT ARE THE ADVANTAGES OF USING THESE 'STANDARDS' This has turned out to have more advantages than we originally anticipated. I'll enumerate: * A package is now a well-formed collection of files under a common directory (/usr/local/pkg-rev). * You no longer have files belonging to one package in directories where the vendor OS is installed, so it is less likely to be deleted or modified when you perform an OS upgrade. * You can instantly determine what version of a package you are using simply by using 'ls -l /usr/local/bin/binary,' which will be a link into the pkg-rev directory. * These packages are painless to distribute. Use tar on the package directory, ship it to the new system, untar it, and run makelinks. * You can install two or more revisions of the same package simultaneously, and test one revision (the SA would use /usr/local/pkg-rev/bin to access it) while the other is in active use by users. Once the SA is satisfied that the new package functions, a "makelinks -r -d /usr/local/pkg-oldrev" followed by "makelinks -d /usr/local/pkg-newrev" makes it immediately available to users with an very (< 2 seconds) brief period of unavailability. * If you are running low on space in /usr/local, it is now very easy to make a symlink from /usr/local/pkg-rev to /bigone/pkg-rev and have the files live on the bigone partition. It's easy to automount packages, now, too, because you can link /usr/local/pkg-rev to the automount point. * From one major release to another, users will not have to change their access habits. I did not mention earlier that the final link made by makelinks is /usr/local/pkg -> /usr/local/pkg-rev. This allows users to add /usr/local/emacs/lib/lisp/... paths to their .emacs files and not have to change that when emacs is upgraded, or to access irc scripts through /usr/local/ircii/scripts. The link /usr/local/pkg always points to the 'supported' version. * Since libraries like xpm, pvm, jpeg, tiff, etc., will exist in /usr/local/lib (as links), users simply have to add one -L option to their command line to link against them. The same as for include. 5 WHAT ARE SOME OF THE DISADVANTAGES Well, I don't want to be accused of being one-sided, so here are some of the disadvantages we've found so far: * X does not work well if you use makelinks on the X distribution. It is best to consider X to be an exception and globally add its bin directory to the user's path. What we generally do is configure the ProjectRoot to be /usr/local/X11R? and "make World ; make install" which deposits X into /usr/local/X11R?. Then we move it to a revision directory with "mv /usr/local/X11R? /usr/local/X11R?-??" and make the link "ln /usr/local/X11R?-?? /usr/local/X11R?." * Some speicial items need their own directory, like local lisp files (vm, gnus) and perl contributions (locally written things). These can go in /usr/local/lisp-rev or perl-rev so that they don't disappear when you upgrade emacs or perl. That's all I can think of at the moment. Please feel free to add your concerns here. 6 HOW IS IT CURRENTLY BEING USED We have this scheme successfully running on the following architectures for about a year now: Machine OS ------- -- Sun SunOS / Solaris SGI IRIX (4 through 6) HP HP/UX 9 DEC OSF/1 Cray Unicos 8 I'm working on setting it up on my FreeBSD box now so that I can distribute binaries to some of the folks here that will be running FreeBSD soon. All I do is build the stuff into /usr/local/pkg-rev, tar it up and put it in our Mass Storage System, and everyone else grabs the tarball, untars it into /usr/local, and runs makelinks on it. 7 FUTURE DIRECTIONS There are some topics that I did not cover in this message that are worth mentioning. I like to consider /usr/local to be very static (no log files are written here) after the initial software population, and sharable among many machines of the same architecture. This means that things like tcp_wrapper, which has local configuration files, should be built to reference itself through /usr/local/pkg-rev like everything else, but should reference it's config files in /local/pkg-rev. The paradigm is that /usr/local is sharable among machines, and that /local is local to one machine. It is *very* desirable to separate vendor supplied files from what is locally installed, which means configuration files don't go in /etc. Separate partitions are nice so that you can just newfs the vendor OS partitions when you upgrade, then mount /local and /usr/local when you are done, and assuming binary compatibility between releases (very common), you're good to go. I've written a paper on this scheme, and it's available via the WWW. Point your browser at: http://ice-www.larc.nasa.gov/ICE/papers/system-software.html if you'd like to see the finer points. You'll find a copy of the makelinks perl script attached. I'd appreciate good and bad comments about the scheme, especially if they expose any weaknesses we haven't seen. I'd really like to see FreeBSD go with this scheme (for personal and practical reasons :-). Feel free to redistribute if the mood strikes you. Travis #!/usr/local/bin/perl -- # -*- Perl -*- # # $Id: makelinks,v 2.6 1994/08/05 13:24:05 priest Exp $ # # # Travis L Priest # NASA Langley Research Center # Information Systems Division # # # -- Configuration Options -- # $BASE = '/usr/local'; # local install tree base # Rather this than extra code to deal with # app-defaults and manual pages @SUBDIRS = ('bin', 'info', 'lib', 'lib/app-defaults', 'include', 'man/man1', 'man/man2', 'man/man3', 'man/man4', 'man/man5', 'man/man6', 'man/man7', 'man/man8', 'man/mann', 'man/manl', 'man/cat1', 'man/cat2', 'man/cat3', 'man/cat4', 'man/cat5', 'man/cat6', 'man/cat7', 'man/cat8', 'man/catn', 'man/catl'); # # -- Standard Definitions -- # $FALSE = 0; $TRUE = 1; @ME = split(/\//, $0); $ME = pop(@ME); undef(@ME); # My name ($REVISION) = q$Revision: 2.6 $ =~ /^Revision:\s+([\d\.]+)/; # RCS version # # -- Process Options -- # require("getopts.pl"); # -b base Change the base installed base to something else # -d dir Make links from the specified directory # -h Print help # -l lib Create links in a system library # -n Simply print what would be done without doing it # -r Remove links (un-install the package) # -s file Create a /bin/sh script named file to do the work instead # -v Print version # -x file Execute script at completion $USAGE = "Usage: $ME -[hnrv] [-b base] [-d dir] [-l lib] [-[sx] script]\n"; do Getopts('b:d:hl:nrrs:vx:') || die $USAGE; $base = $opt_b ? $opt_b : $BASE; $path = $opt_d ? $opt_d : `pwd`; $help = $opt_h ? &printhelp : $FALSE; $lib = $opt_l ? $opt_l : $FALSE; $FAKE = $opt_n ? $TRUE : $FALSE; $remove = $opt_r ? $TRUE : $FALSE; $script = $opt_s ? $opt_s : $FALSE; $revision = $opt_v ? &printrev : $FALSE; $execute = $opt_x ? $opt_x : $FALSE; exit if ($help || $revision ); # already done select(STDERR); $| = $TRUE; # unbuffered I/O select(STDOUT); $| = $TRUE; # unbuffered I/O &startscript if $script; # create a MakeLinks script instead of doing it # # -- Main -- # # # $pkg is the name of the package # $rev is the revision number # $location is the path to the install dir # (ie. /usr/local/$pkg-$rev) # $path is the standard access path # (ie. /usr/local/$pkg) # if ( $lib ) { ($pkg, $location, $path) = &parselib($lib); &linkents("$location", "$path") if -d "$location"; } else { ($pkg, $rev, $location, $path) = &parsedir($path); # link the access dir to the installation dir &dolink($location, "$base/$pkg") unless $lib; while ( $dir = shift @SUBDIRS ) { &linkents("$location/$dir", "$base/$dir") if -d "$location/$dir"; } } &runscript($execute) if $execute; # should we run a closing config program? close(SCRIPT) if $script; exit(0); # end main # # -- Subroutines -- # # # Attempt to determine the package name and revision level based on the # directory name. # sub parsedir { local($path) = @_; local(@path, $pkg, @rev, $rev, $loc); # check the path for validity $path =~ s/\n$//; chdir($path) || die $!; $loc = $path; # save location @path = split(/\//, $path); ($pkg, @rev) = split('-', pop(@path)); do { $rev = shift @rev; # save revision } while ( ($rev !~ /\d/) && ($pkg .= "-$rev") && @rev ); $path = join('/', @path, $pkg); # save path # sanity check die "Can't determine package name" if $pkg =~ /^$/; die "Can't determine revision" if $rev =~ /^$/; return($pkg, $rev, $loc, $path); } # # Attempt to determine similar information as above, but for a system # library # sub parselib { local($path) = @_; local(@path, $pkg, $loc, $lib); # check the path for validity $path =~ s/\n$//; chdir($path) || die "$ME: $!"; $loc = $path; # save location @path = split(/\//, $path); $pkg = pop(@path); $path = join('/', @path, 'lib'); # save path return($pkg, $loc, $path); } # # Create a links for all files in the src directory to the target directory. # sub linkents { local($src, $target) = @_; opendir(DIR, "$src") || die "Can't read $src: $!"; while ( $entry = readdir DIR ) { next if $entry =~ /^\./; # ignore dot files next if $entry =~ /~$/; # ignore ~ (emacs backup) files next if $entry =~ /\.p[hl]$/; # special -- ignore perl .pl and .ph next if $entry =~ /\.(bak)|(orig)|(old)$/; # ignore human backup files next if -d "$src/$entry"; return unless &mkpath($target); &dolink("$src/$entry", "$target/$entry"); } closedir DIR; return; } # # Perform the unlink() and optinally symlink() to a file. # sub dolink { local($old, $new) = @_; local($dir); if ( $script ) { $new =~ s/^$base/\$BASE/; print SCRIPT "\$RM $new\n"; print SCRIPT "\$LN $old $new\n" if ! $remove; } else { if ( -e $new || -l $new ) { if ( $FAKE ) { print "remove $new\n"; } else { unlink $new || die "Can't unlink $new: $!"; } } if ( ! $remove ) { # global var! if ( $FAKE ) { print "link $old -> $new\n"; } else { symlink($old, $new) || warn "linking $old -> $new failed: $!"; } } } return; } sub startscript { if ( $FAKE ) { open(SCRIPT, ">&STDOUT") || die "$ME: $!"; } else { open(SCRIPT, ">$script") || die "$ME: $!"; } print SCRIPT "#!/bin/sh -x\n\n"; print SCRIPT "# Created by $ME-$REVISION\n\n"; print SCRIPT "BASE='$base'\n"; print SCRIPT "RM='rm -f'\n"; print SCRIPT "LN='ln -s'\n"; print SCRIPT "\n"; return; } # # Print out a helpful usage message # sub printhelp { print "$USAGE\n"; while ( ) { print; } return(1); } # # Print out an informative revision message # sub printrev { print "$ME-$REVISION\n"; return(1); } # # Some packages require special things to be run # sub runscript { local($name) = @_; local(@file, $file); @file = split(/\//, $name); $file = pop(@file); undef(@file); unless ( -x $name ) { print STDERR "'$name' not executable by you -- not executed\n"; return($FALSE); } # This is hard to determine. I have chosen # to force the name to be in the physical # directory, so lump it. if ( $script ) { print SCRIPT "\n# Run the configuration script '$name'\n"; print SCRIPT "$location/$file\n"; } else { print `$location/$file`; } return($TRUE); } sub mkpath { # make needed subdirs local(@parts) = split(/\//,@_[0]); local($path) = ''; $! = ''; # reset and check later foreach $part ( @parts ) { next if $part eq ''; $path .= "/$part"; unless ( -d $path ) { unless ( mkdir($path,0777) ) { print STDERR "Failed to make path: $!\n"; return($FALSE); } } } if ( $! ) { } return($TRUE); } # # -- DATA -- # __END__ -b base Change the base installed base to something else -d dir Make links from the specified directory -h Print help -l lib Create links in a system library -n Simply print what would be done without doing it -r Remove links (un-install the package) -s file Create a /bin/sh script named file to do the work instead -v Print version -x file Execute command file after all other work is complete