Date: Sat, 06 Jan 2001 09:35:01 -0500 From: "Bryan J. Smith" <b.j.smith@ieee.org> To: david2000la@yahoo.com Cc: david@nns.ne.jp, freebsd-questions@freebsd.org Subject: RE: How about a free TCSH shell to backup to CDR? -- WAS: CDR Backup Copies of Software for Windows Message-ID: <3A572D15.8B3B7F0F@ieee.org> References: <FOEAIBOAIENFHIPHDDFBOEOICKAA.david@nns.ne.jp>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------96831CFD466F3AD9AB8F1BC0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
RE: How about a free TCSH shell to backup to CDR? -- WAS: CDR Backup
Copies of Software for Windows
CDR Software wrote:
> Hello. I am selling CDR backup copies of software for Windows.
> I charge only $10/CD and I have a huge collection including all
> of the very latest programs. If you are interested, please
> e-mail me and I'll send you a list.
Hi David (assuming you have not masqueraded your E-mail as someone
else)! Appreciate the SPAM! This is a FreeBSD (UNIX) list and we
do not believe in piracy. Futhermore, our software is free and we
have no use for pirated Windows software which won't run on our
computers.
Now if you are intersted in backup software to CDRs, I wrote this
free TCSH shell script (attached) that let's you backup to CD
(including per-file compression). I cover the features of the
script below ...
---
I recently (2000Dec15) created this script after a long user group
discussion on backups, file integrity and recovery of corrupted
backups, etc... For example, while Tar is quite recoverable when
corrupted (where the corrupted section only affects files in that
section), a compressed tarball is not (the whole archive becomes
unrecoverable). Likewise, PKZip has this same issue. We even had
one user who was so new to UNIX, he PKZiped his files and then
burned that to a CD (talk about putting your data at risk, let
alone it would be a pain to restore!).
Since ISO9660 is an archive format in itself, you don't need to put
it in a Tar, Cpio or even PKZip archive format. You just need to
get the files in the ISO9660 master ISO file. Now we want to
maximize storage, so we want to compress files individually. And
we want to minimize temporary space, so we want to compress as we
copy to the temporary area where we will master our CDs from.
The end result is a burnable ISO9660 CD (with RockRidge & Joliet
extensions + Apple/Mac if you use mkhybrid) that is filled with
files that can be directly accessed as normal, with the exception
that they are compressed for maximum capacity. I currently backup
almost 2GB into a 650MB CD using this utility. I wanted a nice
complement for CD backup that I have in afio for tape backup.
It works on Linux, Solaris, FreeBSD and even Cygwin (for NT
systems). Hope everyone enjoy's this Tcsh script. Review of
features:
- Per file compression!
- Uses GZip by default
- User selectable (see CONSTANTS)
- Individually selectable for text and non-text
(since GZip is faster/better for text, BZip2
better, although slower for most non-text)
- Minimizes disk usage by compressing as it
finds files -- but doesn't compress the existing
files -- it compresses them in the temporary
archive space
- Identification of pre-compressed files (so it doesn't
waste time recompressing) in 2 ways:
- Via "file" string
(catches anything compressed with Compress/Gzip/BZip2,
including Dia and files not easily identifyable)
- Via filename "extension"
- Both are extensible (see CONSTANTS)
- Takes one or more relative and/or absolute paths
- You can mix'n match relative and absolute paths
- Builds a hierarchy with the directory name in root
(unlike stock mkisofs/mkhybrid, where the source
directory is stripped -- I hate that!)
- Can take the current directory (via ".") as a parameter
- It checks _all_parameters_ to make sure they are
_valid_ relative/absolute directories (yet another
"niceity" over stock mkisofs/mkhybrid)
- Does not cross filesystem boundaries (for obvious
sanity reasons)
- Symlinks are copied as symlinks and special files are
copied over as special files (i.e. no issues/worries
about dealing with them)
- Supports both mkisofs (UNIX/Windows) and mkhybrid (+Mac)
- If "iso_mac" set (see VARIABLES), myhybrid is used
- Can specify alternative, mkisofs/mkhybrid compatible
binary (see CONSTANTS)
- Can also reset UNIX permissions so all files are
world readable (default is to preserve UID, GID and
bits -- see VARIABLES)
- You can set both the output and temporary directories
(see VARIABLES)
- Great for setting an output path to a Samba shared
directory where you can copy the .iso file over to your
Windows system with a CD burner (and pull up CD Creator
and tell it to burn a CD from an .iso image)
- Does not assume binaries are in path, or are set.
- You can override just about any binary location in
CONSTANTS.
- Only need tcsh and which, it either finds, or you
can set, anything else
- Minimal console clutter (nice simple ticks when
archiving), but maximum logging to files
Limitations (currently):
- Parameters must be directories
- A file cannot be a command-line parameter
- But you can still send "." for the current directory
- Parameters cannot have spaces in name
- Only the actually parameters passed on the command-line
- Spaces in their subdirectories and filenames are fine
- Again, only what you pass on the command-line cannot
have spaces (not even if you use "" or \ -- it's an
internal variable issue I haven't accomodated yet),
but any dirs/files under them are fine
- Symlinks are blindly copied over, two issues:
- It does not check that it points to anything useful
in the CD image (especilly for absolute paths)
[ I will probably never address this, no one does ]
- Even if it points to a relative path that is on the
CD, if that file is now compressed, it will not point
to the new filename [ I'll probably fix this soon ]
- Cannot have both a file and a compressed file in the
same directory
- Otherwise, only the first file will get copied and used
(e.g., if file and file.gz both exist, only the first
one seen by find will get backed up)
-- TheBS
--
Bryan "TheBS" Smith, Engineer CONTACT INFO
***********************************************************
Chat: thebs413 @ AOL/MSN/Yahoo (see http://Everybuddy.com)
Email: mailto:thebs@smithconcepts.com,thebs@theseus.com
--------------96831CFD466F3AD9AB8F1BC0
Content-Type: text/plain; charset=us-ascii;
name="back2cd.tcsh"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="back2cd.tcsh"
#!/bin/tcsh -f
#
# back2cd.tcsh
# $Id: back2cd.tcsh,v 1.4 2000/12/15 16:12:48 bjsmith Exp $
#
# Make a CD-R ISO9660 Backup CD Image with UNIX/Windows/Mac extensions
# compressing individual files in the image for maximum capacity.
#
# Written by Bryan "TheBS" Smith (mailto:b.j.smith@ieee.org)
# Licensed under the GNU Public License (GPL) version 2.1
# All rights reserved to the Free Software Foundation
##########################################################
# VARIABLES
##########################################################
# Directories
#=========================================================
# Temporary Directory (need upto 700MB + biggest, uncompressed file)
#---------------------------------------------------------
set iso_tmp=/tmp
# ISO Output Directory (need upto 700MB)
#---------------------------------------------------------
set iso_dir=/home
# Extensions/Options
#=========================================================
# Set iso_mac to use Mac long filenames (requires Mkhybrid)
#---------------------------------------------------------
#DISABLED#set iso_mac=set
# Set iso_rrr to reset UNIX permissions (so all files are readable by everyone)
#---------------------------------------------------------
#DISABLED#set iso_rrr=set
# Information (for ISO9660 namespace)
#=========================================================
# Publisher
#---------------------------------------------------------
set iso_pub="my name, company, address, phone"
# Preparer
#---------------------------------------------------------
set iso_pre="`whoami`@`uname -n`"
##########################################################
# CONSTANTS
##########################################################
# Binaries
#=========================================================
# Mkisofs/Mkhybrid (defaults to mkisofs in path, mkhybrid if "iso_mac" set)
#---------------------------------------------------------
#OPTION#set bin_mkiso=/usr/bin/mkisofs
#OPTION#set bin_mkiso=/usr/bin/mkhybrid
# Compression for text files (defaults to GZip in path)
#---------------------------------------------------------
#OPTION#set bin_cmptxt=/usr/gzip
#OPTION#set bin_cmptxt=/usr/bin/bzip2
# Compression for non-text files (defaults to GZip in path)
#---------------------------------------------------------
#OPTION#set bin_cmpbin=/usr/bin/gzip
#OPTION#set bin_cmpbin=/usr/bin/bzip2
# Other Binaries (defaults to binary in path)
#---------------------------------------------------------
#OPTION#set bin_cpio=/bin/cpio
#OPTION#set bin_find=/usr/bin/find
#OPTION#set bin_file=/usr/bin/file
#OPTION#set bin_grep=/bin/grep
#OPTION#set bin_mkdir=/bin/mkdir
#OPTION#set bin_rm=/bin/rm
# Files that should NOT be compressed
#=========================================================
# Strings reported by "file" (catches files compressed by Compress, GZip, BZip2, compressed XML apps)
#---------------------------------------------------------
set noc_typ="compress"
# By filename extension
#---------------------------------------------------------
set noc_ext="arc lha lhz pak rpm zip zoo"
##########################################################
# SCRIPT
##########################################################
# Help
#=========================================================
if ( $# < 2 || $1 == "--help" || $1 == "-?" || $1 == "-h" || $1 == "-H" ) then
echo ""
echo "Syntax:"
echo " back2cd <label> <dir> [<dir>...]"
echo ""
echo "Purpose:"
echo " Create a 'ready-to-burn' backup ISO9660 image, including"
echo " individually compressed files for maximum capacity while"
echo " preserving data integrity and ease of access."
echo ""
echo "Where:"
echo " <label> is the ISO volume label and output filename"
echo " (withOUT the .iso extension -- NO SPACES)"
echo " <dir> is the relative or absolute path to directories"
echo " (you can specify one or more paths -- use '.' to pick"
echo " the files/subdirectories in the current directory)"
echo ""
echo "The LOG of the archive process will be '${iso_dir}/<label>_arc.log'"
echo "The LOG of the CD mastering will be '${iso_dir}/<label>_iso.log'"
echo "The ISO file created will be '${iso_dir}/<label>.iso'"
echo ""
exit 0
endif
# Verify an ISO image with the same name does NOT already exist
#=========================================================
set iso_label=$1
if ( -e ${iso_dir}/${iso_label}.iso ) then
echo ""
echo "ERROR: ISO Image '${iso_dir}/${iso_label}.iso' ALREADY EXISTS!"
echo "Please pick another output label/filename."
echo ""
exit 1
endif
# Find binaries and/or verify binaries exists (NOTE: long and mundane, but neccessary)
#=========================================================
# Mkisofs/Mkhybrid Binary
#---------------------------------------------------------
if ( $?bin_mkiso ) then
if ( ! -e ${bin_mkiso} ) then
echo ""
echo "ERROR: Binary '${bin_mkiso}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
if ( $?iso_mac ) then
unalias mkhybrid
set bin_mkiso=`which mkhybrid`
if ( ! $?bin_mkiso ) then
echo ""
echo "ERROR: Cannot locate 'mkhybrid' binary"
echo "Apple long filename option set which requires 'mkhybrid'"
echo "Please set full path in script's CONSTANTS section"
echo "OR disabled Apple long filename option"
echo "(i.e. uses 'mkisofs' instead)"
echo ""
exit 1
endif
else
unalias mkisofs
set bin_mkiso=`which mkisofs`
if ( ! $?bin_mkiso ) then
echo ""
echo "ERROR: Cannot locate 'mkisofs' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
endif
# Compression Binaries
#---------------------------------------------------------
if ( $?bin_cmptxt ) then
if ( ! -e ${bin_cmptxt} ) then
echo ""
echo "ERROR: Binary '${bin_cmptxt}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias gzip
set bin_cmptxt=`which gzip`
if ( ! $?bin_cmptxt ) then
echo ""
echo "ERROR: Cannot locate 'gzip' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
if ( $?bin_cmpbin ) then
if ( ! -e ${bin_cmpbin} ) then
echo ""
echo "ERROR: Binary '${bin_cmpbin}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias gzip
set bin_cmpbin=`which gzip`
if ( ! $?bin_cmpbin ) then
echo ""
echo "ERROR: Cannot locate 'gzip' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
# Other binaries
#---------------------------------------------------------
if ( $?bin_cpio ) then
if ( ! -e ${bin_cpio} ) then
echo ""
echo "ERROR: Binary '${bin_cpio}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias cpio
set bin_cpio=`which cpio`
if ( ! $?bin_cpio ) then
echo ""
echo "ERROR: Cannot locate 'cpio' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
if ( $?bin_file ) then
if ( ! -e ${bin_file} ) then
echo ""
echo "ERROR: Binary '${bin_file}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias file
set bin_file=`which file`
if ( ! $?bin_file ) then
echo ""
echo "ERROR: Cannot locate 'file' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
if ( $?bin_find ) then
if ( ! -e ${bin_find} ) then
echo ""
echo "ERROR: Binary '${bin_find}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias find
set bin_find=`which find`
if ( ! $?bin_find ) then
echo ""
echo "ERROR: Cannot locate 'find' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
if ( $?bin_grep ) then
if ( ! -e ${bin_grep} ) then
echo ""
echo "ERROR: Binary '${bin_grep}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias grep
set bin_grep=`which grep`
if ( ! $?bin_grep ) then
echo ""
echo "ERROR: Cannot locate 'grep' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
if ( $?bin_mkdir ) then
if ( ! -e ${bin_mkdir} ) then
echo ""
echo "ERROR: Binary '${bin_mkdir}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias mkdir
set bin_mkdir=`which mkdir`
if ( ! $?bin_mkdir ) then
echo ""
echo "ERROR: Cannot locate 'mkdir' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
if ( $?bin_rm ) then
if ( ! -e ${bin_rm} ) then
echo ""
echo "ERROR: Binary '${bin_rm}' NOT found!"
echo "Please correct in script's CONSTANTS section"
echo ""
exit 1
endif
else
unalias rm
set bin_rm=`which rm`
if ( ! $?bin_rm ) then
echo ""
echo "ERROR: Cannot locate 'rm' binary"
echo "Please set full path to binary in script's CONSTANTS section"
echo ""
exit 1
endif
endif
# Verify target paths actual exist
#=========================================================
shift
set path_start=`pwd`
foreach path_param ( $* )
if ( ! -d "${path_param}" ) then
echo ""
echo "ERROR: Path '${path_param}'"
echo "is NOT a valid relative or absolute path to a directory."
echo ""
exit 1
endif
end
# Create temporary archive hierarchy
#=========================================================
# Begin Log
#---------------------------------------------------------
set arc_log = ${iso_dir}/${iso_label}_arc.log
echo "" > ${arc_log}
echo "${arc_log}" >> ${arc_log}
echo "`date`" >> ${arc_log}
echo "" >> ${arc_log}
echo " ISO Label: ${iso_label}" >> ${arc_log}
echo " Image to Create: ${iso_dir}/${iso_label}.iso" >> ${arc_log}
echo " Mastering Binary: ${bin_mkiso}" >> ${arc_log}
echo "" >> ${arc_log}
echo " Dir Launched In: ${path_start}" >> ${arc_log}
echo " Dirs to Archive: $*" >> ${arc_log}
echo " Archiving Temporary Dir: ${iso_tmp}/${iso_label}" >> ${arc_log}
echo "" >> ${arc_log}
echo " Text Compression Binary: ${bin_cmptxt}" >> ${arc_log}
echo " Non-Text Compression Binary: ${bin_cmpbin}" >> ${arc_log}
echo "Skip Compress 'file' Strings: ${noc_typ}" >> ${arc_log}
echo " Skip Compress Filename Exts: ${noc_ext}" >> ${arc_log}
echo "" >> ${arc_log}
echo " Other Binary Paths: ${bin_cpio} ${bin_find} ${bin_file} ${bin_grep} ${bin_mkdir} ${bin_rm}" >> ${arc_log}
echo "" >> ${arc_log}
# Clear out any directory from a previous attempt
#---------------------------------------------------------
if ( -e "${iso_tmp}/${iso_label}" ) then
echo "Previous temporary archiving directory '${iso_tmp}/${iso_label}' already exists!" >> ${arc_log}
echo " Deleting ..." >> ${arc_log}
echo "" >> ${arc_log}
${bin_rm} -fr {$iso_tmp}/${iso_label}
endif
# Create new hierarchy
#---------------------------------------------------------
echo "Creating temporary backup hierarchy" >> ${arc_log}
echo " mkdir ${iso_tmp}/${iso_label}" >> ${arc_log}
${bin_mkdir} "${iso_tmp}/${iso_label}"
foreach path_param ( $* )
echo " mkdir ${iso_tmp}/${iso_label}/${path_param}" >> ${arc_log}
echo "${path_param}" | ${bin_cpio} -pmdv ${iso_tmp}/${iso_label} >&! /dev/null
end
echo "" >> ${arc_log}
# Make compressed versions of parameter trees
#=========================================================
echo ""
echo -n "Phase 1: Archiving "
foreach path_param ( $* )
echo "" >> ${arc_log}
echo "***********************************************************" >> ${arc_log}
echo "* Archiving '${path_param}'" >> ${arc_log}
echo "***********************************************************" >> ${arc_log}
echo "" >> ${arc_log}
cd ${path_start}
cd ${path_param}
foreach file_curr ( `${bin_find} . -mount -print` )
# First off, blindly copy file/dir to temporary iso archiving area
#---------------------------------------------------------
echo -n "."
if ( -d "${file_curr}" ) then
set path_curdir=${file_curr}
echo "Creating/Analyzing directory '${path_param}/${path_curdir}'" >> ${arc_log}
echo "${file_curr}" | ${bin_cpio} -pmdv ${iso_tmp}/${iso_label}/${path_param} >&! /dev/null
goto skip_proc
endif
if ( -l "${file_curr}" ) then
echo " ${path_param}/${path_curdir} is a symlink, blindy copying (may be broken if referencing a, now, compressed file)" >> ${arc_log}
echo "${file_curr}" | ${bin_cpio} -pmdv ${iso_tmp}/${iso_label}/${path_param} >&! /dev/null
goto skip_proc
endif
echo "${file_curr}" | ${bin_cpio} -pmdv ${iso_tmp}/${iso_label}/${path_param} >&! /dev/null
# If file is plain file
#---------------------------------------------------------
if ( -f "${file_curr}" ) then
# Check file type for known compressed file types
#---------------------------------------------------------
foreach file_typ ( ${noc_typ} )
if ( `file ${file_curr} | ${bin_grep} -i "${file_typ}"` != "" ) then
echo " ${path_param}/${file_curr} is known to be a '${file_typ}' file type -- NO compression" >> ${arc_log}
goto skip_proc
endif
end
# Check filename extension known compressed filename extensions
#---------------------------------------------------------
foreach file_ext ( ${noc_ext} )
if ( `echo "${file_curr}" | ${bin_grep} -i "\.${file_ext}"` != "" ) then
echo " ${path_param}/${file_curr} has a filename extension of '.${file_ext}' -- NO compression" >> ${arc_log}
goto skip_proc
endif
end
# Check for text or non-text content, compress with program as designated
#---------------------------------------------------------
if ( `file ${file_curr} | ${bin_grep} -i "text"` != "" ) then
echo " ${path_param}/${file_curr} is of a known text type, compressing with '${bin_cmptxt}'" >> ${arc_log}
${bin_cmptxt} ${iso_tmp}/${iso_label}/${path_param}/${file_curr}
else
echo " ${path_param}/${file_curr} is of a non-text type, compressing with '${bin_cmpbin}'" >> ${arc_log}
${bin_cmpbin} ${iso_tmp}/${iso_label}/${path_param}/${file_curr}
endif
# Otherwise, leave alone (compressed files, directories, symlinks, etc...)
# (NOTE: symlinks get broken if they point to a plain file that is subsequently compressed)
#---------------------------------------------------------
skip_proc:
endif
end
end
echo ""
echo "" >> ${arc_log}
echo "***********************************************************" >> ${arc_log}
echo "* Archiving process complete" >> ${arc_log}
echo "***********************************************************" >> ${arc_log}
echo "" >> ${arc_log}
# Finally make the freak'n ISO image!
#=========================================================
# Resolve ISO options
#---------------------------------------------------------
if ( $?iso_mac ) then
set iso_macset="-apple"
else
set iso_macset=""
endif
if ( $?iso_rrr ) then
set iso_rrrset="-r"
else
set iso_rrrset=""
endif
# Master ISO image
#---------------------------------------------------------
echo ""
echo "Phase 2: Mastering (no progress marks)"
echo "" >> ${arc_log}
echo "Making ISO file '${iso_dir}/${iso_label}.iso'" >> ${arc_log}
echo "" >> ${arc_log}
echo "Running command (from ${iso_tmp}/${iso_label}): ${bin_mkiso} -A ${iso_label} -J -log-file ${iso_dir}/${iso_label}_iso.log -o ${iso_dir}/${iso_label}.iso -P '${iso_pub}' -p '${iso_pre}' -R ${iso_rrrset} -V ${iso_label} -v ${iso_macset} ." >> ${arc_log}
echo "" >> ${arc_log}
cd ${iso_tmp}/${iso_label}
${bin_mkiso} -A ${iso_label} -J -log-file ${iso_dir}/${iso_label}_iso.log -o ${iso_dir}/${iso_label}.iso -P "${iso_pub}" -p "${iso_pre}" -R ${iso_rrrset} -V ${iso_label} -v ${iso_macset} .
# Trap errors
#--------------------------------------------------------
if ( $status ) then
echo ""
echo "ERROR in CD creation."
echo "See LOG file '${iso_dir}/${iso_label}.log' for details."
echo ""
exit 1
endif
# Remove temporary archive directory (if successful)
#--------------------------------------------------------
${bin_rm} -fr {$iso_tmp}/${iso_label}
echo ""
echo "PROCESS COMPLETE"
echo ""
echo "The LOG of the archive process is '${iso_dir}/${iso_label}_arc.log'"
echo " The LOG of the CD mastering is '${iso_dir}/${iso_label}_iso.log'"
echo " The ISO file created is '${iso_dir}/${iso_label}.iso'"
echo ""
exit 0
--------------96831CFD466F3AD9AB8F1BC0--
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A572D15.8B3B7F0F>