Date: Wed, 5 Dec 2001 22:17:59 +0100 (MET) From: Joerg Schilling <schilling@fokus.gmd.de> To: obrien@freebsd.org, schilling@fokus.gmd.de Cc: brandt@fokus.gmd.de, freebsd-hackers@freebsd.org, tofergus@yahoo.co.uk Subject: Re: tar and nodump flag (fwd) Message-ID: <200112052117.fB5LHxF19472@burner.fokus.gmd.de>
next in thread | raw e-mail | index | archive | help
>From obrien@NUXI.com Mon Dec 3 20:27:24 2001
>On Thu, Nov 29, 2001 at 09:07:03PM +0100, Joerg Schilling wrote:
>> >> - An idiosyncratic build system.
>>
>> This is really funny....
>Please don't waste time on this issue. I can bmake and shoe-horn
>anything into our build system.
Right, this is definitely not a topic to put effort on....
... but to make it easier to understand problems with star vs. GNU tar
I spend several hours to prepare a comparison (much more than it took to
implement star -nodump ;-)
In order to allow an unbiased discussion, I prepared this comparison.
It contains some hints for people (currently using Gnu tar) who like to like to
switch to 'star' and a list of features missing in Gnu tar, a smaller list of
features missing in star and a list of incompatible options in Gnu tar.
As I am not sure whether I will have the time to keep it in sync with future
changes, it is valid for December 1st 2001.
First some background notes. Star is several years older than Gnu tar and most
of the options that have different meanings in gnu tar have been in use with
star since 1985. This is about two years before the first versions of Gnu tar
(called PD tar or SUG tar in 1987) have been known. Do not expect that the
meaning of those options will be changed in star in the future.
Star tries to be as compatible as possible with the BSD tar implementation
from 1982 and with the POSIX standard. Of course, it implements a lot of
enhancements. Gnu tar in former times (in 1987 when being called
PD tar / SUG tar) has been compatible to the BSD tar implementation but this
changed in 1989 when FSF adopted the program as Gnu tar. Gnu tar is not
very POSIX compliant (in command line options as well as in the archive format).
If you know the standard and thus know where Gnu tar is not standard compliant,
it is easy to switch between Gnu tar and star if you only use the basic
functionality. If you don't know the standard and only know Gnu tar you may
get into some problems but this is really a Gnu tar problem.
Now let us compare functionality and options. Star has a lot of functions
that are usually invisible to the user but make the daily use much easier:
- A FIFO allows to keeps the tape streaming. This gives you faster backups
than you can achieve with ufsdump and with other tar implementations
if the size of the filesystem is > 1 GByte.
- A powerful pattern matcher for a convenient user interface (see manual
page for more details) to archive/extract a subset of files.
- A sophisticated diff with a user tailorable interface for comparing
tar archives against file trees. This is one of the most interesting parts
of the star implementation.
- deals with all 3 times, it stores/restores all 3 times of a file
(even creation time)
On Solaris, it may reset access time after doing the backup without
clobbering the ctime.
- Star does not clobber files by default. More recent copies on disk will
not be clobbered from tape. This may be the main advantage over other
tar implementations. This allows automatically repairing of corruptions
after a crash & fsck (Check for differences after doing this with the
diff option).
- Star does automatic byte swapping if needed. Star automatically detects
swapped archives and transparently reads them the right way.
- Star has automatic compression detection. In the recent case when star
is used to unpack compressed files containing tar archives, star will
transparently detect and call the right decompression program.
- Star has automatic archive format detection. Star automatically detects
several common archive formats and adopts to them. Supported archive
types are:
tar old tar format
star old star format from 1985
gnutar gnu tar format
ustar standard tar (ieee 1003.1) format
xstar extended standard tar format
xustar extended standard tar format without tar signature
exustar extended standard tar format without tar signature (always x-header)
pax extended POSIX.1-200x standard tar format
suntar Sun's extended pre-POSIX.1-200x Solaris 7/8 tar format
- Star is fully ANSI/Posix 1003.1 compatible. See README.otherbugs for a
complete description of bugs found in other tar implementations.
- Support for the new POSIX.1-200x extended tar format. This new tar
format allows to archive many new things in a standard compliant way.
These things are currently are implemented in Gnu tar in a proprietary
way and have in former times been implemented in a proprietary way in
star. To give an example, the most interesting features that are now
handled in a standard compliant way are filenames without name len
limitation and file size without the old 8 GB tar limitation.
An interesting aspect of the new extended header format is that it is
extensible without limitation. Star uses this extension format to
archive Access Control Lists and file flags from BSD-4.4 and Linux.
Comparing options is a more complex task:
- Star has 84 base Options
- Gnu tar has 70 base Options
- Star has 39 Options that are missing in Gnu tar
- Gnu tar has 22 Options that are missing in star
- 9 of the Gnu tar options are not needed or do not make sense to implement
- 2 of the Gnu tar options are considered to be of low importance
- 6 of the Gnu tar options are nice to have
- 2 of the missing options are halfway ready in star
- 5 of the missing options are needed and important
Only two of the Gnu tar options may be knock off criteria (incremental backup
and "ignore failed reads") the latter may not work correctly in Gnu tar so only
the incremental backup remains as important missing point.
STAR Option Description Gnu tar equiv. Remarks
=========== =========== ============= ===================
Cmds:
-c/-u/-r create/update/replace named files to tape -c/-u/-r identical
-x/-t extract/list named files from tape -x/-t identical
-n trace named files from tape --missing--
-diff diff archive against file system (see -xhelp) --diff Gnu --diff is not as useful
as star's -diff
Options:
-help print this help --help close to identical
-xhelp print extended help see above
-version --version close to identical
-debug --missing--
blocks=#,b=# set blocking factor to #x512 Bytes (default 20) -b close to identical
file=nm,f=nm use 'nm' as tape instead of stdin/stdout -f close to identical
-T use $TAPE as tape instead of stdin/stdout --missing--
-fifo/-no-fifo use/don't use a fifo to optimize data flow from/to tape --missing--
-shm use SysV shared memory for fifo --missing--
-v be verbose -v identical
-block-number print the block numbers where the TAR headers start --block-number Not working in Gnu tar!
-tpath use with -t to list path names only --missing--
H=header generate 'header' type archive (see H=help) --missing--
C=dir perform a chdir to 'dir' before storing next file -C identical
-z pipe input/output through gzip, does not work on tapes -z identical
-bz pipe input/output through bzip2, does not work on tapes -j identical
-B perform multiple reads (needed on pipes) -B identical
-i ignore checksum errors -i similar
-d do not store/create directories --missing--
-m do not restore access and modification time -m identical
-nochown do not restore owner and group --no-same-owner similar
-a,-atime reset access time after storing file --atime-preserve partial!!!
-p restore filemodes of directories -p similar ?
-l do not print a message if not all links are dumped --missing--
-h,-L follow symbolic links as if they were files -h identical
-D do not descend directories --no-recursion identical
-M do not descend mounting points -l GNU -l violates POSIX
-I,-w do interactive creation/extraction/renaming -w partial
-O be compatible to old tar (except for checksum bug) -o GNU -o violates POSIX
-P last record may be partial (useful on cartridge tapes) --missing--
-S do not store/create special files --missing--
-F,-FF,-FFF,... do not store/create SCCS/RCS, core and object files --missing--
-U restore files unconditionally --missing--
diffopts=optlst comma separated list of diffopts (see diffopts=help) --missing--
-not,-V use those files which do not match pattern --missing--
VOLHDR=name use name to generate a volume header -V name similar
-xdir extract dir even if the current is never --missing--
-keep-old-files,-k keep existing files -k similar
-refresh-old-files refresh existing files, don't create new files --overwrite ????
-refresh --overwrite ????
-/ don't strip leading '/'s from file names -P similar
list=name read filenames from named file -I --files-from=NAME similar
-dodesc do descend directories found in a list= file --missing--
pattern=p,pat=p set matching pattern PATTERN not as mighty as star
maxsize=# do not store file if it is bigger than # kBytes --missing--
newer=name store only files which are newer than 'name' --never=DATE no find compat with GNUtar!
-ctime use ctime for newer= option --newer-mtime invers idea to star
-nodump do not dump files that have the nodump flag set --missing--
bs=# set (output) block size to # -b de-facto missing
fs=# set fifo size to # --missing--
tsize=# set tape volume size to # 512 byte blocks -L x 1024 bytes
-qic24 set tape volume size to 61440 kBytes --missing--
-qic120 set tape volume size to 128000 kBytes --missing--
-qic150 set tape volume size to 153600 kBytes --missing--
-qic250 set tape volume size to 256000 kBytes --missing--
-nowarn do not print warning messages --missing--
-time print timing info --totals only partially
-no-statistics do not print statistics --missing-- always
-fifostats print fifo statistics --missing--
-numeric don't use user/group name from tape --numeric-owner identical
-newest find newest file on tape --missing--
-newest-file find newest regular file on tape --missing--
-hpdev use HP's non POSIX compliant method to store dev numbers --missing--
-modebits include all 16 bits from stat.st_mode (violates POSIX-1003.1) --missing--
-copylinks Copy hard and symlinks rather than linking --missing--
-hardlinks Extract symlinks as hardlinks --missing--
-symlinks Extract hardlinks as symlinks --missing--
-signed-checksum use signed chars to calculate checksum --missing--
-sparse handle file with holes effectively on store/create -S --sparse similar
-force-hole try to extract all files with holes --missing--
-to-stdout extract files to stdout -O --to-stdout
-wready wait for tape drive to become ready --missing--
-force-remove force to remove non writable files on extraction --missing--
-ask-remove ask to remove non writable files on extraction --missing--
-remove-first remove files before extraction --unlink-first
-remove-recursive remove files recursive --recursive-unlink not yet ready in star
-onull,-nullout simulate creating an achive to compute the size -f/dev/null in GNUtar worse than star (cannot do performance tests)
/*--------------------------------------------------------------------------*/
Gnu tar options that (in the single char variant) are incompatible:
BsS -F, --info-script=FILE run script at end of each tape (implies -M)
s -L, --tape-length=NUM change tape after writing NUM x 1024 bytes
s -M, --multi-volume create/list/extract multi-volume archive
s -O, --to-stdout extract files to standard output
sS (+) -P, --absolute-names don't strip leading `/'s from file names
s -S, --sparse handle sparse files efficiently
s -T, -I, --files-from=NAME get names to extract or create from file NAME
s -U, --unlink-first remove each file prior to extracting over it
s -V, --label=NAME create archive with volume name NAME
s -d, --diff, --compare find differences between archive and file system
sP -l, --one-file-system stay in local file system when creating archive
sP -o, --old-archive, --portability write a V7 format archive
B Incompatible with BSD tar
s Incompatible with star
S Incompatible with Sun's/SVr4 tar
P Incompatible with POSIX
+) This option is the only option where star deviates from other tar
implementations, but as there is no other nice way to have an option to
specify that the last record should be partial and the star option -/
is easy to remember as well as -P for Partial record is I see no need
to change star.
/*--------------------------------------------------------------------------*/
Options only found in GNUtar:
GNUtar option GNUtar description Star remarks
==================== ================================================== =======================================
Main operation mode:
-A, --catenate append tar files to an archive low importance
--delete delete from the archive (not on mag tapes!) low importance
Operation modifiers:
-W, --verify attempt to verify the archive after writing it not needed (may be implemented via rewind
and -diff in star)
--remove-files remove files after adding them to the archive not needed
-U, --unlink-first remove each file prior to extracting over it halfway ready
--recursive-unlink empty hierarchies prior to extracting directory halfway ready
-g, --listed-incremental=FILE --needed-- and important
handle new GNU-format incremental backup
--ignore-failed-read do not exit with nonzero on unreadable files --needed-- and important
Handling of file attributes:
--owner=NAME force NAME as owner for added files --nice-to-have--
--group=NAME force NAME as group for added files --nice-to-have--
--mode=CHANGES force (symbolic) mode CHANGES for added files --nice-to-have--
--no-same-owner extract files as yourself not needed
-s, --same-order sort names to extract to match archive ????
Device selection and switching:
--rsh-command=COMMAND use remote COMMAND instead of rsh makes no sense (rsh is slow)
-[0-7][lmh] specify drive and density not needed
-M, --multi-volume create/list/extract multi-volume archive --needed-- and important
-F, --info-script=FILE run script at end of each tape (implies -M) --needed-- and important
--volno-file=FILE use/update the volume number in FILE --needed-- and important
Archive format selection:
S -V, --label=NAME create archive with volume name NAME
PATTERN at list/extract time, a globbing PATTERN
-Z, --compress, --uncompress filter the archive through compress not needed (compress is rarey used)
--use-compress-program=PROG filter through PROG (must accept -d) not needed (rarely used)
Local file selection:
--null -T reads null-terminated names, disable -C ???
-X, --exclude-from=FILE exclude globbing patterns listed in FILE --nice to have--
-K, --starting-file=NAME begin at file NAME in the archive --nice to have--
--backup[=CONTROL] backup before removal, choose version control not needed
--suffix=SUFFIX backup before removal, override usual suffix not needed
Informative output:
--checkpoint print directory names while reading the archive --nice to have--
Jörg
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
js@cs.tu-berlin.de (uni) If you don't have iso-8859-1
schilling@fokus.gmd.de (work) chars I am J"org Schilling
URL: http://www.fokus.gmd.de/usr/schilling ftp://ftp.fokus.gmd.de/pub/unix
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200112052117.fB5LHxF19472>