Date: Thu, 22 Jan 1998 19:18:15 +1000 From: Stephen McKay <syssgm@dtir.qld.gov.au> To: Chuck <crtb@capecod.net>, chrw <shredder@hack.babel.dk> Cc: freebsd-isp@FreeBSD.ORG, syssgm@dtir.qld.gov.au Subject: Re: recovering tar archieve / ssh Message-ID: <199801220918.TAA01819@ogre.dtir.qld.gov.au> In-Reply-To: <Pine.FBS.3.93.1.980108170741.1518C-100000@capecod.net> from Chuck at "Thu, 08 Jan 1998 22:19:49 %2B0000" References: <Pine.FBS.3.93.1.980108170741.1518C-100000@capecod.net>
next in thread | previous in thread | raw e-mail | index | archive | help
G'day! I'm resurrecting an old thread because I didn't see any conclusion
and because so many things need answering.
On Thursday, 8th January 1998, Chuck wrote:
>
>On Thu, 8 Jan 1998, chrw wrote:
>
>> Ive been using tar thru an ssh pipe to the fbsd streamer host, like
>>
>> tar cvfpl - * | ssh -c blowfish molly 'cat > /dev/nrst0'
>>
>> everything seemed to work fine, but today i tried doing a desaster
>> recovery test.. It start extracting fine, but after 1 or 2 mins tar
>> terminates with "tar: Unexpected EOF on archive file" :(
Someone suggested adding '-e none' to prevent '~.' being recognised as an
excape. This is unnecessary since ssh only allocates pseudo ttys for
interactive connections and (from the manual):
"If no pseudo tty has been allocated, the session is trans-
parent and can be used to reliably transfer binary data."
Someone suggested -B on tar to reblock data passing through a pipe. This
is unnecessary since FreeBSD use gnu tar and it always sets -B when '-' is
used as the file name.
Strangely, no one suggested using 'dd' instead of 'cat'. cat will use a
small block size. Using dd, you can specify a large blocksize. 32KB works
well for me on both QIC and Exabyte tapes. Use the same block size on tar
also unless you know the difference between fixed and variable block size
tape drives.
Once you are happy with 'dd', then you can try 'team', from the ports
collection. This will give you much better throughput and I use it all
the time. Use something like:
tar clbf 64 - . | ssh -c blowfish molly 'team 32k 8 > /dev/nrst0'
In this example, team will write in 32KB blocks and will use 8 cooperating
processes (so the total amount of buffering is 256KB) to smooth the erratic
data flow over the network. I even use it for local tape writing as it
can smooth over the speed dips from tar itself when it hits a patch of
many small files.
>> the setup is, im doing backup from several sgi servers, onto the freebsd
>> streamer host. I cant see this should be a problem.
Have you tried reading the tape on the same machine that has the tape drive?
This will remove doubts about the network and ssh. What errors (if any)
were in the system logs of the machine with the tape drive?
>> Anyway, EVEN if it is a tape error, i cannot rest assure that my backups
>> are good untill i have to verified every single tape i got :(
>>
>> Maybe tar can verify each archieve it makes automatically, in the
>> future..? Ill go check the man page.
Yes, you should be reading back your tapes. Gnu tar has a --compare flag
that might be useful, but some files would have changed since the backup.
I am happy to just read the tape to verify that there are no media errors.
Of course, I have restored some files occasionally to test that my scripts
are not totally broken and to be sure I've used the norewind device when
I should be.
>Sounds familiar. I have a FreeBSD 2.2.2 PC, Tomato, with a TR4 (SCSI)
>drive named nrst0. I run dump on it whenever I feel like it, and have
>restored files from way down the tape. It goes from one end to the other,
>then turns around and goes the other way. I don't know how many tracks
>there are --it takes quite a few end-to-end trips-- but it works.
>
>But, when I run dump over the (thinwire) Ethernet from my other
>FreeBSD 2.2.2 PC, Daisy, to the same nrst0 via rmt, first, it hiccups
>about every two seconds, and averages about 50 kBps. *** AND *** it
>only goes to the end of the tape once and does not turn around. It says
>that Tomato's operator should mount a fresh tape. This in spite of dump's
>-a option.
I had strange failures remote dumping from FreeBSD 2.1.5 box to Solaris 2.5
so I gave up on rmt. Since then I've only used rsh or ssh as the data
transport. Dump to stdout then pipe it to team on the remote box using
rsh or ssh and things should work better and a lot faster.
>So I've taken to running dump from Daisy to a BIG file on Tomato, and
>then writing that file to tape using dd. Very fast.
For my home FreeBSD boxes, I use dump, rsh and team all the time and have
no problems, and it is fast (well as fast as the tape drives can handle
anyway).
> ABHOR SECRECY -- DEFEND PRIVACY
Wise words. Oops, is that the US secret service kicking in my door?
No sir, I had no intention of encrypting my grocery list! Ouch! Ouch!
Stephen.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199801220918.TAA01819>