Date: Fri, 13 Dec 2002 15:33:47 -0500 From: "Charles Swiger" <cswiger@mac.com> To: <freebsd-isp@FreeBSD.ORG> Cc: "Simon" <simon@optinet.com> Subject: Re: network backup Message-ID: <013501c2a2e6$f0d9ab50$0301a8c0@prime> References: <20021213180527.6B64543EA9@mx1.FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Simon wrote:
> I don't *just* want to make a backup. I want to back it up using an
> efficient method because I'm dealing with terabytes of data. I can't
> just back so much data daily over network and compress it. I have
> to do incremental backups and compression on the fly is a must to
> save disk space. I understand that rsync needs to be able to read
> local copy of the backup in order to sync files correctly, however,
> this can be done using on-fly compression, I just don't have/know the
> right tool.
[ ... ]
I have several considerations which you might want to go over in more detail:
1) Is the client representation of the data fixed, or can you do things like
compress the data files beforehand?
2) What changes would be typical in a daily incremental backup? How much data
changes per day? And are old files changing; being appended to; or are new
files being created (ala the output of rotatelogs on a webserver's log files)?
3) Are you using sparse files?
4) Do you want to copy the data to a central server and then backup that one
place, or do you want to backup data directly from the client machines?
Give us some idea of the scope, although the notion of TB worth of data implies
you should be considering commercial backup software-- eg, Legato Networker, or
Veritas as others have mentioned.
5) Is the security of the backup network traffic important?
6) Hardware: You're also probably looking for at least a multitape silo loader,
more probably a full roboticized tape library, if you need to backup all data at
one complete central point. If you can backup individual clients (lets say you
have a dozen webservers), you can divide up the problem and use a dozen tape
drives (say 200 GB capacity per tape), you've got 2.4 TB of local tape storage
available, which would save you from having to architect a backup network to
handle all of the backup traffic.
> Dump is a mess to work with, it doesn't work with directories nor with
> single archive file. You need to keep creating new dumps using different
> backup levels and I don't know how you will restore files for x user
> using all those little dumps when you need to efficiently.
Um. You don't want to do a full backup of terabytes worth of data every night,
correct? The reason why dump has different levels is so that you can take
incremental dumps, which is what you asked for.
Maybe these scripts will help you understand:
In crontab:
1 2 * * * /export/Backups/Scripts/pong_inc.sh
#1 2 * * * /export/Backups/Scripts/pong_full.sh
I switch which script runs when a tape fills up or when I want to take a full
backup (to take offsite or before making a big change, etc).
2-pong# cd /export/Backups/Scripts
backup_freebsd.sh* nohup.out pong_inc.sh*
intent_push.sh* pong_full.sh*
# Here is the full dump at level 0, plus a few useful commands
# commented out so that the other admins I work with don't have
# to figure things out. This is for a Solaris 8 system; under FreeBSD
# one would use dump instead of 'ufsdump', etc.
4-pong# cat pong_full.sh
#! /bin/sh
#mt -f /dev/rmt/0ubn erase
mt -f /dev/rmt/0ubn rewind
/usr/sbin/ufsdump 0cfu /dev/rmt/0ubn /dev/md/rdsk/d10
mt -f /dev/rmt/0ubn asf 1
/usr/sbin/ufsdump 0cfu /dev/rmt/0ubn /dev/md/rdsk/d20
mt -f /dev/rmt/0ubn asf 2
/usr/sbin/ufsdump 0cfu /dev/rmt/0ubn /dev/md/rdsk/d30
mt -f /dev/rmt/0ubn asf 3
/usr/sbin/ufsdump 0cfu /dev/rmt/0ubn /dev/md/rdsk/d40
mt -f /dev/rmt/0ubn asf 4
/usr/sbin/ufsdump 0cfu /dev/rmt/0ubn /dev/md/rdsk/d60
mt -f /dev/rmt/0ubn asf 5
/usr/sbin/ufsdump 0cfu /dev/rmt/0ubn /dev/md/rdsk/d70
# mt -f /dev/rmt/0ubn asf 4
# ufsrestore ib 126 /dev/rmt/0ub
Here's the incremental backup script:
#! /bin/sh
mt -f /dev/rmt/0ubn eom
/usr/sbin/ufsdump 1cfu /dev/rmt/0ubn /dev/md/rdsk/d10
mt -f /dev/rmt/0ubn eom
/usr/sbin/ufsdump 1cfu /dev/rmt/0ubn /dev/md/rdsk/d20
mt -f /dev/rmt/0ubn eom
/usr/sbin/ufsdump 1cfu /dev/rmt/0ubn /dev/md/rdsk/d30
mt -f /dev/rmt/0ubn eom
/usr/sbin/ufsdump 1cfu /dev/rmt/0ubn /dev/md/rdsk/d40
mt -f /dev/rmt/0ubn eom
/usr/sbin/ufsdump 1cfu /dev/rmt/0ubn /dev/md/rdsk/d60
mt -f /dev/rmt/0ubn eom
/usr/sbin/ufsdump 1cfu /dev/rmt/0ubn /dev/md/rdsk/d70
# mt -f /dev/rmt/0ubn asf 4
# ufsrestore ib 126 /dev/rmt/0ub
# There was a problem with non-rewind semantics for my tape drive, so I
# worked around it (thus all of the mt eom and mt asf stuff).
# I think it's been fixed since, but I'm not going to rock that boat again.
Also, a freebsd backup script which handles all of the parts of the servers
which I customize:
6-pong# cat backup_freebsd.sh
#! /bin/sh
PATH=/usr/local/bin:/usr/bin:/usr/sbin:/usr/libexec:/usr/lib:/bin:/sbin
FROM=${1-pi}
echo "Backing up ${FROM}"
DEST=/export/Backups/${FROM}/
mkdir ${DEST}
mkdir ${DEST}var
mkdir ${DEST}usr
mkdir ${DEST}usr/src
mkdir ${DEST}usr/src/sys
mkdir ${DEST}usr/src/sys/i386
mkdir ${DEST}usr/local
mkdir ${DEST}usr/local/etc
rsync -a ${FROM}:/etc ${DEST}
rsync -a ${FROM}:/var/log ${DEST}var/
rsync -a ${FROM}:/var/named ${DEST}var/
rsync -a ${FROM}:/usr/local/etc/rc.d ${DEST}usr/local/etc/
rsync -a ${FROM}:/usr/src/sys/i386/conf ${DEST}usr/src/sys/i386/
For what it's worth, this is backing up about two dozen machines to a backup
server which has 10 disks (four 15K RPM 36GB in a RAID-1,0; four 10K RPM 18GB
@ -1,0; two 10K RPM 9's @ RAID-1 for the OS) and is using a Quantum sDLT 110/220
GB tape drive; via four SCSI channels over 66MHz/wide PCI. A Sun E450,
specificly.
-Chuck
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?013501c2a2e6$f0d9ab50$0301a8c0>