Date: Fri, 12 Mar 2004 00:14:51 -0500 From: Bob Johnson <bob89@bobj.org> To: Dany Nativel <dany_list@natzo.com> Cc: questions@freebsd.org Subject: Re: Backups: rsync, software RAID, other strategies? Message-ID: <200403120014.51555.bob89@bobj.org> In-Reply-To: <404C1F34.3010606@natzo.com> References: <200403070152.55770.bob89@bobj.org> <404C1F34.3010606@natzo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks for pointing me to these tools. I may not use Unison on this particular project, but I think it will be the next project after the backup server is finalized! - Bob On Monday 08 March 2004 02:22 am, Dany Nativel <Dany Nativel <dany_list@natzo.com>> wrote: > Hi Bob, > > I use the following configuration on my file server. > It's a small cube based on a low power mini-ITX EPIA 5000 motherboard > (fanless) and running two 120GB HDDs. > I looked at RAID but it doesn't help solving one of the potential > issue ... me, user removing files that are not supposed to be removed > so incremental backup is a plus. This baby is powered by FreeBSD > 5.2.1 (used to be Debian). > > DISK 0 (live) > 128M ad0s1a / > 512M ad0s1b swap > 128M ad0s1d /var > 200M ad0s1e /tmp > 3200MB ad0s1f /usr > 110GB ad0s2d /data > > DISK 1 (Backup) > ad2s1d 128M /backup/os/root > ad2s0b 512M swap > ad2s1e 128M /backup/os/var > ad2s1f 200M /backup/os/tmp > ad2s1g 3199M /backup/os/usr > ad2s2d 108G /backup/data/backup > > > I use 3 different programs : > - Unison (http://www.cis.upenn.edu/~bcpierce/unison/): 2-way > synchronization using rsync/ssh, multi-platform graphical interface. > I can have the same files on my file server, laptop running win2k as > well as my desktop running BSD. Very convenient especially with > laptops when you can't be connected all the time.Very fast too (only > transmit diffs) - rsync (man rsync) : typical rsync that will mirror > the source to the destination > - rdiff-backup (http://rdiff-backup.stanford.edu/index.html): it's > based on rsync but you get the advantage of incremental backups so > you can restore from a specific date. You can also purge the backup > by removing old stuff. No fancy file format, just .gz for the > diffs. > > Here is how I use those tools : > /data/current/user0_live (DISK 0) <----> UNISON : 2-way > synchronization with laptop/desktop > /data/current/user0_incremental (DISK 0) <----> RDIFF-BACKUP : > incremental backup of user0_live using RDIFF system > /backup/data/backup/user0_incremental (DISK1) <----> RSYNC : quick > mirror of the already incremental backup > > /backup/data/backup/pictures (DISK1) <----> RDIFF-BACKUP > : in this case, rdiff-backup between drive0 and drive1 (no > incremental on disk0) > > /backup/os/root (DISK1) <----> DUMP : 1:1 copy of the live root > fs /backup/os/tmp (DISK1) > /backup/os/var (DISK1) > > /backup/os/usr (DISK1) <----> RSYNC : (with -delete option) > for a quick mirror of current /usr > > PS: for user0, there are two copies of the data on disk0, 1 live > synchronized with Unison and another one which is an incremental of > the first one. For less critical data (like /data/current/pictures) > I use rdiff-backup between disk0 and disk1. In that case I would lose > incremental backups if disk1 fails. > > I've simulated a crisis situation by removing the drive0 and swapping > it with drive1. It worked (except for those entries in fstab > referring to disk1). > > I have a cron job taking of the different backups at night. > > #!/bin/sh > > # Duplicate / > # erase slive before mirroring, any other way? > umount /backup/os/root > newfs /dev/ad2s1a > mount /backup/os/root > # dump with 'live filesystem' option > dump -0 -L -f - /dev/ad0s1a | (cd /backup/os/root ; restore -r -v -f > -) > > # Duplicate /var > umount /backup/os/var > newfs /dev/ad2s1d > mount /backup/os/var > dump -0 -L -f - /dev/ad0s1d | (cd /backup/os/var ; restore -r -v -f > -) > > # Duplicate /tmp (probably a wate of time) > umount /backup/os/tmp > newfs /dev/ad2s1e > mount /backup/os/tmp > dump -0 -L -f - /dev/ad0s1e | (cd /backup/os/tmp ; restore -r -v -f > -) > > # incremental backup of the ./pictures directory on the second drive > rdiff-backup /data/current/pictures /backup/data/backup/pictures > > # First, incremental of the user0_live dir on the same drive then > rsync on the second drive > rdiff-backup /data/current/user0_live /data/current/user0_incremental > rsync -a --delete /data/current/user0_incremental/ > /backup/data/backup/user0_incremental > > > The only I don't like is the NEWFS command. Is there a cleaner way to > do this dump ? > > I use this configuration is a non-critical installation (my house) > but it has been serving its purpose so far. > Dany > > > PS: On the rdiff-backup webpage there is a link to another tool call > duplicity (http://rdiff-backup.stanford.edu/duplicity.html). You can > do remote backup but in that case the image can be stored on a remote > FTP server and encrypted with GPG... sweet if you're planning to use > your ISP's disk space for backups! > > Bob Johnson wrote: > >A bunch of related questions: > > > >I'm setting up a small mail and file server. The mail server part > > will be Courier, while the file server part will primarily be used > > via NFS and Samba to store backups of my desktop and laptop > > computers. > > > >The system has a pair of WD1600JB 160 GB ATA 100 drives in it, both > > on a single Promise PDC20268 UDMA100 controller, but each on a > > separate channel (i.e. both are masters with no slaves). My plan > > is to use one of the drives as a backup for the other. I want to > > use a backup method that creates a mirror of the working drive so > > that if it fails, I can simply mount the backup in place of the > > working drive, and get back in business. The operating system will > > (probably) not be on either of these drives, they will only host > > /home where mail and backup files will reside. > > > >I've tentatively decided to synchronize the mirror to the working > > drive with rsync run once a night (perhaps more often later). This > > risks losing up to a day of mail, but that's probably not the end > > of the world. The reasoning I used was that if I use software > > RAID, an unexpected power failure during a large write operation > > (yes, it will be on a UPS) could corrupt both drives. Running > > rsync once a night would reduce the risk of a failure that damaged > > both drives, since their write activity would not be so strongly > > correlated. > > > >Is my fear of losing both drives in a software RAID (mirrored > > drives) configuration a reasonable one? Or is that not going to > > happen? > > > >If I use rsync with -delete to maintain a mirror of the primary > > drive, what happens when the primary drive fails? Is there a > > scenario that causes rsync to duplicate all the missing data on the > > primary drive by deleting it from the mirror drive (I've heard of > > this happening to someone, but I believe he was using a homegrown > > perl script rather than rsync). > > > >Is Courier compatible with this scheme? Or does it care about inode > >numbers or some such thing that will make the backup copy useless? > > > >Is there any chance it would make sense to use the Coda file system > > for this? > > > >Do any of the answers change if the mail server ends up on a remote > >system, but I still want the maildirs backed up on the file server? > > > >Any other suggestions that don't involve buying new hardware? An > > IDE RAID controller would be nice, but buying one isn't on my list > > of things to do. But if I DID break down and buy a new controller > > card, what should it be? > > > >Thanks, > > > >- Bob > >_______________________________________________ > >freebsd-questions@freebsd.org mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-questions > >To unsubscribe, send any mail to > > "freebsd-questions-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200403120014.51555.bob89>