Date: Fri, 13 Dec 2002 21:16:08 -0500 From: George Georgalis <george@galis.org> To: Simon <simon@optinet.com> Cc: Marcus Reid <marcus@blazingdot.com>, "freebsd-isp@FreeBSD.ORG" <freebsd-isp@FreeBSD.ORG> Subject: Re: network backup Message-ID: <20021214021608.GA29015@trot.local> In-Reply-To: <20021213164339.5420343ED1@mx1.FreeBSD.org> References: <20021213142728.GA92171@blazingdot.com> <20021213164339.5420343ED1@mx1.FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi - I'd use an actual db or a flat text file index on the source side. For the daily backups archive and compress the files that have changed, send them over, and update the db. At some threshold, extract the remote and rsync the whole thing (ugh), it might be sensible to rsync sections of the filesystem. Some versions of rsync require --block-size on command line to make that work properly, I think the default is way too small too. Another approach might be compressed file systems. In linux I think there is a module/mount parameter so the kernel compresses/extracts as it writes to disk, then you could just rsync -z the whole thing and have a compressed remote too. Not sure if that is available in FBSD. I've never used this but another feature you might want to incorporate are snapshots, the rsync might take a 'long' time... http://www.au.freebsd.org/doc/en_US.ISO8859-1/books/handbook/snapshots.html // George On Fri, Dec 13, 2002 at 11:45:31AM -0500, Simon wrote: > >They don't change a whole lot, but they do change and new ones are >uploaded. What I'm trying to do is build massive network backup server, >backing up 6+ servers simultaneously. Each server has about 50-100gb >worth of data to be backed up daily. I did look at dump, looks great, the >only problem is that it can't backup individual directories, rather whole >filesystems, if I didn't misread something. > >-Simon > > >On Fri, 13 Dec 2002 06:27:29 -0800, Marcus Reid wrote: > >>On Fri, Dec 13, 2002 at 03:22:06AM -0500, Simon wrote: >>> >>> rsync is nice, but it can't (afaik) compress data being synced on the fly to >>> save disk space :-( Is there anything out there which works like rsync and >>> can compress on the fly to space disk space? having 100GB of text files >>> compressed can save quite a few gigs. >> >>The -z option for rsync will compress for the transfer, but the file will be >>decompressed before it is written to the disk (having the same file at both >>ends being the point of rsync..) >> >>If the files change a lot and you need to do a lot of incremental backups, >>I'd use rsync and just give up the space to store the uncompressed files if >>you can. >> >>Otherwise, if the files are generally small or rarely change, you could pipe >>the output of dump over ssh and through gzip and into an archive, and then do >>incremental dumps the same way. Can be automated nicely if you set up an >>authorized keypair so you don't need to enter a password. >> >>Note that I've removed the hackers list from the recipients. This is totally >>off-topic for that list. >> >>Marcus >> >>> >>> Thanks, >>> Simon >> > > > > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-isp" in the body of the message > -- GEORGE GEORGALIS, System Admin/Architect cell: 347-451-8229 Security Services, Web, Mail, mailto:george@galis.org Multimedia, DB, DNS and Metrics. http://www.galis.org/george To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021214021608.GA29015>