From owner-freebsd-stable@FreeBSD.ORG Thu Oct 30 05:26:40 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5E50106564A for ; Thu, 30 Oct 2008 05:26:40 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.225]) by mx1.freebsd.org (Postfix) with ESMTP id 91C6F8FC0C for ; Thu, 30 Oct 2008 05:26:40 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so376804rvf.43 for ; Wed, 29 Oct 2008 22:26:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=uU4OPK4LLCDm4PJzr797gSKc8ceUMyUXhUJqeMuliME=; b=Go4AN4n8UArojf4DQYXXViGovpHGQxW4BWIKhQkcPYDmII7mUfNSyH9VY9a/j5bGxb qcHiuarlSC95bIkIEucK5EFD7s/7SZ4F48BhdQkJ+1ELj2c36fMDPFOtWpiJj8KI48Kp L17GpKAhS/Oaom1Er2xVyXdEn0InHv6AnBZNg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=X6/SvT/Q8js81cvmQshtph5O/XBzr1X2K4foqrUM/BPzPe9M7ITm0xc86ZebkDAl8q KSOrnNxLDmysfvNEkvT/haGIZhIwU2WM74vS2segaBP4BBVuM3HGK57eusKOwSzIYMVt ELucKByQTx2qFxqIiFs74PabIePV0Q2P0Nu3g= Received: by 10.115.73.20 with SMTP id a20mr7965451wal.142.1225342835267; Wed, 29 Oct 2008 22:00:35 -0700 (PDT) Received: by 10.114.121.12 with HTTP; Wed, 29 Oct 2008 22:00:33 -0700 (PDT) Message-ID: Date: Thu, 30 Oct 2008 05:00:33 +0000 From: "Freddie Cash" To: freebsd-stable@freebsd.org In-Reply-To: <490912CC.6000406@modulus.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20081029231926.GA35188@0lsen.net> <490907AC.5070303@freebsd.org> <490912CC.6000406@modulus.org> Subject: Re: Anyone used rsync scriptology for incremental backup? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Oct 2008 05:26:40 -0000 On Thu, Oct 30, 2008 at 1:50 AM, Andrew Snow wrote: > In this way, each day we generate a batch file that lets us step back one > day. The diffs themselves, compressed with gzip, and extremely space > efficient. We can step back potentially hundreds of days, though it seems > to throw errors sometimes when backing up Windows boxes, which I haven't > tracked down yet. > > But to be honest, soon you can save yourself a lot of hassle by simply using > ZFS and taking snapshots. It'll be faster, and with compression very space > efficient. That's exactly what we do, use ZFS and RSync. We have a ZFS /storage/backup filesystem, with directories for each remote site, and sub-directories for each server to be backed up. Each night we snapshot the directory, then run rsync to backup each server. Snapshots are named with the current date. For 80 FreeBSD and Linux servers, we average 10 GB of changed data a night. No muss, no fuss. We've used it to restore entire servers (boot off Knoppix/Frenzy CD, format partitions, rsync back), individual files (no mounting required, just cd into the .zfs/snapshot/snapshotname directory and scp the file), and even once to restore the permissions on a pair of servers where a clueless admin "chmod -R user /home" and "chmod -R 777 /home". Our backup script is pretty much just a double-for loop that scans a set of site-name directories for server config files, and runs rsync in parallel (1 per remote site). We we looking into using variations on rsnapshot, custom squashfs/hardlink stuff, and other solutions, but once we started using ZFS, we stopped looking down those roads. We were able to do in 3 days of testing and scripting what we hadn't been able to do in almost a month of research and testing. -- Freddie Cash fjwcash@gmail.com