From owner-freebsd-stable@FreeBSD.ORG Thu Oct 30 05:19:51 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17A0B1065673 for ; Thu, 30 Oct 2008 05:19:51 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.237]) by mx1.freebsd.org (Postfix) with ESMTP id D7A198FC08 for ; Thu, 30 Oct 2008 05:19:50 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so374262rvf.43 for ; Wed, 29 Oct 2008 22:19:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=uU4OPK4LLCDm4PJzr797gSKc8ceUMyUXhUJqeMuliME=; b=ohYz/XZNpL3RKg8XDIVjsMj9XiM1lZ0SBNbedWp1TgPe/bfj9s2lSSvzYrDQXYqRIX hqgVTTR214mjIgOciuw32xZ16OM0TaFhOIrvojoDuCCZQswKL5SIb9IqWLocIqC9+/3V 50Fyz7UtVBr5fwy+6ElxC3JfuHi3IjSkrMawU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=ugJATifNnz3YY9qLbGrIjXV5UK51TADaUCPHY8OBRjkdhitIygii4Vgb5c3tom2Z1M vPvnCRj5h8z4vgjWqFOj4d4kBcGBru2EBKXqeGaKuJDvfcTgVIWqyrPsX5YDhlJHhoQZ NxGT2VriLYVBaRk3NdZ9Gbj2zMQ+4UdTXQevU= Received: by 10.141.18.10 with SMTP id v10mr1371735rvi.3.1225342732887; Wed, 29 Oct 2008 21:58:52 -0700 (PDT) Received: by 10.140.200.11 with HTTP; Wed, 29 Oct 2008 21:58:52 -0700 (PDT) Message-ID: Date: Thu, 30 Oct 2008 04:58:52 +0000 From: "Freddie Cash" To: freebsd-stable@freebsd.org In-Reply-To: <490912CC.6000406@modulus.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20081029231926.GA35188@0lsen.net> <490907AC.5070303@freebsd.org> <490912CC.6000406@modulus.org> Subject: Re: Anyone used rsync scriptology for incremental backup? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Oct 2008 05:19:51 -0000 On Thu, Oct 30, 2008 at 1:50 AM, Andrew Snow wrote: > In this way, each day we generate a batch file that lets us step back one > day. The diffs themselves, compressed with gzip, and extremely space > efficient. We can step back potentially hundreds of days, though it seems > to throw errors sometimes when backing up Windows boxes, which I haven't > tracked down yet. > > But to be honest, soon you can save yourself a lot of hassle by simply using > ZFS and taking snapshots. It'll be faster, and with compression very space > efficient. That's exactly what we do, use ZFS and RSync. We have a ZFS /storage/backup filesystem, with directories for each remote site, and sub-directories for each server to be backed up. Each night we snapshot the directory, then run rsync to backup each server. Snapshots are named with the current date. For 80 FreeBSD and Linux servers, we average 10 GB of changed data a night. No muss, no fuss. We've used it to restore entire servers (boot off Knoppix/Frenzy CD, format partitions, rsync back), individual files (no mounting required, just cd into the .zfs/snapshot/snapshotname directory and scp the file), and even once to restore the permissions on a pair of servers where a clueless admin "chmod -R user /home" and "chmod -R 777 /home". Our backup script is pretty much just a double-for loop that scans a set of site-name directories for server config files, and runs rsync in parallel (1 per remote site). We we looking into using variations on rsnapshot, custom squashfs/hardlink stuff, and other solutions, but once we started using ZFS, we stopped looking down those roads. We were able to do in 3 days of testing and scripting what we hadn't been able to do in almost a month of research and testing. -- Freddie Cash fjwcash@gmail.com