From owner-freebsd-stable@FreeBSD.ORG Thu Oct 30 15:18:07 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4AE991065686 for ; Thu, 30 Oct 2008 15:18:07 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from gv-out-0910.google.com (gv-out-0910.google.com [216.239.58.186]) by mx1.freebsd.org (Postfix) with ESMTP id C063B8FC14 for ; Thu, 30 Oct 2008 15:18:06 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by gv-out-0910.google.com with SMTP id n8so136813gve.39 for ; Thu, 30 Oct 2008 08:18:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-pgp-agent:x-mailer; bh=4cGxjupXdSgrH/oIJlSDwbhJvrYjjm9YJTJD3WeTakg=; b=qomhAzcvjLYIjHmUYkZxAt6blS8rFzQR6XS9lO9QYGa6FVg/RnPi5UQi2OnoZXTek4 /7lFC3R1vZageMwNMxa3F9MBN0W1ZotOhLkYeAOt9MctyYbu3dsdhcttk0crwoA3GHx1 0yRZhf/S8vIgl9b2jGbMlukgfbLHOZv+i3WdU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-pgp-agent:x-mailer; b=LE6a1as1s/THMSDjJtUQCzYJpuwEAxSnAP1G1ViL0Qx+m/sl4B/6b25Jtjkenh+brF qvIenWcKVqB0r+9hfqSXMXOuQw9TRtNvamzaF1oCUinQBUXRLDKMImpfDjbFkhAroCax wstxIGTZrCqmTW7bBapbFLLC84GpBUJMOk5gY= Received: by 10.103.11.5 with SMTP id o5mr4925133mui.74.1225379884999; Thu, 30 Oct 2008 08:18:04 -0700 (PDT) Received: from ndenev.cmotd.com (blah.sun-fish.com [217.18.249.150]) by mx.google.com with ESMTPS id w5sm5877386mue.10.2008.10.30.08.18.02 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 30 Oct 2008 08:18:03 -0700 (PDT) Message-Id: <444B4891-057B-4E09-99A1-A50A1187109E@gmail.com> From: Nikolay Denev To: Freddie Cash In-Reply-To: <200810300804.31186.fjwcash@gmail.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Date: Thu, 30 Oct 2008 17:18:00 +0200 References: <20081029231926.GA35188@0lsen.net> <95550BEC-DB92-4C68-8409-3DFF7C0B86C0@gmail.com> <200810300804.31186.fjwcash@gmail.com> X-Pgp-Agent: GPGMail d53 (v53, Leopard) X-Mailer: Apple Mail (2.929.2) Cc: freebsd-stable@freebsd.org Subject: Re: Anyone used rsync scriptology for incremental backup? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Oct 2008 15:18:07 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30 Oct, 2008, at 17:04 , Freddie Cash wrote: > On October 30, 2008 01:25 am Nikolay Denev wrote: >> On 30 Oct, 2008, at 07:00 , Freddie Cash wrote: >>> On Thu, Oct 30, 2008 at 1:50 AM, Andrew Snow >>> >>> wrote: >>>> In this way, each day we generate a batch file that lets us step >>>> back one >>>> day. The diffs themselves, compressed with gzip, and extremely >>>> space efficient. We can step back potentially hundreds of days, >>>> though it seems >>>> to throw errors sometimes when backing up Windows boxes, which I >>>> haven't >>>> tracked down yet. >>>> >>>> But to be honest, soon you can save yourself a lot of hassle by >>>> simply using >>>> ZFS and taking snapshots. It'll be faster, and with compression >>>> very space >>>> efficient. >>> >>> That's exactly what we do, use ZFS and RSync. We have a ZFS >>> /storage/backup filesystem, with directories for each remote site, >>> and sub-directories for each server to be backed up. >>> >>> Each night we snapshot the directory, then run rsync to backup each >>> server. Snapshots are named with the current date. For 80 FreeBSD >>> and Linux servers, we average 10 GB of changed data a night. >>> >>> No muss, no fuss. We've used it to restore entire servers (boot off >>> Knoppix/Frenzy CD, format partitions, rsync back), individual files >>> (no mounting required, just cd into the .zfs/snapshot/snapshotname >>> directory and scp the file), and even once to restore the >>> permissions >>> on a pair of servers where a clueless admin "chmod -R user /home" >>> and >>> "chmod -R 777 /home". >>> >>> Our backup script is pretty much just a double-for loop that scans a >>> set of site-name directories for server config files, and runs rsync >>> in parallel (1 per remote site). >>> >>> We we looking into using variations on rsnapshot, custom >>> squashfs/hardlink stuff, and other solutions, but once we started >>> using ZFS, we stopped looking down those roads. We were able to do >>> in 3 days of testing and scripting what we hadn't been able to do in >>> almost a month of research and testing. > >> Do you experience problems with the snapshots? >> Last time I tried something similiar for backups the bachine >> began to spit errors after a few days of snapshots. >> >> http://lists.freebsd.org/pipermail/freebsd-fs/2008-February/004413.html > > We have 72 daily snapshots so far. Have had up to 30 of them mounted > read-only while looking for the right version of a file to restore. > > These are ZFS snapshots, very different from UFS snapshots. > > -- > Freddie Cash > fjwcash@gmail.com Yes, Mine were zfs snapshots too, and I've never managed to create more than a few days worth of snapshots before the machine start to print "bad file descriptor" errors while trying to access the snapshot directory. But I guess (hope) this problem does not exist anymore when you are able to do 72 snapshots. - -- Regards, Nikolay Denev -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (Darwin) iEYEARECAAYFAkkJ0CgACgkQHNAJ/fLbfrn9pACfSVFPyiHDosaK6FdOdfgo8onL Ia4An1qUoSnOq/yjIGC5fMngT+PPkEKk =bWqT -----END PGP SIGNATURE-----