From owner-freebsd-stable@FreeBSD.ORG Fri Mar 1 17:55:33 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3EF46270 for ; Fri, 1 Mar 2013 17:55:33 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-ee0-f50.google.com (mail-ee0-f50.google.com [74.125.83.50]) by mx1.freebsd.org (Postfix) with ESMTP id CA24DD2D for ; Fri, 1 Mar 2013 17:55:32 +0000 (UTC) Received: by mail-ee0-f50.google.com with SMTP id e51so2633723eek.37 for ; Fri, 01 Mar 2013 09:55:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=uCWyI+xvYUgehAukdUcKbVVFw7uCGivX29AhRu1GOY4=; b=VuRXWAAtFjIHPhlTeqTeXNXYP0HGa3u/iE2BA+MR3Np5Mbe0nsE8RsGRMMNFHPro82 gPxkI+wnMc5zPstNAhYF2HzEhJMkBKgsSGXtr8fwMAS5wFy4xmz+xsUE8bF/4bqWhyIx l4NIuhdotv4QytXLEnIONAMOIn3ePoFElbtCmg/UGDIIpuQLyiljDLAxt531YfELJL1p vyngeuzKySVAXwIkJTQBg5EvBNq04/ma03o/xN5Fyehe+CN/1TyvOrYx+RLEtZ2FJoVK TBWfJUkWzCZtP4sRHLXSMSfQrG2MsAoUKyPkE5RzWLiyuHaKifIXa8UVuwY1Vq+xxkHh v+5A== X-Received: by 10.14.216.198 with SMTP id g46mr29962827eep.30.1362160526011; Fri, 01 Mar 2013 09:55:26 -0800 (PST) Received: from [192.168.1.128] ([91.196.229.122]) by mx.google.com with ESMTPS id s3sm18303048eem.4.2013.03.01.09.55.24 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 01 Mar 2013 09:55:24 -0800 (PST) Message-ID: <5130EB8A.7060706@gmail.com> Date: Fri, 01 Mar 2013 19:55:22 +0200 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:19.0) Gecko/20100101 Firefox/19.0 SeaMonkey/2.16 MIME-Version: 1.0 To: Karl Denninger , freebsd-stable@freebsd.org Subject: Re: Musings on ZFS Backup strategies References: <5130BA35.5060809@denninger.net> In-Reply-To: <5130BA35.5060809@denninger.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Mar 2013 17:55:33 -0000 01.03.2013 16:24, Karl Denninger: > Dabbling with ZFS now, and giving some thought to how to handle backup > strategies. > > ZFS' snapshot capabilities have forced me to re-think the way that I've > handled this. Previously near-line (and offline) backup was focused on > being able to handle both disasters (e.g. RAID adapter goes nuts and > scribbles on the entire contents of the array), a double-disk (or worse) > failure, or the obvious (e.g. fire, etc) along with the "aw crap, I just > rm -rf'd something I'd rather not!" > > ZFS makes snapshots very cheap, which means you can resolve the "aw > crap" situation without resorting to backups at all. This turns the > backup situation into a disaster recovery one. > > And that in turn seems to say that the ideal strategy looks more like: > > Take a base snapshot immediately and zfs send it to offline storage. > Take an incremental at some interval (appropriate for disaster recovery) > and zfs send THAT to stable storage. > > If I then restore the base and snapshot, I get back to where I was when > the latest snapshot was taken. I don't need to keep the incremental > snapshot for longer than it takes to zfs send it, so I can do: > > zfs snapshot pool/some-filesystem@unique-label > zfs send -i pool/some-filesystem@base pool/some-filesystem@unique-label > zfs destroy pool/some-filesystem@unique-label > > and that seems to work (and restore) just fine. Yes, I'm working with backups the same way, I wrote a simple script that synchronizes two filesystems between distant servers. I also use the same script to synchronize bushy filesystems (with hundred thousands of files) where rsync produces a too big load for synchronizing. https://github.com/kworr/zfSnap/commit/08d8b499dbc2527a652cddbc601c7ee8c0c23301 I left it where it was but I was also planning to write some purger for snapshots that would automatically purge snapshots when pool gets low on space. Never hit that yet. > Am I looking at this the right way here? Provided that the base backup > and incremental are both readable, it appears that I have the disaster > case covered, and the online snapshot increments and retention are > easily adjusted and cover the "oops" situations without having to resort > to the backups at all. > > This in turn means that keeping more than two incremental dumps offline > has little or no value; the second merely being taken to insure that > there is always at least one that has been written to completion without > error to apply on top of the base. That in turn makes the backup > storage requirement based only on entropy in the filesystem and not time > (where the "tower of Hanoi" style dump hierarchy imposed both a time AND > entropy cost on backup media.) Well, snapshots can pose a value in a longer timeframe depending on data. Being able to restore some file accidentally deleted two month ago already saved 2k$ for one of our customers. -- Sphinx of black quartz, judge my vow.