From owner-freebsd-fs@FreeBSD.ORG Sat Dec 3 05:29:31 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16679106566C for ; Sat, 3 Dec 2011 05:29:31 +0000 (UTC) (envelope-from techchavez@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 94E908FC0C for ; Sat, 3 Dec 2011 05:29:30 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so2694947wgb.31 for ; Fri, 02 Dec 2011 21:29:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=h3tUU5558abFVwA20wQMjvcuQUYGC5a++JCA/sFK82k=; b=sXF8lahFCWSkYIW163nvFuOyweXoTQ/FNXV+zZ6ax6F+Q6zIlLpFID5snnvV9KJU6s UbJ0akNJo+M6pl8404rCwFSOqrqFEOkY0TWiWSXv5XGVlnQe1R7ZbmW6XSfaiMoxHpGw rRFY8C5ANivQO0u/kzvLzVH6ZEh+gdbE0rOtg= MIME-Version: 1.0 Received: by 10.216.14.37 with SMTP id c37mr242177wec.86.1322890169281; Fri, 02 Dec 2011 21:29:29 -0800 (PST) Received: by 10.180.94.197 with HTTP; Fri, 2 Dec 2011 21:29:29 -0800 (PST) In-Reply-To: <4ED92139.6010900@brockmann-consult.de> References: <4ED77B09.1090709@brockmann-consult.de> <4ED87FA6.6010408@brockmann-consult.de> <20111202142656.GA7104@e4310> <20111202145008.GA27853@icarus.home.lan> <4ED92139.6010900@brockmann-consult.de> Date: Fri, 2 Dec 2011 22:29:29 -0700 Message-ID: From: Techie To: Peter Maloney Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: ZFS dedup and replication X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 05:29:31 -0000 Hey Peter, Thanks for your informative response. This is odd, I have been waiting for a response to this question for a few days and these messages just came through..I am glad they did. Anyhow please allow me to explain the whole "tar" thing. I regret it because no one even addressed the DDT, Dedup Table part of it. You see I wanted to use ZFS as a deduplication disk target for my backup applications and use the native replication capabilities of ZFS to replicate the virtual backup cartridges. All modern backup apps leverage disk as a backup target but some don't do replication. My idea was to use ZFS to do this. However after testing I came to the realization that ZFS deduplication is NOT ideal for "deduping" third party backup streams. From what I read this is due to the fact that backup applications put their own metadata in the streams and throw off the block alignment. Products like Data Domain and Quantum DXi use a variable length block and are developed towards deduplicating backup application streams. ZFS does OK but nothing in comparison to the dedup ratio seen on these aforementioned appliances. I used tar as an example and should have been more specific. I understand what you are saying about replicating every 15 minutes etc.. However since backup application create huge files, an incremental send would need to send the newly created huge file..At least that is how I understand, I may be correctly. In my testing this was the case but perhaps my syntax was not correct. In any case deduplication appliances when replicating only send the changed blocks that don't exist on the target side. To do this they have to have knowledge of what exists in the target side "block pool", "dedup hash table",or whatever it may be called. >From what I understand a ZFS file system on the source side has no idea of what exists on the target side. I also understand and maybe incorrectly, that the zfs send -D only eliminates duplicate blocks in the stream it is sending and does not account for a block that may already exist at the target. As an example let's say I am using a backup app like Amanda.. I do a full backup every day to a ZFS based disk target..Every day after the backup completes I do a -- "zfs send -D -i {snap_yesterday} {snap_today} | ssh DESTINATION zfs recv DEST_FS". Now each day's full backup will only have maybe a 1% change rate and this will be reflected on the source side file system. So if I had 5 days of 2 GB full backups, the source file system will show maybe 3GB Alloc in the zpool list output. However since the source does not know about duplicate blocks on the target side from yesterday's backup, it sends the entire 2GB full backup from today only removing any duplicate blocks that exist in the stream it is sending. The difference with a dedup appliance is that it is aware of duplicate blocks on the target side and won't send them. This is the reason my original question was asking if there were any plans to implement a "global DDT" or dedup table to make the target aware of the destination duplicate blocks so that only unique blocks are transferred. Am I incorrect in my understanding of the ZFS DDT being unique to each ZFS file system/pool? Thanks Jimmy On Fri, Dec 2, 2011 at 12:04 PM, Peter Maloney wrote: > Am 02.12.2011 15:50, schrieb Jeremy Chadwick: >> On Fri, Dec 02, 2011 at 03:27:03PM +0100, Michel Le Cocq wrote: >>> it's just me or there is no attachment ? >> The mailing list stripped the attachment. =C2=A0The previous individual = will >> need to put it up on the web somewhere. >> > It is possible that I forgot to attach it. I assumed it would be > stripped off but the ones in the to/cc would get it. > > Here it is on the company website: > > http://www.brockmann-consult.de/peter2/zfs.tgz > > > > Disclaimer/notes: > -provided as is... might destroy your system, furthermore, I am not > responsible for bodily injury nor nuclear war that may result from misuse > -there are no unit tests, and no documentation other than a few comments > that are possibly only coherent when I read them. For example, it says > that it does it recursively and rolls back the destination dataset, but > there are a few undocumented cases I can't remember when I needed to do > something manual like delete a snapshot, or destroy a dataset. Maybe > that is all in the past. I don't know. > -the zfs_repl2.bash is the one that makes snapshots and replicates which > I wrote myself. The other ksh one is the Oracle one I linked above, and > the .sh version of it was just what I was working on to try to make it > work reliably, before redoing it all myself (reinventing the wheel is > indeed fun). > -especially beware of the deleteOldSnapshots.bash which is not well > tested and not used yet (and deleteEmptySnapshots.bash which does not > work and I believe cannot work). > -granted transferable your choice of any present or future version of > the BSD or GPL license > > and another note, I meant to study these which might be better versions > of the same thing, or something different, but never got around to it: > =C2=A0 =C2=A0/usr/ports/sysutils/zfs-replicate/ > =C2=A0 =C2=A0/usr/ports/sysutils/zfsnap/ > =C2=A0 =C2=A0/usr/ports/sysutils/zfs-periodic > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"