From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 31 16:35:36 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A95F106566C for ; Sun, 31 Oct 2010 16:35:36 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 860098FC17 for ; Sun, 31 Oct 2010 16:35:35 +0000 (UTC) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.4/8.14.1) with ESMTP id o9VGZGPY046167; Sun, 31 Oct 2010 09:35:16 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.4/8.13.4/Submit) id o9VGZG1O046164; Sun, 31 Oct 2010 09:35:16 -0700 (PDT) Date: Sun, 31 Oct 2010 09:35:16 -0700 (PDT) From: Matthew Dillon Message-Id: <201010311635.o9VGZG1O046164@apollo.backplane.com> To: perryh@pluto.rain.com References: <4ccceb10.4n2iAQ/sY/YrDSI2%perryh@pluto.rain.com> Cc: cronfy@gmail.com, freebsd-hackers@freebsd.org Subject: Re: Slow disk access while rsync - what should I tune? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Oct 2010 16:35:36 -0000 :cronfy wrote: : :> And also, maybe there are other ways to create incremental backups :> instead of using rsync/hardlinks? : :Yes. Use dump(8) -- that's what it's for. It reads the inodes, :directories, and files directly from the disk device, thereby :eliminating stat() overhead entirely. : :Any replication mechanism -- rsync, tar, even dd -- can be used :as a backup mechanism, but dump was specifically designed for the :purpose. Well, dump is 25+ years old and has some serious issues when it comes to reliably backing data up. On a modern (large) filesystem you are virtually guaranteed to get corruption due to the asynchronous nature of the dump. This can be partially mitigated by using a block level snapshot on the UFS source filesystem and then dumping the snapshot instead of the live filesystem, but that opens up a pandora's box of other issues (such as whether issuing the snapshot itself will destabilize the machine), plus the live system will stall while it is making the snapshot, assuming you want a consistent snapshot, plus the snapshot itself may not be entirely consistent, depending on how it is made. Plus dump uses mtime to detect changes, which is unreliable, and the output produced by dump is not live-accessible whereas a snapshot / live filesystem copy is. That makes the dump fairly worthless for anything other than catastrophic recovery. From experience, most people need to access backups to pull out small bits of information rather than whole filesystems, such as to restore a user's home directory or web pages, and dump/restore is really unsuitable for that sort of thing. Live backups are far, far, far superior to dump/restore. -Matt