From owner-freebsd-current@FreeBSD.ORG Sat Jan 27 23:03:44 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 966A016A400 for ; Sat, 27 Jan 2007 23:03:44 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (mail.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 787F613C48C for ; Sat, 27 Jan 2007 23:03:44 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 6CFCE5B32; Sat, 27 Jan 2007 14:41:20 -0800 (PST) To: Peter Jeremy In-reply-to: Your message of "Sun, 28 Jan 2007 08:42:34 +1100." <20070127214234.GO927@turion.vk2pj.dyndns.org> Date: Sat, 27 Jan 2007 14:41:20 -0800 From: Bakul Shah Message-Id: <20070127224120.6CFCE5B32@mail.bitblocks.com> Cc: freebsd-current@freebsd.org Subject: Re: Interesting speed benchmarks X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2007 23:03:44 -0000 Peter Jeremy wrote: > On Sat, 2007-Jan-27 11:54:37 -0500, Brian Reichert wrote: > >On Sat, Jan 27, 2007 at 09:43:52AM +1100, Peter Jeremy wrote: > >> Note that dump/restore is the only tool that can correctly reproduce > >> sparse files. tar, cpio and pax also have filename and file size > >> restrictions. I don't think that cpio or pax support ACLs or file > >> flags. > > > >I thought 'star' handled sparse files and all the extra magic? Not > >that this is germaine to the topic at hand... > > You can detect a sparse file by comparing the length of the file > with the number of allocated blocks. You can't determine whether a > specific block is all NULLs or not allocated without either bypassing > the filesystem (which dump does) or using a syscall to retrieve a > bitmap of allocated blocks (which FreeBSD doesn't have). =20 Both rsync and gtar create a sparse file by not writing blocks of zeroes given -S option. You are correct that there is no special help for reading sparse files. I recall pjd@freebsd.org saying he plans to implement solaris specific extensions to lseek() in zfs (lseek's whence arg can take SEEK_HOLE/SEEK_DATA values, to seek to next null/nonnull block). It would be nice if this was done for all filesystems. Returning a bitmap is far more work than adding SEEK_{HOLE,DATA}. For backups I use vbackup & venti from plan9ports. Like dump vbackup knows ufs1 & ufs2 structure to walk the filesytem and write only live data to venti (a SHA1-addressed block storage server) so it correctly handles sparse files. A full backup is faster than dump. All subseqeuent backups are incremental and generally run very fast (for example I can backup a UFS2 system at about 10MB/s to a USB2 ATA disk. dd shows write speed of 23MB/s). You can get at backed up data in a number of different ways: 1) vcat (makes an identical disk image given a snapshot's `score') 2) vftp (you can browse/copy files individually) 3) vnfs + vmount (make one or more snapshots accessible in filesystem namespace) AFAIK flags, attributes & ACLs are maintained but are only available if you use vcat. See http://swtch.com/plan9ports