From owner-cvs-all@FreeBSD.ORG  Fri Apr 23 13:53:35 2004
Return-Path: <owner-cvs-all@FreeBSD.ORG>
Delivered-To: cvs-all@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP
	id 21D1F16A4CE; Fri, 23 Apr 2004 13:53:35 -0700 (PDT)
Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net
	[66.166.149.50])	by mx1.FreeBSD.org (Postfix) with ESMTP
	id 2103B43D46; Fri, 23 Apr 2004 13:53:34 -0700 (PDT)
	(envelope-from tim@kientzle.com)
Received: from kientzle.com (p54.kientzle.com [66.166.149.54])
	by kientzle.com (8.12.9/8.12.9) with ESMTP id i3NKqw90090410;
	Fri, 23 Apr 2004 13:52:58 -0700 (PDT)
	(envelope-from tim@kientzle.com)
Message-ID: <40898228.8040502@kientzle.com>
Date: Fri, 23 Apr 2004 13:52:56 -0700
From: Tim Kientzle <tim@kientzle.com>
User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4) Gecko/20031006
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Julian Elischer <julian@elischer.org>
References: <Pine.BSF.4.21.0404231145150.6894-100000@InterJet.elischer.org>
In-Reply-To: <Pine.BSF.4.21.0404231145150.6894-100000@InterJet.elischer.org>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
cc: Tim Kientzle <kientzle@FreeBSD.org>
cc: cvs-all@FreeBSD.org
Subject: Re: bad news for bsdtar..
X-BeenThere: cvs-all@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: CVS commit messages for the entire tree <cvs-all.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/cvs-all>,
	<mailto:cvs-all-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/cvs-all>
List-Post: <mailto:cvs-all@freebsd.org>
List-Help: <mailto:cvs-all-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/cvs-all>,
	<mailto:cvs-all-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 23 Apr 2004 20:53:35 -0000

Julian Elischer wrote:
> done on a small subdirectory of our data...
> (only accounts starting with 849xxx)
> total data as reported by 'du': 9394486 KB  (9GB)
> (note du takes links into account and doesn't count the same data twice)
> number of files with > 1 link: 
> # find 849* -type f -links +1 | wc -l
>   240395
> #
> total number of file directory entries..
> # find 849* -type f | wc -l
>   598018
> #
> total number of directories:
> %find 849* -type d | wc -l
>    51245
> 
> average filesize 23k
> but major file types are: jpeg images.. 100kb
> 			jpeg thumbnails, 3kb
> 			transaction descriptors 1kb
> ########################################################
> the bad news:
> 
> old (FreeBSD 2.x) tar with symlink hack
> %time ~/tar cf /dev/null 849*
> 5.309u 73.584s 21:15.44 6.1%    329+5150k 786697+0io 0pf+0w
> 
> freebsd 4.8 tar
> %time tar cf /dev/null 849
> 5.159u 17.724s 4:08.97 9.1%     364+5863k 66144+0io 1pf+0w
> 
> bsdtar
> %time ~/bsdtar cf /dev/null 849*
> 16.423u 68.141s 31:40.82 4.4%   402+1639k 784651+0io 0pf+0w
> 
> It looks to me that the tar in 4.8 is doing something very
> sneeky with it's IOs..
> maybe using mmap?
> maybe detecting /dev/null and instead discarding the data?
> maybe reading the entire file in a single read?

Very interesting.  bsdtar, I note, used about 1/4 as
much memory as either of the others.

I'd be interested to know what happens if you edit
write.c to change the buffer used in write_file_data
from the current miserly 8k to something more reasonable,
like 128k.

I love a challenge... ;-)

Tim