From owner-freebsd-hackers@FreeBSD.ORG  Fri Oct 13 17:19:51 2006
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
X-Original-To: freebsd-hackers@freebsd.org
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 1224916A403
	for <freebsd-hackers@freebsd.org>; Fri, 13 Oct 2006 17:19:51 +0000 (UTC)
	(envelope-from dillon@apollo.backplane.com)
Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2])
	by mx1.FreeBSD.org (Postfix) with ESMTP id C092343D45
	for <freebsd-hackers@freebsd.org>; Fri, 13 Oct 2006 17:19:50 +0000 (GMT)
	(envelope-from dillon@apollo.backplane.com)
Received: from apollo.backplane.com (localhost [127.0.0.1])
	by apollo.backplane.com (8.13.7/8.13.4) with ESMTP id k9DHJllJ025556;
	Fri, 13 Oct 2006 10:19:47 -0700 (PDT)
Received: (from dillon@localhost)
	by apollo.backplane.com (8.13.7/8.13.4/Submit) id k9DHJak1025551;
	Fri, 13 Oct 2006 10:19:37 -0700 (PDT)
Date: Fri, 13 Oct 2006 10:19:37 -0700 (PDT)
From: Matthew Dillon <dillon@apollo.backplane.com>
Message-Id: <200610131719.k9DHJak1025551@apollo.backplane.com>
To: "Steven Hartland" <killing@multiplay.co.uk>
References: <17707.64434.913943.549852@bhuda.mired.org>
	<200610110934.k9B9YASW081294@lurza.secnetix.de>
	<20061011133250.GA483@britannica.bec.de>
	<01a301c6ede8$061e9160$b3db87d4@multiplay.co.uk>
Cc: freebsd-hackers@freebsd.org, Joerg Sonnenberger <joerg@britannica.bec.de>
Subject: Re: "tar -c|gzip" faster than "tar -cz"?!?
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 13 Oct 2006 17:19:51 -0000


:
:Just a silly one but are you guys using the same
:version of gzip, would be worth just checking?

    It could also simply be a piplining issue.  If the pipe inbetween the
    'tar' and the 'gzip' is too small (whether gzip is internal to tar
    or not), then the 'tar' portion could wind up getting blocked by the
    'gzip' portion and not do disk I/O in parallel with the cpu that the gzip
    portion uses.

    Here I am presuming that there is in fact a fork internal to tar when
    using the built-in gzip.  There had better be, or performance would be
    horrible!

    In anycase, the pipe buffer needs to be at least 2x the block size gzip
    uses internally when compressing.  I would even recommend making it 
    very large, like several hundred kilobytes (at least).   It is the
    same problem one faces when, say, streaming data to a slow device such
    as a tape drive.  You want a large pipe buffer to avoid unsightly stalls
    of the code scanning the filesystem.

					-Matt