From owner-freebsd-hackers@FreeBSD.ORG  Wed Oct 10 04:53:31 2012
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id 10339C17
 for <freebsd-hackers@freebsd.org>; Wed, 10 Oct 2012 04:53:31 +0000 (UTC)
 (envelope-from tim@kientzle.com)
Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net
 [99.115.135.74])
 by mx1.freebsd.org (Postfix) with ESMTP id D6CA08FC08
 for <freebsd-hackers@freebsd.org>; Wed, 10 Oct 2012 04:53:30 +0000 (UTC)
Received: (from root@localhost)
 by monday.kientzle.com (8.14.4/8.14.4) id q9A4rM96032111;
 Wed, 10 Oct 2012 04:53:22 GMT (envelope-from tim@kientzle.com)
Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65])
 by kientzle.com with SMTP id b7sp22idag68zchregtgjzrste;
 Wed, 10 Oct 2012 04:53:22 +0000 (UTC)
 (envelope-from tim@kientzle.com)
Subject: Re: SMP Version of tar
Mime-Version: 1.0 (Apple Message framework v1278)
Content-Type: text/plain; charset=us-ascii
From: Tim Kientzle <tim@kientzle.com>
In-Reply-To: <alpine.BSF.2.00.1210081219300.4673@wojtek.tensor.gdynia.pl>
Date: Tue, 9 Oct 2012 21:54:03 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <15DBA1A9-A4B6-4F7D-A9DC-3412C4BE3517@kientzle.com>
References: <5069C9FC.6020400@brandonfa.lk>
 <alpine.BSF.2.00.1210071859430.15957@wojtek.tensor.gdynia.pl>
 <324B736D-8961-4E44-A212-2ECF3E60F2A0@kientzle.com>
 <alpine.BSF.2.00.1210080838170.3664@wojtek.tensor.gdynia.pl>
 <20121008083814.GA5830@straylight.m.ringlet.net>
 <alpine.BSF.2.00.1210081219300.4673@wojtek.tensor.gdynia.pl>
To: Wojciech Puchar <wojtek@wojtek.tensor.gdynia.pl>
X-Mailer: Apple Mail (2.1278)
Cc: freebsd-hackers@freebsd.org, Brandon Falk <bfalk_bsd@brandonfa.lk>
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 10 Oct 2012 04:53:31 -0000


On Oct 8, 2012, at 3:21 AM, Wojciech Puchar wrote:

>> Not necessarily.  If I understand correctly what Tim means, he's =
talking
>> about an in-memory compression of several blocks by several separate
>> threads, and then - after all the threads have compressed their
>=20
> but gzip format is single stream. dictionary IMHO is not reset every X =
kilobytes.
>=20
> parallel gzip is possible but not with same data format.

Yes, it is.

The following creates a compressed file that
is completely compatible with the standard
gzip/gunzip tools:

   * Break file into blocks
   * Compress each block into a gzip file (with gzip header and trailer =
information)
   * Concatenate the result.

This can be correctly decoded by gunzip.

In theory, you get slightly worse compression.  In practice, if your =
blocks are reasonably large (a megabyte or so each), the difference is =
negligible.

Tim