Date: Tue, 9 Oct 2012 21:54:03 -0700 From: Tim Kientzle <tim@kientzle.com> To: Wojciech Puchar <wojtek@wojtek.tensor.gdynia.pl> Cc: freebsd-hackers@freebsd.org, Brandon Falk <bfalk_bsd@brandonfa.lk> Subject: Re: SMP Version of tar Message-ID: <15DBA1A9-A4B6-4F7D-A9DC-3412C4BE3517@kientzle.com> In-Reply-To: <alpine.BSF.2.00.1210081219300.4673@wojtek.tensor.gdynia.pl> References: <5069C9FC.6020400@brandonfa.lk> <alpine.BSF.2.00.1210071859430.15957@wojtek.tensor.gdynia.pl> <324B736D-8961-4E44-A212-2ECF3E60F2A0@kientzle.com> <alpine.BSF.2.00.1210080838170.3664@wojtek.tensor.gdynia.pl> <20121008083814.GA5830@straylight.m.ringlet.net> <alpine.BSF.2.00.1210081219300.4673@wojtek.tensor.gdynia.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
On Oct 8, 2012, at 3:21 AM, Wojciech Puchar wrote: >> Not necessarily. If I understand correctly what Tim means, he's talking >> about an in-memory compression of several blocks by several separate >> threads, and then - after all the threads have compressed their > > but gzip format is single stream. dictionary IMHO is not reset every X kilobytes. > > parallel gzip is possible but not with same data format. Yes, it is. The following creates a compressed file that is completely compatible with the standard gzip/gunzip tools: * Break file into blocks * Compress each block into a gzip file (with gzip header and trailer information) * Concatenate the result. This can be correctly decoded by gunzip. In theory, you get slightly worse compression. In practice, if your blocks are reasonably large (a megabyte or so each), the difference is negligible. Tim
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?15DBA1A9-A4B6-4F7D-A9DC-3412C4BE3517>