Date: Tue, 2 Oct 2012 08:36:34 +0200 From: Yamagi Burmeister <lists@yamagi.org> To: tim@kientzle.com Cc: freebsd-hackers@freebsd.org, bfalk_bsd@brandonfa.lk Subject: Re: SMP Version of tar Message-ID: <20121002083634.3103fe958508a4026384ac96@yamagi.org> In-Reply-To: <87549776-9051-4B4B-8D53-DAE6D51C2A94@kientzle.com> References: <5069C9FC.6020400@brandonfa.lk> <87549776-9051-4B4B-8D53-DAE6D51C2A94@kientzle.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Mon, 1 Oct 2012 22:16:53 -0700 Tim Kientzle <tim@kientzle.com> wrote: > There are a few different parallel command-line compressors and decompressors in ports; experiment a lot (with large files being read from and/or written to disk) and see what the real effect is. In particular, some decompression algorithms are actually faster than memcpy() when run on a single processor. Parallelizing such algorithms is not likely to help much in the real world. > > The two popular algorithms I would expect to benefit most are bzip2 compression and lzma compression (targeting xz or lzip format). For decompression, bzip2 is block-oriented so fits SMP pretty naturally. Other popular algorithms are stream-oriented and less amenable to parallelization. > > Take a careful look at pbzip2, which is a parallelized bzip2/bunzip2 implementation that's already under a BSD license. You should be able to get a lot of ideas about how to implement a parallel compression algorithm. Better yet, you might be able to reuse a lot of the existing pbzip2 code. > > Mark Adler's pigz is also worth studying. It's also license-friendly, and is built on top of regular zlib, which is a nice technique when it's feasible. Just a small note: There's a parallel implementation of xz called "pixz". It's build atop of liblzma and libarchiv and stands under a BSD style license. See: https://github.com/vasi/pixz Maybe it's possible to reuse most of the code. -- Homepage: www.yamagi.org XMPP: yamagi@yamagi.org GnuPG/GPG: 0xEFBCCBCB [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlBqi3cACgkQWTjlg++8y8u5uwCfWMSoOYdxB2ZDV+qDbfEX3z76 6kcAn0bJRHV47a4i2po55UTQKnrXBulb =fMmx -----END PGP SIGNATURE-----help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121002083634.3103fe958508a4026384ac96>