From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 2 06:36:44 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 47C611065674 for ; Tue, 2 Oct 2012 06:36:44 +0000 (UTC) (envelope-from lists@yamagi.org) Received: from mail.yamagi.org (mail.yamagi.org [IPv6:2a01:4f8:121:2102:1::7]) by mx1.freebsd.org (Postfix) with ESMTP id CFCF08FC0A for ; Tue, 2 Oct 2012 06:36:43 +0000 (UTC) Received: from happy.home.yamagi.org (hmbg-5f77e48d.pool.mediaWays.net [95.119.228.141]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.yamagi.org (Postfix) with ESMTPSA id 29B061666312; Tue, 2 Oct 2012 08:36:40 +0200 (CEST) Date: Tue, 2 Oct 2012 08:36:34 +0200 From: Yamagi Burmeister To: tim@kientzle.com Message-Id: <20121002083634.3103fe958508a4026384ac96@yamagi.org> In-Reply-To: <87549776-9051-4B4B-8D53-DAE6D51C2A94@kientzle.com> References: <5069C9FC.6020400@brandonfa.lk> <87549776-9051-4B4B-8D53-DAE6D51C2A94@kientzle.com> X-Mailer: Sylpheed 3.2.0 (GTK+ 2.24.6; amd64-portbld-freebsd9.0) Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA1"; boundary="Signature=_Tue__2_Oct_2012_08_36_34_+0200_EDLuBl3TxLr/FPZX" Cc: freebsd-hackers@freebsd.org, bfalk_bsd@brandonfa.lk Subject: Re: SMP Version of tar X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Oct 2012 06:36:44 -0000 --Signature=_Tue__2_Oct_2012_08_36_34_+0200_EDLuBl3TxLr/FPZX Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, 1 Oct 2012 22:16:53 -0700 Tim Kientzle wrote: > There are a few different parallel command-line compressors and decompres= sors in ports; experiment a lot (with large files being read from and/or wr= itten to disk) and see what the real effect is. In particular, some decomp= ression algorithms are actually faster than memcpy() when run on a single p= rocessor. Parallelizing such algorithms is not likely to help much in the = real world. >=20 > The two popular algorithms I would expect to benefit most are bzip2 compr= ession and lzma compression (targeting xz or lzip format). For decompressi= on, bzip2 is block-oriented so fits SMP pretty naturally. Other popular al= gorithms are stream-oriented and less amenable to parallelization. >=20 > Take a careful look at pbzip2, which is a parallelized bzip2/bunzip2 impl= ementation that's already under a BSD license. You should be able to get a= lot of ideas about how to implement a parallel compression algorithm. Bet= ter yet, you might be able to reuse a lot of the existing pbzip2 code. >=20 > Mark Adler's pigz is also worth studying. It's also license-friendly, an= d is built on top of regular zlib, which is a nice technique when it's feas= ible. Just a small note: There's a parallel implementation of xz called "pixz". It's build atop of liblzma and libarchiv and stands under a=20 BSD style license. See: https://github.com/vasi/pixz Maybe it's possible to reuse most of the code. --=20 Homepage: www.yamagi.org XMPP: yamagi@yamagi.org GnuPG/GPG: 0xEFBCCBCB --Signature=_Tue__2_Oct_2012_08_36_34_+0200_EDLuBl3TxLr/FPZX Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlBqi3cACgkQWTjlg++8y8u5uwCfWMSoOYdxB2ZDV+qDbfEX3z76 6kcAn0bJRHV47a4i2po55UTQKnrXBulb =fMmx -----END PGP SIGNATURE----- --Signature=_Tue__2_Oct_2012_08_36_34_+0200_EDLuBl3TxLr/FPZX--