From owner-freebsd-questions@freebsd.org Wed May 31 02:27:49 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE93ABF352B for ; Wed, 31 May 2017 02:27:49 +0000 (UTC) (envelope-from 0101015c5c5498c3-84377171-9e30-40da-a58b-1c4af755f602-000000@us-west-2.amazonses.com) Received: from a27-35.smtp-out.us-west-2.amazonses.com (a27-35.smtp-out.us-west-2.amazonses.com [54.240.27.35]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C265183E81 for ; Wed, 31 May 2017 02:27:49 +0000 (UTC) (envelope-from 0101015c5c5498c3-84377171-9e30-40da-a58b-1c4af755f602-000000@us-west-2.amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=stdm37fqbypyvw6iolbo3nkyd73es3w7; d=vmeta.jp; t=1496197667; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:In-Reply-To; bh=OGZKEOpGu9g6egw3CU0aXB4Tr0g3WTZoR08CawBVreA=; b=FAJeTCtItXDX5H3C+ul8ZDy6VBru6n+zJ1j49yUbzwoE/uHMKdCsHyrA9pScLqmE F6GA4483IFRewe7ng3ybvsuAYVK5tNjg0THfDzM74tl3kYIXLGCMhfJp6KsBRKfBcB0 sqYSkhSRw/NbT13LxVPuAqk9KdqQB2ZZ/7/3yUaQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1496197667; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type:In-Reply-To:Feedback-ID; bh=OGZKEOpGu9g6egw3CU0aXB4Tr0g3WTZoR08CawBVreA=; b=SGrxrH6oIZ8jBdUW6KEd6hhENTH5v7KiaL5Ni7bgzB9+G98WPdwXdySAitYDVaSM QfETp+eLkcdzsnZWs+Xu24q/lgcSlI/aHqaG+D18XFbkgEYjgWDoSok4kSBhBBJPXCn kWAPl0nH6UtDtriWsE7UreH2PsfqPxV9mzPX9W8U= Date: Wed, 31 May 2017 02:27:47 +0000 From: meta To: "Chad J. Milios" Cc: freebsd-questions Subject: Re: Is it possible to use more than 1 CPUs during pkg create? Message-ID: <0101015c5c5498c3-84377171-9e30-40da-a58b-1c4af755f602-000000@us-west-2.amazonses.com> References: <0101015c58b3dc58-1b936a1a-8412-472b-a257-c436a40e89f2-000000@us-west-2.amazonses.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 10.3-RELEASE-p18 amd64 User-Agent: Mutt/1.8.2 (2017-04-18) X-SES-Outgoing: 2017.05.31-54.240.27.35 Feedback-ID: 1.us-west-2.ngRt4x2U/cWqug8pbfjwMxB6pcDw1fmN73bGmMLYyRI=:AmazonSES X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 May 2017 02:27:50 -0000 On Tue, May 30, 2017 at 06:55:02AM -0400, Chad J. Milios wrote: > Threading forfeits a bit of xz’s great compression ratio. An xz file created with -T greater than 1 will necessarily be some amount larger. Operating with just one thread achieves the maximal compression ratio. (The multiple compression threads fail to share the dictionaries they are currently building.) > Yes, compressing with one thread achieves the best ratio. However, it is really a little bit. I created a package of x11-fonts as example of a big package. Its size is 813230580 bytes. Then I recompressed the package with -T0 option, the archive is just 1% larger. # make -C /usr/ports/x11-fonts/noto package $ ls noto* -rw-r--r-- 1 meta wheel 813230580 May 30 21:30 noto-1.0.5_1.txz $ zcat noto-1.0.5_txz | xz -T0 -c > noto-1.0.5_1.txz.T0 $ ls noto* -rw-r--r-- 1 meta wheel 813230580 May 30 21:30 noto-1.0.5_1.txz -rw-r--r-- 1 meta wheel 821921768 May 30 21:45 noto-1.0.5_1.txz.T0 I understand the merit of compressing package in single thread distributed by pkg.freebsd.org. However, I'd rather save time than storage or network bandwidth. I want to make `make package` faster on my host. The build can use multi cores but pkg create process uses only one so package stage is the bottle neck. > When you are transferring GBytes to yourself once, use -T0. When you are compressing something one time that hundreds or thousands of people will potentially download and/or store, use -T1 (the default). Do you mean I can use -T0 during pkg create? AFAIK it is not possible but if possible, how? -- `whois vmeta.jp | nkf -w` meta