From owner-freebsd-ports@freebsd.org Wed Apr 12 14:11:18 2017 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57BEAD3BFB1 for ; Wed, 12 Apr 2017 14:11:18 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 44C4FDF for ; Wed, 12 Apr 2017 14:11:18 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 441D2D3BFB0; Wed, 12 Apr 2017 14:11:18 +0000 (UTC) Delivered-To: ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43B61D3BFAF for ; Wed, 12 Apr 2017 14:11:18 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06AD9DE; Wed, 12 Apr 2017 14:11:18 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1354) id 385CB5C8F; Wed, 12 Apr 2017 14:11:17 +0000 (UTC) From: Jan Beich To: Michael Gmelin Cc: Alexey Dokuchaev , ports@FreeBSD.org Subject: Re: LLVM port(s) take very long time to package In-Reply-To: <79A96B5B-8E4C-4E2E-8108-BA4F05CBFA4F@freebsd.org> (Michael Gmelin's message of "Wed, 12 Apr 2017 10:43:14 +0200") References: <20170412034308.GA56148@FreeBSD.org> <79A96B5B-8E4C-4E2E-8108-BA4F05CBFA4F@freebsd.org> Date: Wed, 12 Apr 2017 16:12:45 +0200 Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Apr 2017 14:11:18 -0000 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Michael Gmelin writes: >> On 12. Apr 2017, at 05:43, Alexey Dokuchaev wrote: >>=20 >> Hi there porters, >>=20 >> It had been annoying me for a while, so I figured I try asking here: >> LLVM ports (particularly, llvm39-3.9.1) take abnormally long time to >> package, e.g. right now it took 43 minutes (tmpfs-backed build area, >> pkg-1.10.0_2.txz, AMD A8 APU @1900 MHz, plenty of RAM, no swap). >>=20 >> pkg was eating 100% (of a single core) during that time, so it looks >> like it's actually busy (not waiting for I/O or something), but with >> what? And can it be fixed? LLVM ports are real PITA to (re)build, >> and while ccache helps to greatly reduce the build times, 3/4-hour >> package time still cripples it. >>=20 > > It's probably busy compressing the package. Try setting > PKG_NOCOMPRESS=3D1 in /etc/make.conf (works ok when just building a > package with make, didn't test with poudriere[0]). When creating > packages, the current compression algorithm can't take advantage of > multiple cores, so it's kind of a bottleneck on modern machines. Nah, libarchive just defaults to 1 thread instead of using hw.ncpu. The tricky part is propagating the fix into old releases or setting=20 compression options from within pkg(8). Index: lib/libarchive/config_freebsd.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =2D-- lib/libarchive/config_freebsd.h (revision 316719) +++ lib/libarchive/config_freebsd.h (working copy) @@ -139,6 +139,7 @@ #define HAVE_LONG_LONG_INT 1 #define HAVE_LSTAT 1 #define HAVE_LUTIMES 1 +#define HAVE_LZMA_STREAM_ENCODER_MT 1 #define HAVE_MBRTOWC 1 #define HAVE_MEMMOVE 1 #define HAVE_MEMORY_H 1 Index: contrib/libarchive/libarchive/archive_write_add_filter_xz.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =2D-- contrib/libarchive/libarchive/archive_write_add_filter_xz.c (revision= 316719) +++ contrib/libarchive/libarchive/archive_write_add_filter_xz.c (working co= py) @@ -152,7 +152,11 @@ common_setup(struct archive_write_filter *f) } f->data =3D data; data->compression_level =3D LZMA_PRESET_DEFAULT; +#ifdef HAVE_LZMA_STREAM_ENCODER_MT + data->threads =3D lzma_cputhreads(); +#else data->threads =3D 1; +#endif f->open =3D &archive_compressor_xz_open; f->close =3D archive_compressor_xz_close; f->free =3D archive_compressor_xz_free; Index: contrib/libarchive/libarchive/archive_write_set_format_xar.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =2D-- contrib/libarchive/libarchive/archive_write_set_format_xar.c (revisio= n 316719) +++ contrib/libarchive/libarchive/archive_write_set_format_xar.c (working c= opy) @@ -384,7 +384,11 @@ archive_write_set_format_xar(struct archive *_a) /* Set default compression type, level, and number of threads. */ xar->opt_compression =3D GZIP; xar->opt_compression_level =3D 6; +#ifdef HAVE_LZMA_STREAM_ENCODER_MT + xar->opt_threads =3D lzma_cputhreads(); +#else xar->opt_threads =3D 1; +#endif =20 a->format_data =3D xar; =20 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQF8BAEBCgBmBQJY7jXdXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXREQjQ0MzY3NEM3RDIzNTc4NkUxNDkyQ0VF NEM3Nzg4MzQ3OURCRERCAAoJEOTHeINHnb3brhQH/3+v5PGotcvMNei5x/udZm6n 6oU0vaULD65HpZib06Y0BT2xKPT2F5PoWpp+E+eZQU4NEADI+YzYR/S4+K1aW8Q0 sbS1J5kvuubg1r4E1A3XKgz1U/pStAEmMDxvc9pwBlZakxXXHlzsGD+YAm5trfgT wQnT0cghcAvM7XrVgslj9TDG97nzf3xX9CJQLHpCgG5hTLeGPbcGMBknXxEAIZGy fkXQ7bJ35jvUqrxwirJYV9fhAc0NLNizuKM10+n6GZrMRYRx/Q/SxEOpHg5NITEy 5fRRVDDy9yJoCinXFVALx42LifjAbwkOp8OOQuElucD8AwtfPwYpaAotPyF5vm0= =h6RG -----END PGP SIGNATURE----- --=-=-=--