Date: Wed, 12 Apr 2017 16:12:45 +0200 From: Jan Beich <jbeich@FreeBSD.org> To: Michael Gmelin <grembo@freebsd.org> Cc: Alexey Dokuchaev <danfe@FreeBSD.org>, ports@FreeBSD.org Subject: Re: LLVM port(s) take very long time to package Message-ID: <mvbl-g1eq-wny@FreeBSD.org> In-Reply-To: <79A96B5B-8E4C-4E2E-8108-BA4F05CBFA4F@freebsd.org> (Michael Gmelin's message of "Wed, 12 Apr 2017 10:43:14 %2B0200") References: <20170412034308.GA56148@FreeBSD.org> <79A96B5B-8E4C-4E2E-8108-BA4F05CBFA4F@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Michael Gmelin <grembo@freebsd.org> writes: >> On 12. Apr 2017, at 05:43, Alexey Dokuchaev <danfe@FreeBSD.org> wrote: >>=20 >> Hi there porters, >>=20 >> It had been annoying me for a while, so I figured I try asking here: >> LLVM ports (particularly, llvm39-3.9.1) take abnormally long time to >> package, e.g. right now it took 43 minutes (tmpfs-backed build area, >> pkg-1.10.0_2.txz, AMD A8 APU @1900 MHz, plenty of RAM, no swap). >>=20 >> pkg was eating 100% (of a single core) during that time, so it looks >> like it's actually busy (not waiting for I/O or something), but with >> what? And can it be fixed? LLVM ports are real PITA to (re)build, >> and while ccache helps to greatly reduce the build times, 3/4-hour >> package time still cripples it. >>=20 > > It's probably busy compressing the package. Try setting > PKG_NOCOMPRESS=3D1 in /etc/make.conf (works ok when just building a > package with make, didn't test with poudriere[0]). When creating > packages, the current compression algorithm can't take advantage of > multiple cores, so it's kind of a bottleneck on modern machines. Nah, libarchive just defaults to 1 thread instead of using hw.ncpu. The tricky part is propagating the fix into old releases or setting=20 compression options from within pkg(8). Index: lib/libarchive/config_freebsd.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =2D-- lib/libarchive/config_freebsd.h (revision 316719) +++ lib/libarchive/config_freebsd.h (working copy) @@ -139,6 +139,7 @@ #define HAVE_LONG_LONG_INT 1 #define HAVE_LSTAT 1 #define HAVE_LUTIMES 1 +#define HAVE_LZMA_STREAM_ENCODER_MT 1 #define HAVE_MBRTOWC 1 #define HAVE_MEMMOVE 1 #define HAVE_MEMORY_H 1 Index: contrib/libarchive/libarchive/archive_write_add_filter_xz.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =2D-- contrib/libarchive/libarchive/archive_write_add_filter_xz.c (revision= 316719) +++ contrib/libarchive/libarchive/archive_write_add_filter_xz.c (working co= py) @@ -152,7 +152,11 @@ common_setup(struct archive_write_filter *f) } f->data =3D data; data->compression_level =3D LZMA_PRESET_DEFAULT; +#ifdef HAVE_LZMA_STREAM_ENCODER_MT + data->threads =3D lzma_cputhreads(); +#else data->threads =3D 1; +#endif f->open =3D &archive_compressor_xz_open; f->close =3D archive_compressor_xz_close; f->free =3D archive_compressor_xz_free; Index: contrib/libarchive/libarchive/archive_write_set_format_xar.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =2D-- contrib/libarchive/libarchive/archive_write_set_format_xar.c (revisio= n 316719) +++ contrib/libarchive/libarchive/archive_write_set_format_xar.c (working c= opy) @@ -384,7 +384,11 @@ archive_write_set_format_xar(struct archive *_a) /* Set default compression type, level, and number of threads. */ xar->opt_compression =3D GZIP; xar->opt_compression_level =3D 6; +#ifdef HAVE_LZMA_STREAM_ENCODER_MT + xar->opt_threads =3D lzma_cputhreads(); +#else xar->opt_threads =3D 1; +#endif =20 a->format_data =3D xar; =20 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQF8BAEBCgBmBQJY7jXdXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXREQjQ0MzY3NEM3RDIzNTc4NkUxNDkyQ0VF NEM3Nzg4MzQ3OURCRERCAAoJEOTHeINHnb3brhQH/3+v5PGotcvMNei5x/udZm6n 6oU0vaULD65HpZib06Y0BT2xKPT2F5PoWpp+E+eZQU4NEADI+YzYR/S4+K1aW8Q0 sbS1J5kvuubg1r4E1A3XKgz1U/pStAEmMDxvc9pwBlZakxXXHlzsGD+YAm5trfgT wQnT0cghcAvM7XrVgslj9TDG97nzf3xX9CJQLHpCgG5hTLeGPbcGMBknXxEAIZGy fkXQ7bJ35jvUqrxwirJYV9fhAc0NLNizuKM10+n6GZrMRYRx/Q/SxEOpHg5NITEy 5fRRVDDy9yJoCinXFVALx42LifjAbwkOp8OOQuElucD8AwtfPwYpaAotPyF5vm0= =h6RG -----END PGP SIGNATURE----- --=-=-=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?mvbl-g1eq-wny>