Date: Thu, 22 Jul 2021 10:46:07 +0200 From: Stefan Esser <se@freebsd.org> To: "Matthew D. Fuller" <fullermd@over-yonder.net>, Alexey Dokuchaev <danfe@freebsd.org> Cc: Baptiste Daroussin <bapt@freebsd.org>, ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Subject: Re: git: ec2764d5ec49 - main - Mk/bsd.port.mk: prepare the land for pkg 1.17 Message-ID: <e17015aa-eaf7-0e8a-fdab-e4c2b7c19f04@freebsd.org> In-Reply-To: <YPjZhks2qvKuUU7t@over-yonder.net> References: <202107211223.16LCNDjf070492@gitrepo.freebsd.org> <YPgxkAglZ2GRunni@FreeBSD.org> <20210721145201.5jcynsooy4hva7ir@aniel.nours.eu> <YPjT%2BMuwECRSMe4X@FreeBSD.org> <YPjZhks2qvKuUU7t@over-yonder.net>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --OK6F7Lx6pySpdaXcd7k6FAD3z5tXBgj51 Content-Type: multipart/mixed; boundary="kntKnBcj15fl8N9EsOR9JonOpK2c4s3Ak"; protected-headers="v1" From: Stefan Esser <se@freebsd.org> To: "Matthew D. Fuller" <fullermd@over-yonder.net>, Alexey Dokuchaev <danfe@freebsd.org> Cc: Baptiste Daroussin <bapt@freebsd.org>, ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Message-ID: <e17015aa-eaf7-0e8a-fdab-e4c2b7c19f04@freebsd.org> Subject: Re: git: ec2764d5ec49 - main - Mk/bsd.port.mk: prepare the land for pkg 1.17 References: <202107211223.16LCNDjf070492@gitrepo.freebsd.org> <YPgxkAglZ2GRunni@FreeBSD.org> <20210721145201.5jcynsooy4hva7ir@aniel.nours.eu> <YPjT+MuwECRSMe4X@FreeBSD.org> <YPjZhks2qvKuUU7t@over-yonder.net> In-Reply-To: <YPjZhks2qvKuUU7t@over-yonder.net> --kntKnBcj15fl8N9EsOR9JonOpK2c4s3Ak Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Am 22.07.21 um 04:35 schrieb Matthew D. Fuller: > On Thu, Jul 22, 2021 at 02:12:08AM +0000 I heard the voice of > Alexey Dokuchaev, and lo! it spake thus: >> >> I've switched to PKG_NOCOMPRESS and >> PKG_DEPENDS+=3Dlbzip2:archivers/lbzip2 since 2017 when I got >> dissatisfied with ridiculously long LLVM packaging times* >=20 > I've wished (not enough yet to see if I could manage it, but) to be > able to use something like `gzip -1` for compression on some poudriere > instances; all the clients pulling from it are on the same LAN if not > the same physical machine, so bandwidth is trivially enough, and > compression/decompression times eat up a significant amount of the > cost of updates... I have used .tar without compression on ZFS file systems that use zstd-2 compression (my default for all file systems not meant to hold already compressed distfiles or media files). This is the fastest and most efficient configuration IMHO (if you are using ZFS anyway), with an effective compression ratio of more than 3 in practice and the compression time hidden in the filesystem layer. If you are transferring these files over slow networks, that is not a good setup, but within a system or on a LAN it does not matter. Example of a randomly selected old package file of significant size: $ cd /usr/ports/portmaster-backups $ time cp llvm60-6.0.1_1.tar /tmp/ real 0m0.928s user 0m0.000s sys 0m0.930s $ time cp /tmp/llvm60-6.0.1_1.tar . real 0m0.572s user 0m0.000s sys 0m0.575s $ ls -ls llvm60-6.0.1_1.tar /tmp/llvm60-6.0.1_1.tar 835724 -rw-r--r-- 1 root wheel 855781376 Jul 22 09:57 /tmp/llvm60-6.0.= 1_1.tar 290613 -rw-r--r-- 1 root wheel 855781376 Jul 22 09:58 llvm60-6.0.1_1.t= ar The compression ratio is about 2.9, and the system time required for reading the file (the copy to /tmp on tmpfs) is about doubled due to the decompression performed by ZFS (compared to copying from a ZFS file system without compression). The compression time is not (fully) accounted to the process writing the file to the compressed file system, but the real time is what you observe from the point of view of the build process. For comparison with user land compression $ cd /tmp $ time bzip2 -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.tbz real 0m57.487s user 0m57.318s sys 0m0.169s $ ls -ls llvm60-6.0.1_1.tbz 241348 -rw-r--r-- 1 root wheel 247136911 Jul 22 10:15 llvm60-6.0.1_1.t= bz $ time xz -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz real 0m28.899s user 0m28.714s sys 0m0.183s $ ls -ls llvm60-6.0.1_1.txz 191412 -rw-r--r-- 1 root wheel 196004988 Jul 22 10:17 llvm60-6.0.1_1.t= xz $ time xz -9 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz real 3m41.291s user 3m40.973s sys 0m0.314s $ ls -ls /tmp/llvm60-6.0.1_1.txz 104196 -rw-r--r-- 1 root wheel 106694296 Jul 22 10:25 llvm60-6.0.1_1.t= xz The compression achieved by using XZ is much higher than that of zstd-2, but the time saving of using tar instead of txz is substantial, too. I have just performed a test with zstd-19 for /usr/packages and found, th= at the compression ratio is within 33% of that of xz -1 at much higher speed= (and nearly as good as bzip -1), and I'm considering to keep this setting= for /usr/packages: $ ls -ls llvm60-6.0.1_1.tar 254307 -rw-r--r-- 1 root wheel 855781376 Jul 22 10:28 llvm60-6.0.1_1.t= ar Regards, STefan --kntKnBcj15fl8N9EsOR9JonOpK2c4s3Ak-- --OK6F7Lx6pySpdaXcd7k6FAD3z5tXBgj51 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAmD5ME8FAwAAAAAACgkQR+u171r99USR Xwf/dH/gLYyyw/UO89qfPWMvD32pDW1c1kr7yVlcoRLCHPZuHcUM/01Hx77T9w0trN5lU8NzpDI5 23wI8NBozV0nhNSRvKiRoiARIXoXaEIuL76ti2UqZfAOlK/bKetIUBFTdujaIvxcErMGukqrL+v2 MXFJTQYq3ocgf5TVaO2TEexCve0sEob9BUTh/v31s2SHtvApIIU/W3AGjtjQjSbUeD1kwPv4ibZw 8jJveNAuaz0zUR+wOn5cV/xvWafUvJtDGAghRj6ChMPDt4IjHnTcmuA/y5vL3Z+FGwNlj7hBpzAF OnBmegoIBQ8fYrElHkeMr5Ii2dPOCZUvzB4JuRs1yQ== =LcV5 -----END PGP SIGNATURE----- --OK6F7Lx6pySpdaXcd7k6FAD3z5tXBgj51--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e17015aa-eaf7-0e8a-fdab-e4c2b7c19f04>