Date: Thu, 22 Jul 2021 07:59:55 -0700 From: Kevin Bowling <kevin.bowling@kev009.com> To: Stefan Esser <se@freebsd.org> Cc: Alexey Dokuchaev <danfe@freebsd.org>, Baptiste Daroussin <bapt@freebsd.org>, "Matthew D. Fuller" <fullermd@over-yonder.net>, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org, ports-committers@freebsd.org Subject: Re: git: ec2764d5ec49 - main - Mk/bsd.port.mk: prepare the land for pkg 1.17 Message-ID: <CAK7dMtB6V0DB0iTG7yT2TzBSjGm8AdUJNMGjm67idEAOFP26hQ@mail.gmail.com> In-Reply-To: <e17015aa-eaf7-0e8a-fdab-e4c2b7c19f04@freebsd.org> References: <202107211223.16LCNDjf070492@gitrepo.freebsd.org> <YPgxkAglZ2GRunni@FreeBSD.org> <20210721145201.5jcynsooy4hva7ir@aniel.nours.eu> <YPjT%2BMuwECRSMe4X@FreeBSD.org> <YPjZhks2qvKuUU7t@over-yonder.net> <e17015aa-eaf7-0e8a-fdab-e4c2b7c19f04@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jul 22, 2021 at 1:46 AM Stefan Esser <se@freebsd.org> wrote: > Am 22.07.21 um 04:35 schrieb Matthew D. Fuller: > > On Thu, Jul 22, 2021 at 02:12:08AM +0000 I heard the voice of > > Alexey Dokuchaev, and lo! it spake thus: > >> > >> I've switched to PKG_NOCOMPRESS and > >> PKG_DEPENDS+=3Dlbzip2:archivers/lbzip2 since 2017 when I got > >> dissatisfied with ridiculously long LLVM packaging times* > > > > I've wished (not enough yet to see if I could manage it, but) to be > > able to use something like `gzip -1` for compression on some poudriere > > instances; all the clients pulling from it are on the same LAN if not > > the same physical machine, so bandwidth is trivially enough, and > > compression/decompression times eat up a significant amount of the > > cost of updates... > > I have used .tar without compression on ZFS file systems that use zstd-2 > compression (my default for all file systems not meant to hold already > compressed distfiles or media files). > > This is the fastest and most efficient configuration IMHO (if you are > using ZFS anyway), with an effective compression ratio of more than 3 > in practice and the compression time hidden in the filesystem layer. > I=E2=80=99m skeptical of the performance of the zstd implementation in open= zfs as it stands, you may lose out on overall system performance by using it right now. I believe mjg@ did some tests with buildworld and poudriere and found the pool threading design in it doesn=E2=80=99t scale up well (see https://github.com/openzfs/zfs/blob/master/module/zstd/zfs_zstd.c). So a user of poudriere would probably not want to do it like this, if you can build up parallelism userspace is the right place to do the compression. > If you are transferring these files over slow networks, that is not a > good setup, but within a system or on a LAN it does not matter. > > Example of a randomly selected old package file of significant size: > > $ cd /usr/ports/portmaster-backups > $ time cp llvm60-6.0.1_1.tar /tmp/ > > real 0m0.928s > user 0m0.000s > sys 0m0.930s > > $ time cp /tmp/llvm60-6.0.1_1.tar . > > real 0m0.572s > user 0m0.000s > sys 0m0.575s > > $ ls -ls llvm60-6.0.1_1.tar /tmp/llvm60-6.0.1_1.tar > 835724 -rw-r--r-- 1 root wheel 855781376 Jul 22 09:57 > /tmp/llvm60-6.0.1_1.tar > 290613 -rw-r--r-- 1 root wheel 855781376 Jul 22 09:58 llvm60-6.0.1_1.t= ar > > The compression ratio is about 2.9, and the system time required for > reading the file (the copy to /tmp on tmpfs) is about doubled due > to the decompression performed by ZFS (compared to copying from a > ZFS file system without compression). > > The compression time is not (fully) accounted to the process writing > the file to the compressed file system, but the real time is what > you observe from the point of view of the build process. > > For comparison with user land compression > > $ cd /tmp > $ time bzip2 -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.tbz > > real 0m57.487s > user 0m57.318s > sys 0m0.169s > > $ ls -ls llvm60-6.0.1_1.tbz > 241348 -rw-r--r-- 1 root wheel 247136911 Jul 22 10:15 llvm60-6.0.1_1.t= bz > > $ time xz -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz > > real 0m28.899s > user 0m28.714s > sys 0m0.183s > > $ ls -ls llvm60-6.0.1_1.txz > 191412 -rw-r--r-- 1 root wheel 196004988 Jul 22 10:17 llvm60-6.0.1_1.t= xz > > $ time xz -9 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz > > real 3m41.291s > user 3m40.973s > sys 0m0.314s > > $ ls -ls /tmp/llvm60-6.0.1_1.txz > 104196 -rw-r--r-- 1 root wheel 106694296 Jul 22 10:25 llvm60-6.0.1_1.t= xz > > The compression achieved by using XZ is much higher than that of zstd-2, > but the time saving of using tar instead of txz is substantial, too. > > I have just performed a test with zstd-19 for /usr/packages and found, th= at > the compression ratio is within 33% of that of xz -1 at much higher speed > (and nearly as good as bzip -1), and I'm considering to keep this setting > for /usr/packages: > > $ ls -ls llvm60-6.0.1_1.tar > 254307 -rw-r--r-- 1 root wheel 855781376 Jul 22 10:28 llvm60-6.0.1_1.t= ar > > Regards, STefan > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAK7dMtB6V0DB0iTG7yT2TzBSjGm8AdUJNMGjm67idEAOFP26hQ>