From nobody Tue Oct 3 23:14:47 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S0YZ86yNkz4w8h7 for ; Tue, 3 Oct 2023 23:14:56 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4S0YZ80m5xz4P50 for ; Tue, 3 Oct 2023 23:14:56 +0000 (UTC) (envelope-from allanjude@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 209.51.186.6 is neither permitted nor denied by domain of allanjude@freebsd.org) smtp.mailfrom=allanjude@freebsd.org; dmarc=none Received: from [10.1.1.110] (unknown [10.2.89.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 544B811B96 for ; Tue, 3 Oct 2023 23:14:49 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 544B811B96 Message-ID: Date: Tue, 3 Oct 2023 19:14:47 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: something magic about the size of a ports tree Content-Language: en-US To: freebsd-current@freebsd.org References: <86y1gjhdx7.fsf@ltc.des.no> From: Allan Jude In-Reply-To: <86y1gjhdx7.fsf@ltc.des.no> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.95 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.976]; NEURAL_HAM_SHORT(-0.97)[-0.972]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_TLS_ALL(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; R_SPF_SOFTFAIL(0.00)[~all:c]; FREEFALL_USER(0.00)[allanjude]; TO_DOM_EQ_FROM_DOM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4S0YZ80m5xz4P50 On 2023-10-03 12:24, Dag-Erling Smørgrav wrote: > Matthias Apitz writes: >> I have on my poudriere build host a ports tree and wanted to move it to >> the host where the resulting packages are installed: >> >> root@jet:/usr/local/poudriere/ports # du -sh ports20230806 >> 397M ports20230806 >> root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806 >> root@jet:/usr/local/poudriere/ports # ls -lh p.tar >> -rw-r--r-- 1 root wheel 672M Oct 3 18:00 p.tar >> >> already the size of the tar file is somewhat magic; but if you un-tar it >> on the other host I will get: >> >> [guru@c720-1400094 ~]$ ls -lh p.tar >> -rw-r--r-- 1 guru wheel 672M 3 oct. 18:00 p.tar >> [guru@c720-1400094 ~]$ tar xf p.tar >> [guru@c720-1400094 ~]$ du -sh ports20230806 >> 1,2G ports20230806 >> >> How this is possible? > > Most files in the ports tree are very small. On disk, each file gets > rounded up to the nearest multiple of the filesystem block size, which > could be as small as 512 bytes or as large as 8 kB (or even more in > pathological cases). In a tarball, they get rounded up to the nearest > multiple of 512 bytes plus an additional 512 bytes per file for > metadata. > > For instance, your average distinfo file (of which there are 30k in the > ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an > FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem. > As an interesting side note to this, if ZFS is able to compress the file to under 112 bytes, ZFS will not allocate a sector, but instead store the file in an "embedded blockpointer", basically using the space it would normally store the LBAs and checksum of the file, to store the actual file data, resulting in a file that appears to use 0 bytes of space, because it entirely fits in the indirect block that would have pointed to the block itself. > Note that if the target system is FreeBSD 14 or newer, you can simply > mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`). > > DES -- Allan Jude