Date: Tue, 3 Oct 2023 11:56:11 -0600 From: Warner Losh <imp@bsdimp.com> To: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= <des@freebsd.org> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: something magic about the size of a ports tree Message-ID: <CANCZdfrR1tHRdiNiQ1kk0%2BQMLt4gegkO0Xocp=1B5a-WR6cCCg@mail.gmail.com> In-Reply-To: <86y1gjhdx7.fsf@ltc.des.no> References: <ZRw8x58bxtp26A8e@c720-1400094.fritz.box> <86y1gjhdx7.fsf@ltc.des.no>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000e0c1b10606d39b60 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 3, 2023, 10:24 AM Dag-Erling Sm=C3=B8rgrav <des@freebsd.org> wr= ote: > Matthias Apitz <guru@unixarea.de> writes: > > I have on my poudriere build host a ports tree and wanted to move it to > > the host where the resulting packages are installed: > > > > root@jet:/usr/local/poudriere/ports # du -sh ports20230806 > > 397M ports20230806 > > root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806 > > root@jet:/usr/local/poudriere/ports # ls -lh p.tar > > -rw-r--r-- 1 root wheel 672M Oct 3 18:00 p.tar > > > > already the size of the tar file is somewhat magic; but if you un-tar i= t > > on the other host I will get: > > > > [guru@c720-1400094 ~]$ ls -lh p.tar > > -rw-r--r-- 1 guru wheel 672M 3 oct. 18:00 p.tar > > [guru@c720-1400094 ~]$ tar xf p.tar > > [guru@c720-1400094 ~]$ du -sh ports20230806 > > 1,2G ports20230806 > > > > How this is possible? > > Most files in the ports tree are very small. On disk, each file gets > rounded up to the nearest multiple of the filesystem block size, which > could be as small as 512 bytes or as large as 8 kB (or even more in > pathological cases). In a tarball, they get rounded up to the nearest > multiple of 512 bytes plus an additional 512 bytes per file for > metadata. > > For instance, your average distinfo file (of which there are 30k in the > ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an > FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem. > > Note that if the target system is FreeBSD 14 or newer, you can simply > mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`). > Do we support any compression on top of that? Has support for poudriere been added for it? Aldo I want a pony.... I'm mostly curious... I have no immediate plans here (though aligning with the boot loader and supporting this on a block device to support rootfs would be cool). Maybe some or all of these wishes would make good GSOC projects? Warner DES > -- > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org > > --000000000000e0c1b10606d39b60 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Tue, Oct 3, 2023, 10:24 AM Dag-Erling Sm=C3=B8rgrav= <<a href=3D"mailto:des@freebsd.org">des@freebsd.org</a>> wrote:<br><= /div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-le= ft:1px #ccc solid;padding-left:1ex">Matthias Apitz <<a href=3D"mailto:gu= ru@unixarea.de" target=3D"_blank" rel=3D"noreferrer">guru@unixarea.de</a>&g= t; writes:<br> > I have on my poudriere build host a ports tree and wanted to move it t= o<br> > the host where the resulting packages are installed:<br> ><br> > root@jet:/usr/local/poudriere/ports # du -sh ports20230806<br> > 397M=C2=A0 =C2=A0 ports20230806<br> > root@jet:/usr/local/poudriere/ports # tar cf p.tar ports20230806<br> > root@jet:/usr/local/poudriere/ports # ls -lh p.tar<br> > -rw-r--r--=C2=A0 1 root wheel=C2=A0 672M Oct=C2=A0 3 18:00 p.tar<br> ><br> > already the size of the tar file is somewhat magic; but if you un-tar = it<br> > on the other host I will get:<br> ><br> > [guru@c720-1400094 ~]$ ls -lh p.tar<br> > -rw-r--r--=C2=A0 1 guru wheel=C2=A0 672M=C2=A0 3 oct.=C2=A0 18:00 p.ta= r<br> > [guru@c720-1400094 ~]$ tar xf p.tar<br> > [guru@c720-1400094 ~]$ du -sh ports20230806<br> > 1,2G=C2=A0 ports20230806<br> ><br> > How this is possible?<br> <br> Most files in the ports tree are very small.=C2=A0 On disk, each file gets<= br> rounded up to the nearest multiple of the filesystem block size, which<br> could be as small as 512 bytes or as large as 8 kB (or even more in<br> pathological cases).=C2=A0 In a tarball, they get rounded up to the nearest= <br> multiple of 512 bytes plus an additional 512 bytes per file for<br> metadata.<br> <br> For instance, your average distinfo file (of which there are 30k in the<br> ports tree) is only 200-250 bytes long, but it occupies 512 bytes on an<br> FFS filesystem, 1 kB in a tarball, and 4 kB on a typical ZFS filesystem.<br= > <br> Note that if the target system is FreeBSD 14 or newer, you can simply<br> mount the tarball (`sudo mount -rt tarfs p.tar /usr/ports`).<br></blockquot= e></div></div><div dir=3D"auto"><br></div><div dir=3D"auto">Do we support a= ny compression on top of that? Has support for poudriere been added for it?= </div><div dir=3D"auto"><br></div><div dir=3D"auto">Aldo I want a pony.... = I'm mostly curious... I have no immediate plans here (though aligning w= ith the boot loader and supporting this on a block device to support rootfs= would be cool). Maybe some or all of these wishes would make good GSOC pro= jects?</div><div dir=3D"auto"><br></div><div dir=3D"auto">Warner</div><div = dir=3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockq= uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"> DES<br> -- <br> Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org<br> <br> </blockquote></div></div></div> --000000000000e0c1b10606d39b60--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfrR1tHRdiNiQ1kk0%2BQMLt4gegkO0Xocp=1B5a-WR6cCCg>