Date: Mon, 25 Mar 2024 19:42:35 -0400 From: Mark Saad <nonesuch@longcount.org> To: FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: TarFS Message-ID: <CAMXt9NYNmaCJhSmFg4QAS2d7Lia1oEVAgTeas50w=%2BohsdrwCw@mail.gmail.com> In-Reply-To: <86edbyh0od.fsf@ltc.des.dev> References: <CAMXt9NYOA4KJu86gnCwOZeDp31gsxwwrzdb%2Bc6KLsOQxxD6r5A@mail.gmail.com> <86edbyh0od.fsf@ltc.des.dev>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000065f0a061484bb93 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Mar 25, 2024 at 9:59=E2=80=AFAM Dag-Erling Sm=C3=B8rgrav <des@freeb= sd.org> wrote: > Mark Saad <nonesuch@longcount.org> writes: > > I was wondering if anyone has started to play with tarfs in FreeBSD 14? > > I wrote it, does that count? > > > It appears to puke with larger tarballs. > > On the contrary, one of the use cases I tested was buildworld from a > tarball of the source tree, which it handled just fine. > > Dag let me just say , great work. I really like this. > > For example, if I fetch a copy of ports or pkgsrc and decompress it to > > just a posix tar archive I can't get it to mount. > > > > root@mono:/home/nonesuch # file ports.tar > > ports.tar: POSIX tar archive > > root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports > > mount: ./ports.tar: Inappropriate file type or format > > % fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst > ports.tar.zst 47 MB 5431 kBps > 09s > % sudo mount -rt tarfs $PWD/ports.tar.zst /mnt > mount: /home/des/ports.tar.zst: Inappropriate file type or format > % dmesg | grep tarfs > tarfs_alloc_one: unsupported global extended header at 0 > % zcat ports.tar.zst | hexdump -C | head -3 > 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 > |pax_global_heade| > 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |r...............| > 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > > This is a non-standard extension header emitted by `git archive` which > contains metadata about the git tree from which the archive was created. > I believe you can safely strip it off (zcat ports.tar.zst | dd bs=3D1k > skip=3D1 >ports.tar), but you'll run into issues with very long paths in > devel/electron*. I will take a closer look when I find the time. > > Note that tarfs can mount tarballs compressed with zstd, but performance > will be poor unless you create a multi-frame archive (try using --zstd > --options=3Dzstd:frame-per-file,zstd:min-frame-size=3D65536). > > DES > So I am not sure exactly what tarfs is yelling about with pkgsrc.tar , but I wonder if it's a by-product of netbsd using pax . NetBSD tar's dont work , OpenBSD stuff does and they use a ustar format. root@mono:/home/nonesuch # cat pkgsrc.tar |hexdump -C |head -3 00000000 70 6b 67 73 72 63 2f 00 00 00 00 00 00 00 00 00 |pkgsrc/.........| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * root@mono:/home/nonesuch # cat ports.tar |hexdump -C |head -3 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_heade| 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r...............| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| root@mono:/home/nonesuch # cat xenocara.tar |hexdump -C |head -3 00000000 2e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * Two other questions ( I have not tried to read the code , please forgive my assumptions ) behind the scenes are the tarballs extracted over some fs, or mounted directly and read ? Second are there any contradictions with tarfs ; can they be nfs exported, mounted on a tmpfs etc . Again good work. C an we expect to see a libarchive-fs next :) > -- > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org > --=20 mark saad | nonesuch@longcount.org --000000000000065f0a061484bb93 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Mon, Mar 25, 2024 at 9:59=E2=80=AF= AM Dag-Erling Sm=C3=B8rgrav <<a href=3D"mailto:des@freebsd.org">des@free= bsd.org</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"= margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef= t:1ex">Mark Saad <<a href=3D"mailto:nonesuch@longcount.org" target=3D"_b= lank">nonesuch@longcount.org</a>> writes:<br> > I was wondering if anyone has started to play with tarfs in FreeBSD 14= ?<br> <br> I wrote it, does that count?<br> <br> > It appears to puke with larger tarballs.<br> <br> On the contrary, one of the use cases I tested was buildworld from a<br> tarball of the source tree, which it handled just fine.<br> <br></blockquote><div><br></div><div>Dag let me just say , great work. I re= ally like this. <br></div><div>=C2=A0</div><blockquote class=3D"gmail_quote= " style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);= padding-left:1ex"> > For example, if I fetch a copy of ports or pkgsrc and decompress it to= <br> > just a posix tar archive I can't get it to mount.<br> ><br> > root@mono:/home/nonesuch # file ports.tar<br> > ports.tar: POSIX tar archive<br> > root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports<br> > mount: ./ports.tar: Inappropriate file type or format<br> <br> % fetch <a href=3D"ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.= zst" rel=3D"noreferrer" target=3D"_blank">ftp://ftp.freebsd.org/pub/FreeBSD= /ports/ports/ports.tar.zst</a><br> ports.tar.zst=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A047 MB 5431 kBps=C2=A0 =C2=A0 09s<br> % sudo mount -rt tarfs $PWD/ports.tar.zst /mnt<br> mount: /home/des/ports.tar.zst: Inappropriate file type or format<br> % dmesg | grep tarfs=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 <br> tarfs_alloc_one: unsupported global extended header at 0<br> % zcat ports.tar.zst | hexdump -C | head -3<br> 00000000=C2=A0 70 61 78 5f 67 6c 6f 62=C2=A0 61 6c 5f 68 65 61 64 65=C2=A0 = |pax_global_heade|<br> 00000010=C2=A0 72 00 00 00 00 00 00 00=C2=A0 00 00 00 00 00 00 00 00=C2=A0 = |r...............|<br> 00000020=C2=A0 00 00 00 00 00 00 00 00=C2=A0 00 00 00 00 00 00 00 00=C2=A0 = |................|<br> <br> This is a non-standard extension header emitted by `git archive` which<br> contains metadata about the git tree from which the archive was created.<br= > I believe you can safely strip it off (zcat ports.tar.zst | dd bs=3D1k<br> skip=3D1 >ports.tar), but you'll run into issues with very long path= s in<br> devel/electron*.=C2=A0 I will take a closer look when I find the time.<br> <br> Note that tarfs can mount tarballs compressed with zstd, but performance<br= > will be poor unless you create a multi-frame archive (try using --zstd<br> --options=3Dzstd:frame-per-file,zstd:min-frame-size=3D65536).<br> <br> DES<br> </blockquote><div><br></div><div>So I am not sure exactly what tarfs is yel= ling about with pkgsrc.tar , but I wonder if it's a by-product of netbs= d using pax . <br></div><div>NetBSD tar's dont work , OpenBSD stuff doe= s and they use a ustar format. <br></div><div><br></div><div>root@mono:/hom= e/nonesuch # cat pkgsrc.tar |hexdump -C |head -3 <br>00000000 =C2=A070 6b 6= 7 73 72 63 2f 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|pkgsrc/.........|<br>= 00000010 =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0= |................|<br>*<br>root@mono:/home/nonesuch # cat ports.tar =C2=A0|= hexdump -C |head -3<br>00000000 =C2=A070 61 78 5f 67 6c 6f 62 =C2=A061 6c 5= f 68 65 61 64 65 =C2=A0|pax_global_heade|<br>00000010 =C2=A072 00 00 00 00 = 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|r...............|<br>00000020= =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|.......= .........|<br></div><div><br></div><div><br></div><div>root@mono:/home/none= such # cat xenocara.tar =C2=A0 |hexdump -C |head -3<br>00000000 =C2=A02e 00= 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|................|<b= r>00000010 =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2= =A0|................|<br>*<br></div></div><div class=3D"gmail_quote"><div><= br></div><div><br></div><div>Two other questions ( I have not tried to read= the code , please forgive my assumptions )=C2=A0 behind the scenes are the= tarballs extracted over some fs, or mounted directly and read ?</div><div>= Second are there any contradictions with tarfs ; can they be nfs exported, = mounted on a tmpfs etc . <br></div><div><br></div><div>Again good work. C a= n we expect to see a libarchive-fs next :)<br></div><div>=C2=A0</div><block= quote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1= px solid rgb(204,204,204);padding-left:1ex">-- <br> Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org<br> </blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre= fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">mark saad | <= a href=3D"mailto:nonesuch@longcount.org" target=3D"_blank">nonesuch@longcou= nt.org</a><br></div></div> --000000000000065f0a061484bb93--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMXt9NYNmaCJhSmFg4QAS2d7Lia1oEVAgTeas50w=%2BohsdrwCw>