Date: Mon, 25 Mar 2024 19:42:35 -0400 From: Mark Saad <nonesuch@longcount.org> To: FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: TarFS Message-ID: <CAMXt9NYNmaCJhSmFg4QAS2d7Lia1oEVAgTeas50w=%2BohsdrwCw@mail.gmail.com> In-Reply-To: <86edbyh0od.fsf@ltc.des.dev> References: <CAMXt9NYOA4KJu86gnCwOZeDp31gsxwwrzdb%2Bc6KLsOQxxD6r5A@mail.gmail.com> <86edbyh0od.fsf@ltc.des.dev>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Mon, Mar 25, 2024 at 9:59 AM Dag-Erling Smørgrav <des@freebsd.org> wrote: > Mark Saad <nonesuch@longcount.org> writes: > > I was wondering if anyone has started to play with tarfs in FreeBSD 14? > > I wrote it, does that count? > > > It appears to puke with larger tarballs. > > On the contrary, one of the use cases I tested was buildworld from a > tarball of the source tree, which it handled just fine. > > Dag let me just say , great work. I really like this. > > For example, if I fetch a copy of ports or pkgsrc and decompress it to > > just a posix tar archive I can't get it to mount. > > > > root@mono:/home/nonesuch # file ports.tar > > ports.tar: POSIX tar archive > > root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports > > mount: ./ports.tar: Inappropriate file type or format > > % fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst > ports.tar.zst 47 MB 5431 kBps > 09s > % sudo mount -rt tarfs $PWD/ports.tar.zst /mnt > mount: /home/des/ports.tar.zst: Inappropriate file type or format > % dmesg | grep tarfs > tarfs_alloc_one: unsupported global extended header at 0 > % zcat ports.tar.zst | hexdump -C | head -3 > 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 > |pax_global_heade| > 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |r...............| > 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > > This is a non-standard extension header emitted by `git archive` which > contains metadata about the git tree from which the archive was created. > I believe you can safely strip it off (zcat ports.tar.zst | dd bs=1k > skip=1 >ports.tar), but you'll run into issues with very long paths in > devel/electron*. I will take a closer look when I find the time. > > Note that tarfs can mount tarballs compressed with zstd, but performance > will be poor unless you create a multi-frame archive (try using --zstd > --options=zstd:frame-per-file,zstd:min-frame-size=65536). > > DES > So I am not sure exactly what tarfs is yelling about with pkgsrc.tar , but I wonder if it's a by-product of netbsd using pax . NetBSD tar's dont work , OpenBSD stuff does and they use a ustar format. root@mono:/home/nonesuch # cat pkgsrc.tar |hexdump -C |head -3 00000000 70 6b 67 73 72 63 2f 00 00 00 00 00 00 00 00 00 |pkgsrc/.........| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * root@mono:/home/nonesuch # cat ports.tar |hexdump -C |head -3 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_heade| 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r...............| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| root@mono:/home/nonesuch # cat xenocara.tar |hexdump -C |head -3 00000000 2e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * Two other questions ( I have not tried to read the code , please forgive my assumptions ) behind the scenes are the tarballs extracted over some fs, or mounted directly and read ? Second are there any contradictions with tarfs ; can they be nfs exported, mounted on a tmpfs etc . Again good work. C an we expect to see a libarchive-fs next :) > -- > Dag-Erling Smørgrav - des@FreeBSD.org > -- mark saad | nonesuch@longcount.org [-- Attachment #2 --] <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 25, 2024 at 9:59 AM Dag-Erling Smørgrav <<a href="mailto:des@freebsd.org">des@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Mark Saad <<a href="mailto:nonesuch@longcount.org" target="_blank">nonesuch@longcount.org</a>> writes:<br> > I was wondering if anyone has started to play with tarfs in FreeBSD 14?<br> <br> I wrote it, does that count?<br> <br> > It appears to puke with larger tarballs.<br> <br> On the contrary, one of the use cases I tested was buildworld from a<br> tarball of the source tree, which it handled just fine.<br> <br></blockquote><div><br></div><div>Dag let me just say , great work. I really like this. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> > For example, if I fetch a copy of ports or pkgsrc and decompress it to<br> > just a posix tar archive I can't get it to mount.<br> ><br> > root@mono:/home/nonesuch # file ports.tar<br> > ports.tar: POSIX tar archive<br> > root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports<br> > mount: ./ports.tar: Inappropriate file type or format<br> <br> % fetch <a href="ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst" rel="noreferrer" target="_blank">ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst</a><br>; ports.tar.zst 47 MB 5431 kBps 09s<br> % sudo mount -rt tarfs $PWD/ports.tar.zst /mnt<br> mount: /home/des/ports.tar.zst: Inappropriate file type or format<br> % dmesg | grep tarfs <br> tarfs_alloc_one: unsupported global extended header at 0<br> % zcat ports.tar.zst | hexdump -C | head -3<br> 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_heade|<br> 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r...............|<br> 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|<br> <br> This is a non-standard extension header emitted by `git archive` which<br> contains metadata about the git tree from which the archive was created.<br> I believe you can safely strip it off (zcat ports.tar.zst | dd bs=1k<br> skip=1 >ports.tar), but you'll run into issues with very long paths in<br> devel/electron*. I will take a closer look when I find the time.<br> <br> Note that tarfs can mount tarballs compressed with zstd, but performance<br> will be poor unless you create a multi-frame archive (try using --zstd<br> --options=zstd:frame-per-file,zstd:min-frame-size=65536).<br> <br> DES<br> </blockquote><div><br></div><div>So I am not sure exactly what tarfs is yelling about with pkgsrc.tar , but I wonder if it's a by-product of netbsd using pax . <br></div><div>NetBSD tar's dont work , OpenBSD stuff does and they use a ustar format. <br></div><div><br></div><div>root@mono:/home/nonesuch # cat pkgsrc.tar |hexdump -C |head -3 <br>00000000 70 6b 67 73 72 63 2f 00 00 00 00 00 00 00 00 00 |pkgsrc/.........|<br>00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|<br>*<br>root@mono:/home/nonesuch # cat ports.tar |hexdump -C |head -3<br>00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_heade|<br>00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r...............|<br>00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|<br></div><div><br></div><div><br></div><div>root@mono:/home/nonesuch # cat xenocara.tar |hexdump -C |head -3<br>00000000 2e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|<br>00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|<br>*<br></div></div><div class="gmail_quote"><div><br></div><div><br></div><div>Two other questions ( I have not tried to read the code , please forgive my assumptions ) behind the scenes are the tarballs extracted over some fs, or mounted directly and read ?</div><div>Second are there any contradictions with tarfs ; can they be nfs exported, mounted on a tmpfs etc . <br></div><div><br></div><div>Again good work. C an we expect to see a libarchive-fs next :)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">-- <br> Dag-Erling Smørgrav - des@FreeBSD.org<br> </blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">mark saad | <a href="mailto:nonesuch@longcount.org" target="_blank">nonesuch@longcount.org</a><br></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMXt9NYNmaCJhSmFg4QAS2d7Lia1oEVAgTeas50w=%2BohsdrwCw>