From nobody Mon Mar 25 01:10:01 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V2vxP5rvRz5DvrH for ; Mon, 25 Mar 2024 01:10:17 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (pdx.rh.CN85.dnsmgr.net [65.75.216.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4V2vxP01KVz3wcZ for ; Mon, 25 Mar 2024 01:10:16 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Authentication-Results: mx1.freebsd.org; none Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 42P1A2hA035046; Sun, 24 Mar 2024 18:10:02 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 42P1A1nF035045; Sun, 24 Mar 2024 18:10:01 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <202403250110.42P1A1nF035045@gndrsh.dnsmgr.net> Subject: Re: Removing or changing the ping interval restriction for non-root users In-Reply-To: <4f8b035c-b2cc-4606-a691-f1d86827282b@app.fastmail.com> To: Tom Forbes Date: Sun, 24 Mar 2024 18:10:01 -0700 (PDT) CC: Cy Schubert , freebsd-hackers@FreeBSD.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:10494, ipnet:65.75.216.0/23, country:US] X-Rspamd-Queue-Id: 4V2vxP01KVz3wcZ > I've personally never come across this limitation on any system that I've used, however it is a good point that there are bound to be systems that have the same limitation. After digging a bit more into the history the limitation was added in this commit[1] in 1998 with the explicit intention to "secure options from user-level D.O.S attacks". > > A lot of things have changed since 1998, and setting an arbitrary high limit to prevent "ping" and "ping6" from being used to DOS networked devices would be a pretty suspect decision if it was suggested today. I expect a few other distributions have inherited this limit from the original contribution, but to me that doesn't lend a strong argument to keeping it if the underlying reason it exists doesn't make sense anymore _and_ if removing/reducing it is a backwards-compatible, simple and non-invasive change. IIRC the history on this is a bit more complex, originally only an interger value of interval was allowed, and hince the lower limit WAS 1 second, then fractional intervals got added. Flood pinging (-f) has always been restricted to root. A debian bookworm ping(1) has minimal interval of 200mS per the manual page, so I would say there are other major systems showing that a restriciton on ping rates is a reasonable situation. > > Tom > 1. https://github.com/freebsd/freebsd-src/commit/526f06b278d9252add168aa18b60242c08771165 > > On Sun, 24 Mar 2024, at 2:48 PM, Cy Schubert wrote: > > On March 24, 2024 5:57:01 AM PDT, Tom Forbes wrote: > > >Hello, > > >I maintain a small project called gping[1] that recently added support for FreeBSD. One of the issues I ran into with running this on FreeBSD was that the `ping` command seems to disallow intervals of less than 1 second if you are not running as root[2]. This check was last touched 23 years ago and I'm curious as to why this restriction exists? I assume it's from an earlier time in the internets history, and perhaps is related to potential misuse of the command to flood targets with packets via ping? What other systems have you tried gping on that does NOT have your "issue?" > > > > > >If it is then I'd like to suggest that this limitation be removed or is reduced to `0.1` seconds instead? Using `ping` for this kind of thing isn't a viable attack today, and the 1 second limitation seems like it would get in the way of useful uses of the ping command. I can not agree with your 0.1 second limit, I could probably be convinced the lower limit of 200mS though. > > > > > >Also this is my first post to any *BSD mailing list, so please let me know if this is not the right place to ask this question or propose this! > > > > > >Thanks, > > >Tom > > > > > >1. https://github.com/orf/gping > > >2. https://github.com/freebsd/freebsd-src/blame/8a56ef8d75b42ee7228247466c8c1712de6e3b6f/sbin/ping/ping6.c#L441 > > Other UNIX-like systems have the same restriction. At $JOB we use Solaris and various Linux systems. All maintain the same restriction. Other BSDs are the same.I don't think FreeBSD should be an outlier. > > > > Maybe setgid bit or a capability to remove the restriction may be a better solution. But to reduce the timeout to essentially remove it is IMO unwise. I concur. > > -- > > Cy Schubert > > FreeBSD UNIX: Web: https://FreeBSD.org -- Rod Grimes rgrimes@freebsd.org From nobody Mon Mar 25 13:59:46 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3F1K5Wd6z5F0ql for ; Mon, 25 Mar 2024 13:59:49 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3F1K53tTz4DyG; Mon, 25 Mar 2024 13:59:49 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711375189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6i9lSmjo0POIXEEYEgl9rQwR+PHcG2UbIXjyCKd3JFY=; b=W+k6ipKPgTYg/ojAGnKNmYvda914qJbRZcZfg26S0lWo3Fht0mxDoF8Qw1qfnOB4TPrbeJ VVK/A7dD2VtWYUsqWw52TKlGtWrieNnH2eGSaB+Tv6Bu0GZWmN5TeYjwvY41nOXKWD2MlC BQXIBAYU7ozFcDSuhKlKg4DCdW/yGIXSIp3WvOLGMo+wAYCPHMwN/HoO3jK9d1TJ+rsfAk 3JH+WadJOjTaucSdG9y19u8kLZ10bdHHKSaMOgnnuvCkLupJU6qnmfhkFkm0tiY0cicC4X tS7H/+KG3hv/FQDxvUksZbLA5InwFTbbK14vTN0KkBEZwdqfLr+NJVPyyCdi8A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711375189; a=rsa-sha256; cv=none; b=mxn4TOyOBoZRKnOZlrpJcg8lqKzKUuun6hHBfqP7whOXIFYvWUb88XJSJUg5jSlz4DjB3V X/OJG3FGYRZKXpdavNRP2lP1Q3C3TPT+r9P4hp8Xwjuo06hXag8h8DO0W4KSEOHKOrdXN0 V868sQ56Gs3X/Exuo+CrvQj9kxAtD/ZXrII0o/D3Iys2t7G8o+hWF5/A/dTUF6qiQvoqoJ tKXjGzn7TkaHLoOF2xlzJ19X38inOsasct+CzRF9HEDLqCVNGTR57eQmZUF2EDRVrlwX8G /SQI4lFgqNrbJu1p0NOmvxYNPhocpnozyTOns7kRHbY//YYgFA97d+In7iyXAA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711375189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6i9lSmjo0POIXEEYEgl9rQwR+PHcG2UbIXjyCKd3JFY=; b=hjPbsGZb3hpeoxBF9wHscPGaMEPp9MxXPYEwkUOGJtq1ezEpFuyrOyLZJkPGaJ0GU1G4vb K/NCdYDPppRx6t85jiMEL/fgfwqoLU2PGPWjTlAAxsJfypcqTcNkJ6j6B3swLKTRAq8Spz aQL7OSeE0AxGvpsEevO3DuRU/d4HnESfpJ7e4ZVqhdjyYe9/H09LnjBN1gxBwS/NmLvY+c c2lp7hjbfqb2wZsvUGNASDyfZJYbpruWkYtLatIiPLA4f37yvOWuvqkmNLlY7NcZrZ7Veg yXxAhXKkyU1w7xJIIOePaQvWdxyAg82GhJo5OCwD1h14PJAqacZVbR5gCRGtPg== Received: from ltc.des.dev (163.23.65.37.rev.sfr.net [37.65.23.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4V3F1K3qz2zV6B; Mon, 25 Mar 2024 13:59:49 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.dev (Postfix, from userid 1001) id D592E781B9; Mon, 25 Mar 2024 14:59:46 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Mark Saad Cc: FreeBSD Hackers Subject: Re: TarFS In-Reply-To: (Mark Saad's message of "Fri, 22 Mar 2024 11:48:21 -0400") References: User-Agent: Gnus/5.13 (Gnus v5.13) Date: Mon, 25 Mar 2024 14:59:46 +0100 Message-ID: <86edbyh0od.fsf@ltc.des.dev> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mark Saad writes: > I was wondering if anyone has started to play with tarfs in FreeBSD 14? I wrote it, does that count? > It appears to puke with larger tarballs. On the contrary, one of the use cases I tested was buildworld from a tarball of the source tree, which it handled just fine. > For example, if I fetch a copy of ports or pkgsrc and decompress it to > just a posix tar archive I can't get it to mount. > > root@mono:/home/nonesuch # file ports.tar > ports.tar: POSIX tar archive > root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports > mount: ./ports.tar: Inappropriate file type or format % fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst ports.tar.zst 47 MB 5431 kBps = 09s % sudo mount -rt tarfs $PWD/ports.tar.zst /mnt mount: /home/des/ports.tar.zst: Inappropriate file type or format % dmesg | grep tarfs=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 tarfs_alloc_one: unsupported global extended header at 0 % zcat ports.tar.zst | hexdump -C | head -3 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_hea= de| 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r.............= ..| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |..............= ..| This is a non-standard extension header emitted by `git archive` which contains metadata about the git tree from which the archive was created. I believe you can safely strip it off (zcat ports.tar.zst | dd bs=3D1k skip=3D1 >ports.tar), but you'll run into issues with very long paths in devel/electron*. I will take a closer look when I find the time. Note that tarfs can mount tarballs compressed with zstd, but performance will be poor unless you create a multi-frame archive (try using --zstd --options=3Dzstd:frame-per-file,zstd:min-frame-size=3D65536). DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org From nobody Mon Mar 25 18:12:35 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3LdD5cT3z4nmTH for ; Mon, 25 Mar 2024 18:12:48 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3LdC6t1Qz4k6D for ; Mon, 25 Mar 2024 18:12:47 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.17.1/8.15.2) with ESMTPSA id 42PICZmY069031 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Mon, 25 Mar 2024 14:12:41 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: Date: Mon, 25 Mar 2024 14:12:35 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: George Mitchell Subject: xfce4-terminal fonts disappeared To: FreeBSD Hackers Content-Language: en-US Autocrypt: addr=george+freebsd@m5p.com; keydata= xjMEZaHDbxYJKwYBBAHaRw8BAQdA2W6oBfS8haXY0/Ft4zS1OTLYfC8EBIADPTgMQdh85C3N KEdlb3JnZSBNaXRjaGVsbCA8Z2VvcmdlK2ZyZWVic2RAbTVwLmNvbT7CmQQTFgoAQRYhBDpv v9n4+UzMLAJ8EZocD3futmd9BQJlocSiAhsDBQkFo5qABQsJCAcCAiICBhUKCQgLAgQWAgMB Ah4HAheAAAoJEJocD3futmd9SxwBAJUi6DNdVhWCZBTv5XGy1g0JgApLWe/3S0M0zz9sn7/L AQCcJcV5k5s2rt9J5C1AUm6XVsuneVvIWXO5j1GKWk0NC844BGWhw28SCisGAQQBl1UBBQEB B0AaFz/6B95RRvjOdLZr5fSdhuIHvwr24H3ePDZSw6wlUwMBCAfCfgQYFgoAJhYhBDpvv9n4 +UzMLAJ8EZocD3futmd9BQJlocNvAhsMBQkFo5qAAAoJEJocD3futmd9RXsBANwRD9RE56F6 /jeZOrujHICLcgPiOt50Y6866v9OUTjUAP9GlC1aopfBpNwuPLJBam7oBaGqvY98VDhzOjoT 7DNbCQ== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------sJgVfKKDFcDfHBMB06yDtqaD" X-Spam-Status: No, score=0.2 required=10.0 tests=HELO_MISC_IP,HELO_NO_DOMAIN autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on mattapan.m5p.com X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.23 / 15.00]; SIGNED_PGP(-2.00)[]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.99)[-0.989]; NEURAL_HAM_MEDIUM(-0.96)[-0.957]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; MIME_BASE64_TEXT(0.10)[]; XM_UA_NO_VERSION(0.01)[]; TAGGED_FROM(0.00)[freebsd]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[m5p.com]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; HAS_ATTACHMENT(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~] X-Rspamd-Queue-Id: 4V3LdC6t1Qz4k6D This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------sJgVfKKDFcDfHBMB06yDtqaD Content-Type: multipart/mixed; boundary="------------eyslyzLULsKcYbPv7Qv0bdb6"; protected-headers="v1" From: George Mitchell To: FreeBSD Hackers Message-ID: Subject: xfce4-terminal fonts disappeared --------------eyslyzLULsKcYbPv7Qv0bdb6 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 QWZ0ZXIgdGhlIG1vc3QgcmVjZW50IHVwZ3JhZGUgdG8gdGhlICJub3RvIiBmYW1pbHkgb2Yg Zm9udHMgKGJ1aWx0DQpsb2NhbGx5IGZyb20gc291cmNlKSwgeGZjZTQtdGVybWluYWwncyBm b250IHNlbGVjdGlvbiB0aGluZ3kgc2hvd3MgbWFueSwNCm1hbnkgZmV3ZXIgZm9udHMgdGhh biBiZWZvcmUgSSB1cGdyYWRlZC4gIFNwZWNpZmljYWxseSwgTm90byBDb2xvciBFbW9qaQ0K aXMgdGhlIE9OTFkgZm9udCBhdmFpbGFibGUgZnJvbSB0aGUgbm90byBmYW1pbHkuDQoNCkFu eSBvdGhlciBhcHBsaWNhdGlvbiAobW91c2VwYWQsIHRodW5kZXJiaXJkLCBjaHJvbWl1bSwg ZmlyZWZveCkgc3RpbGwNCmhhcyBhbGwgdGhlIGluc3RhbGxlZCBmb250cyBhdmFpbGFibGUs IGp1c3QgYXMgYmVmb3JlICgxMTUgeDExLWZvbnRzKS4NCk9ubHkgeGZjZTQtdGVybWluYWwg aXMgY29uZnVzZWQuDQoNCkZyZWVCU0QgMTMuMi1SRUxFQVNFLXAxMCBhbWQ2NA0KeG9yZy03 LjdfMw0KeGZjZTQtdGVybWluYWwtMS4wLjRfMSAoanVzdCBub3cgdXBkYXRlZCB0byAxLjEu Mzsgbm8gaW1wcm92ZW1lbnQpDQoNCi0tIEdlb3JnZQ0K --------------eyslyzLULsKcYbPv7Qv0bdb6-- --------------sJgVfKKDFcDfHBMB06yDtqaD Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQQ6b7/Z+PlMzCwCfBGaHA937rZnfQUCZgG+kwUDAAAAAAAKCRCaHA937rZnfZ2c AP44RNoJreQmFCPbhX2IFSrc8SpXWh9bIiXdNOiAbI17pQD9HAMT0oESsEO6vR2I1okOVMuSWIkr fUy+jw1uhO2/bwk= =3OqY -----END PGP SIGNATURE----- --------------sJgVfKKDFcDfHBMB06yDtqaD-- From nobody Mon Mar 25 23:42:35 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3Ty22t4nz5FYds for ; Mon, 25 Mar 2024 23:42:50 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-vk1-xa35.google.com (mail-vk1-xa35.google.com [IPv6:2607:f8b0:4864:20::a35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3Ty13Fx2z4K5c for ; Mon, 25 Mar 2024 23:42:49 +0000 (UTC) (envelope-from nonesuch@longcount.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount.org header.s=google header.b=sHaT0Cwz; dmarc=none; spf=pass (mx1.freebsd.org: domain of nonesuch@longcount.org designates 2607:f8b0:4864:20::a35 as permitted sender) smtp.mailfrom=nonesuch@longcount.org Received: by mail-vk1-xa35.google.com with SMTP id 71dfb90a1353d-4d435a60217so1404181e0c.0 for ; Mon, 25 Mar 2024 16:42:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; t=1711410168; x=1712014968; darn=freebsd.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=DN4E7jcj6IQ3jM85YpbpdH5uQoJSEi9XnaZUmgqxXGM=; b=sHaT0CwzN7FzpCHIaC8aEMz6FEPd9PvNsotB5Jx2mIbc6TeBq0JfuS42LBdrwUPZlE +HsWo7hP5SUhG2Xwr7z72VVRgVsekkXyV1idIxgbUcy7YlOygJlfazYzWTPvPrOwp5xJ N4gTlL84M0X4SmlsSu3/p5oPejcNv6+psgDxAr93Hshqg2eI4tmt545xor2jAQ1XkGrI 4KDwXvv39DodIpywYoKYrVZhr7++7i3Nzq8xe5aOfCKFSVx0XZdJO6AED9cJCqEqmC6w dbePeyGNxA/v7a3Jlz0X5KvTpx9J41RhyNwYFaZQVr2kgHkHCvp1EuL6u4Wpkj0gwPsi sncg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410168; x=1712014968; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DN4E7jcj6IQ3jM85YpbpdH5uQoJSEi9XnaZUmgqxXGM=; b=drgb/UQSdRbvNZVo2kQXEyU/+UR+T2RMnjL8sSbgA1nHRKL3PMzcN4DNh6v+REOEU7 NmH/5faEcKjaJfQ2htDMPXw22RcuqUN7352ra+TkdAjSwBfwmaGRFL/K0Vm27Q6g4fev dpqqWNpZeAwI+Q+PH54pke7sVa6xRv/KeAbC8J+jOnyA/iRPAqHF8eigImmQQBZThHxQ JgZTF1m53wHsgr9mGjmI/xBsyx4X4KNXPflke1DLyD0XRnF5KbV5HBVvnAFKiNS5VMCa ceTrCH8kfMrmhrH2mZ5bhOwkA7jhdYIwUy6qESZ6kMU0w5f8KGmxConaj/II7DX60iBQ A1ww== X-Gm-Message-State: AOJu0Yy9u5tpqTwOgxHRT75HLlrHh1EuHcgqRzxIXsoK2HpoVU+yXGmD uaHcvMvabiAKK3dR75nb514jjw5cQY/XtQqCjzE2zGc60cgfU4vcKXzRhHS7E0n3MzIqjTz7l5P 7ZvnORz3DMpEqIGoskXCtEPdeMDtrKQISkULJDfI/+lT4kr5P2BW13w== X-Google-Smtp-Source: AGHT+IEUgswXQferwM4/x46iHJkJ4ULbH9EAefNpdZzFexAPmJ79GdAPejuS10epxV6mbOE344dfM7pTP2Nl+cz5pvk= X-Received: by 2002:a05:6122:1682:b0:4d3:3adc:b639 with SMTP id 2-20020a056122168200b004d33adcb639mr6254511vkl.6.1711410166250; Mon, 25 Mar 2024 16:42:46 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <86edbyh0od.fsf@ltc.des.dev> In-Reply-To: <86edbyh0od.fsf@ltc.des.dev> From: Mark Saad Date: Mon, 25 Mar 2024 19:42:35 -0400 Message-ID: Subject: Re: TarFS To: FreeBSD Hackers Content-Type: multipart/alternative; boundary="000000000000065f0a061484bb93" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[longcount.org:s=google]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[longcount.org]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::a35:from]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[longcount.org:+] X-Rspamd-Queue-Id: 4V3Ty13Fx2z4K5c --000000000000065f0a061484bb93 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Mar 25, 2024 at 9:59=E2=80=AFAM Dag-Erling Sm=C3=B8rgrav wrote: > Mark Saad writes: > > I was wondering if anyone has started to play with tarfs in FreeBSD 14? > > I wrote it, does that count? > > > It appears to puke with larger tarballs. > > On the contrary, one of the use cases I tested was buildworld from a > tarball of the source tree, which it handled just fine. > > Dag let me just say , great work. I really like this. > > For example, if I fetch a copy of ports or pkgsrc and decompress it to > > just a posix tar archive I can't get it to mount. > > > > root@mono:/home/nonesuch # file ports.tar > > ports.tar: POSIX tar archive > > root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports > > mount: ./ports.tar: Inappropriate file type or format > > % fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst > ports.tar.zst 47 MB 5431 kBps > 09s > % sudo mount -rt tarfs $PWD/ports.tar.zst /mnt > mount: /home/des/ports.tar.zst: Inappropriate file type or format > % dmesg | grep tarfs > tarfs_alloc_one: unsupported global extended header at 0 > % zcat ports.tar.zst | hexdump -C | head -3 > 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 > |pax_global_heade| > 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |r...............| > 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > > This is a non-standard extension header emitted by `git archive` which > contains metadata about the git tree from which the archive was created. > I believe you can safely strip it off (zcat ports.tar.zst | dd bs=3D1k > skip=3D1 >ports.tar), but you'll run into issues with very long paths in > devel/electron*. I will take a closer look when I find the time. > > Note that tarfs can mount tarballs compressed with zstd, but performance > will be poor unless you create a multi-frame archive (try using --zstd > --options=3Dzstd:frame-per-file,zstd:min-frame-size=3D65536). > > DES > So I am not sure exactly what tarfs is yelling about with pkgsrc.tar , but I wonder if it's a by-product of netbsd using pax . NetBSD tar's dont work , OpenBSD stuff does and they use a ustar format. root@mono:/home/nonesuch # cat pkgsrc.tar |hexdump -C |head -3 00000000 70 6b 67 73 72 63 2f 00 00 00 00 00 00 00 00 00 |pkgsrc/.........| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * root@mono:/home/nonesuch # cat ports.tar |hexdump -C |head -3 00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_heade| 00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r...............| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| root@mono:/home/nonesuch # cat xenocara.tar |hexdump -C |head -3 00000000 2e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * Two other questions ( I have not tried to read the code , please forgive my assumptions ) behind the scenes are the tarballs extracted over some fs, or mounted directly and read ? Second are there any contradictions with tarfs ; can they be nfs exported, mounted on a tmpfs etc . Again good work. C an we expect to see a libarchive-fs next :) > -- > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org > --=20 mark saad | nonesuch@longcount.org --000000000000065f0a061484bb93 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Mar 25, 2024 at 9:59=E2=80=AF= AM Dag-Erling Sm=C3=B8rgrav <des@free= bsd.org> wrote:
Mark Saad <nonesuch@longcount.org> writes:
> I was wondering if anyone has started to play with tarfs in FreeBSD 14= ?

I wrote it, does that count?

> It appears to puke with larger tarballs.

On the contrary, one of the use cases I tested was buildworld from a
tarball of the source tree, which it handled just fine.


Dag let me just say , great work. I re= ally like this.
=C2=A0
> For example, if I fetch a copy of ports or pkgsrc and decompress it to=
> just a posix tar archive I can't get it to mount.
>
> root@mono:/home/nonesuch # file ports.tar
> ports.tar: POSIX tar archive
> root@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports
> mount: ./ports.tar: Inappropriate file type or format

% fetch ftp://ftp.freebsd.org/pub/FreeBSD= /ports/ports/ports.tar.zst
ports.tar.zst=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A047 MB 5431 kBps=C2=A0 =C2=A0 09s
% sudo mount -rt tarfs $PWD/ports.tar.zst /mnt
mount: /home/des/ports.tar.zst: Inappropriate file type or format
% dmesg | grep tarfs=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0
tarfs_alloc_one: unsupported global extended header at 0
% zcat ports.tar.zst | hexdump -C | head -3
00000000=C2=A0 70 61 78 5f 67 6c 6f 62=C2=A0 61 6c 5f 68 65 61 64 65=C2=A0 = |pax_global_heade|
00000010=C2=A0 72 00 00 00 00 00 00 00=C2=A0 00 00 00 00 00 00 00 00=C2=A0 = |r...............|
00000020=C2=A0 00 00 00 00 00 00 00 00=C2=A0 00 00 00 00 00 00 00 00=C2=A0 = |................|

This is a non-standard extension header emitted by `git archive` which
contains metadata about the git tree from which the archive was created. I believe you can safely strip it off (zcat ports.tar.zst | dd bs=3D1k
skip=3D1 >ports.tar), but you'll run into issues with very long path= s in
devel/electron*.=C2=A0 I will take a closer look when I find the time.

Note that tarfs can mount tarballs compressed with zstd, but performance will be poor unless you create a multi-frame archive (try using --zstd
--options=3Dzstd:frame-per-file,zstd:min-frame-size=3D65536).

DES

So I am not sure exactly what tarfs is yel= ling about with pkgsrc.tar , but I wonder if it's a by-product of netbs= d using pax .
NetBSD tar's dont work , OpenBSD stuff doe= s and they use a ustar format.

root@mono:/hom= e/nonesuch # cat pkgsrc.tar |hexdump -C |head -3
00000000 =C2=A070 6b 6= 7 73 72 63 2f 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|pkgsrc/.........|
= 00000010 =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0= |................|
*
root@mono:/home/nonesuch # cat ports.tar =C2=A0|= hexdump -C |head -3
00000000 =C2=A070 61 78 5f 67 6c 6f 62 =C2=A061 6c 5= f 68 65 61 64 65 =C2=A0|pax_global_heade|
00000010 =C2=A072 00 00 00 00 = 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|r...............|
00000020= =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|.......= .........|


root@mono:/home/none= such # cat xenocara.tar =C2=A0 |hexdump -C |head -3
00000000 =C2=A02e 00= 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2=A0|................|00000010 =C2=A000 00 00 00 00 00 00 00 =C2=A000 00 00 00 00 00 00 00 =C2= =A0|................|
*
<= br>

Two other questions ( I have not tried to read= the code , please forgive my assumptions )=C2=A0 behind the scenes are the= tarballs extracted over some fs, or mounted directly and read ?
= Second are there any contradictions with tarfs ; can they be nfs exported, = mounted on a tmpfs etc .

Again good work. C a= n we expect to see a libarchive-fs next :)
=C2=A0
--
Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org


--
mark saad | <= a href=3D"mailto:nonesuch@longcount.org" target=3D"_blank">nonesuch@longcou= nt.org
--000000000000065f0a061484bb93-- From eugen@grosbein.net Tue Mar 26 00:38:46 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3WBg5v3Fz5FfSW for ; Tue, 26 Mar 2024 00:38:51 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from mail.rdtc.ru (ns3.rdtc.ru [62.231.190.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3WBg4XSrz4PF2 for ; Tue, 26 Mar 2024 00:38:51 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: by mail.rdtc.ru (RDTC Post Office Server, from userid 1000) id 254DC1CF08; Tue, 26 Mar 2024 07:38:50 +0700 (+07) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: egrosbein@rdtc.ru) by mail.rdtc.ru (RDTC Post Office Server) with ESMTPSA id AEF291CC48 for ; Tue, 26 Mar 2024 07:38:48 +0700 (+07) X-Envelope-From: eugen@grosbein.net X-Envelope-To: freebsd-hackers@freebsd.org Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.17.1/8.17.1) with ESMTPS id 42Q0ck2h039738 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Tue, 26 Mar 2024 07:38:47 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: TarFS To: Mark Saad , FreeBSD Hackers References: <86edbyh0od.fsf@ltc.des.dev> From: Eugene Grosbein Message-ID: Date: Tue, 26 Mar 2024 07:38:46 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,LOCAL_FROM, NICE_REPLY_A,SPF_PASS,T_DATE_IN_FUTURE_96_Q autolearn=no autolearn_force=no version=4.0.0 X-Spam-Report: * -0.0 SPF_PASS SPF: sender matches SPF record * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 T_DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * 2.6 LOCAL_FROM From my domains * -2.5 NICE_REPLY_A Looks like a legit reply (A) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on eg.sd.rdtc.ru X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29072, ipnet:62.231.184.0/21, country:RU] X-Rspamd-Queue-Id: 4V3WBg4XSrz4PF2 26.03.2024 6:42, Mark Saad wrote: > Again good work. C an we expect to see a libarchive-fs next :) We have sysutils/archivemount in ports. From nobody Tue Mar 26 02:20:57 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3YSX1sR1z5FpR0 for ; Tue, 26 Mar 2024 02:21:00 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3YSX1MZSz4b8M; Tue, 26 Mar 2024 02:21:00 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711419660; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7hqgF5WMqWJ4pNB8Axsmu+5Jm8b0w3VNXalWhWVmu2I=; b=AXWGOuPrU7vaSIgUY/ZEVhiXMjBDa9SL7XFgTAKbncA8bBZ+Um4o7RCi97p+7pdobO2k7L 2OS9lLB8aqEx0JM4wBNsN5FX+I1Qb6SYVkmN2EAb9xek5Ja0pa7G2wJNXNNbMjC3pASWhu SZ1sIrzwr5cXh2zlQoqwROxXWZaNo8QGNoFqW2gaDzY2S9urVCx2N7mB84ktQHrbNVosx6 7oFXPqL7QxB0erW3ONiejifQjxEeZtiTvgMoVGwmown9VyI2F8+77ZH9J7NdWTFakBrSaf BrwWgWv7HfoNN9OeLkO4OzMhtB5tkkzhbheia47LMZHKhbVzgN66U/Lwc/wUJQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711419660; a=rsa-sha256; cv=none; b=PjwCLGanoK9G99/MxSHx2tG83SNpmZqMCjA9i92eANm5fEuX2kpyvnx6HIGGvn3ZWOkRFg SBwTXEq+miBoDH9W2UQsROQW9wgOPnIRsjlzSk5G38eeu7/H9ZKYPlCVI82mYI1R1PZl3e ogAmMB4A6BLlOhDrL494GYgI8OMcyMphyKbZTBU+obTkH4b17JKFUalQweIeTWqGixZoBM VjHSE0nwkOjKSAKPrGUZDkmvgYHyjURzs2MLEP4+CSDiYDzikAnTRTaFAcUqgLUNoG0jer D/MjGxrU5gMv7k1foulaWSxHJwEeknHoxziyGtMWusthF8tW5XNEN71wLicCfg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711419660; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7hqgF5WMqWJ4pNB8Axsmu+5Jm8b0w3VNXalWhWVmu2I=; b=mSkMWza0vkFpuq+mRmzs6PCwSDbB1I8Pux/pmrkKBNIEySqq8IHanr9JlnzPHomLOzQM9H yLUKeq7FiQOYtK9nMwMfTl5vkFLGwPIwTaZHQS16BVypI45tmhLzujvAGhE4b9MjwxjhMo saCWD6LVlvo/IzcwjwxJFKSoD9sVls8sG0+6taJwVkgKquIayek0oSIh3YoRxw4Iv3sO29 WbCj6If0mLT84Q4fHvtZIFHGS+b8FaY72KVQPT2rLADFvvdlgaMSG2KDXL+lN5JqFwTG4n ej+nXSEYLcIjhFF/iLnvgXENs9R8XAaHdBPFeDzdJ+wbdK5yXOOv4aIB7EshzQ== Received: from ltc.des.dev (2a02-8428-0993-f001-922e-16ff-fef1-acef.rev.sfr.net [IPv6:2a02:8428:993:f001:922e:16ff:fef1:acef]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4V3YSX0MRzzyn7; Tue, 26 Mar 2024 02:21:00 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.dev (Postfix, from userid 1001) id 866C1780C0; Tue, 26 Mar 2024 03:20:57 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Mark Saad Cc: FreeBSD Hackers Subject: Re: TarFS In-Reply-To: (Mark Saad's message of "Mon, 25 Mar 2024 19:42:35 -0400") References: <86edbyh0od.fsf@ltc.des.dev> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Tue, 26 Mar 2024 03:20:57 +0100 Message-ID: <867chpg2d2.fsf@ltc.des.dev> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mark Saad writes: > Two other questions ( I have not tried to read the code , please > forgive my assumptions ) behind the scenes are the tarballs extracted > over some fs, or mounted directly and read ? It scans the archive, builds an index of its contents, and reads what you ask for when you ask for it. No additional disk space is consumed. This is why compressed archives can be very slow. > Second are there any contradictions with tarfs ; can they be nfs > exported, mounted on a tmpfs etc . They can be exported, but I'm not sure what you mean by =E2=80=9Cmounted on= a tmpfs=E2=80=9D. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org From nobody Tue Mar 26 08:33:51 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3jl4413fz5F7lm for ; Tue, 26 Mar 2024 08:34:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3jl42Fs8z49ph for ; Tue, 26 Mar 2024 08:34:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-568c714a9c7so6059879a12.2 for ; Tue, 26 Mar 2024 01:34:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1711442044; x=1712046844; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pt6RDAok6jCeVX16GiGHfhtESRz14QFCtA6Mc6v9P/o=; b=zB9LPiKLOCQ/bKQfU45LgVsmRRBpwmo64Un6MgixL89OKw0xCUSKlTM+LCWWIhvWZg JIKS8CkQY4AdBOLkENYizxrzbnbM6ddsHMsb45xsXwcOV33i84AbH68yGGyDIrbK+q3z pUjcyZ576t9NA8xIU96Qiy/6eVNWYHY/hU7xz4QpNXC5/f+UN5SKmTwi3ZeCjolWhnLN 0GCqDVLfxQiw3iomakWCS7PV9elDlqHE+bWrg/H/KIJdYJSHhMkL3UXN8Ao9Fm8xRSls 4Pjx751DxoNVGL3oVcpot9BG9M5f9QAw/JKu5X23d5zE+BxgkD1hQcOaM+9cS/lgvwlZ 4cKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711442044; x=1712046844; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pt6RDAok6jCeVX16GiGHfhtESRz14QFCtA6Mc6v9P/o=; b=bWvP8TTxh4vBkfQiqXCLcrPHVTxCvMUcnFx5ZqAJEyYpyeTC9m0uE2BI9tzWWBGYen 67yh6TpBIjrRRGPA+Uun1RQO/FSNCiExOxrpj5Xabu3/9SFVy6g1kgIXdwxi5MVxsdrg zwY+YMx5liJUMcBzk2qTkjE9dRB+4DSekyt3NtKGnS30t7eW49TBSX/bRIXDN8NM3x3s n1fVIo7xr+cNt22LnahRfXvX6pSA8HdPzLQfEt7JD3Dqd2wx/jJMjWNdjC7hA45Ese1N fMehJ51Nw3PZEBrJVYxz41Zb1DchblGjrp9Rlbe2NTjLBQ0QFRSHlNSw2gq+kjht4gtt ZEmQ== X-Forwarded-Encrypted: i=1; AJvYcCVs922wdnUbs96LP7obxvyOUU0M9D7+gCnrz0rlVHrdjHLkd80EurL6voOBxeJyMZ6E82AimiH82I5Okd2fQJliTdv/CeBWh+tVBm8= X-Gm-Message-State: AOJu0Yx2rliL4WxnDLxyhsc+2lmzr+QRblErEMeiQQHs3Thp81Y2pZrY XWt6y35XVArSWZfvhIYSx2t95ofppTp004ERw1FL/Zw5t+t2KtxwmIA666sLia+XobuQ6RCjI0i JU1TJreVnM43pJUkswpu4vzX6e59cYY4v328uBPGVWzPnfokiDdo= X-Google-Smtp-Source: AGHT+IEO4scE/t3yJuSLTxfTMI+mtzs9vj1V29HTzZaMoqh0TBnUtWzgBoz7bCe4KN8i7NNhQnMXYjwf+TBsKF7WZTI= X-Received: by 2002:a17:907:7283:b0:a47:4c8d:a950 with SMTP id dt3-20020a170907728300b00a474c8da950mr5420235ejc.34.1711442044073; Tue, 26 Mar 2024 01:34:04 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <86edbyh0od.fsf@ltc.des.dev> <867chpg2d2.fsf@ltc.des.dev> In-Reply-To: <867chpg2d2.fsf@ltc.des.dev> From: Warner Losh Date: Tue, 26 Mar 2024 09:33:51 +0100 Message-ID: Subject: Re: TarFS To: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= Cc: Mark Saad , FreeBSD Hackers Content-Type: multipart/alternative; boundary="00000000000017520606148c270d" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4V3jl42Fs8z49ph --00000000000017520606148c270d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Mar 26, 2024, 3:21=E2=80=AFAM Dag-Erling Sm=C3=B8rgrav wrote: > Mark Saad writes: > > Two other questions ( I have not tried to read the code , please > > forgive my assumptions ) behind the scenes are the tarballs extracted > > over some fs, or mounted directly and read ? > > It scans the archive, builds an index of its contents, and reads what > you ask for when you ask for it. No additional disk space is consumed. > This is why compressed archives can be very slow. > > > Second are there any contradictions with tarfs ; can they be nfs > > exported, mounted on a tmpfs etc . > > They can be exported, but I'm not sure what you mean by =E2=80=9Cmounted = on a > tmpfs=E2=80=9D. > I think he means does it work witg unionfs. Side question: how hard is it to add and offset of tarball mount option? Warner > DES > -- > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org > > --00000000000017520606148c270d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Mar 26, 2024, 3:21=E2=80=AFAM Dag-Erling Sm=C3= =B8rgrav <des@freebsd.org> wro= te:
Mark Saad <nonesuch@long= count.org> writes:
> Two other questions ( I have not tried to read the code , please
> forgive my assumptions ) behind the scenes are the tarballs extracted<= br> > over some fs, or mounted directly and read ?

It scans the archive, builds an index of its contents, and reads what
you ask for when you ask for it.=C2=A0 No additional disk space is consumed= .
This is why compressed archives can be very slow.

> Second are there any contradictions with tarfs ; can they be nfs
> exported, mounted on a tmpfs etc .

They can be exported, but I'm not sure what you mean by =E2=80=9Cmounte= d on a
tmpfs=E2=80=9D.


I think he means does it work wi= tg unionfs.

Side questio= n: how hard is it to add and offset of tarball mount option?

Warner
DES
--
Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org

--00000000000017520606148c270d-- From nobody Tue Mar 26 12:10:08 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3pXM1xNqz5FTcK for ; Tue, 26 Mar 2024 12:10:11 +0000 (UTC) (envelope-from des@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3pXM0Vsxz4SM4; Tue, 26 Mar 2024 12:10:11 +0000 (UTC) (envelope-from des@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711455011; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e14uZzRFNGi/cq2WzCTxePvu3EIagjDupYD5Q1kIVUQ=; b=R8ryNWnMHuS/SVNA7IWl84ED7FvR+XxpEES7/23JuurUkdgD9grp91Y8RTBafY5AeVH9pw qAOraj+X0B/X4u6VX738f0HCJ+0gwEng8xUnr2xcYH6FtUQZopvE4jzVHAmUqA21CESaRn 6TGPcH/9/hx9Z5DjHZw3D8NFGWAK2Hdsv2LUUvIgTlTNQjELp39bCIUXDKocaSvl+EciA7 nPkHsvh8IeL0hnv1jzbUXuTWDp5wN5JFrwpJJ6gNl7U+/Hc7Rfe3mamnoadZ1VL6/TQpu0 Mwe920Bo7mOj8q1tQV3buSN7a0lQIs5U+sQp0Cs+mHDmjofewm8ggqNa2EZC7A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711455011; a=rsa-sha256; cv=none; b=ta+vMoTKe8bviX6E70VDiFZmJPPOERTufqaqLZpLyFdb6tZQWFyfpGCD+yLgLhmHgrv0zZ vS+IaP0ww1D3RkVylk1S4SKBWjGG/4Ix0TO+7ez18e/WqroeR+7CYJv28d3kCVlKsJBkir LIU+1GRDs3alBex9v7QXIv8TxVCXd+YBRbsCcQhSROIlPKPIPPXPhNw67IWFgO1PAmkXAv /QaJUhciRJh9nUPMYf6OgC3sHIiQmoi/rwNHJpzR0SOEOaLpiuyZ6tY5gSjwAZ45Xe9jTL laVyYDtQfVVWIu2e5KBu6z1mCeugYtCmARLT0EloxBoyBxqz27tvZlInrzoayA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711455011; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e14uZzRFNGi/cq2WzCTxePvu3EIagjDupYD5Q1kIVUQ=; b=JBROe+LL8p8UeCL5xQU9HteNTpoZTWSFokjAyWFCjgIMxibYC1GiC//DtZ7zisd+d2X0h5 RTUTBOgTyxSd7rQDV24M3FQlBhbPs89TG/IrZLof6RBK2DS8/IVkRhvHKRy/dLzb2tPOB8 fB7x60+zIVDl9a9NlcPiozUhAyPO9il+bh9ZKRIK7cFG/re1NVLRggMTVM9ukTu6BjfXTc gFjkQ+45gDMfbAm/SPziOT6hd9UIjYyViJXtxrTWRT6CSWkSAIDVeL0p4TWTPky51vSd7f iIVR+Y3aOmU40IFKujyqsOIQRv8l7W/TyqeCqofFm/Kwhncd29tob9QyOUaotw== Received: from ltc.des.dev (163.23.65.37.rev.sfr.net [37.65.23.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: des) by smtp.freebsd.org (Postfix) with ESMTPSA id 4V3pXL6Sxnz1Bxx; Tue, 26 Mar 2024 12:10:10 +0000 (UTC) (envelope-from des@freebsd.org) Received: by ltc.des.dev (Postfix, from userid 1001) id 966C178366; Tue, 26 Mar 2024 13:10:08 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Warner Losh Cc: Mark Saad , FreeBSD Hackers Subject: Re: TarFS In-Reply-To: (Warner Losh's message of "Tue, 26 Mar 2024 09:33:51 +0100") References: <86edbyh0od.fsf@ltc.des.dev> <867chpg2d2.fsf@ltc.des.dev> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Tue, 26 Mar 2024 13:10:08 +0100 Message-ID: <8634sdfb33.fsf@ltc.des.dev> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Warner Losh writes: > Dag-Erling Sm=C3=B8rgrav writes: > > They can be exported, but I'm not sure what you mean by =E2=80=9Cmounte= d on a > > tmpfs=E2=80=9D. > I think he means does it work witg unionfs. There's no reason why it shouldn't, although I haven't tested it. Unlike exportability, which requires explicit support in each and every filesystem, it's handled transparently by VFS. > Side question: how hard is it to add and offset of tarball mount > option? It shouldn't be hard, since all I/O goes through tarfs_io_read(), but why would you want to? DES --=20 Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org From nobody Tue Mar 26 13:23:01 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3r8b157Jz5FbD8 for ; Tue, 26 Mar 2024 13:23:11 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3r8Z2lBjz4ZwZ; Tue, 26 Mar 2024 13:23:10 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Authentication-Results: mx1.freebsd.org; none Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 6F9C589293; Tue, 26 Mar 2024 13:23:02 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.18.1/8.16.1) with ESMTPS id 42QDN16U099058 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 26 Mar 2024 13:23:02 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.18.1/8.16.1/Submit) id 42QDN1Kh099057; Tue, 26 Mar 2024 13:23:01 GMT (envelope-from phk) Message-Id: <202403261323.42QDN1Kh099057@critter.freebsd.dk> To: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= cc: Warner Losh , Mark Saad , FreeBSD Hackers Subject: Re: TarFS In-reply-to: <8634sdfb33.fsf@ltc.des.dev> From: "Poul-Henning Kamp" References: <86edbyh0od.fsf@ltc.des.dev> <867chpg2d2.fsf@ltc.des.dev> <8634sdfb33.fsf@ltc.des.dev> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-ID: <99055.1711459381.1@critter.freebsd.dk> Content-Transfer-Encoding: 8bit Date: Tue, 26 Mar 2024 13:23:01 +0000 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU] X-Rspamd-Queue-Id: 4V3r8Z2lBjz4ZwZ -------- Dag-Erling Smørgrav writes: > > Side question: how hard is it to add and offset of tarball mount > > option? > > It shouldn't be hard, since all I/O goes through tarfs_io_read(), but > why would you want to? To skip headers TarFS does not grok ? And a big thanks for TarFS from me too! -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nobody Tue Mar 26 13:44:04 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3rd135M5z5FcrN for ; Tue, 26 Mar 2024 13:44:21 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3rcz2NgMz4dqL for ; Tue, 26 Mar 2024 13:44:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-566e869f631so5948483a12.0 for ; Tue, 26 Mar 2024 06:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1711460657; x=1712065457; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=SRss/MbpPhXNARKYjYFhujPK9m1ASCQoyZKxMWyLjZI=; b=d4tcradKM3w4NxJFQI12waX1oL6w3SH/hBRBPf/hi/BU3ABpwG+2ganD4u+vuTFOiA 0/QEeK7R/0Ssl3LDYfDkqTRMqS0xNagwvEfJgwmgwzM0cPW30BoDa8UIbvmSDWf9wLOU NDWP2AE/BVWRazw97rukR5NOy0iwQslMVsf0Kqz1Rgksh9lFZzPatyYaHx9yMT91NI/y uQTerAO3kFTLzM0yUYDki5vsuodnGxP1WW0KIlueAHMBLzc/fuaN/bW6Xz2tpU8sUi02 K+wU5e646IuEHSYL9qwRiMUaqjbT8RyIKSJy1qorlXNPkms8VJ/cRX8535FzViJgXeyL ONgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711460657; x=1712065457; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SRss/MbpPhXNARKYjYFhujPK9m1ASCQoyZKxMWyLjZI=; b=LW+wfe9sJ/a3XknWR82SBlsRjsGXD2nAHdXRLcb6ncbTLMmPhKraoifn5cdqT52xzQ VIKZp2LPBTro9YnOowivo/RA/P2BY6U/TxT3lX928AHaNYYAaTIQp0m7Q10r6/zoNiMo 4X6v2NljkoI3zGjnC+tfbMtpJjmmUv/drKAOsUBJJ3YFCx67Ol12Gpvtb53JtDvc3hx8 OHRTUpykLIiekScrb1jrXBxZ9YgtrBa6rzTw0sDFzpEfCXZNBLIP047lLLBGEmbhf5TV 6H3k1DWMYiofnU749EeMCVQ7z7qrnWcsIaOfT8WtMbsvF3d/Uu/EXKfW7OM5MRn161iN iM8Q== X-Forwarded-Encrypted: i=1; AJvYcCXWixa/a4OZfaR717A1pZkVKc3XNlVu25NkRGRLTjujN1sSIB6xvOZc4DGN0XlcRIcEZuWY4C/Srsd/XdQEUdYDvsl5kd4yddlSRLw= X-Gm-Message-State: AOJu0Ywodn2w121Xny6Ywv16yK6nG6BTtQL5d63kubcgausJogaBfoq6 SIR3u7+y1L+g3UN461qeSFRLz7kvZukrwV9dn14KS3borzPAZAqyPyGO+4xF/xbnPR/fAsCYJE7 UPNwGGqYUdSTsx7o42npu2ZE5wbk/y6Fu8xFWMw== X-Google-Smtp-Source: AGHT+IGIb8Lg91fDDWV3fMBqbs8gvlrjPLAiwJ/2OXw9nWYhuaj7lxfywFOrgHQaoRsYiRdHvEDSFX06cotqfFwcHJA= X-Received: by 2002:a17:906:53ce:b0:a4d:f7ca:1856 with SMTP id p14-20020a17090653ce00b00a4df7ca1856mr971268ejo.61.1711460657297; Tue, 26 Mar 2024 06:44:17 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <86edbyh0od.fsf@ltc.des.dev> <867chpg2d2.fsf@ltc.des.dev> <8634sdfb33.fsf@ltc.des.dev> <202403261323.42QDN1Kh099057@critter.freebsd.dk> In-Reply-To: <202403261323.42QDN1Kh099057@critter.freebsd.dk> From: Warner Losh Date: Tue, 26 Mar 2024 14:44:04 +0100 Message-ID: Subject: Re: TarFS To: Poul-Henning Kamp Cc: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= , Mark Saad , FreeBSD Hackers Content-Type: multipart/alternative; boundary="0000000000008699900614907c2e" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4V3rcz2NgMz4dqL --0000000000008699900614907c2e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Mar 26, 2024, 2:23=E2=80=AFPM Poul-Henning Kamp wrote: > -------- > Dag-Erling Sm=C3=B8rgrav writes: > > > > Side question: how hard is it to add and offset of tarball mount > > > option? > > > > It shouldn't be hard, since all I/O goes through tarfs_io_read(), but > > why would you want to? > > To skip headers TarFS does not grok ? > It would make it possible to have all the files for an application in an ELF section formatted as a tarball... AppImage https://docs.appimage.org/reference/specification.html is one thing that does in Linux... with user mount enabled the wrapper could mount the app's files in a TarFS... Though Linux has a generic "at a given offset" flag to mount which might be better.... And a big thanks for TarFS from me too! > Definitely cool. Warner Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk@FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetenc= e. > --0000000000008699900614907c2e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Mar 26, 2024, 2:23=E2=80=AFPM Poul-Henning Kam= p <phk@phk.freebsd.dk> wrot= e:
--------
Dag-Erling Sm=C3=B8rgrav writes:

> > Side question: how hard is it to add and offset of tarball mount<= br> > > option?
>
> It shouldn't be hard, since all I/O goes through tarfs_io_read(), = but
> why would you want to?

To skip headers TarFS does not grok ?

It would make it possible to have all= the files for an application in an ELF section formatted as a tarball... A= ppImage = https://docs.appimage.org/reference/specification.html is one thing tha= t does in Linux... with user mount enabled the wrapper could mount the app&= #39;s files in a TarFS... Though Linux has a generic "at a given offse= t" flag to mount which might be better....

=
And a big thanks for TarFS from me too!

Definitely cool.

Warner

Poul-Henning Kamp=C2=A0 =C2=A0 =C2=A0 =C2=A0| UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| TCP/IP since RFC 956
FreeBSD committer=C2=A0 =C2=A0 =C2=A0 =C2=A0| BSD since 4.3-tahoe=C2=A0 =C2= =A0
Never attribute to malice what can adequately be explained by incompetence.=
--0000000000008699900614907c2e-- From nobody Tue Mar 26 13:49:40 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V3rlS4j5cz5FdZj for ; Tue, 26 Mar 2024 13:49:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4V3rlQ5xsTz4fv6; Tue, 26 Mar 2024 13:49:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 42QDneVF099184; Tue, 26 Mar 2024 15:49:43 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 42QDneVF099184 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 42QDne11099183; Tue, 26 Mar 2024 15:49:40 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 26 Mar 2024 15:49:40 +0200 From: Konstantin Belousov To: Poul-Henning Kamp Cc: Dag-Erling =?utf-8?B?U23DuHJncmF2?= , Warner Losh , Mark Saad , FreeBSD Hackers Subject: Re: TarFS Message-ID: References: <86edbyh0od.fsf@ltc.des.dev> <867chpg2d2.fsf@ltc.des.dev> <8634sdfb33.fsf@ltc.des.dev> <202403261323.42QDN1Kh099057@critter.freebsd.dk> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <202403261323.42QDN1Kh099057@critter.freebsd.dk> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4V3rlQ5xsTz4fv6 On Tue, Mar 26, 2024 at 01:23:01PM +0000, Poul-Henning Kamp wrote: > -------- > Dag-Erling Smørgrav writes: > > > > Side question: how hard is it to add and offset of tarball mount > > > option? > > > > It shouldn't be hard, since all I/O goes through tarfs_io_read(), but > > why would you want to? > > To skip headers TarFS does not grok ? Then tarfs should skip unsupported headers, enabled with an option. If such header appears in the middle of known and important blocks, an offset does not help. From nobody Tue Mar 26 19:46:25 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V40g845Ptz5Fh1t for ; Tue, 26 Mar 2024 19:46:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-20.consmr.mail.gq1.yahoo.com (sonic305-20.consmr.mail.gq1.yahoo.com [98.137.64.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4V40g74Qh9z44dn for ; Tue, 26 Mar 2024 19:46:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="Y4/ZJM92"; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1711482401; bh=LCI47Uzfqt/shcd31ZqC/Y7ICfL1KPPiCx43f6dQyx0=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=Y4/ZJM92GEmJp173Jp8lS+dOtaIrvk+If716Z7h+OPAcgnptJhntl11hxpbXbZmvff6oe0lFaVBW5sJvt0z01KFg5XbdWILfJXOmWk8vQIr5BWatml3p09b5GZqordOPhgxrnedc5C7V+dQAGYPvAz7cwfYkRFyKqyl0VONxhfD6VcQkliQm31fK9e5i07vITHI4J6z9RE4LXLCjxzjWgeVbDuG3U8HnmmYo0jfrQL07UIv3zqoAJCEORTZvh6vcgLMeq8qANkkXhD7f/sdgargu0IZzzf+fbK1JAHJBXcLmvYGMLj3oCYdlgpJBFQmnFfHFwK9eHOBOw0nAzy1iSA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1711482401; bh=sVAIxJ2VJ7AAZuOrA+X2HMDLk26BIdXUUqTiaUnNgZL=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=eBAlbS+mE8QQX5jMDc8YttMKLrbqgaWme4Wk+tzzrjpMC4S9f8Dy7CmdwCPXO15mCdDdPNwk/LQOMZFOi93WcZp2ELNqSRJfEk4uGdyvMd48zTbGBFj2efbaXdmeHoVAJ8IABQwB7/6pyaPOfmKbBbiqaWMm+Cr9yKjn8JmK7WYBVjtiJ0hn6pERtL6QAaDW4Y8b7/WyS43XT96JqqpVPMd22vkf+7Jr+XBxmlHpBZE932CqSEYkTedHieDZ6VkxS2NGFBLebQw7ZGsIyjvRZ9/t26xX4c3k5Qu7MflS20q5lplqrkXyoz/lYlL9648jCGw7MJbc2AnADS69llKEig== X-YMail-OSG: vFe58XEVM1nLD.omO3vZ7N0xogLte1xcXR_MueJK5.ZtPe7VBEzL2rFrZZ3vdCg yhQhgBSAYs5jXMoUyu288Fn5Sg3AbYGzE07M8VHe_sAiC5PNgy.1gtgGEwKzF65hb3SgDDrTYfrr GTR8qhynoJMZDllW39O9235Q4bJ6jx73P2SAJIF_Ey3SL_i2wQ3EZuf8BiGV7Fgx8KjTG26huod2 KqIbrgpsnl.Od_aUec3MZczgLVNdUuZBnDatxA6sjQiqBLt5D_8JkYP9_ssmnz34d89wm9NMOp0u ABTKGtfqMcmdtsIm0oCrOcUDMbJhsDky79dwgp.AAFhkAiAYLh.dYB7Z.tQ2nDr0Heif_WAsQ3dO Fg7RKNPmBkfcfKsvaCzp.mn7xxmI3sTGHSLYn.fEMf2EBpcRaRZb4svunRy.lFyhuHCLDdUjdgF4 hmKueRMUQ0_jQiTZcIacn90BM20rCFmqy1FqNXZRGLEv2XbHvU6I30g_m0TcPFX.oP0nDbX_noWZ AqgHDaIoXJos3sWJNkPmFKiHqEtOFIXbGjik1LLNtBl8kxrRd5BqE2Fy0OnTOIvEKNhj5UCEwHn4 3Q_F_arB7hP9d1pN4eg9xEgr6IV2SrYNq.5vIkFSdINKl7KL8SCPY4uURhF9A425_Wz0UJPL0Vrw QqL1d_0w1Prbx0EuxRJuZJpJRirxseOmM.6.KYjZHNffTFsfg5wKJvLMA6B95YEHYtM4YRQzK1J_ Au__Dzqy5mgv1zbVO6PhTdeKDvbDqclQNgTKm9_y01R.UBqmaYHWbYILkn6KDZL7T3jIVM6n9NYH C7mqFuCRryFq3LOEEJ707Dp7XQklPtjWJHjkgZx10dY.ebs7K2Vmmx0o5rjNma4S.Q01uELdJur5 glSqmK9_yjBOt53M2_1SWOZ6Yzd_.tfytx1nNXK5hHTE35_B13YmlQf7l7G2jDCY0GU28VQ1l0dp 8UIMOtOClUiZd6IpyviQbR5BXHmuS19XIz4burkXnT7K2lq5.9_Ag48bhil1LTSCn26em.jckaC_ 6kdM3J3TKK0r_rUuTNOHDoINVw8wIxV9cvjHPmxg3RroD_PdnKtgOy.AxwsxhaoDhklZUMbfVi43 _b61xLqbOuWjv5vLw4VHfZ6VoOlzHE1MeYCzGWdyueD_CXPDoa0TcZLFIegQfG1KJlLnfb8v_J8t Wll.e_mm1SjYJxhL6NYYjtJZKjtfp5wIs1BpRrXzbd5S4JlWgQH8cn2tgiZqoR9FMjFcV8.E3jlR f1otnXzLbdu90pcGmJW25PrXk0tJZG_N2wrtecRwuFFr5f6GcIl4LGOP25ZVhjuN0A.i0y.cX1ON j40Vre8aiQCjXMIJQidpw8L8BtkDNhDNkM8gfcWGag3TyGUGmTf1ZkTDJ8iLq4uO11zKuUajfK.e FHV9qsNpgLK6rvoBs50.iGet0hZKaDftNUn_2_pdwhaPq8tPbU6dzJvY4Kuje495hLt0RW8VLpTs TteABVAotUaGkJBCK2C0oxvec.0QObyEZGbV1SrntDffl3IYR.Q6EmO8phcUFoVLHqGWkgIHAMUq c2uk9zLpWvbR5toSGKFrT2dhXKeo5iHVWpSyTryCNPdbvxZivqsGZXLwAZHXCbWFUTeSHhFN_1_S Tj6.lb9UZiCWLTtXKnMeBah7O0ARmaHP_EkpuG1Q2C8Weh_iKDAJ0s.goN12Db8O8ubz_X9W6NF5 HzcdX2pbIgdr927a6MauJIVH0HwNVkx5pIIS68_Barinp6zrG56p4TdOGZYvtRCBBbY8jlal1m2J mZz3hcxYDMAzR3x9duydMhYP8EJYAtWsuvbQg0awLHvbe4jy034BNk5IFe3MF5l.Jqk.XCLWhQRv T.OKtOFqOC5kLeybp6QgFHEUMhwt4YCQ2HVV5lOOjx2pUqtD8GPYWRdYKa5H3JEBEyBdrhok9FGX J4jzwotFHzKucU2g4v6o4ScOTtE_GCPNL1wCBMrGPbHBb8oCYQGxIZsABR9y4RV3ZYC_byqcdiQr 4PXgiDw_A2ZpZraoGAbnIvyeo18XmT7sQIHDic1au3wxV4.eoMya6T1VCjYJBxZRmK5gJcbVOfwM 2a1Ob2poG.Qw93o8XS8IdhQNBupnxjC68BdH8Syndm20zJayK3lmNUpRTd6j3zwWDeLZNTpS9we0 SIKzjcOlxGJToPbq4fzxTbJxYw.1n_pamuX2NMjyGd0UYyEBB37nbA2i_lzFGMp9zW9IVaJ3dMaF dutNW4xtaS8aKoSLV9PYsYQVKiZK7BY7p._UiXYeE3UxKjXaX6wxMH.vla8jFOI6NRIBtNYrUInE - X-Sonic-MF: X-Sonic-ID: ebc3b4c7-8e43-44a2-84cf-c2f7f6ece846 Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Tue, 26 Mar 2024 19:46:41 +0000 Received: by hermes--production-gq1-5c57879fdf-27p5r (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 2e4686134e8f9dd1a749f41fa9530510; Tue, 26 Mar 2024 19:46:36 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: devel/llvm18 building example: building with BE_AMDGPU can greatly increase RAM+SWAP use compared to not Message-Id: Date: Tue, 26 Mar 2024 12:46:25 -0700 To: freebsd-hackers , FreeBSD Mailing List , Nuno Teixeira X-Mailer: Apple Mail (2.3774.400.31) References: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.96 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.959]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.83:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.83:from] X-Rspamd-Queue-Id: 4V40g74Qh9z44dn I've been doing some poudriere bulk testing of RAM+SWAP use by some package build activity recently. The below is based on using amd64 Hyper-V to run FreeBSD with 2 GiBYTes of RAM and 4 single-thread cores. But it should give a reasonable idea for aarch64 and indicate the direction for armv7 for such core counts. (The amd64 Hyper-V context in use takes far less time for an experiment, like hours instead of days for the RPi4B overclocked and for the OrangePi+2ed .) I started with my normal build context analogous to what I use on aarch64 and armv7 systems: A) BE_NATIVE instead of BE_STANDARD B) No MLIR The examples here are poudriere bulk single-builders, effectively with MAKE_JOBS_NUMBER_LIMIT=4. I use a modified version of top that tracks and reports various MAXimum OBServed ("MaxObs") figures and a little more. For building with BE_AMDGPU enabled: RAM: 1402Mi MaxObsActive 807392Ki MaxObsWired 1944Mi MaxObs(Act+Wir+Lndry) SWAP: 5329Mi MaxObsUsed RAM+SWAP: 6534Mi MaxObs(Act+Lndry+SwapUsed) 7089Mi MaxObs(A+Wir+L+SU) [Abbreviating more what is not additional] 7254Mi (A+W+L+SU+InAct) [Abbreviating more what is not additional] (The 7254Mi (A+W+L+SU+InAct) is from when 7089Mi MaxObs(A+Wir+L+SU) was live but is not a MaxObs [MAXimum OBServed] figure itself.) The peak RAM+SWAP is during llvm-tblgen for AMDGPU related materials. For building with BE_AMDGPU disabled: RAM: 1401Mi MaxObsActive 786872Ki MaxObsWired 1944Mi MaxObs(Act+Wir+Lndry) SWAP: 1252Mi MaxObsUsed RAM+SWAP: 2613Mi MaxObs(Act+Lndry+SwapUsed) 3160Mi MaxObs(A+Wir+L+SU) [Abbreviating more what is not additional] 3190Mi (A+W+L+SU+InAct) [Abbreviating more what is not additional] (The 3190Mi (A+W+L+SU+InAct) is from when 3160Mi MaxObs(A+Wir+L+SU) was live but is not a MaxObs [MAXimum OBServed] figure itself.) The around 7254Mi/3190Mi =approx.= 2.27 suggests that low-end build-machine activity that does not need to build BE_AMDGPU possibly should have devel/llvm18 configured to avoid BE_AMDGPU . Notes: The 2 GiBytes of RAM forces memory pressure required to observe reasonable RAM+SWAP usage figures, at least for package builds needing a sufficient about RAM+SWAP. === Mark Millard marklmi at yahoo.com From nobody Wed Mar 27 15:55:20 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V4WV32YjPz5FjdD; Wed, 27 Mar 2024 15:55:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4V4WV26HSMz4Cvl; Wed, 27 Mar 2024 15:55:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 42RFtK40088677; Wed, 27 Mar 2024 17:55:23 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 42RFtK40088677 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 42RFtK9H088674; Wed, 27 Mar 2024 17:55:20 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 27 Mar 2024 17:55:20 +0200 From: Konstantin Belousov To: ticso@cicely.de Cc: freebsd-current@freebsd.org, freebsd-hackers@freebsd.org, Bernd Walter Subject: Re: Only 1TB RAM out of 1.5TB on amd64 Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4V4WV26HSMz4Cvl On Wed, Mar 27, 2024 at 03:01:11AM +0100, Bernd Walter wrote: > Same Problem with current: > reeBSD 15.0-CURRENT #0 main-n268793-220ee18f1964: Thu Mar 14 02:58:39 UTC 2024 > > real memory = 1649265344512 (1572862 MB) > avail memory = 1057118396416 (1008146 MB) Real memory is really the max physical address of the valid byte. If you have holes in the phys map, real memory should reflect these holes. Avail memory is the total memory as reported by EFI map. Verify all that by looking at sysctl machdep.efi_map output. > > On Fri, Mar 22, 2024 at 07:54:06PM +0100, Bernd Walter wrote: > > .. > > real memory = 1649265344512 (1572862 MB) > > avail memory = 1057138888704 (1008166 MB) > > .. > > > > I suspect address space issues, but don't know for sure. > > Changing NUMA nodes per socket in the BIOS had an influence to make it > > worse, but not better. > > > > > > Copyright (c) 1992-2021 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > The Regents of the University of California. All rights reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 13.3-RELEASE releng/13.3-n257428-80d2b634ddf0 GENERIC amd64 > > FreeBSD clang version 17.0.6 (https://github.com/llvm/llvm-project.git llvmorg-17.0.6-0-g6009708b4367) > > VT(efifb): resolution 1024x768 > > CPU: AMD EPYC 7352 24-Core Processor (2300.07-MHz K8-class CPU) > > Origin="AuthenticAMD" Id=0x830f10 Family=0x17 Model=0x31 Stepping=0 > > Features=0x178bfbff > > Features2=0x7ed8320b > > AMD Features=0x2e500800 > > AMD Features2=0x75c237ff > > Structured Extended Features=0x219c91a9 > > Structured Extended Features2=0x400004 > > XSAVE Features=0xf > > AMD Extended Feature Extensions ID EBX=0x18cf757 > > SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 > > TSC: P-state invariant, performance statistics > > real memory = 1649265344512 (1572862 MB) > > avail memory = 1057138888704 (1008166 MB) > > Event timer "LAPIC" quality 600 > > ACPI APIC Table: > > FreeBSD/SMP: Multiprocessor System Detected: 96 CPUs > > FreeBSD/SMP: 2 package(s) x 8 cache groups x 3 core(s) x 2 hardware threads > > random: registering fast source Intel Secure Key RNG > > random: fast provider: "Intel Secure Key RNG" > > random: unblocking device. > > ioapic0 irqs 0-23 > > ioapic1 irqs 24-55 > > ioapic2 irqs 56-87 > > ioapic3 irqs 88-119 > > ioapic4 irqs 120-151 > > ioapic5 irqs 152-183 > > ioapic6 irqs 184-215 > > ioapic7 irqs 216-247 > > ioapic8 irqs 248-279 > > Launching APs: 32 30 34 13 14 16 23 21 19 24 28 9 86 10 87 7 26 37 39 41 45 46 43 66 71 69 54 57 59 80 82 78 49 53 51 60 65 62 84 88 91 95 74 73 76 92 33 40 11 2 27 22 20 5 44 18 15 36 6 8 25 89 79 85 93 56 48 70 31 81 12 50 38 77 55 63 75 58 1 90 29 94 52 72 67 83 35 42 61 64 68 47 4 17 3 > > random: entropy device external interface > > kbd1 at kbdmux0 > > .. > > > > kern.maxphys: 1048576 > > hw.physmem: 1099322220544 > > hw.usermem: 1091323355136 > > hw.realmem: 1649265344512 > > vm.phys_locality: > > 0: 10 32 > > 1: 32 10 > > > > vm.phys_segs: > > SEGMENT 0: > > > > start: 0x10000 > > end: 0xa0000 > > domain: 0 > > free list: 0xffffffff81ec7ea0 > > > > SEGMENT 1: > > > > start: 0x100000 > > end: 0x1000000 > > domain: 0 > > free list: 0xffffffff81ec7ea0 > > > > SEGMENT 2: > > > > start: 0x1000000 > > end: 0x74000000 > > domain: 0 > > free list: 0xffffffff81ec7c30 > > > > SEGMENT 3: > > > > start: 0x74043000 > > end: 0x75db0000 > > domain: 0 > > free list: 0xffffffff81ec7c30 > > > > SEGMENT 4: > > > > start: 0x76000000 > > end: 0x963b8000 > > domain: 0 > > free list: 0xffffffff81ec7c30 > > > > SEGMENT 5: > > > > start: 0x963c2000 > > end: 0x963f9000 > > domain: 0 > > free list: 0xffffffff81ec7c30 > > > > SEGMENT 6: > > > > start: 0x96500000 > > end: 0xa57f7000 > > domain: 0 > > free list: 0xffffffff81ec7c30 > > > > SEGMENT 7: > > > > start: 0xa8e96000 > > end: 0xac000000 > > domain: 0 > > free list: 0xffffffff81ec7c30 > > > > SEGMENT 8: > > > > start: 0x10000e000 > > end: 0x7d07800000 > > domain: 0 > > free list: 0xffffffff81ec79c0 > > > > SEGMENT 9: > > > > start: 0x10000019000 > > end: 0x1797de00000 > > domain: 1 > > free list: 0xffffffff81ec8110 > > > > SEGMENT 10: > > > > start: 0x17ffdc00000 > > end: 0x17ffdde7000 > > domain: 1 > > free list: 0xffffffff81ec8110 > > > > > > > > -- > > B.Walter https://www.bwct.de > > Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. > > > > -- > B.Walter https://www.bwct.de > Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.