From nobody Sun Jan 21 07:38:56 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4THlbT3hlgz56rBx for ; Sun, 21 Jan 2024 07:39:01 +0000 (UTC) (envelope-from robert@rrbrussell.com) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4THlbS3Zd7z4t0n for ; Sun, 21 Jan 2024 07:39:00 +0000 (UTC) (envelope-from robert@rrbrussell.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=rrbrussell.com header.s=fm2 header.b=RjfSrXI8; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="t 4Z+z5D"; dmarc=pass (policy=quarantine) header.from=rrbrussell.com; spf=pass (mx1.freebsd.org: domain of robert@rrbrussell.com designates 66.111.4.29 as permitted sender) smtp.mailfrom=robert@rrbrussell.com Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 167B85C00F8 for ; Sun, 21 Jan 2024 02:39:00 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Sun, 21 Jan 2024 02:39:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rrbrussell.com; h=cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1705822740; x=1705909140; bh=ceEXBjomqo1jCVg6kknsORf+42RCOtFHrMOse8Em230=; b= RjfSrXI8OMHyt8JtXUF95f2IFw5M+9JHTKOeDddHuwJgWKIzjZmIiTTQdZiDW9Z7 ducPBOXCqqxSl6GaTpkicEpT5cBZzgI2RX/XKVZgdTibXLEl9dMq/cWcZXucFRLq gmYYawPC75sVe4e/Yy+5qFB1Fz2RrS4uQxgU3dA3GOKf1udpDT1mpO2PDAjMu6IG ygTS/I0gUw+sVPFIFpRjo9jsXVceGxgBrwqJ5MfCH1lOo2AMfyb3V3GyWa0HJ3Jt 92Cn0yqxFxFAsI1zb6qTcSoeSGz/eYDZ4maLd8fZKab+dHisXLaopShjHr/UgB2g xbb4Iq9M6LWa6GsQIkafeg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1705822740; x= 1705909140; bh=ceEXBjomqo1jCVg6kknsORf+42RCOtFHrMOse8Em230=; b=t 4Z+z5DyJbf3k86J1P2MH+n+0GrOwMMgxcEPtzWY9vGbkjebpD/j6nNqk/Jovfaab evRSDkWjO90LFs0aWgyAOWGGx0Wbzw/c+1pwaeX4eQn5hEGoIAirR9AgFtuTVXLG +Hpm9yZ80p2FiZQ1of0bqFEsGeqtd7za9rrCBvdNKhxdfZCRFg9d6gzNMGWGQ5iK LrLXI68lRsWvuqQHrCOUSM+dQH7LX8dTFlz2yi7WibrnrUCqK5LZco2SvkrqffPb 9LhyGPaau2SBKQFyj0A1CegvDjHv1JDtZBySjlZqV0A4m75dtTCmS8hW3Wmvhwua Jd/J6BsK2FNrwc5Fpr7Wg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdekfedguddtlecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfgjfhfogggtgfesth hqredtredtjeenucfhrhhomhepfdftohgsvghrthcutfdrucftuhhsshgvlhhlfdcuoehr ohgsvghrthesrhhrsghruhhsshgvlhhlrdgtohhmqeenucggtffrrghtthgvrhhnpeefke duvdeftdfftdfhffegleelfedvjefggeelhefhuddtleeftdffgeejveevudenucffohhm rghinhepghhithhhuhgsrdgtohhmpdhmrghinhdrrhhspdhruhhsthdqlhgrnhhgrdhorh hgpdhfrhgvvggsshgurdhorhhgpdhruhhsthdrmhhkpdhfvgguohhrrghprhhojhgvtght rdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh eprhhosggvrhhtsehrrhgsrhhushhsvghllhdrtghomh X-ME-Proxy: Feedback-ID: ie421460a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sun, 21 Jan 2024 02:38:59 -0500 (EST) Date: Sun, 21 Jan 2024 01:38:56 -0600 From: "Robert R. Russell" To: freebsd-hackers@freebsd.org Subject: Re: The Case for Rust (in the base system) Message-ID: <20240121013856.77ad3767@venus.private.rrbrussell.com> In-Reply-To: <20240121141024.e275343c0e7e3f86a50f4490@dec.sakura.ne.jp> References: <1673801705774097@mail.yandex.ru> <20240121110611.af567b0ac3a8fd8593ffcb7f@dec.sakura.ne.jp> <20240121141024.e275343c0e7e3f86a50f4490@dec.sakura.ne.jp> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.50 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[rrbrussell.com,quarantine]; RWL_MAILSPIKE_EXCELLENT(-0.40)[66.111.4.29:from]; R_DKIM_ALLOW(-0.20)[rrbrussell.com:s=fm2,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.29]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.29:from]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[rrbrussell.com:+,messagingengine.com:+]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[robert]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim] X-Rspamd-Queue-Id: 4THlbS3Zd7z4t0n On Sun, 21 Jan 2024 14:10:24 +0900 Tomoaki AOKI wrote: > On Sat, 20 Jan 2024 21:14:59 -0700 > Alan Somers wrote: >=20 > > On Sat, Jan 20, 2024, 8:44=E2=80=AFPM Warner Losh wrot= e: > > =20 > > > > > > > > > On Sat, Jan 20, 2024, 7:20=E2=80=AFPM Alan Somers > > > wrote:=20 > > >> On Sat, Jan 20, 2024 at 7:06=E2=80=AFPM Tomoaki AOKI > > >> wrote: =20 > > >> > > > >> > On Sat, 20 Jan 2024 15:31:23 -0700 > > >> > Warner Losh wrote: > > >> > =20 > > >> > > On Sat, Jan 20, 2024 at 11:45=E2=80=AFAM Aleksandr Fedorov < =20 > > >> wigneddoom@yandex.ru> =20 > > >> > > wrote: > > >> > > =20 > > >> > > > What about external dependencies? > > >> > > > > > >> > > > =20 > > >> https://github.com/Axcient/freebsd-nfs-exporter/blob/master/Cargo.to= ml#L19 > > >> =20 > > >> > > > =20 > > >> https://github.com/asomers/gstat-rs/blob/master/gstat/src/main.rs#L20 > > >> =20 > > >> > > > > > >> > > > Is there any plan for which crates we should take into the > > >> > > > base =20 > > >> system? =20 > > >> > > > > > >> > > > We have had C++ in base for many years, but I don=E2=80=99t see > > >> > > > any good =20 > > >> libraries =20 > > >> > > > for CLI, logging, JSON, etc. > > >> > > > > > >> > > > > > >> > > > =20 > > >> https://doc.rust-lang.org/rustc/platform-support.html#tier-1-with-ho= st-tools > > >> =20 > > >> > > > > > >> > > > Where is the support for Freebsd as a primary platform? > > >> > > > ARM, RISC-V, Power? Should we rewrite devd? > > >> > > > > > >> > > > I think we need to start by providing official > > >> > > > repositories (e.g git.FreeBSD.org/rust.git or > > >> > > > git.FreeBSD.org/go.git) for different languages that > > >> > > > include stable bindings to the system =20 > > >> API: =20 > > >> > > > - sysctl > > >> > > > - libgeom > > >> > > > - libifconfig > > >> > > > - netgraph > > >> > > > - jail > > >> > > > - etc. > > >> > > > > > >> > > > So that it=E2=80=99s not just some anonymous on crates.io that > > >> > > > represents =20 > > >> these =20 > > >> > > > bindings, but our community. > > >> > > > Officially, with support for a stable ABI for releases, > > >> > > > security =20 > > >> patches, =20 > > >> > > > etc. > > >> > > > > > >> > > > After this, it will be possible to think about which > > >> > > > components to =20 > > >> include =20 > > >> > > > in the base system. > > >> > > > > > >> > > > I would be glad to see a more modern language than C in > > >> > > > the =20 > > >> database, but =20 > > >> > > > I=E2=80=99m afraid that it will be like with C++, > > >> > > > that we will get a couple of daemons and utilities and > > >> > > > that=E2=80=99s all.=20 > > >> > > > > >> > > These are all good questions that need good answers, though =20 > > >> necessarily to =20 > > >> > > get started. > > >> > > > > >> > > But the other question that occured to me after my last > > >> > > posting was =20 > > >> "What =20 > > >> > > about build integration?" > > >> > > How much of the rust automation do we take in vs how much do > > >> > > we drive =20 > > >> from =20 > > >> > > a future bsd.rust.mk. > > >> > > I can sketch out bsd.rust.mk (to pick an arbitrary name, > > >> > > we'd likely =20 > > >> need =20 > > >> > > one for what we traditionally > > >> > > think of as libraries (which may or may not map 1:1 onto > > >> > > crates: we =20 > > >> could =20 > > >> > > have c callable libraries > > >> > > written in rust in the future, for example) and one for > > >> > > binaries. Initially, though, if we go with the > > >> > > 'make rust tests possible' then we'd likely need the > > >> > > appropriate =20 > > >> packages =20 > > >> > > installed for whatever > > >> > > dependencies we'd have in the tests. This would give us a > > >> > > taste for =20 > > >> what =20 > > >> > > we'd need to do for > > >> > > base, I'd think. Once we had that notion, I can easily see > > >> > > there =20 > > >> needing to =20 > > >> > > be some sort of > > >> > > rust bindings for ATF/kyua as one of the first libraries / > > >> > > crates that would test that aspect of > > >> > > the build system. That all would be up to the people writing > > >> > > the =20 > > >> tests in =20 > > >> > > rust, I'd imagine. > > >> > > > > >> > > While I could jot out the basics of this integration (so one > > >> > > could =20 > > >> just add =20 > > >> > > the rust > > >> > > tools to a subdir or subdirs, include the bsd.rust.mk or > > >> > > whatever =20 > > >> and then =20 > > >> > > it would build > > >> > > if rust is enabled, and would emit a warning it was skipped > > >> > > because =20 > > >> rust =20 > > >> > > was disabled). > > >> > > We'd find out if this is workable or not and iterate from > > >> > > there. But =20 > > >> that =20 > > >> > > would also require > > >> > > active participation from the rust advocates to make it a > > >> > > reality: I =20 > > >> can =20 > > >> > > put together the > > >> > > build infrastructure for the disabled case, but likely can't > > >> > > on my =20 > > >> own do =20 > > >> > > the rust enabled > > >> > > case. I'd be happy to work with someone to do that, but I'm > > >> > > not going =20 > > >> to be =20 > > >> > > able to do > > >> > > that myself: my need for rust is slight, my knowledge of > > >> > > rust is =20 > > >> weak, etc. =20 > > >> > > Working with > > >> > > someone (or ideally several someones), though it could > > >> > > become =20 > > >> reality. So =20 > > >> > > please contact > > >> > > me if you'd like to work on this. > > >> > > > > >> > > Warner =20 > > >> > > > >> > One way to go could be moving programs rewritten with rust to > > >> > ports. There are some programs (not in rust, though) moved to > > >> > ports, like rcs. =20 > > >> > > >> I've already done this with a few, though I didn't delete the C > > >> versions from base. > > >> usr.bin/gstat =3D> sysutils/gstat-rs > > >> tools/regression/fsx =3D> devel/fsx > > >> =20 > > > > > > So > > > % size `which gstat-rs` `which gstat` > > > text data bss dec hex filename > > > 2094442 176472 568 2271482 0x22a8fa > > > /usr/local/sbin/gstat-rs 19350 1180 41 20571 > > > 0x505b /usr/sbin/gstat % file `which gstat-rs` `which gstat` > > > /usr/local/sbin/gstat-rs: ELF 64-bit LSB pie executable, ARM > > > aarch64, version 1 (FreeBSD), dynamically linked, interpreter > > > /libexec/ld-elf.so.1, FreeBSD-style, stripped > > > /usr/sbin/gstat: ELF 64-bit LSB pie executable, ARM > > > aarch64, version 1 (FreeBSD), dynamically linked, interpreter > > > /libexec/ld-elf.so.1, for FreeBSD 15.0 (1500008), FreeBSD-style, > > > stripped 8:36pm brazos:[3826]> ldd `which gstat-rs` `which gstat` > > > /usr/local/sbin/gstat-rs: > > > libgeom.so.5 =3D> /lib/libgeom.so.5 (0x60fd38647000) > > > libthr.so.3 =3D> /lib/libthr.so.3 (0x60fd38b57000) > > > libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x60fd39af1000) > > > libc.so.7 =3D> /lib/libc.so.7 (0x60fd3be6f000) > > > libbsdxml.so.4 =3D> /lib/libbsdxml.so.4 (0x60fd3a009000) > > > libsbuf.so.6 =3D> /lib/libsbuf.so.6 (0x60fd3a55e000) > > > /usr/sbin/gstat: > > > libdevstat.so.7 =3D> /lib/libdevstat.so.7 (0x448867cd000) > > > libgeom.so.5 =3D> /lib/libgeom.so.5 (0x4488710b000) > > > libedit.so.8 =3D> /lib/libedit.so.8 (0x44887f8d000) > > > libtinfow.so.9 =3D> /lib/libtinfow.so.9 (0x44888aab000) > > > libncursesw.so.9 =3D> /lib/libncursesw.so.9 (0x44889c60000) > > > libc.so.7 =3D> /lib/libc.so.7 (0x4488aaf4000) > > > libkvm.so.7 =3D> /lib/libkvm.so.7 (0x44888f77000) > > > libbsdxml.so.4 =3D> /lib/libbsdxml.so.4 (0x4488ba02000) > > > libsbuf.so.6 =3D> /lib/libsbuf.so.6 (0x4488c68d000) > > > libelf.so.2 =3D> /lib/libelf.so.2 (0x4488ca45000) > > > > > > So that looks scary, like rust is 100x larger binaries... But at > > > runtime it's about the same: > > > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME > > > COMMAND imp 14735 0.0 0.0 14140 4828 0 S+ 20:38 > > > 0:00.04 gstat imp 14766 1.3 0.0 15772 6256 0 S+ 20:39 > > > 0:00.02 gstat-rs > > > > > > So the runtime size is at least in the same ballpark (still > > > larger, but not crazy larger). More CPU too, > > > but that's just a polling artifact I think (other times gstat had > > > some, and gstat-rs didn't). > > > > > > Why is the rust binary so much larger? Are the rust runtime and > > > dependencies statically linked? > > > =20 > >=20 > > Yes, that's a large part of it. Rust libraries are usually > > statically linked (though they don't have to be). For example, in > > the output above, notice that gstat-rs does not link to ncurses. > > That's because the equivalent library is statically linked in > > instead. Also, rust programs by default include goodies like stack > > unwinding on panic, which takes extra code too. But that can be > > turned off if you really want to save space. -Alan =20 >=20 > Because of these, I feel rust fits best for commercial DOS apps, not > Unix-based OS'es. In DOS era, AFAIK, most programs installed required > libraries at the same directory which its executable is installed, > regarless exactly same libraries are installed in other directory by > other softwares. This looks very similar as current rust default, > except that libraries are statically linked in rust. >=20 > But rust seems to have (non-default) option to create shared libraries > by --crate-type=3Ddylib and --crate-type=3Dcdylib options [1]. >=20 > Charlie Li pointed me this page before [2]. >=20 > Without this kind of options, linking rust codes with C/C++ codes > would be nonsense. (Whichever should have this kind of options, but > rust ABI seems to be too unstalbe [changing rapidly] to implement it > in C/C++ side.) >=20 > I'd not yet dug in deep enough to conclude or write rust code myself, > so just a thought for now, but I currently think we would better wait > for rust ABI to be stabilized. Until then, rust codes in ports may > increase and maintainers good at rust would increase, too. > It would greatly help to investigate how to introduce rust codes in > base. >=20 > [1] https://doc.rust-lang.org/reference/linkage.html >=20 > [2] > https://lists.freebsd.org/archives/freebsd-ports/2023-September/004546.ht= ml >=20 I honestly don't expect many new or current languages to get their own dedicated FFI/ABIs in the future. Especially, if they don't want a full language specific runtime involved. Enforcing a requirement of must export to the C ABI for any libraries going into the base system is very reasonable. On the other side of things restricting the dependency list to packaged stuff is also a good idea. Cargo already includes an option for enforcing the use of a local mirror as its package source.[1] Fedora uses this to guarantee they can build any rust code they package repeatably.[2] [1] https://doc.rust-lang.org/cargo/reference/source-replacement.html [2] https://docs.fedoraproject.org/en-US/packaging-guidelines/Rust/