From nobody Sat Jan 20 22:31:23 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4THWRs5jmjz57fH5 for ; Sat, 20 Jan 2024 22:31:37 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4THWRr3RRSz53cm for ; Sat, 20 Jan 2024 22:31:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20230601.gappssmtp.com header.s=20230601 header.b=g+46lFko; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::52b) smtp.mailfrom=wlosh@bsdimp.com Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-556c3f0d6c5so2558993a12.2 for ; Sat, 20 Jan 2024 14:31:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1705789893; x=1706394693; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=fT12rTF04Yv9iC6W45aGYzJ/ZECzsBePXYNdOVw8bfk=; b=g+46lFkofYfGYO2p9YYLRA/lb2ND3gna1ZHOByffc49Ra3oq8SJdBw67COYbX+liiz 0qnaPCCOL0HiA/QsILVO6pXvF0SuvzSr8frZoHeNjX7ng0h18zikeyCGX3d5JqQ0wiRi P7+bOkaWhkrheo6rwOe9rbCiF5AuF+rUGaKuvWmu8a/TuQVVu+VoDFX5e4tDXexIhlMp 7r3GV7hvFnYNbX2I/JVnj6XrHUPhR3vdB3n1JD32zdNBjZQEu/mJbTYm5+PHTppMqG6y RWyYkz5Nq8FBd41prfH30KqDxnZevv1R1srRRlacZhmsnaZLvvkH2ggHLZXxGF9t3jtw ASRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705789893; x=1706394693; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=fT12rTF04Yv9iC6W45aGYzJ/ZECzsBePXYNdOVw8bfk=; b=bR5OtpQfef7Y73kRjWJRfNw94WG1Pyx9NgoaxTUgrq7GAMwWoVQodvOul6vMknVmao xx3UH48P2vE95DyljFthREugB4syAMbqVdhLBgHj/zLM4LL2BLwWEkSW2Rrzm4kJqzpi 1qUrJwUnrlTYTtCuKmJ0mqZOaI8G4S4XrjgzDaVIAE/8Uc35uL5IOwRSbpfaeI+o7c7x nI8dmyr8/btpQiz6rbNwC1I1ke83ExpYZ0IqrT98Xq4kdUdayoZR3Nw6a99/Aw2czoab 92I+j5N0RjaW4Zs8a2uJ+QNET+Fy2LFkIEGI5ULfcR/uiLc5qcKTqIu9IUBbWJgCrAAm Iv+Q== X-Gm-Message-State: AOJu0YxgakKl5yjrasR194SaJV6KtCAK5PA5UFTGZr4T3v5lP6KiAOzb EAkxnOMEYVEd1FtEwKUZWPA8vzhnwi8nV6dGjrzqEHGGIGMpt5hw28ES4NF7yhtknpq1YhyXqh7 LEw8AEUmTdftofJ7ZrgbWni6g3mUfJwwf6MouhA== X-Google-Smtp-Source: AGHT+IGpCU1qKZt+/bnhlA1ZO7I3oxVIlOmQjCipEop2fq0D0TCbAT8IiPWmK8HSVym5uqNsN4wKW20zeZ0lKX+mvQE= X-Received: by 2002:a05:6402:2202:b0:559:fbdf:6d79 with SMTP id cq2-20020a056402220200b00559fbdf6d79mr511235edb.114.1705789893083; Sat, 20 Jan 2024 14:31:33 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <1673801705774097@mail.yandex.ru> In-Reply-To: <1673801705774097@mail.yandex.ru> From: Warner Losh Date: Sat, 20 Jan 2024 15:31:23 -0700 Message-ID: Subject: Re: The Case for Rust (in the base system) To: Aleksandr Fedorov Cc: Alan Somers , FreeBSD Hackers , Scott Long , "meka@tilda.center" Content-Type: multipart/alternative; boundary="000000000000a39865060f682804" X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_EQ_ADDR_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[bsdimp.com]; RCVD_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52b:from]; FREEMAIL_TO(0.00)[yandex.ru]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20230601.gappssmtp.com:+] X-Rspamd-Queue-Id: 4THWRr3RRSz53cm --000000000000a39865060f682804 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Jan 20, 2024 at 11:45=E2=80=AFAM Aleksandr Fedorov wrote: > What about external dependencies? > > https://github.com/Axcient/freebsd-nfs-exporter/blob/master/Cargo.toml#L1= 9 > https://github.com/asomers/gstat-rs/blob/master/gstat/src/main.rs#L20 > > Is there any plan for which crates we should take into the base system? > > We have had C++ in base for many years, but I don=E2=80=99t see any good = libraries > for CLI, logging, JSON, etc. > > > https://doc.rust-lang.org/rustc/platform-support.html#tier-1-with-host-to= ols > > Where is the support for Freebsd as a primary platform? ARM, RISC-V, > Power? Should we rewrite devd? > > I think we need to start by providing official repositories (e.g > git.FreeBSD.org/rust.git or git.FreeBSD.org/go.git) > for different languages that include stable bindings to the system API: > - sysctl > - libgeom > - libifconfig > - netgraph > - jail > - etc. > > So that it=E2=80=99s not just some anonymous on crates.io that represents= these > bindings, but our community. > Officially, with support for a stable ABI for releases, security patches, > etc. > > After this, it will be possible to think about which components to includ= e > in the base system. > > I would be glad to see a more modern language than C in the database, but > I=E2=80=99m afraid that it will be like with C++, > that we will get a couple of daemons and utilities and that=E2=80=99s all= . > These are all good questions that need good answers, though necessarily to get started. But the other question that occured to me after my last posting was "What about build integration?" How much of the rust automation do we take in vs how much do we drive from a future bsd.rust.mk. I can sketch out bsd.rust.mk (to pick an arbitrary name, we'd likely need one for what we traditionally think of as libraries (which may or may not map 1:1 onto crates: we could have c callable libraries written in rust in the future, for example) and one for binaries. Initially, though, if we go with the 'make rust tests possible' then we'd likely need the appropriate packages installed for whatever dependencies we'd have in the tests. This would give us a taste for what we'd need to do for base, I'd think. Once we had that notion, I can easily see there needing to be some sort of rust bindings for ATF/kyua as one of the first libraries / crates that would test that aspect of the build system. That all would be up to the people writing the tests in rust, I'd imagine. While I could jot out the basics of this integration (so one could just add the rust tools to a subdir or subdirs, include the bsd.rust.mk or whatever and then it would build if rust is enabled, and would emit a warning it was skipped because rust was disabled). We'd find out if this is workable or not and iterate from there. But that would also require active participation from the rust advocates to make it a reality: I can put together the build infrastructure for the disabled case, but likely can't on my own do the rust enabled case. I'd be happy to work with someone to do that, but I'm not going to be able to do that myself: my need for rust is slight, my knowledge of rust is weak, etc. Working with someone (or ideally several someones), though it could become reality. So please contact me if you'd like to work on this. Warner > 20.01.2024, 19:51, "Alan Somers" : > > In a recent thread on src-committers, we discussed the costs and > benefits of including Rust code in the FreeBSD base system. To > summarize, the cost is that it would double our build times. imp > suggested adding an additional step after buildworld for stuff that > requires an external toolchain. That would ease the build time pain. > The benefit is that some tools would become easier to write, or even > become possible. Here is a list of actual and potential Rust projects > that could benefit from being in-tree. If anybody else has items to > add, I suggest moving this into the project wiki: > > Stuff that could only be written in Rust if it were in base > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > * ctl-exporter (I started this, but discovered that the CTL stats API is > unstable, so it can't live in ports. Instead, I had to do it in C). > > https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a86401= 469181a67ec34 > > * fusefs tests. Absolutely impossible to do in C. I considered Rust, but > went > with C++ so they could live in base. They are too closely coupled to > fusefs(5) to live out-of-tree. > https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs > > * devd. Currently C++, but imp suggested a rewrite. > https://github.com/freebsd/freebsd-src/tree/main/sbin/devd > > * zfsd. Currently C++, but I've long pondered a rewrite. Using Rust would > make it more testable. > https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd > > * nscd. Currently C, but confusing and with no test coverage. I've > contemplated a rewrite myself, but I don't want to do it in C. > https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd > > * The userland portion of the 802.11ac and Lightning stacks. scottl > suggested > that these were good candidates for Rust. > > * freebsd-kpi-r14-0 . https://crates.io/crates/freebsd-kpi-r14-0 > > Stuff that can live in ports, but would be nicer in base > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > * gstat-rs https://crates.io/crates/gstat > > * geom-exporter (I've started this, but haven't published it) > > * nfs-exporter https://crates.io/crates/freebsd-nfs-exporter > > * virtiofsd-rs . Nobody has yet tried to port it to FreeBSD. But if the > connection to bhyve(8) is too intimate, it might be hard to do in ports= . > https://gitlab.com/virtio-fs/virtiofsd > > * jail-exporter https://crates.io/crates/jail_exporter > > * Various jail managers have been attempted in Rust. I think these are > fine in > ports, but others like Goran Mekic have opined that they should be move= d > to > base instead. > > * musikid's pjdfstest rewrite. I think it would be great to start using > this > to test the base system's file systems. If the tests themselves lived i= n > base, they would be easier to sync with file system development. > https://github.com/musikid/pjdfstest > > * pf-rs. I suspect that the API isn't very stable. > https://crates.io/crates/pf-rs > > * benchpmc. The pmc counter names changes between releases. > https://crates.io/crates/benchpmc > > FreeBSD-related applications that are just fine in ports > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > * fsx-rs. Unlike pjdfstest, this only tests datapath APIs. Those are > usually > more stable than control path APIs, so I think there's little to be > gained by > moving this into base. https://crates.io/crates/fsx > > * ztop. It uses ZFS's kstats sysctl interface, which is pretty stable. > https://crates.io/crates/ztop > > * iocage-provision https://crates.io/crates/iocage-provision > > * rsblk https://crates.io/crates/rsblk > > * xfuse https://github.com/KhaledEmaraDev/xfuse > > Other FreeBSD-related libraries in Rust > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Just see the list at https://crates.io/keywords/freebsd > > --000000000000a39865060f682804 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sat, Jan 20, 2024 at 11:45=E2=80= =AFAM Aleksandr Fedorov <wignedd= oom@yandex.ru> wrote:
https://gith= ub.com/asomers/gstat-rs/blob/master/gstat/src/main.rs#L20

Is the= re any plan for which crates we should take into the base system?
=C2=A0
We have had C++ in base for many years, but I don=E2= =80=99t see any good libraries for CLI, logging, JSON, etc.
=C2=A0
=C2=A0
Where is the support for Freebsd as a primary platform? = ARM, RISC-V, Power? Should we rewrite devd?
=C2=A0
I think we need to start by providing official repositories (e.g git.FreeBSD.org/r= ust.git or = git.FreeBSD.org/go.git)
for different languages that include = stable bindings to the system API:
- sysctl
- libgeom
- libifconfig
- netgraph
- jail
- et= c.
=C2=A0
So that it=E2=80=99s not just some anonymous = on crates.io that repres= ents these bindings, but our community.
Officially, with support = for a stable ABI for releases, security patches, etc.
=C2=A0
After this, it will be possible to think about which components to in= clude in the base system.
=C2=A0
I would be glad t= o see a more modern language than C in the database, but I=E2=80=99m afraid= that it will be like with C++,
that we will get a couple of daem= ons and utilities and that=E2=80=99s all.

These are all good questions that need g= ood answers, though necessarily to get started.

Bu= t the other question that occured to me after my last posting was "Wha= t about build integration?"
How much of the rust automation = do we take in vs how much do we drive from a future bsd.rust.mk.
I can sketch out bsd.rust.mk (to pick an arbitrary name, we'd likely need on= e for what we traditionally
think of as libraries (which may or m= ay not map 1:1 onto crates: we could have c callable libraries
wr= itten in rust in the future, for example) and one for binaries.=C2=A0 Initi= ally, though, if we go with the
'make rust tests possible'= ; then we'd likely need the appropriate packages installed for whatever=
dependencies we'd have in the tests. This would give us a ta= ste for what we'd need to do for
base, I'd think. Once we= had that notion, I can easily see there needing to be some sort of
rust bindings for ATF/kyua as one of the first libraries / crates that w= ould test that aspect of
the build system. That all would be up t= o the people writing the tests in rust, I'd imagine.

=
While I could jot out the basics of this integration (so one cou= ld just add the rust
tools to a subdir or subdirs, include the bsd.rust.mk or whatever and then it would = build
if rust is enabled, and would emit a warning it was skipped= because rust was disabled).
We'd find out if this is workabl= e or not and iterate from there. But that would also require
acti= ve participation from the rust advocates to make it a reality: I can put to= gether the
build infrastructure for the disabled case, but likely= can't on my own do the rust enabled
case. I'd be happy t= o work with someone to do that, but I'm not going to be able to do
that myself: my need for rust is slight, my knowledge of rust is weak= , etc. Working with
someone (or ideally several someones), though= it could become reality. So please contact
me if you'd like = to work on this.

Warner
=C2=A0
20.01.2024, 19:5= 1, "Alan Somers" <asomers@freebsd.org>:

In a recent thread = on src-committers, we discussed the costs and
benefits of including Rust= code in the FreeBSD base system. To
summarize, the cost is that it woul= d double our build times. imp
suggested adding an additional step after = buildworld for stuff that
requires an external toolchain. That would eas= e the build time pain.
The benefit is that some tools would become easie= r to write, or even
become possible. Here is a list of actual and potent= ial Rust projects
that could benefit from being in-tree. If anybody else= has items to
add, I suggest moving this into the project wiki:

S= tuff that could only be written in Rust if it were in base
=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D

* ctl-exporter (I started this, but discovered that = the CTL stats API is
=C2=A0=C2=A0unstable, so it can't live in ports= . Instead, I had to do it in C).
=C2=A0=C2=A0https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f5= 04f6c48a86401469181a67ec34

* fusefs tests. Absolutely impossible= to do in C. I considered Rust, but went
=C2=A0=C2=A0with C++ so they co= uld live in base. They are too closely coupled to
=C2=A0=C2=A0fusefs(5) = to live out-of-tree.
=C2=A0=C2=A0https://github.= com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs

* devd. Cu= rrently C++, but imp suggested a rewrite.
=C2=A0=C2=A0http= s://github.com/freebsd/freebsd-src/tree/main/sbin/devd

* zfsd. C= urrently C++, but I've long pondered a rewrite. Using Rust would
=C2= =A0=C2=A0make it more testable.
=C2=A0=C2=A0https= ://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd

*= nscd. Currently C, but confusing and with no test coverage. I've
= =C2=A0=C2=A0contemplated a rewrite myself, but I don't want to do it in= C.
=C2=A0=C2=A0https://github.com/freebsd/freebsd-src= /tree/main/usr.sbin/nscd

* The userland portion of the 802.11ac = and Lightning stacks. scottl suggested
=C2=A0=C2=A0that these were good = candidates for Rust.

* freebsd-kpi-r14-0 . https://crates.io/crates/fr= eebsd-kpi-r14-0

Stuff that can live in ports, but would be nicer= in base
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

* gstat-rs https://crates.io/crates/gstat
* geom-exporter (I've started this, but haven't published it)
=
* nfs-exporter https://crates.io/crates/freebsd-nfs-exporter
* virtiofsd-rs . Nobody has yet tried to port it to FreeBSD. But if the=C2=A0=C2=A0connection to bhyve(8) is too intimate, it might be hard to d= o in ports.
=C2=A0=C2=A0https://gitlab.com/virtio-fs/virtiofsd

* ja= il-exporter https://crates.io/crates/jail_exporter

* Various jail mana= gers have been attempted in Rust. I think these are fine in
=C2=A0=C2=A0= ports, but others like Goran Mekic have opined that they should be moved to=
=C2=A0=C2=A0base instead.

* musikid's pjdfstest rewrite. I t= hink it would be great to start using this
=C2=A0=C2=A0to test the base = system's file systems. If the tests themselves lived in
=C2=A0=C2=A0= base, they would be easier to sync with file system development.
=C2=A0= =C2=A0ht= tps://github.com/musikid/pjdfstest

* pf-rs. I suspect that the A= PI isn't very stable.
=C2=A0=C2=A0https://crates.io/crates/pf-rs

* benc= hpmc. The pmc counter names changes between releases.
=C2=A0=C2=A0https://crates.io= /crates/benchpmc

FreeBSD-related applications that are just fine= in ports
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

* fsx-rs. Unlike pjdfstest, thi= s only tests datapath APIs. Those are usually
=C2=A0=C2=A0more stable th= an control path APIs, so I think there's little to be gained by
=C2= =A0=C2=A0moving this into base. https://crates.io/crates/fsx

* ztop. It uses ZFS= 's kstats sysctl interface, which is pretty stable.
=C2=A0=C2=A0https://crates.io/c= rates/ztop

* iocage-provision https://crates.io/crates/iocage-provi= sion

* rsblk https://crates.io/crates/rsblk

* xfuse https://github.com= /KhaledEmaraDev/xfuse

Other FreeBSD-related libraries in Rust=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Just see the list at https://crates.= io/keywords/freebsd
=C2=A0
--000000000000a39865060f682804--