From nobody Wed Jul 31 17:01:17 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WYz1k0TC8z5SDW8 for ; Wed, 31 Jul 2024 17:03:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WYz1j5mVbz4FVM for ; Wed, 31 Jul 2024 17:03:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-2cb4c4de4cbso3962127a91.1 for ; Wed, 31 Jul 2024 10:03:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1722445384; x=1723050184; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=inTcaeSwBnSI8SKQHYaKN61i1wOqZffoACUbgd4Y+ME=; b=Nhn/SCdzGSdzPkanq9F991ZAh6TLmKqLv2mBV/y3yyQ+SyZC4XtuOEY7P30Us3vlpG QMv6hBbGTSJ3K/Az8ioNObgVyZXejXnzPaU3yPOOUwrjybrL9dpRptGJ/9T8vwJZV9wu 5pN2qHv8ws+LvOzTKRfaj83hpvWyLYR1hE+EjJL9J7yNm8WU4arJaUXhYlmq0tKXa8EH dyh/rGPSTD3IwqayCbbYPgpAWKCzOuIoytBEMsM0dce/mt6n7AOaOLUXicpzy56wpoTK L/lH9cHf/r4V1e9DE6T7iC9+5I8qRVtiPU5iq1DHSO1mCDPrpZrMRPhvSxXryAX4ptUN yhdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722445384; x=1723050184; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=inTcaeSwBnSI8SKQHYaKN61i1wOqZffoACUbgd4Y+ME=; b=KdngEVdRQHwDsTNowW1M1zEK0o8Xri8V6tMmUsb/K48Zye/4O8FMiMZOOZlA70HU/T ZERHG45pQFU1ZLyWVsMdkEgD7a0rJ06A3Elu1gNyaHGjM2LwddsNPR7eIMKUKVeXxnBt K9iY1waZXK0gqjaG5A/buBRaOAaJGRrsWCGTyk7RdG1FvHS0TQtzOtAKwaX1fchAb5e7 7dzErLvJu/g6156y6T8Yo+zvfsfATKt0bxax5cekCR1DOReXB5sQ9PjZwL3dY0fLMk2t Px4qjOhai+sdm47/G0BZZdBeFPByqVGfBbZc83Oi11enDfPz8doRXnTiKUvTUP3DuESL e+Lw== X-Forwarded-Encrypted: i=1; AJvYcCWZ/9q+a7PIpU8s3tQ7b35zZgVj4hF4h+RC1/ZUEFyVu3DvRl/izmyYUMD8/1YsAYg04efzjAAj9EwegVgjDhKowHFpcxCvHuxLbSE= X-Gm-Message-State: AOJu0YwVpcGgPw7AVDT2vS0pOeUH6sSJ6SQcR7itrlmDVUAUnJV1QHQn FBLiBu4uf46hV3sB3X1mb+PAkVsTT/q87Cy5CXQlXT0j/L5Z9O7ZNCVAQI6bVhgGPiLyQ/ETGqU voMGkGkK8iuMbYUnt/4qiRVOvVtfCXEGQo7GotLCtXwo2DZMH X-Google-Smtp-Source: AGHT+IETHIow8ZE20L0IY7SLDTy3Qlk7huf4Ea6LC0dh49rd1JqKoEUP+a45LK/BmrPbICot3XhqEnrqTgP9eB+OAoo= X-Received: by 2002:a17:90a:b397:b0:2c7:e46e:f8b7 with SMTP id 98e67ed59e1d1-2cf7e0976admr14319938a91.4.1722445384101; Wed, 31 Jul 2024 10:03:04 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Wed, 31 Jul 2024 11:01:17 -0600 Message-ID: Subject: Re: The Case for Rust (in the base system) To: Alan Somers Cc: Shawn Webb , FreeBSD Hackers , Scott Long , "Goran Meki??" Content-Type: multipart/alternative; boundary="00000000000043da1a061e8e11f5" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WYz1j5mVbz4FVM --00000000000043da1a061e8e11f5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jul 31, 2024, 9:40=E2=80=AFAM Alan Somers wro= te: > On Wed, Jul 31, 2024 at 8:37=E2=80=AFAM Shawn Webb > wrote: > > > > On Sat, Jan 20, 2024 at 09:51:25AM -0700, Alan Somers wrote: > > > In a recent thread on src-committers, we discussed the costs and > > > benefits of including Rust code in the FreeBSD base system. To > > > summarize, the cost is that it would double our build times. imp > > > suggested adding an additional step after buildworld for stuff that > > > requires an external toolchain. That would ease the build time pain. > > > The benefit is that some tools would become easier to write, or even > > > become possible. Here is a list of actual and potential Rust project= s > > > that could benefit from being in-tree. If anybody else has items to > > > add, I suggest moving this into the project wiki: > > > > > > Stuff that could only be written in Rust if it were in base > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > * ctl-exporter (I started this, but discovered that the CTL stats API > is > > > unstable, so it can't live in ports. Instead, I had to do it in C)= . > > > > https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a86401= 469181a67ec34 > > > > > > * fusefs tests. Absolutely impossible to do in C. I considered Rust= , > but went > > > with C++ so they could live in base. They are too closely coupled = to > > > fusefs(5) to live out-of-tree. > > > https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusef= s > > > > > > * devd. Currently C++, but imp suggested a rewrite. > > > https://github.com/freebsd/freebsd-src/tree/main/sbin/devd > > > > > > * zfsd. Currently C++, but I've long pondered a rewrite. Using Rust > would > > > make it more testable. > > > https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd > > > > > > * nscd. Currently C, but confusing and with no test coverage. I've > > > contemplated a rewrite myself, but I don't want to do it in C. > > > https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd > > > > > > * The userland portion of the 802.11ac and Lightning stacks. scottl > suggested > > > that these were good candidates for Rust. > > > > > > * freebsd-kpi-r14-0 . https://crates.io/crates/freebsd-kpi-r14-0 > > > > > > Stuff that can live in ports, but would be nicer in base > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > > > > * gstat-rs https://crates.io/crates/gstat > > > > > > * geom-exporter (I've started this, but haven't published it) > > > > > > * nfs-exporter https://crates.io/crates/freebsd-nfs-exporter > > > > > > * virtiofsd-rs . Nobody has yet tried to port it to FreeBSD. But if > the > > > connection to bhyve(8) is too intimate, it might be hard to do in > ports. > > > https://gitlab.com/virtio-fs/virtiofsd > > > > > > * jail-exporter https://crates.io/crates/jail_exporter > > > > > > * Various jail managers have been attempted in Rust. I think these > are fine in > > > ports, but others like Goran Mekic have opined that they should be > moved to > > > base instead. > > > > > > * musikid's pjdfstest rewrite. I think it would be great to start > using this > > > to test the base system's file systems. If the tests themselves > lived in > > > base, they would be easier to sync with file system development. > > > https://github.com/musikid/pjdfstest > > > > > > * pf-rs. I suspect that the API isn't very stable. > > > https://crates.io/crates/pf-rs > > > > > > * benchpmc. The pmc counter names changes between releases. > > > https://crates.io/crates/benchpmc > > > > > > FreeBSD-related applications that are just fine in ports > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > * fsx-rs. Unlike pjdfstest, this only tests datapath APIs. Those ar= e > usually > > > more stable than control path APIs, so I think there's little to be > gained by > > > moving this into base. https://crates.io/crates/fsx > > > > > > * ztop. It uses ZFS's kstats sysctl interface, which is pretty stabl= e. > > > https://crates.io/crates/ztop > > > > > > * iocage-provision https://crates.io/crates/iocage-provision > > > > > > * rsblk https://crates.io/crates/rsblk > > > > > > * xfuse https://github.com/KhaledEmaraDev/xfuse > > > > > > Other FreeBSD-related libraries in Rust > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > Just see the list at https://crates.io/keywords/freebsd > > > > > > > One new data point: DARPA is looking to rewrite a significant amount > > of C code to Rust with their "Translating All C to Rust (TRACTOR)" > > project: > > https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view > > Interesting. And since you bring it up, I have two new data points mysel= f: > > * ctld: while working on some bugs in ctld, I had trouble > understanding the config file parsing. So I rewrote that part in > Rust, just to help my understanding. Later, I rewrote the XML > parsing, too. Then I rewrote the LUN creation and deletion, just to > see how hard it would be. All of those parts take about 5x fewer SLOC > in Rust than in C, and they're less buggy, too. Config file parsing > is more consistent, no memory leaks, etc. Alas, I'm not planning to > finish this project, since the base system doesn't allow Rust and ctld > is too tightly coupled to ctl to live in ports. > Cool. Still waiting for anybody to take me up on the offer to do build system integration. Since the Rust advocates can't get even this basic step done for review, it's going to be impossible to have Rust in the base. This isn't even integrate rust compiler like we do with llvm, but with external Rust toolchain. Until somebody steps up for this task, the status quo can't possibly change= . Warner * geom-exporter: I mentioned this project up above. It's finished > now, and you can find it in ports at net-mgmt/geom-exporter . > --00000000000043da1a061e8e11f5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Jul 31, 2024, 9:40=E2=80=AFAM Alan Somers <= asomers@freebsd.org> wrote:
On Wed, Jul 31, 2024 at 8:37=E2=80= =AFAM Shawn Webb <shawn.webb@hardenedbsd.org> wrote: >
> On Sat, Jan 20, 2024 at 09:51:25AM -0700, Alan Somers wrote:
> > In a recent thread on src-committers, we discussed the costs and<= br> > > benefits of including Rust code in the FreeBSD base system.=C2=A0= To
> > summarize, the cost is that it would double our build times.=C2= =A0 imp
> > suggested adding an additional step after buildworld for stuff th= at
> > requires an external toolchain.=C2=A0 That would ease the build t= ime pain.
> > The benefit is that some tools would become easier to write, or e= ven
> > become possible.=C2=A0 Here is a list of actual and potential Rus= t projects
> > that could benefit from being in-tree.=C2=A0 If anybody else has = items to
> > add, I suggest moving this into the project wiki:
> >
> > Stuff that could only be written in Rust if it were in base
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >
> > * ctl-exporter (I started this, but discovered that the CTL stats= API is
> >=C2=A0 =C2=A0unstable, so it can't live in ports.=C2=A0 Instea= d, I had to do it in C).
> >=C2=A0 =C2=A0https://github.com/freebsd/freebsd-src/commit/1a7f22d9c2= 11f504f6c48a86401469181a67ec34
> >
> > * fusefs tests.=C2=A0 Absolutely impossible to do in C.=C2=A0 I c= onsidered Rust, but went
> >=C2=A0 =C2=A0with C++ so they could live in base.=C2=A0 They are t= oo closely coupled to
> >=C2=A0 =C2=A0fusefs(5) to live out-of-tree.
> >=C2=A0 =C2=A0https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs > >
> > * devd.=C2=A0 Currently C++, but imp suggested a rewrite.
> >=C2=A0 =C2=A0https://g= ithub.com/freebsd/freebsd-src/tree/main/sbin/devd
> >
> > * zfsd.=C2=A0 Currently C++, but I've long pondered a rewrite= .=C2=A0 Using Rust would
> >=C2=A0 =C2=A0make it more testable.
> >=C2=A0 =C2=A0= https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd
> >
> > * nscd.=C2=A0 Currently C, but confusing and with no test coverag= e.=C2=A0 I've
> >=C2=A0 =C2=A0contemplated a rewrite myself, but I don't want t= o do it in C.
> >=C2=A0 =C2=A0https= ://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd
> >
> > * The userland portion of the 802.11ac and Lightning stacks.=C2= =A0 scottl suggested
> >=C2=A0 =C2=A0that these were good candidates for Rust.
> >
> > * freebsd-kpi-r14-0 .=C2=A0 https://c= rates.io/crates/freebsd-kpi-r14-0
> >
> > Stuff that can live in ports, but would be nicer in base
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >
> > * gstat-rs https://crates.io/crates/gstat
> >
> > * geom-exporter (I've started this, but haven't published= it)
> >
> > * nfs-exporter https://crates.io/c= rates/freebsd-nfs-exporter
> >
> > * virtiofsd-rs .=C2=A0 Nobody has yet tried to port it to FreeBSD= .=C2=A0 But if the
> >=C2=A0 =C2=A0connection to bhyve(8) is too intimate, it might be h= ard to do in ports.
> >=C2=A0 =C2=A0https://gitlab.com/virtio-fs/= virtiofsd
> >
> > * jail-exporter https://crates.io/crates/= jail_exporter
> >
> > * Various jail managers have been attempted in Rust.=C2=A0 I thin= k these are fine in
> >=C2=A0 =C2=A0ports, but others like Goran Mekic have opined that t= hey should be moved to
> >=C2=A0 =C2=A0base instead.
> >
> > * musikid's pjdfstest rewrite.=C2=A0 I think it would be grea= t to start using this
> >=C2=A0 =C2=A0to test the base system's file systems.=C2=A0 If = the tests themselves lived in
> >=C2=A0 =C2=A0base, they would be easier to sync with file system d= evelopment.
> >=C2=A0 =C2=A0https://github.com/musikid/pjd= fstest
> >
> > * pf-rs.=C2=A0 I suspect that the API isn't very stable.
> >=C2=A0 =C2=A0https://crates.io/crates/pf-rs > >
> > * benchpmc.=C2=A0 The pmc counter names changes between releases.=
> >=C2=A0 =C2=A0https://crates.io/crates/benchpmc<= /a>
> >
> > FreeBSD-related applications that are just fine in ports
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >
> > * fsx-rs.=C2=A0 Unlike pjdfstest, this only tests datapath APIs.= =C2=A0 Those are usually
> >=C2=A0 =C2=A0more stable than control path APIs, so I think there&= #39;s little to be gained by
> >=C2=A0 =C2=A0moving this into base. https://crates.i= o/crates/fsx
> >
> > * ztop.=C2=A0 It uses ZFS's kstats sysctl interface, which is= pretty stable.
> >=C2=A0 =C2=A0https://crates.io/crates/ztop
> >
> > * iocage-provision=C2=A0 https://crate= s.io/crates/iocage-provision
> >
> > * rsblk https://crates.io/crates/rsblk
> >
> > * xfuse=C2=A0 https://github.com/KhaledE= maraDev/xfuse
> >
> > Other FreeBSD-related libraries in Rust
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > Just see the list at https://crates.io/keywor= ds/freebsd
> >
>
> One new data point: DARPA is looking to rewrite a significant amount > of C code to Rust with their "Translating All C to Rust (TRACTOR)= "
> project:
> https://sam.gov/opp/1e45d64= 8886b4e9ca91890285af77eb7/view

Interesting.=C2=A0 And since you bring it up, I have two new data points my= self:

* ctld: while working on some bugs in ctld, I had trouble
understanding the config file parsing.=C2=A0 So I rewrote that part in
Rust, just to help my understanding.=C2=A0 Later, I rewrote the XML
parsing, too.=C2=A0 Then I rewrote the LUN creation and deletion, just to see how hard it would be.=C2=A0 All of those parts take about 5x fewer SLOC=
in Rust than in C, and they're less buggy, too.=C2=A0 Config file parsi= ng
is more consistent, no memory leaks, etc.=C2=A0 Alas, I'm not planning = to
finish this project, since the base system doesn't allow Rust and ctld<= br> is too tightly coupled to ctl to live in ports.

Cool. Still waiting for anyb= ody to take me up on the offer to do build system integration. Since the Ru= st advocates can't get even this basic step done for review, it's g= oing to be impossible to have Rust in the base. This isn't even integra= te rust compiler like we do with llvm, but with external Rust toolchain.

Until somebody steps up fo= r this task, the status quo can't possibly change.

Warner

* geom-exporter: I mentioned this project up above.=C2=A0 It's finished=
now, and you can find it in ports at net-mgmt/geom-exporter .
--00000000000043da1a061e8e11f5--