Date: Wed, 31 Jul 2024 11:01:17 -0600 From: Warner Losh <imp@bsdimp.com> To: Alan Somers <asomers@freebsd.org> Cc: Shawn Webb <shawn.webb@hardenedbsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, Scott Long <scottl@freebsd.org>, "Goran Meki??" <meka@tilda.center> Subject: Re: The Case for Rust (in the base system) Message-ID: <CANCZdfqB1%2B-8BkpKwKoCM%2BzM4mCOFy63yHr1Pco7MnT1DFkb4w@mail.gmail.com> In-Reply-To: <CAOtMX2iDK3uN_oQgzzZAdoOZCfNsnvpefeZvKoTCRmPBhZywzA@mail.gmail.com> References: <CAOtMX2hAUiWdGPtpaCJLPZB%2Bj2yzNw5DSjUmkwTi%2B%2BmyemehCA@mail.gmail.com> <vdmg5zocd6wqcwc2bvzvzqn4bii2pwdc2r4mgnisukfkboj6nf@f7lv5quu4fjx> <CAOtMX2iDK3uN_oQgzzZAdoOZCfNsnvpefeZvKoTCRmPBhZywzA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000043da1a061e8e11f5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jul 31, 2024, 9:40=E2=80=AFAM Alan Somers <asomers@freebsd.org> wro= te: > On Wed, Jul 31, 2024 at 8:37=E2=80=AFAM Shawn Webb <shawn.webb@hardenedbs= d.org> > wrote: > > > > On Sat, Jan 20, 2024 at 09:51:25AM -0700, Alan Somers wrote: > > > In a recent thread on src-committers, we discussed the costs and > > > benefits of including Rust code in the FreeBSD base system. To > > > summarize, the cost is that it would double our build times. imp > > > suggested adding an additional step after buildworld for stuff that > > > requires an external toolchain. That would ease the build time pain. > > > The benefit is that some tools would become easier to write, or even > > > become possible. Here is a list of actual and potential Rust project= s > > > that could benefit from being in-tree. If anybody else has items to > > > add, I suggest moving this into the project wiki: > > > > > > Stuff that could only be written in Rust if it were in base > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > * ctl-exporter (I started this, but discovered that the CTL stats API > is > > > unstable, so it can't live in ports. Instead, I had to do it in C)= . > > > > https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a86401= 469181a67ec34 > > > > > > * fusefs tests. Absolutely impossible to do in C. I considered Rust= , > but went > > > with C++ so they could live in base. They are too closely coupled = to > > > fusefs(5) to live out-of-tree. > > > https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusef= s > > > > > > * devd. Currently C++, but imp suggested a rewrite. > > > https://github.com/freebsd/freebsd-src/tree/main/sbin/devd > > > > > > * zfsd. Currently C++, but I've long pondered a rewrite. Using Rust > would > > > make it more testable. > > > https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd > > > > > > * nscd. Currently C, but confusing and with no test coverage. I've > > > contemplated a rewrite myself, but I don't want to do it in C. > > > https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd > > > > > > * The userland portion of the 802.11ac and Lightning stacks. scottl > suggested > > > that these were good candidates for Rust. > > > > > > * freebsd-kpi-r14-0 . https://crates.io/crates/freebsd-kpi-r14-0 > > > > > > Stuff that can live in ports, but would be nicer in base > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > > > > * gstat-rs https://crates.io/crates/gstat > > > > > > * geom-exporter (I've started this, but haven't published it) > > > > > > * nfs-exporter https://crates.io/crates/freebsd-nfs-exporter > > > > > > * virtiofsd-rs . Nobody has yet tried to port it to FreeBSD. But if > the > > > connection to bhyve(8) is too intimate, it might be hard to do in > ports. > > > https://gitlab.com/virtio-fs/virtiofsd > > > > > > * jail-exporter https://crates.io/crates/jail_exporter > > > > > > * Various jail managers have been attempted in Rust. I think these > are fine in > > > ports, but others like Goran Mekic have opined that they should be > moved to > > > base instead. > > > > > > * musikid's pjdfstest rewrite. I think it would be great to start > using this > > > to test the base system's file systems. If the tests themselves > lived in > > > base, they would be easier to sync with file system development. > > > https://github.com/musikid/pjdfstest > > > > > > * pf-rs. I suspect that the API isn't very stable. > > > https://crates.io/crates/pf-rs > > > > > > * benchpmc. The pmc counter names changes between releases. > > > https://crates.io/crates/benchpmc > > > > > > FreeBSD-related applications that are just fine in ports > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > * fsx-rs. Unlike pjdfstest, this only tests datapath APIs. Those ar= e > usually > > > more stable than control path APIs, so I think there's little to be > gained by > > > moving this into base. https://crates.io/crates/fsx > > > > > > * ztop. It uses ZFS's kstats sysctl interface, which is pretty stabl= e. > > > https://crates.io/crates/ztop > > > > > > * iocage-provision https://crates.io/crates/iocage-provision > > > > > > * rsblk https://crates.io/crates/rsblk > > > > > > * xfuse https://github.com/KhaledEmaraDev/xfuse > > > > > > Other FreeBSD-related libraries in Rust > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > Just see the list at https://crates.io/keywords/freebsd > > > > > > > One new data point: DARPA is looking to rewrite a significant amount > > of C code to Rust with their "Translating All C to Rust (TRACTOR)" > > project: > > https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view > > Interesting. And since you bring it up, I have two new data points mysel= f: > > * ctld: while working on some bugs in ctld, I had trouble > understanding the config file parsing. So I rewrote that part in > Rust, just to help my understanding. Later, I rewrote the XML > parsing, too. Then I rewrote the LUN creation and deletion, just to > see how hard it would be. All of those parts take about 5x fewer SLOC > in Rust than in C, and they're less buggy, too. Config file parsing > is more consistent, no memory leaks, etc. Alas, I'm not planning to > finish this project, since the base system doesn't allow Rust and ctld > is too tightly coupled to ctl to live in ports. > Cool. Still waiting for anybody to take me up on the offer to do build system integration. Since the Rust advocates can't get even this basic step done for review, it's going to be impossible to have Rust in the base. This isn't even integrate rust compiler like we do with llvm, but with external Rust toolchain. Until somebody steps up for this task, the status quo can't possibly change= . Warner * geom-exporter: I mentioned this project up above. It's finished > now, and you can find it in ports at net-mgmt/geom-exporter . > --00000000000043da1a061e8e11f5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Wed, Jul 31, 2024, 9:40=E2=80=AFAM Alan Somers <= <a href=3D"mailto:asomers@freebsd.org">asomers@freebsd.org</a>> wrote:<b= r></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border= -left:1px #ccc solid;padding-left:1ex">On Wed, Jul 31, 2024 at 8:37=E2=80= =AFAM Shawn Webb <<a href=3D"mailto:shawn.webb@hardenedbsd.org" target= =3D"_blank" rel=3D"noreferrer">shawn.webb@hardenedbsd.org</a>> wrote:<br= > ><br> > On Sat, Jan 20, 2024 at 09:51:25AM -0700, Alan Somers wrote:<br> > > In a recent thread on src-committers, we discussed the costs and<= br> > > benefits of including Rust code in the FreeBSD base system.=C2=A0= To<br> > > summarize, the cost is that it would double our build times.=C2= =A0 imp<br> > > suggested adding an additional step after buildworld for stuff th= at<br> > > requires an external toolchain.=C2=A0 That would ease the build t= ime pain.<br> > > The benefit is that some tools would become easier to write, or e= ven<br> > > become possible.=C2=A0 Here is a list of actual and potential Rus= t projects<br> > > that could benefit from being in-tree.=C2=A0 If anybody else has = items to<br> > > add, I suggest moving this into the project wiki:<br> > ><br> > > Stuff that could only be written in Rust if it were in base<br> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > ><br> > > * ctl-exporter (I started this, but discovered that the CTL stats= API is<br> > >=C2=A0 =C2=A0unstable, so it can't live in ports.=C2=A0 Instea= d, I had to do it in C).<br> > >=C2=A0 =C2=A0<a href=3D"https://github.com/freebsd/freebsd-src/com= mit/1a7f22d9c211f504f6c48a86401469181a67ec34" rel=3D"noreferrer noreferrer"= target=3D"_blank">https://github.com/freebsd/freebsd-src/commit/1a7f22d9c2= 11f504f6c48a86401469181a67ec34</a><br> > ><br> > > * fusefs tests.=C2=A0 Absolutely impossible to do in C.=C2=A0 I c= onsidered Rust, but went<br> > >=C2=A0 =C2=A0with C++ so they could live in base.=C2=A0 They are t= oo closely coupled to<br> > >=C2=A0 =C2=A0fusefs(5) to live out-of-tree.<br> > >=C2=A0 =C2=A0<a href=3D"https://github.com/freebsd/freebsd-src/tre= e/main/tests/sys/fs/fusefs" rel=3D"noreferrer noreferrer" target=3D"_blank"= >https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs</a><b= r> > ><br> > > * devd.=C2=A0 Currently C++, but imp suggested a rewrite.<br> > >=C2=A0 =C2=A0<a href=3D"https://github.com/freebsd/freebsd-src/tre= e/main/sbin/devd" rel=3D"noreferrer noreferrer" target=3D"_blank">https://g= ithub.com/freebsd/freebsd-src/tree/main/sbin/devd</a><br> > ><br> > > * zfsd.=C2=A0 Currently C++, but I've long pondered a rewrite= .=C2=A0 Using Rust would<br> > >=C2=A0 =C2=A0make it more testable.<br> > >=C2=A0 =C2=A0<a href=3D"https://github.com/freebsd/freebsd-src/tre= e/main/cddl/usr.sbin/zfsd" rel=3D"noreferrer noreferrer" target=3D"_blank">= https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd</a><br>; > ><br> > > * nscd.=C2=A0 Currently C, but confusing and with no test coverag= e.=C2=A0 I've<br> > >=C2=A0 =C2=A0contemplated a rewrite myself, but I don't want t= o do it in C.<br> > >=C2=A0 =C2=A0<a href=3D"https://github.com/freebsd/freebsd-src/tre= e/main/usr.sbin/nscd" rel=3D"noreferrer noreferrer" target=3D"_blank">https= ://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd</a><br> > ><br> > > * The userland portion of the 802.11ac and Lightning stacks.=C2= =A0 scottl suggested<br> > >=C2=A0 =C2=A0that these were good candidates for Rust.<br> > ><br> > > * freebsd-kpi-r14-0 .=C2=A0 <a href=3D"https://crates.io/crates/f= reebsd-kpi-r14-0" rel=3D"noreferrer noreferrer" target=3D"_blank">https://c= rates.io/crates/freebsd-kpi-r14-0</a><br> > ><br> > > Stuff that can live in ports, but would be nicer in base<br> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > ><br> > > * gstat-rs <a href=3D"https://crates.io/crates/gstat" rel=3D"nore= ferrer noreferrer" target=3D"_blank">https://crates.io/crates/gstat</a><br>; > ><br> > > * geom-exporter (I've started this, but haven't published= it)<br> > ><br> > > * nfs-exporter <a href=3D"https://crates.io/crates/freebsd-nfs-ex= porter" rel=3D"noreferrer noreferrer" target=3D"_blank">https://crates.io/c= rates/freebsd-nfs-exporter</a><br> > ><br> > > * virtiofsd-rs .=C2=A0 Nobody has yet tried to port it to FreeBSD= .=C2=A0 But if the<br> > >=C2=A0 =C2=A0connection to bhyve(8) is too intimate, it might be h= ard to do in ports.<br> > >=C2=A0 =C2=A0<a href=3D"https://gitlab.com/virtio-fs/virtiofsd" re= l=3D"noreferrer noreferrer" target=3D"_blank">https://gitlab.com/virtio-fs/= virtiofsd</a><br> > ><br> > > * jail-exporter <a href=3D"https://crates.io/crates/jail_exporter= " rel=3D"noreferrer noreferrer" target=3D"_blank">https://crates.io/crates/= jail_exporter</a><br> > ><br> > > * Various jail managers have been attempted in Rust.=C2=A0 I thin= k these are fine in<br> > >=C2=A0 =C2=A0ports, but others like Goran Mekic have opined that t= hey should be moved to<br> > >=C2=A0 =C2=A0base instead.<br> > ><br> > > * musikid's pjdfstest rewrite.=C2=A0 I think it would be grea= t to start using this<br> > >=C2=A0 =C2=A0to test the base system's file systems.=C2=A0 If = the tests themselves lived in<br> > >=C2=A0 =C2=A0base, they would be easier to sync with file system d= evelopment.<br> > >=C2=A0 =C2=A0<a href=3D"https://github.com/musikid/pjdfstest" rel= =3D"noreferrer noreferrer" target=3D"_blank">https://github.com/musikid/pjd= fstest</a><br> > ><br> > > * pf-rs.=C2=A0 I suspect that the API isn't very stable.<br> > >=C2=A0 =C2=A0<a href=3D"https://crates.io/crates/pf-rs" rel=3D"nor= eferrer noreferrer" target=3D"_blank">https://crates.io/crates/pf-rs</a><br= > > ><br> > > * benchpmc.=C2=A0 The pmc counter names changes between releases.= <br> > >=C2=A0 =C2=A0<a href=3D"https://crates.io/crates/benchpmc" rel=3D"= noreferrer noreferrer" target=3D"_blank">https://crates.io/crates/benchpmc<= /a><br> > ><br> > > FreeBSD-related applications that are just fine in ports<br> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > ><br> > > * fsx-rs.=C2=A0 Unlike pjdfstest, this only tests datapath APIs.= =C2=A0 Those are usually<br> > >=C2=A0 =C2=A0more stable than control path APIs, so I think there&= #39;s little to be gained by<br> > >=C2=A0 =C2=A0moving this into base. <a href=3D"https://crates.io/c= rates/fsx" rel=3D"noreferrer noreferrer" target=3D"_blank">https://crates.i= o/crates/fsx</a><br> > ><br> > > * ztop.=C2=A0 It uses ZFS's kstats sysctl interface, which is= pretty stable.<br> > >=C2=A0 =C2=A0<a href=3D"https://crates.io/crates/ztop" rel=3D"nore= ferrer noreferrer" target=3D"_blank">https://crates.io/crates/ztop</a><br>; > ><br> > > * iocage-provision=C2=A0 <a href=3D"https://crates.io/crates/ioca= ge-provision" rel=3D"noreferrer noreferrer" target=3D"_blank">https://crate= s.io/crates/iocage-provision</a><br> > ><br> > > * rsblk <a href=3D"https://crates.io/crates/rsblk" rel=3D"norefer= rer noreferrer" target=3D"_blank">https://crates.io/crates/rsblk</a><br>; > ><br> > > * xfuse=C2=A0 <a href=3D"https://github.com/KhaledEmaraDev/xfuse"= rel=3D"noreferrer noreferrer" target=3D"_blank">https://github.com/KhaledE= maraDev/xfuse</a><br> > ><br> > > Other FreeBSD-related libraries in Rust<br> > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> > > Just see the list at <a href=3D"https://crates.io/keywords/freebs= d" rel=3D"noreferrer noreferrer" target=3D"_blank">https://crates.io/keywor= ds/freebsd</a><br> > ><br> ><br> > One new data point: DARPA is looking to rewrite a significant amount<b= r> > of C code to Rust with their "Translating All C to Rust (TRACTOR)= "<br> > project:<br> > <a href=3D"https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view" = rel=3D"noreferrer noreferrer" target=3D"_blank">https://sam.gov/opp/1e45d64= 8886b4e9ca91890285af77eb7/view</a><br> <br> Interesting.=C2=A0 And since you bring it up, I have two new data points my= self:<br> <br> * ctld: while working on some bugs in ctld, I had trouble<br> understanding the config file parsing.=C2=A0 So I rewrote that part in<br> Rust, just to help my understanding.=C2=A0 Later, I rewrote the XML<br> parsing, too.=C2=A0 Then I rewrote the LUN creation and deletion, just to<b= r> see how hard it would be.=C2=A0 All of those parts take about 5x fewer SLOC= <br> in Rust than in C, and they're less buggy, too.=C2=A0 Config file parsi= ng<br> is more consistent, no memory leaks, etc.=C2=A0 Alas, I'm not planning = to<br> finish this project, since the base system doesn't allow Rust and ctld<= br> is too tightly coupled to ctl to live in ports.<br></blockquote></div></div= ><div dir=3D"auto"><br></div><div dir=3D"auto">Cool. Still waiting for anyb= ody to take me up on the offer to do build system integration. Since the Ru= st advocates can't get even this basic step done for review, it's g= oing to be impossible to have Rust in the base. This isn't even integra= te rust compiler like we do with llvm, but with external Rust toolchain.</d= iv><div dir=3D"auto"><br></div><div dir=3D"auto">Until somebody steps up fo= r this task, the status quo can't possibly change.</div><div dir=3D"aut= o"><br></div><div dir=3D"auto">Warner</div><div dir=3D"auto"><br></div><div= dir=3D"auto"><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" = style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> * geom-exporter: I mentioned this project up above.=C2=A0 It's finished= <br> now, and you can find it in ports at net-mgmt/geom-exporter .<br> </blockquote></div></div></div> --00000000000043da1a061e8e11f5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfqB1%2B-8BkpKwKoCM%2BzM4mCOFy63yHr1Pco7MnT1DFkb4w>