Date: Sat, 20 Jan 2024 11:37:25 -0700 From: Warner Losh <imp@bsdimp.com> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>, "Goran Meki??" <meka@tilda.center> Subject: Re: The Case for Rust (in the base system) Message-ID: <CANCZdfrdwJPjVObeHweQnZa5KeqYXdkj%2BwFZ=36nBNKmX%2Bk64g@mail.gmail.com> In-Reply-To: <CAOtMX2hAUiWdGPtpaCJLPZB%2Bj2yzNw5DSjUmkwTi%2B%2BmyemehCA@mail.gmail.com> References: <CAOtMX2hAUiWdGPtpaCJLPZB%2Bj2yzNw5DSjUmkwTi%2B%2BmyemehCA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000009d07c060f64e4be Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Jan 20, 2024, 9:51=E2=80=AFAM Alan Somers <asomers@freebsd.org> wro= te: > In a recent thread on src-committers, we discussed the costs and > benefits of including Rust code in the FreeBSD base system. To > summarize, the cost is that it would double our build times. imp > suggested adding an additional step after buildworld for stuff that > requires an external toolchain. That would ease the build time pain. > The benefit is that some tools would become easier to write, or even > become possible. Here is a list of actual and potential Rust projects > that could benefit from being in-tree. If anybody else has items to > add, I suggest moving this into the project wiki: > While a good list, it's just a list. Rust is interesting but without experience in the FreeBSD context we won't know if it materially improves things. The why of Rust is currently somewhere between hype and reality. Those that love it, love it. Those that don't, don't see how this fad is different from all the other fads. That's part of why it's hard to sell: the rust toolchain is difficult and different; it can do things automatically with crates that are insecure and unrepeatable; etc. With that, though comes a lot of benefits in safety of the language in more ways than just security. What is needed to convince people it's not just a fad is to bring some things into the tree that aren't critical to the tree, per se, but that can let people opt in to see just how bad the downside is in a buildworld context and just what the upsides are in practice. If it really is a good fit, it will sell itself... So this isn't about what could be rust: a lot can be. But what's the point of rewriting things that haven't had security bugs that rust prevents? The benefits of using rust have to be documented and demonstrated rather than just hyped, asserted and sold. That's why I suggested we start with better tests in rust. That's the petfect way to make sure we get the external tool chain working. We get the integration right. We fight with whatever pain there is and fix it as needed. We learn if it actually fits in and if we can keep up the infrastructure or if that's not something we can do. Once that works, we can rewrite other things as the need or desire arises. So no, we aren't going to rush headlong into fullscale rust replacement of all the things. However I'll note Linux hasn't done that either. They are taking a one step at a time approach. Step one is build integration with an external toolchain, imho. Until we have that, 0 Rust will be in the system. We already can do ports... but if you want it more central the first steps will be in making it possible to build rust binaries at all in a repeatable way. Warner Stuff that could only be written in Rust if it were in base > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > * ctl-exporter (I started this, but discovered that the CTL stats API is > unstable, so it can't live in ports. Instead, I had to do it in C). > > https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a86401= 469181a67ec34 > > * fusefs tests. Absolutely impossible to do in C. I considered Rust, bu= t > went > with C++ so they could live in base. They are too closely coupled to > fusefs(5) to live out-of-tree. > https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs > > * devd. Currently C++, but imp suggested a rewrite. > https://github.com/freebsd/freebsd-src/tree/main/sbin/devd > > * zfsd. Currently C++, but I've long pondered a rewrite. Using Rust wou= ld > make it more testable. > https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd > > * nscd. Currently C, but confusing and with no test coverage. I've > contemplated a rewrite myself, but I don't want to do it in C. > https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd > > * The userland portion of the 802.11ac and Lightning stacks. scottl > suggested > that these were good candidates for Rust. > > * freebsd-kpi-r14-0 . https://crates.io/crates/freebsd-kpi-r14-0 > > Stuff that can live in ports, but would be nicer in base > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > * gstat-rs https://crates.io/crates/gstat > > * geom-exporter (I've started this, but haven't published it) > > * nfs-exporter https://crates.io/crates/freebsd-nfs-exporter > > * virtiofsd-rs . Nobody has yet tried to port it to FreeBSD. But if the > connection to bhyve(8) is too intimate, it might be hard to do in ports= . > https://gitlab.com/virtio-fs/virtiofsd > > * jail-exporter https://crates.io/crates/jail_exporter > > * Various jail managers have been attempted in Rust. I think these are > fine in > ports, but others like Goran Mekic have opined that they should be move= d > to > base instead. > > * musikid's pjdfstest rewrite. I think it would be great to start using > this > to test the base system's file systems. If the tests themselves lived = in > base, they would be easier to sync with file system development. > https://github.com/musikid/pjdfstest > > * pf-rs. I suspect that the API isn't very stable. > https://crates.io/crates/pf-rs > > * benchpmc. The pmc counter names changes between releases. > https://crates.io/crates/benchpmc > > FreeBSD-related applications that are just fine in ports > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > * fsx-rs. Unlike pjdfstest, this only tests datapath APIs. Those are > usually > more stable than control path APIs, so I think there's little to be > gained by > moving this into base. https://crates.io/crates/fsx > > * ztop. It uses ZFS's kstats sysctl interface, which is pretty stable. > https://crates.io/crates/ztop > > * iocage-provision https://crates.io/crates/iocage-provision > > * rsblk https://crates.io/crates/rsblk > > * xfuse https://github.com/KhaledEmaraDev/xfuse > > Other FreeBSD-related libraries in Rust > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Just see the list at https://crates.io/keywords/freebsd > --00000000000009d07c060f64e4be Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Sat, Jan 20, 2024, 9:51=E2=80=AFAM Alan Somers <= <a href=3D"mailto:asomers@freebsd.org">asomers@freebsd.org</a>> wrote:<b= r></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border= -left:1px #ccc solid;padding-left:1ex">In a recent thread on src-committers= , we discussed the costs and<br> benefits of including Rust code in the FreeBSD base system.=C2=A0 To<br> summarize, the cost is that it would double our build times.=C2=A0 imp<br> suggested adding an additional step after buildworld for stuff that<br> requires an external toolchain.=C2=A0 That would ease the build time pain.<= br> The benefit is that some tools would become easier to write, or even<br> become possible.=C2=A0 Here is a list of actual and potential Rust projects= <br> that could benefit from being in-tree.=C2=A0 If anybody else has items to<b= r> add, I suggest moving this into the project wiki:<br></blockquote></div></d= iv><div dir=3D"auto"><br></div><div dir=3D"auto">While a good list, it'= s just a list. Rust is interesting but without experience in the FreeBSD co= ntext we won't know if it materially improves things. The why of Rust i= s currently somewhere between hype and reality. Those that love it, love it= . Those that don't, don't see how this fad is different from all th= e other fads. That's part of why it's hard to sell: the rust toolch= ain is difficult and different; it can do things automatically with crates = that are insecure and unrepeatable; etc. With that, though comes a lot of b= enefits in safety of the language in more ways than just security. What is = needed to convince people it's not just a fad is to bring some things i= nto the tree that aren't critical to the tree, per se, but that can let= people opt in to see just how bad the downside is in a buildworld context = and just what the upsides are in practice. If it really is a good fit, it w= ill sell itself...</div><div dir=3D"auto"><br></div><div dir=3D"auto">So th= is isn't about what could be rust: a lot can be. But what's the poi= nt of rewriting things that haven't had security bugs that rust prevent= s? The benefits of using rust have to be documented and demonstrated rather= than just hyped, asserted and sold.</div><div dir=3D"auto"><br></div><div = dir=3D"auto">That's why I suggested we start with better tests in rust.= That's the petfect way to make sure we get the external tool chain wor= king. We get the integration right. We fight with whatever pain there is an= d fix it as needed. We learn if it actually fits in and if we can keep up t= he infrastructure or if that's not something we can do.=C2=A0 Once that= works, we can rewrite other things as the need or desire arises.</div><div= dir=3D"auto"><br></div><div dir=3D"auto">So no, we aren't going to rus= h headlong into fullscale rust replacement of all the things. However I'= ;ll note Linux hasn't done that either. They are taking a one step at a= time approach. Step one is build integration with an external toolchain, i= mho. Until we have that, 0 Rust will be in the system. We already can do po= rts... but if you want it more central the first steps will be in making it= possible to build rust binaries at all in a repeatable way.</div><div dir= =3D"auto"><br></div><div dir=3D"auto">Warner</div><div dir=3D"auto"><br></d= iv><div dir=3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quote"= ><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1= px #ccc solid;padding-left:1ex"> Stuff that could only be written in Rust if it were in base<br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D<br> <br> * ctl-exporter (I started this, but discovered that the CTL stats API is<br= > =C2=A0 unstable, so it can't live in ports.=C2=A0 Instead, I had to do = it in C).<br> =C2=A0 <a href=3D"https://github.com/freebsd/freebsd-src/commit/1a7f22d9c21= 1f504f6c48a86401469181a67ec34" rel=3D"noreferrer noreferrer" target=3D"_bla= nk">https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a864= 01469181a67ec34</a><br> <br> * fusefs tests.=C2=A0 Absolutely impossible to do in C.=C2=A0 I considered = Rust, but went<br> =C2=A0 with C++ so they could live in base.=C2=A0 They are too closely coup= led to<br> =C2=A0 fusefs(5) to live out-of-tree.<br> =C2=A0 <a href=3D"https://github.com/freebsd/freebsd-src/tree/main/tests/sy= s/fs/fusefs" rel=3D"noreferrer noreferrer" target=3D"_blank">https://github= .com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs</a><br> <br> * devd.=C2=A0 Currently C++, but imp suggested a rewrite.<br> =C2=A0 <a href=3D"https://github.com/freebsd/freebsd-src/tree/main/sbin/dev= d" rel=3D"noreferrer noreferrer" target=3D"_blank">https://github.com/freeb= sd/freebsd-src/tree/main/sbin/devd</a><br> <br> * zfsd.=C2=A0 Currently C++, but I've long pondered a rewrite.=C2=A0 Us= ing Rust would<br> =C2=A0 make it more testable.<br> =C2=A0 <a href=3D"https://github.com/freebsd/freebsd-src/tree/main/cddl/usr= .sbin/zfsd" rel=3D"noreferrer noreferrer" target=3D"_blank">https://github.= com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd</a><br> <br> * nscd.=C2=A0 Currently C, but confusing and with no test coverage.=C2=A0 I= 've<br> =C2=A0 contemplated a rewrite myself, but I don't want to do it in C.<b= r> =C2=A0 <a href=3D"https://github.com/freebsd/freebsd-src/tree/main/usr.sbin= /nscd" rel=3D"noreferrer noreferrer" target=3D"_blank">https://github.com/f= reebsd/freebsd-src/tree/main/usr.sbin/nscd</a><br> <br> * The userland portion of the 802.11ac and Lightning stacks.=C2=A0 scottl s= uggested<br> =C2=A0 that these were good candidates for Rust.<br> <br> * freebsd-kpi-r14-0 .=C2=A0 <a href=3D"https://crates.io/crates/freebsd-kpi= -r14-0" rel=3D"noreferrer noreferrer" target=3D"_blank">https://crates.io/c= rates/freebsd-kpi-r14-0</a><br> <br> Stuff that can live in ports, but would be nicer in base<br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D<br> <br> * gstat-rs <a href=3D"https://crates.io/crates/gstat" rel=3D"noreferrer nor= eferrer" target=3D"_blank">https://crates.io/crates/gstat</a><br>; <br> * geom-exporter (I've started this, but haven't published it)<br> <br> * nfs-exporter <a href=3D"https://crates.io/crates/freebsd-nfs-exporter" re= l=3D"noreferrer noreferrer" target=3D"_blank">https://crates.io/crates/free= bsd-nfs-exporter</a><br> <br> * virtiofsd-rs .=C2=A0 Nobody has yet tried to port it to FreeBSD.=C2=A0 Bu= t if the<br> =C2=A0 connection to bhyve(8) is too intimate, it might be hard to do in po= rts.<br> =C2=A0 <a href=3D"https://gitlab.com/virtio-fs/virtiofsd" rel=3D"noreferrer= noreferrer" target=3D"_blank">https://gitlab.com/virtio-fs/virtiofsd</a><b= r> <br> * jail-exporter <a href=3D"https://crates.io/crates/jail_exporter" rel=3D"n= oreferrer noreferrer" target=3D"_blank">https://crates.io/crates/jail_expor= ter</a><br> <br> * Various jail managers have been attempted in Rust.=C2=A0 I think these ar= e fine in<br> =C2=A0 ports, but others like Goran Mekic have opined that they should be m= oved to<br> =C2=A0 base instead.<br> <br> * musikid's pjdfstest rewrite.=C2=A0 I think it would be great to start= using this<br> =C2=A0 to test the base system's file systems.=C2=A0 If the tests thems= elves lived in<br> =C2=A0 base, they would be easier to sync with file system development.<br> =C2=A0 <a href=3D"https://github.com/musikid/pjdfstest" rel=3D"noreferrer n= oreferrer" target=3D"_blank">https://github.com/musikid/pjdfstest</a><br>; <br> * pf-rs.=C2=A0 I suspect that the API isn't very stable.<br> =C2=A0 <a href=3D"https://crates.io/crates/pf-rs" rel=3D"noreferrer norefer= rer" target=3D"_blank">https://crates.io/crates/pf-rs</a><br>; <br> * benchpmc.=C2=A0 The pmc counter names changes between releases.<br> =C2=A0 <a href=3D"https://crates.io/crates/benchpmc" rel=3D"noreferrer nore= ferrer" target=3D"_blank">https://crates.io/crates/benchpmc</a><br>; <br> FreeBSD-related applications that are just fine in ports<br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D<br> <br> * fsx-rs.=C2=A0 Unlike pjdfstest, this only tests datapath APIs.=C2=A0 Thos= e are usually<br> =C2=A0 more stable than control path APIs, so I think there's little to= be gained by<br> =C2=A0 moving this into base. <a href=3D"https://crates.io/crates/fsx" rel= =3D"noreferrer noreferrer" target=3D"_blank">https://crates.io/crates/fsx</= a><br> <br> * ztop.=C2=A0 It uses ZFS's kstats sysctl interface, which is pretty st= able.<br> =C2=A0 <a href=3D"https://crates.io/crates/ztop" rel=3D"noreferrer noreferr= er" target=3D"_blank">https://crates.io/crates/ztop</a><br>; <br> * iocage-provision=C2=A0 <a href=3D"https://crates.io/crates/iocage-provisi= on" rel=3D"noreferrer noreferrer" target=3D"_blank">https://crates.io/crate= s/iocage-provision</a><br> <br> * rsblk <a href=3D"https://crates.io/crates/rsblk" rel=3D"noreferrer norefe= rrer" target=3D"_blank">https://crates.io/crates/rsblk</a><br>; <br> * xfuse=C2=A0 <a href=3D"https://github.com/KhaledEmaraDev/xfuse" rel=3D"no= referrer noreferrer" target=3D"_blank">https://github.com/KhaledEmaraDev/xf= use</a><br> <br> Other FreeBSD-related libraries in Rust<br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> Just see the list at <a href=3D"https://crates.io/keywords/freebsd" rel=3D"= noreferrer noreferrer" target=3D"_blank">https://crates.io/keywords/freebsd= </a><br> </blockquote></div></div></div> --00000000000009d07c060f64e4be--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfrdwJPjVObeHweQnZa5KeqYXdkj%2BwFZ=36nBNKmX%2Bk64g>