Date: Fri, 6 Sep 2024 08:48:59 -0600 From: Warner Losh <imp@bsdimp.com> To: Kyle Evans <kevans@freebsd.org> Cc: Baptiste Daroussin <bapt@freebsd.org>, Mark Johnston <markj@freebsd.org>, freebsd-hackers@freebsd.org, imp@freebsd.org Subject: Re: adding new flua libraries Message-ID: <CANCZdfrDkUm8j1LrdcH7rpBkTS-ottZ%2BpPZqQMU6b3y1KeJj2A@mail.gmail.com> In-Reply-To: <2f3039bc-883c-4b11-ab20-d1873c6cc555@FreeBSD.org> References: <Zto2Cf0Rp3kIsuFH@nuc> <qwingl2edkh655hjlfxhssez4ntqrit6cw7xkzj57jnmfbgzv7@jbaucgbobapr> <2f3039bc-883c-4b11-ab20-d1873c6cc555@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Fri, Sep 6, 2024 at 8:45 AM Kyle Evans <kevans@freebsd.org> wrote: > On 9/6/24 04:29, Baptiste Daroussin wrote: > > On Thu 05 Sep 18:51, Mark Johnston wrote: > >> FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It's > >> intended for use by the base system, and so far has a few consumers, but > >> not many so far. (As an aside, flua's intended scope is not totally > >> clear to me. Is it only for use by the base system? What compatibility > >> guarantees does it provide, if any?) > >> > >> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are > >> available, but they don't provide enough to make flua useful as a > >> general purpose programming tool. It lacks interfaces for interacting > >> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as > >> standard programming facilities (e.g., classes, higher-order functions, > >> etc.). Here I'm mostly interested in discussing the former. > >> > >> I think flua could be a very useful alternative to shell scripts and C > >> code where performance is not critical. It's very good at wrapping C > >> interfaces and thus could be used to make FreeBSD features (jails, > >> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who > >> don't want to interact with C. > >> > >> It's a lot of work to build up a set of flua modules that provide enough > >> functionality to be generally useful. My feeling is that the easiest > >> way to get there is to wrap C interfaces directly (to the extent that > >> that's possible, but it's usually easy) and expose them in flua as a > >> stable interface. Libraries written in pure Lua (or other languages > >> that interoperate well with Lua) could be built on top of that. > >> > >> I'm inclined to start by wrapping libc and libsys interfaces in a manner > >> similar to luaposix. There, the namespace is partitioned by C headers, > >> so posix.unistd contains identifiers from unistd.h, posix.sys.stat > >> contains identifiers from sys/stat.h, and so on. In fact, flua already > >> has a small subset of luaposix's functionality. Wrapping C interfaces > >> isn't much fun, but it's easy, and it saves us having to come up with > >> names and interfaces (naming things is hard), and helps ensure that the > >> glue code is relatively small and doesn't impose a large testing burden. > >> Moreover, C interfaces don't change much and are subject to > >> well-understood compatibility constraints, which should mean that Lua > >> interfaces built on top of them are subject to the same constraints. > >> > >> Assuming there's general agreement on that approach, the question I'd > >> really like to answer is, what should the namespace look like? It would > >> be useful to provide a posix module compatible with luaposix, but that > >> obviously won't contain FreeBSD-specific functionality. > >> > >> I propose having a top-level "freebsd" namespace for all modules > >> implemented in the base system, excluding posix and contrib libraries > >> which already define a Lua interface (libucl is the one example I see so > >> far; we could import sqlite bindings as well). Then, if we follow > >> luaposix's convention, we could have freebsd.sys.capsicum.* for > >> Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent > >> wrappers, and so on. The posix module could simply provide a subset of > >> freebsd.*. > >> > >> Maybe it's better to separate C wrappers from native Lua modules though, > >> so it could be better to have freebsd.c.sys.capsicum.*, etc., and > >> provide higher-level interfaces for FreeBSD features under freebsd.pf, > >> freebsd.zfs, freebsd.jail, etc.. I'm not really sure. > >> > >> In the interest of prompting discussion a bit, I posted some patches to > >> add some example wrappers to flua: > >> https://reviews.freebsd.org/D46553 > >> https://reviews.freebsd.org/D46554 > >> https://reviews.freebsd.org/D46556 > >> > >> Does anyone have opinions on anything I've brought up above? I'm pretty > >> happy to write a lot of this glue code or find ways to automate it, as > >> I've already done a fair bit of it, but it's hard to make progress > >> without having some rigourous conventions for the flua namespace (again, > >> naming things is hard). > > > > I like all of what I read and I am fully aligned. > > > > What I don't see here, is I think we should have dynamic modules libucl, > fbsd & > > friends are not dynamic and the reviews I see for the freebsd module > reviews, > > this is still bundled. > > > > Right, this is an artifact of how flua's evolved that we need to move > away from. We didn't have module support at the beginning- that came > later, but we didn't really move things out into modules. We still > can't move some things because of the bootstrap flua problem I mentioned > elsewhere (doing so would block work to do `make sysent` type work as > part of the build as it would break cross-builds), but at least > libucl/fbsd would be fine. > Yea, if the sysent.lua code doesn't use modules, would we still have the same bootstrapping issues? Today, I don't think here's anything but vanilla lua code in there. Warner > > my proposal for unbundling, I need kldload for a project of mine, so I > did: > > https://reviews.freebsd.org/D46558 > > > > Best regards, > > Bapt > > [-- Attachment #2 --] <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 6, 2024 at 8:45 AM Kyle Evans <<a href="mailto:kevans@freebsd.org">kevans@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 9/6/24 04:29, Baptiste Daroussin wrote:<br> > On Thu 05 Sep 18:51, Mark Johnston wrote:<br> >> FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It's<br> >> intended for use by the base system, and so far has a few consumers, but<br> >> not many so far. (As an aside, flua's intended scope is not totally<br> >> clear to me. Is it only for use by the base system? What compatibility<br> >> guarantees does it provide, if any?)<br> >><br> >> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are<br> >> available, but they don't provide enough to make flua useful as a<br> >> general purpose programming tool. It lacks interfaces for interacting<br> >> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as<br> >> standard programming facilities (e.g., classes, higher-order functions,<br> >> etc.). Here I'm mostly interested in discussing the former.<br> >><br> >> I think flua could be a very useful alternative to shell scripts and C<br> >> code where performance is not critical. It's very good at wrapping C<br> >> interfaces and thus could be used to make FreeBSD features (jails,<br> >> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who<br> >> don't want to interact with C.<br> >><br> >> It's a lot of work to build up a set of flua modules that provide enough<br> >> functionality to be generally useful. My feeling is that the easiest<br> >> way to get there is to wrap C interfaces directly (to the extent that<br> >> that's possible, but it's usually easy) and expose them in flua as a<br> >> stable interface. Libraries written in pure Lua (or other languages<br> >> that interoperate well with Lua) could be built on top of that.<br> >><br> >> I'm inclined to start by wrapping libc and libsys interfaces in a manner<br> >> similar to luaposix. There, the namespace is partitioned by C headers,<br> >> so posix.unistd contains identifiers from unistd.h, posix.sys.stat<br> >> contains identifiers from sys/stat.h, and so on. In fact, flua already<br> >> has a small subset of luaposix's functionality. Wrapping C interfaces<br> >> isn't much fun, but it's easy, and it saves us having to come up with<br> >> names and interfaces (naming things is hard), and helps ensure that the<br> >> glue code is relatively small and doesn't impose a large testing burden.<br> >> Moreover, C interfaces don't change much and are subject to<br> >> well-understood compatibility constraints, which should mean that Lua<br> >> interfaces built on top of them are subject to the same constraints.<br> >><br> >> Assuming there's general agreement on that approach, the question I'd<br> >> really like to answer is, what should the namespace look like? It would<br> >> be useful to provide a posix module compatible with luaposix, but that<br> >> obviously won't contain FreeBSD-specific functionality.<br> >><br> >> I propose having a top-level "freebsd" namespace for all modules<br> >> implemented in the base system, excluding posix and contrib libraries<br> >> which already define a Lua interface (libucl is the one example I see so<br> >> far; we could import sqlite bindings as well). Then, if we follow<br> >> luaposix's convention, we could have freebsd.sys.capsicum.* for<br> >> Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent<br> >> wrappers, and so on. The posix module could simply provide a subset of<br> >> freebsd.*.<br> >><br> >> Maybe it's better to separate C wrappers from native Lua modules though,<br> >> so it could be better to have freebsd.c.sys.capsicum.*, etc., and<br> >> provide higher-level interfaces for FreeBSD features under <a href="http://freebsd.pf" rel="noreferrer" target="_blank">freebsd.pf</a>,<br> >> freebsd.zfs, freebsd.jail, etc.. I'm not really sure.<br> >><br> >> In the interest of prompting discussion a bit, I posted some patches to<br> >> add some example wrappers to flua:<br> >> <a href="https://reviews.freebsd.org/D46553" rel="noreferrer" target="_blank">https://reviews.freebsd.org/D46553</a><br>; >> <a href="https://reviews.freebsd.org/D46554" rel="noreferrer" target="_blank">https://reviews.freebsd.org/D46554</a><br>; >> <a href="https://reviews.freebsd.org/D46556" rel="noreferrer" target="_blank">https://reviews.freebsd.org/D46556</a><br>; >><br> >> Does anyone have opinions on anything I've brought up above? I'm pretty<br> >> happy to write a lot of this glue code or find ways to automate it, as<br> >> I've already done a fair bit of it, but it's hard to make progress<br> >> without having some rigourous conventions for the flua namespace (again,<br> >> naming things is hard).<br> > <br> > I like all of what I read and I am fully aligned.<br> > <br> > What I don't see here, is I think we should have dynamic modules libucl, fbsd &<br> > friends are not dynamic and the reviews I see for the freebsd module reviews,<br> > this is still bundled.<br> > <br> <br> Right, this is an artifact of how flua's evolved that we need to move <br> away from. We didn't have module support at the beginning- that came <br> later, but we didn't really move things out into modules. We still <br> can't move some things because of the bootstrap flua problem I mentioned <br> elsewhere (doing so would block work to do `make sysent` type work as <br> part of the build as it would break cross-builds), but at least <br> libucl/fbsd would be fine.<br></blockquote><div><br></div><div>Yea, if the sysent.lua code doesn't use modules, would we still have the same</div><div>bootstrapping issues? Today, I don't think here's anything but vanilla lua code</div><div>in there.</div><div><br></div><div>Warner</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> > my proposal for unbundling, I need kldload for a project of mine, so I did:<br> > <a href="https://reviews.freebsd.org/D46558" rel="noreferrer" target="_blank">https://reviews.freebsd.org/D46558</a><br>; > <br> > Best regards,<br> > Bapt<br> <br> </blockquote></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfrDkUm8j1LrdcH7rpBkTS-ottZ%2BpPZqQMU6b3y1KeJj2A>