Date: Fri, 6 Sep 2024 08:47:21 -0600 From: Warner Losh <imp@bsdimp.com> To: Mark Johnston <markj@freebsd.org> Cc: freebsd-hackers@freebsd.org, kevans@freebsd.org, imp@freebsd.org, bapt@freebsd.org Subject: Re: adding new flua libraries Message-ID: <CANCZdfqOt2se-28GXTh%2BDhwi7nzCfcm906q4icoKt0CSwjyyMA@mail.gmail.com> In-Reply-To: <Zto2Cf0Rp3kIsuFH@nuc> References: <Zto2Cf0Rp3kIsuFH@nuc>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000c2d9ea0621747c4b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Sep 5, 2024 at 4:52=E2=80=AFPM Mark Johnston <markj@freebsd.org> wr= ote: > FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It's > intended for use by the base system, and so far has a few consumers, but > not many so far. (As an aside, flua's intended scope is not totally > clear to me. Is it only for use by the base system? What compatibility > guarantees does it provide, if any?) > To date, it's only provided for FreeBSD provided things. That means there's close coordination between what we provide and it. In general, it should remain compatible, but I think we've taken some liberties when it wasn't widely used to, for example, go from 5.3 to 5.4 w/o providing a fully compatible 5.3 env. > A few flua modules wrapping FreeBSD libraries (e.g., libjail) are > available, but they don't provide enough to make flua useful as a > general purpose programming tool. It lacks interfaces for interacting > with the system (e.g., libc/libsys/libutil/etc wrappers) as well as > standard programming facilities (e.g., classes, higher-order functions, > etc.). Here I'm mostly interested in discussing the former. > Yea, LUA is a language you can build OO concepts on, but doesn't directly provide more than the building blocks. Most librairies I've seen either pick a set of wrappers, or expand the wrappers in-line. We should pick one: I'm partial to the provide a common set of wrappers model when the time comes. > I think flua could be a very useful alternative to shell scripts and C > code where performance is not critical. It's very good at wrapping C > interfaces and thus could be used to make FreeBSD features (jails, > bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who > don't want to interact with C. > Yes. Last summer's GSoC code to test all the boot loader operations went great in lua until the very end when the desire to test all 50 combinations in parallel rather than serially (to reduce the run time from sneaking up on an hour to about a minute) couldn't do it in Lua, so things were rewritten in Python... > It's a lot of work to build up a set of flua modules that provide enough > functionality to be generally useful. My feeling is that the easiest > way to get there is to wrap C interfaces directly (to the extent that > that's possible, but it's usually easy) and expose them in flua as a > stable interface. Libraries written in pure Lua (or other languages > that interoperate well with Lua) could be built on top of that. > Yes. I tend to agree, though I think we may want to find a way to provide some things written in Lua and some in C for these wrapping. > I'm inclined to start by wrapping libc and libsys interfaces in a manner > similar to luaposix. There, the namespace is partitioned by C headers, > so posix.unistd contains identifiers from unistd.h, posix.sys.stat > contains identifiers from sys/stat.h, and so on. In fact, flua already > has a small subset of luaposix's functionality. Wrapping C interfaces > isn't much fun, but it's easy, and it saves us having to come up with > names and interfaces (naming things is hard), and helps ensure that the > glue code is relatively small and doesn't impose a large testing burden. > Moreover, C interfaces don't change much and are subject to > well-understood compatibility constraints, which should mean that Lua > interfaces built on top of them are subject to the same constraints. > Generally I agree, though with the caveat that I think posix module tends to have too-deep a namespace, and I'm hesitant about using that for FreeBSD, given how much clutter it would add to the lua code. > Assuming there's general agreement on that approach, the question I'd > really like to answer is, what should the namespace look like? It would > be useful to provide a posix module compatible with luaposix, but that > obviously won't contain FreeBSD-specific functionality. > > I propose having a top-level "freebsd" namespace for all modules > implemented in the base system, excluding posix and contrib libraries > which already define a Lua interface (libucl is the one example I see so > far; we could import sqlite bindings as well). Then, if we follow > luaposix's convention, we could have freebsd.sys.capsicum.* for > Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent > wrappers, and so on. The posix module could simply provide a subset of > freebsd.*. > I think that may be too hierarchical and provide little benefit. What would we have in freebsd.* that would conflict with freebsd.sys.*? I barely see the benefit for the posix module, but I think it was a mistake to do it there... but we're stuck with the design. And depending on how compatible we want to be with the posix module, this may tie our hands a bit in providing more info, or info in a better format if we try to make it strictly speaking, a subset with the implicatio= n that the behavior is identical. > Maybe it's better to separate C wrappers from native Lua modules though, > so it could be better to have freebsd.c.sys.capsicum.*, etc., and > provide higher-level interfaces for FreeBSD features under freebsd.pf, > freebsd.zfs, freebsd.jail, etc.. I'm not really sure. > I think that's going even further into the weeds. There's no reason that freebsd.zfs couldn't provide the zfs system calls. freebsd.jail the jail ones, etc, in addition to the higher levels. We're not likely going to nest multiple modules that do jail things on top of a base jail system call-only module. I'm having trouble seeing the benefit for all the extra clutter this would bring to the lua code we write. > In the interest of prompting discussion a bit, I posted some patches to > add some example wrappers to flua: > https://reviews.freebsd.org/D46553 > https://reviews.freebsd.org/D46554 > https://reviews.freebsd.org/D46556 > > Does anyone have opinions on anything I've brought up above? I'm pretty > happy to write a lot of this glue code or find ways to automate it, as > I've already done a fair bit of it, but it's hard to make progress > without having some rigourous conventions for the flua namespace (again, > naming things is hard). > We should agree on the namespace stuff, then put the conversation in our rearview mirror. :). But generally, I love the idea and any comments about how to design "the crinkly bits on the fjords" shouldn't be taken to my not liking the fjords. Warner with apologies to douglas adams --000000000000c2d9ea0621747c4b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Thu, Sep 5, 2024 at 4:52=E2=80=AFP= M Mark Johnston <<a href=3D"mailto:markj@freebsd.org">markj@freebsd.org<= /a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0= px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">F= reeBSD ships a Lua 5.4 implementation, flua, in the base system.=C2=A0 It&#= 39;s<br> intended for use by the base system, and so far has a few consumers, but<br= > not many so far.=C2=A0 (As an aside, flua's intended scope is not total= ly<br> clear to me.=C2=A0 Is it only for use by the base system?=C2=A0 What compat= ibility<br> guarantees does it provide, if any?)<br></blockquote><div><br></div><div>To= date, it's only provided for FreeBSD provided things. That means there= 's</div><div>close coordination between what we provide and it. In gene= ral, it should remain</div><div>compatible, but I think we've taken som= e liberties when it wasn't widely used</div><div>to, for example, go fr= om 5.3 to 5.4 w/o providing a fully compatible 5.3 env.</div><div>=C2=A0</d= iv><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord= er-left:1px solid rgb(204,204,204);padding-left:1ex"> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are<br> available, but they don't provide enough to make flua useful as a<br> general purpose programming tool.=C2=A0 It lacks interfaces for interacting= <br> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as<br> standard programming facilities (e.g., classes, higher-order functions,<br> etc.).=C2=A0 Here I'm mostly interested in discussing the former.<br></= blockquote><div><br></div><div>Yea, LUA is a language you can build OO conc= epts on, but doesn't</div><div>directly provide more than the building = blocks. Most librairies I've seen</div><div>either pick a set of wrappe= rs, or expand the wrappers in-line. We should</div><div>pick one: I'm p= artial to the provide a common set of wrappers model</div><div>when the tim= e comes.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"m= argin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left= :1ex"> I think flua could be a very useful alternative to shell scripts and C<br> code where performance is not critical.=C2=A0 It's very good at wrappin= g C<br> interfaces and thus could be used to make FreeBSD features (jails,<br> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who<br> don't want to interact with C.<br></blockquote><div><br></div><div>Yes.= Last summer's GSoC code to test all the boot loader operations</div><d= iv>went great in lua until the very end when the desire to test all 50</div= ><div>combinations in parallel rather than serially (to reduce the run time= </div><div>from sneaking up on an hour to about a minute) couldn't do i= t in Lua,</div><div>so things were rewritten in Python...</div><div>=C2=A0<= /div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bo= rder-left:1px solid rgb(204,204,204);padding-left:1ex"> It's a lot of work to build up a set of flua modules that provide enoug= h<br> functionality to be generally useful.=C2=A0 My feeling is that the easiest<= br> way to get there is to wrap C interfaces directly (to the extent that<br> that's possible, but it's usually easy) and expose them in flua as = a<br> stable interface.=C2=A0 Libraries written in pure Lua (or other languages<b= r> that interoperate well with Lua) could be built on top of that.<br></blockq= uote><div><br></div><div>Yes. I tend to agree, though I think we may want t= o find a way to</div><div>provide some things written in Lua and some in C = for these</div><div>wrapping.</div><div>=C2=A0</div><blockquote class=3D"gm= ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,= 204,204);padding-left:1ex"> I'm inclined to start by wrapping libc and libsys interfaces in a manne= r<br> similar to luaposix.=C2=A0 There, the namespace is partitioned by C headers= ,<br> so posix.unistd contains identifiers from unistd.h, posix.sys.stat<br> contains identifiers from sys/stat.h, and so on.=C2=A0 In fact, flua alread= y<br> has a small subset of luaposix's functionality.=C2=A0 Wrapping C interf= aces<br> isn't much fun, but it's easy, and it saves us having to come up wi= th<br> names and interfaces (naming things is hard), and helps ensure that the<br> glue code is relatively small and doesn't impose a large testing burden= .<br> Moreover, C interfaces don't change much and are subject to<br> well-understood compatibility constraints, which should mean that Lua<br> interfaces built on top of them are subject to the same constraints.<br></b= lockquote><div><br></div><div>Generally I agree, though with the caveat tha= t I think posix module</div><div>tends to have too-deep a namespace, and I&= #39;m hesitant about using</div><div>that for FreeBSD, given how much clutt= er it would add to the lua code.</div><div>=C2=A0</div><blockquote class=3D= "gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(2= 04,204,204);padding-left:1ex"> Assuming there's general agreement on that approach, the question I'= ;d<br> really like to answer is, what should the namespace look like?=C2=A0 It wou= ld<br> be useful to provide a posix module compatible with luaposix, but that<br> obviously won't contain FreeBSD-specific functionality.<br> <br> I propose having a top-level "freebsd" namespace for all modules<= br> implemented in the base system, excluding posix and contrib libraries<br> which already define a Lua interface (libucl is the one example I see so<br= > far; we could import sqlite bindings as well).=C2=A0 Then, if we follow<br> luaposix's convention, we could have freebsd.sys.capsicum.* for<br> Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent<br> wrappers, and so on.=C2=A0 The posix module could simply provide a subset o= f<br> freebsd.*.<br></blockquote><div><br></div><div>I think that may be too hier= archical and provide little benefit. What would</div><div>we have in freebs= d.* that would conflict with freebsd.sys.*? I barely see the</div><div>bene= fit for the posix module, but I think it was a mistake to do it there... bu= t</div><div>we're stuck with the design.</div><div><br></div><div>And d= epending on how compatible we want to be with the posix module,</div><div>t= his may tie our hands a bit in providing more info, or info in a better</di= v><div>format if we try to make it strictly speaking, a subset with the imp= lication</div><div>that the behavior is identical.</div><div>=C2=A0<br></di= v><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;borde= r-left:1px solid rgb(204,204,204);padding-left:1ex"> Maybe it's better to separate C wrappers from native Lua modules though= ,<br> so it could be better to have freebsd.c.sys.capsicum.*, etc., and<br> provide higher-level interfaces for FreeBSD features under <a href=3D"http:= //freebsd.pf" rel=3D"noreferrer" target=3D"_blank">freebsd.pf</a>,<br> freebsd.zfs, freebsd.jail, etc..=C2=A0 I'm not really sure.<br></blockq= uote><div><br></div><div>I think that's going even further into the wee= ds. There's no reason</div><div>that freebsd.zfs couldn't provide t= he zfs system calls. freebsd.jail the jail</div><div>ones, etc, in addition= to the higher levels. We're not likely going to nest</div><div>multipl= e modules that do jail things on top of a base jail system call-only</div><= div>module. I'm having trouble seeing the benefit for all the extra clu= tter</div><div>this would bring to the lua code we write.</div><div>=C2=A0<= /div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bo= rder-left:1px solid rgb(204,204,204);padding-left:1ex"> In the interest of prompting discussion a bit, I posted some patches to<br> add some example wrappers to flua:<br> <a href=3D"https://reviews.freebsd.org/D46553" rel=3D"noreferrer" target=3D= "_blank">https://reviews.freebsd.org/D46553</a><br>; <a href=3D"https://reviews.freebsd.org/D46554" rel=3D"noreferrer" target=3D= "_blank">https://reviews.freebsd.org/D46554</a><br>; <a href=3D"https://reviews.freebsd.org/D46556" rel=3D"noreferrer" target=3D= "_blank">https://reviews.freebsd.org/D46556</a><br>; <br> Does anyone have opinions on anything I've brought up above?=C2=A0 I= 9;m pretty<br> happy to write a lot of this glue code or find ways to automate it, as<br> I've already done a fair bit of it, but it's hard to make progress<= br> without having some rigourous conventions for the flua namespace (again,<br= > naming things is hard).<br></blockquote><div><br></div><div>We should agree= on the namespace stuff, then put the conversation in our</div><div>rearvie= w mirror. :).</div><div><br></div><div>But generally, I love the idea and a= ny comments about how to design "the crinkly=C2=A0bits on the fjords&q= uot;</div><div>shouldn't be taken to my not liking the fjords.</div><di= v><br></div><div>Warner</div><div><br></div><div>with apologies to douglas = adams</div></div></div> --000000000000c2d9ea0621747c4b--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfqOt2se-28GXTh%2BDhwi7nzCfcm906q4icoKt0CSwjyyMA>