From owner-freebsd-hackers@freebsd.org Thu Oct 29 01:03:12 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7A4C94545D4 for ; Thu, 29 Oct 2020 01:03:12 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CM6cw2hjPz3Z0S for ; Thu, 29 Oct 2020 01:03:12 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 398601DEC0 for ; Thu, 29 Oct 2020 01:03:12 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f172.google.com with SMTP id p3so773052qkk.7 for ; Wed, 28 Oct 2020 18:03:12 -0700 (PDT) X-Gm-Message-State: AOAM530EU4ICrO29IvmSiN+kUIjxJpWSvPguKYrzDUNQmT9Lty2LIUkd ukBvK5aNQUn/Nr7LE3oSaeQSKlMxWGszfATlWw8= X-Received: by 2002:a37:6309:: with SMTP id x9mt678847qkb.493.1603933391820; Wed, 28 Oct 2020 18:03:11 -0700 (PDT) MIME-Version: 1.0 References: <0217233F-9E0D-47EC-AAAA-1C19551C2FF9@gmail.com> In-Reply-To: From: Kyle Evans Date: Wed, 28 Oct 2020 20:03:00 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Testing with lua/atf-lua reviews Cc: Enji Cooper , freebsd-testing@freebsd.org, FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2020 01:03:12 -0000 On Wed, Oct 28, 2020 at 7:43 PM Kyle Evans wrote: > > On Wed, Oct 28, 2020 at 7:23 PM Enji Cooper wrote= : > > > > > > On Oct 24, 2020, at 9:09 AM, Kyle Evans wrote: > > > > Hello! > > > > I've just put up for review some work I've done to allow us to write > > tests in lua, primarily intended to test the lua libs we're writing. > > Please feel free to add yourself or drop in for some commentary: > > > > - https://reviews.freebsd.org/D26928 - atf-lua(1)/atf-lua(3) itself > > - https://reviews.freebsd.org/D26929 - Build glue for atf-lua > > - https://reviews.freebsd.org/D26930 - atf.tests.mk infrastructure for > > adding tests > > - https://reviews.freebsd.org/D26931 - Build glue for atf-lua tests > > - https://reviews.freebsd.org/D26932 - jail(3lua) tests, as a sample > > > > Note that D26932 has an additional hard dependency on the libjail > > bindings and some additions I've made to them, notably: D26080, > > D26756, and D26927. > > > > > > Hi Kyle, > > > > I realize that I haven=E2=80=99t been fully in the loop lately due to t= ime and focusing on other things, but I=E2=80=99m not fully onboard with th= is approach. > > > > In particular, one of the things that jmmv was more onboard with was li= miting atf, not extending it, and I agree with his desire to not do that. > > > > Sure- > > > Furthermore, why isn=E2=80=99t this using the luaunit framework instead= and the support being added to kyua to support luaunit: http://lua-users.= org/wiki/UnitTesting ? There are a ton of caveats with ATF that I would rat= her not support longer than necessary and having to teach folks how to use = a homegrown test infrastructure instead of leveraging an open source test i= nfrastructure which is supported by an external group. Doing the latter mak= es maintenance easy for us and improves the utility of the support better. > > > > I had actually considered this, but ruled it out due to a couple > factors. The main one was that there's really no benefit with adopting > yet another test framework for base -- I think we're much less likely > to get people that are already familiar with luaunit contributing to > our base tests than we are to get people in one of two other camps: > > 1. They already work with our vast array of other ATF !lua tests and > would find themselves staring at a really familiar interface, or > 2. They don't, they want to work on Lua stuff and Lua tests in base, > then an ATF lua interface will be at least somewhat applicable to > other areas of our test infrastructure > > I haven't dumped all that much time into this, though, so I have no > problem taking another aerial glance at it. Oh, oh dear. luaunit already supports TAP output: https://luaunit.readthedocs.io/en/latest/#output-formats I kinda prefer the minimal verbosity text format that it defaults to, but as far as effort to get off the ground goes... that's likely impossible to beat.