From owner-freebsd-testing@freebsd.org Thu Oct 29 00:43:31 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B133C45410A; Thu, 29 Oct 2020 00:43:31 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CM6BC4KVjz3YMR; Thu, 29 Oct 2020 00:43:31 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 735661D466; Thu, 29 Oct 2020 00:43:31 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qt1-f176.google.com with SMTP id i7so882130qti.6; Wed, 28 Oct 2020 17:43:31 -0700 (PDT) X-Gm-Message-State: AOAM531i0akd20oITqNrlVwd62qh9JMMywRevMqsySo6cYPPAcHoIUB5 Fo1xV1VRICUG/FXzwIkovHF0XzI259pT4cEgpyQ= X-Google-Smtp-Source: ABdhPJzplWLEfkhjJKo3yPp6c9zZN9WiSgH+WTMjnrd6ozKttXfJwEmJfqq1o1tUnL58I26h4AxfrfIV+dVak8UbtD4= X-Received: by 2002:ac8:3975:: with SMTP id t50mr1503038qtb.53.1603932210998; Wed, 28 Oct 2020 17:43:30 -0700 (PDT) MIME-Version: 1.0 References: <0217233F-9E0D-47EC-AAAA-1C19551C2FF9@gmail.com> In-Reply-To: <0217233F-9E0D-47EC-AAAA-1C19551C2FF9@gmail.com> From: Kyle Evans Date: Wed, 28 Oct 2020 19:43:20 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Testing with lua/atf-lua reviews To: Enji Cooper Cc: freebsd-testing@freebsd.org, FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2020 00:43:31 -0000 On Wed, Oct 28, 2020 at 7:23 PM Enji Cooper wrote: > > > On Oct 24, 2020, at 9:09 AM, Kyle Evans wrote: > > Hello! > > I've just put up for review some work I've done to allow us to write > tests in lua, primarily intended to test the lua libs we're writing. > Please feel free to add yourself or drop in for some commentary: > > - https://reviews.freebsd.org/D26928 - atf-lua(1)/atf-lua(3) itself > - https://reviews.freebsd.org/D26929 - Build glue for atf-lua > - https://reviews.freebsd.org/D26930 - atf.tests.mk infrastructure for > adding tests > - https://reviews.freebsd.org/D26931 - Build glue for atf-lua tests > - https://reviews.freebsd.org/D26932 - jail(3lua) tests, as a sample > > Note that D26932 has an additional hard dependency on the libjail > bindings and some additions I've made to them, notably: D26080, > D26756, and D26927. > > > Hi Kyle, > > I realize that I haven=E2=80=99t been fully in the loop lately due to tim= e and focusing on other things, but I=E2=80=99m not fully onboard with this= approach. > > In particular, one of the things that jmmv was more onboard with was limi= ting atf, not extending it, and I agree with his desire to not do that. > Sure- > Furthermore, why isn=E2=80=99t this using the luaunit framework instead a= nd the support being added to kyua to support luaunit: http://lua-users.or= g/wiki/UnitTesting ? There are a ton of caveats with ATF that I would rathe= r not support longer than necessary and having to teach folks how to use a = homegrown test infrastructure instead of leveraging an open source test inf= rastructure which is supported by an external group. Doing the latter makes= maintenance easy for us and improves the utility of the support better. > I had actually considered this, but ruled it out due to a couple factors. The main one was that there's really no benefit with adopting yet another test framework for base -- I think we're much less likely to get people that are already familiar with luaunit contributing to our base tests than we are to get people in one of two other camps: 1. They already work with our vast array of other ATF !lua tests and would find themselves staring at a really familiar interface, or 2. They don't, they want to work on Lua stuff and Lua tests in base, then an ATF lua interface will be at least somewhat applicable to other areas of our test infrastructure I haven't dumped all that much time into this, though, so I have no problem taking another aerial glance at it.