From nobody Wed Jan 17 01:27:23 2024 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TF7XY3bQvz56Wd8 for <freebsd-arch@mlmmj.nyi.freebsd.org>; Wed, 17 Jan 2024 01:27:25 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TF7XY3981z4cBf; Wed, 17 Jan 2024 01:27:25 +0000 (UTC) (envelope-from kevans@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705454845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WU3/CQ05UTL8asxuoccjEUqVdvN6+1Fd/lVxyPq7IqY=; b=QCE2+X5evee0jA0T12yWEg5TmyjgZgwJ0ikhvbVSqn0j3f89SszXG0sNzvUG63UrWBrfbf avN3+cTnedJsuusjh3xJtyJ7K4uUTPZNBa6+H3ktPw25r6Xv2c2QAXeMOMWBOQsjPBIFBn 4GqVa3yNn9tseRUpeMgiV/zs8LQHG3JBESv2SnQS7NISBaoRieCxmG8fPhfqh49oKm55d2 B5tn0eFwC5EnQYEu3tq0QswriJZvby53gH1Z4dv3NvfAERbmdiDMe3iyF8yjf54vNYQepo Sq/qmoALak7jW4wiLLZ27/8ADoYz8KDpwf07TzvG6ib5gH9fI7kpuenkC73jkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705454845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WU3/CQ05UTL8asxuoccjEUqVdvN6+1Fd/lVxyPq7IqY=; b=YDUtX1hl82MK4AZu9IJ+D2LBp29kEMJPR1xzFvxsWOBrdm7Cxm4ylI/ezzCWqVH1l2voQN ARsLg7P8UbKvBhZyOWFzXXQG914hCFhwLWPethl9GqhakyLrD/2IACFvvAsErWMQp/rBLG zepMh18oMXAi5rwgoRvd08VKfbbqBanLRPq6TAYsm/E5ltpZrjD7M+YYSG3M9mlrQ5JMe6 GYDXxvevyr0/EaHRx/nF/Kts8uvpm8WizyKdSM/hSub3XhmO9aK+H8I8793AjTNZpk1Ruz rhdUYkSHTZbXk+tWa6btEyvjMXiNVH5ums7/p6cJPz9ZrRoDDahJABgGzheF4A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705454845; a=rsa-sha256; cv=none; b=nrqXrPVnuSJYiN9Uj2XK7qHvLX5QvupOl0xBV2xmtyRwZn32y1CGUXC/+jEssQRGOk9O9P GlUuGXhtG0Vm6Rg3Ry346gZsncGDN95SYujtQiiVfeR+QqnYRNpn6Rd5Yb2O99CstknbXx Uq6rloBkq1602VijeQxro9V7NtvgSdHEvTu7/n71BNKqiUUPFs/GlMzwaYkcda48HMVSMK dXoQJ7dZIcV28Vn1IRAt/wrgz1VKzAPWBpxHSQA0LXsUkHhX0vhg/7MnW+HYLFLAAPtmtC eqsL2FdHpeg0FezXlc/0sBYBOA8O4dX0sN7PsQTEhQLND1UMEQhAfKgQujWJ/w== Received: from [10.9.4.95] (unknown [209.182.120.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: kevans/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TF7XY0STLz1CyH; Wed, 17 Jan 2024 01:27:24 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Message-ID: <be058507-b1c9-418e-bf5f-88b986f39f82@FreeBSD.org> Date: Tue, 16 Jan 2024 19:27:23 -0600 List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org> List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: <mailto:freebsd-arch+help@freebsd.org> List-Post: <mailto:freebsd-arch@freebsd.org> List-Subscribe: <mailto:freebsd-arch+subscribe@freebsd.org> List-Unsubscribe: <mailto:freebsd-arch+unsubscribe@freebsd.org> Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: orch(1) for tty testing Content-Language: en-US To: Mark Johnston <markj@freebsd.org> Cc: freebsd-arch@freebsd.org References: <478cf238-ab2e-4185-a5eb-4230b5efe735@FreeBSD.org> <ZabXgs4kWzVBFlBT@nuc> From: Kyle Evans <kevans@FreeBSD.org> In-Reply-To: <ZabXgs4kWzVBFlBT@nuc> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 1/16/24 13:22, Mark Johnston wrote: > On Mon, Jan 15, 2024 at 01:54:54PM -0600, Kyle Evans wrote: >> Hi, >> >> I wrote a tool for tty testing w/ lua scripts, inspired by expect(1) and >> with some similarities, but explicitly without the goal of trying to be >> compatible: >> >> https://git.kevans.dev/kevans/orch >> >> An example of its use is here: >> >> https://git.kevans.dev/kevans/tty-tests/src/branch/main/test_canon.orch >> >> Each spawn() creates a new pts(4) and allows us to drive I/O to the process >> with write/match statements. I have one or two tests written with it so far >> and the end result is, IMO, much less painful looking and less error prone >> than the equivalent in C. >> >> It still needs a bit of polish (e.g., improved diagnostics when a match >> fails), but I'd like to consider importing this somewhere for use in the >> near future for tty testing given how much it simplifies. I can push it to >> ports, but we have a private liblua in base so it could be built and >> banished off into, e.g., /usr/tests/sys/kern/tty/, to make my life just a >> tad easier re: getting these to run in CI. It's pretty lightweight; the >> orch binary on my amd64 system clocks in at ~27K, and orch.lua at 13K. >> >> Any objections to importing this new tool into /usr/tests or maybe even >> /usr/libexec? > > This looks quite handy. I have some lua scripts which I use to checkout > a src tree, build a VM image, boot the image and run tests, then > download the resulting kyua DB, all with a single CLI command. I > currently run the hypervisor under expect(1) and send commands over a > pipe, but that was an expedient hack and I'd rather use something > implemented natively. There are some existing lua expect ports, but > orch looks simpler and I'll try it out to see if it's missing anything > for my use-case. More generally, we lack a test framework which lets > one boot and interact with VMs, and orch might be a useful component of > that. That seems like a good exercise for it- feel free to get in touch. I wouldn't mind porting whatever you've written in expect(1) over and take the opportunity to fill in the feature gaps, regardless of whether you use the end result or not.