From nobody Tue Jan 16 19:22:42 2024 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TDzRr04H4z57s5d for ; Tue, 16 Jan 2024 19:22:48 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TDzRq4XhKz3xfC; Tue, 16 Jan 2024 19:22:47 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qk1-x731.google.com with SMTP id af79cd13be357-7831ed13d39so819954485a.0; Tue, 16 Jan 2024 11:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705432965; x=1706037765; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=g9iq+XbeD3soB6t0CPh6QnYDcJ/JkN9GdyVm6PkZMm4=; b=bap69O/2Aftl97Rd2WoHmsddjvJ0pxNvdA8wfO6oOWsbOHj3R9V3pqvCBqlDArApaM E+yVODnUW8iyR9+7pps+zp/osK2Wx93Hv2RylW3RUl9qOWgt68l9wJT3V4SawTdN4sSd LOgTZ0lcocY/MwV7qmZLHRCaTqaJ+ikfa3xJxO+Lpq16+aipmRnnLA3vk/JQc4PaTpRR HMND/cdiLECDi1iaBp0uYhy8ZwFKejziOYz8O13IPYjtbUQeCBKzFejNpiDS9qNwnkl1 5PO+XgK4Hui2q6OB7cgSJSP1FYO5nwgYGYNuGj6MgNPgp8JnTFCfJafq0kF9okalJKYx VubQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705432965; x=1706037765; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g9iq+XbeD3soB6t0CPh6QnYDcJ/JkN9GdyVm6PkZMm4=; b=O3+RufAQAdztxV4peiCG6pI/Q2Ib8SIzqPOfen5bmM0uegpbOxub07aiSVbh5Wu5z3 0CKECckkde1t8Artm5hT1FOqmsnb6RMS/DUZ/Pxlux9EyLdr/iM82/iMIp1PgKMzWC7/ FZpNjYgTR+3QtztLnVjPy3qmWwVBrkLLWcI3Orqz6uH/9HaElDBr8jUIKvMilQvj8IwG aUOsuMCvRRxMJ/LQgdCKKZnwxj0Pz1AOK5u4i8YN7RL12dHpXH/vZPmTq0nMXyww4Z5W pZOt6b4birgery9SVJ4WAnTJ98Df/r+IKlSFHGHxg9XuNFlVGKYIM6CQJpmLsnLOjCMk AmTA== X-Gm-Message-State: AOJu0YzYlz+OHXf88bFhfuC+9aJzCiNKO/WlqACqwNG7MkoGT9SV3bpr +A77tsg+d0nss0cOnbnnM1rUihLgskQs9Q== X-Google-Smtp-Source: AGHT+IH1KSg08NBHZYmcH5Rcn3b7MAs85njba7sjX5dc5SlZYv0xefnVMp5UQQ9dfwIVcNZarimEYw== X-Received: by 2002:ae9:c10a:0:b0:781:5efd:403b with SMTP id z10-20020ae9c10a000000b007815efd403bmr8769985qki.13.1705432965012; Tue, 16 Jan 2024 11:22:45 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id z25-20020a05620a261900b00781ae860c31sm3943151qko.70.2024.01.16.11.22.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 11:22:44 -0800 (PST) Date: Tue, 16 Jan 2024 14:22:42 -0500 From: Mark Johnston To: Kyle Evans Cc: freebsd-arch@freebsd.org Subject: Re: orch(1) for tty testing Message-ID: References: <478cf238-ab2e-4185-a5eb-4230b5efe735@FreeBSD.org> List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <478cf238-ab2e-4185-a5eb-4230b5efe735@FreeBSD.org> X-Rspamd-Queue-Id: 4TDzRq4XhKz3xfC X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Mon, Jan 15, 2024 at 01:54:54PM -0600, Kyle Evans wrote: > Hi, > > I wrote a tool for tty testing w/ lua scripts, inspired by expect(1) and > with some similarities, but explicitly without the goal of trying to be > compatible: > > https://git.kevans.dev/kevans/orch > > An example of its use is here: > > https://git.kevans.dev/kevans/tty-tests/src/branch/main/test_canon.orch > > Each spawn() creates a new pts(4) and allows us to drive I/O to the process > with write/match statements. I have one or two tests written with it so far > and the end result is, IMO, much less painful looking and less error prone > than the equivalent in C. > > It still needs a bit of polish (e.g., improved diagnostics when a match > fails), but I'd like to consider importing this somewhere for use in the > near future for tty testing given how much it simplifies. I can push it to > ports, but we have a private liblua in base so it could be built and > banished off into, e.g., /usr/tests/sys/kern/tty/, to make my life just a > tad easier re: getting these to run in CI. It's pretty lightweight; the > orch binary on my amd64 system clocks in at ~27K, and orch.lua at 13K. > > Any objections to importing this new tool into /usr/tests or maybe even > /usr/libexec? This looks quite handy. I have some lua scripts which I use to checkout a src tree, build a VM image, boot the image and run tests, then download the resulting kyua DB, all with a single CLI command. I currently run the hypervisor under expect(1) and send commands over a pipe, but that was an expedient hack and I'd rather use something implemented natively. There are some existing lua expect ports, but orch looks simpler and I'll try it out to see if it's missing anything for my use-case. More generally, we lack a test framework which lets one boot and interact with VMs, and orch might be a useful component of that.