From owner-freebsd-hackers@freebsd.org Mon Apr 19 21:22:59 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A633E5FF742 for ; Mon, 19 Apr 2021 21:22:59 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FPKXy1t2Wz4sds; Mon, 19 Apr 2021 21:22:58 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mail-pj1-x1031.google.com with SMTP id kb13-20020a17090ae7cdb02901503d67f0beso222439pjb.0; Mon, 19 Apr 2021 14:22:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=nXvZXZfclQ+t0BD46L30g+bO4tW5bvofLcA7K1EJAK8=; b=m8NRyU1gLrnUdr/BjdhEaCBgHTJowvsLkPxUwmBscGqiomj0xdcXi/sTuAVFjRCfLt SFCQeu+KBfdjWFD2bUPi0lt7P+T0Jnz7kYZVPukbvipNre/aTybgGlti1jBEjcomYOMU 9N3oXHVxj6Rs0AU/MyDVV9BSZlFcsGstbDe8ofByWDzNGbCSJrSqwBIKauN8cge3rI+2 uDLtEYYtdNPKkoEjq6o9Mjw5abxFwnLP2UOod45SwT/dFfIStJHYjf8whgn+O0BjuuOO OoG/NFkpUiVjJlIeLmpT2aeW8rEweR7UaPVWJkO0F546y6AhK7qRh09jLVel/d6B/Vpp 5VNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=nXvZXZfclQ+t0BD46L30g+bO4tW5bvofLcA7K1EJAK8=; b=lC4F6r8AMiY4JdQKl6m3xvFknVlX+hzVFdVYE/J2W5UKc2Pf2wazY+7TDpvO4Jz74k JBNrV/GpryHhn5gDwkICIiwIqPRBlFXNm242W9KNeU++5jMWRviMgEdCvcN25E0F+Bkr mbPiCvRPfuIF+vScaRILX+MNDbQIquq3hfeZlXP1fUC0e/eHl/l+sW1mVQOsnWlGrPYg Xdo38fxRYgg5jUjUPuOhwXrxeC8o395RgjXQ3ErbcmRQDAnqeEpK8AwaKoxhUfh3CJJQ ykmJ5qm+WbCmcmoCczn3PzV9IB6K5kqLwCi5cHc8krd3g7wOeA2Ym0NHtBxOub2olpkv aneA== X-Gm-Message-State: AOAM531oniEzCjvV4IpxbHNt94kDT60HsukgmSTUzNLISRixV/4h0VKD 25CZP4LD7Lq8xDToK9JHukeXqUSS07UaY4UQ X-Google-Smtp-Source: ABdhPJzbgWVgubs7J5FxwPDRrk1Ewjb0tx1EmpmVgDF2agWDWZsfZR2Azv4IFuaHtFXRVsSikz5Wmw== X-Received: by 2002:a17:90b:714:: with SMTP id s20mr1179787pjz.62.1618867376786; Mon, 19 Apr 2021 14:22:56 -0700 (PDT) Received: from [192.168.20.29] (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id t21sm3793017pfg.211.2021.04.19.14.22.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Apr 2021 14:22:56 -0700 (PDT) From: Enji Cooper Message-Id: <23AB05ED-BF78-47CD-82B1-2CB7EB65F6B2@gmail.com> Content-Type: multipart/signed; boundary="Apple-Mail=_3E326F09-A3D5-4F1B-AA50-AC100C313AA5"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: How do I write test cases for interactive commands? Date: Mon, 19 Apr 2021 14:22:54 -0700 In-Reply-To: Cc: Gleb Popov , Yoshihiro Ota , freebsd-hackers To: Alexander Richardson References: <20210414190252.69d4bb503fafc73ca1721eb2@j.email.ne.jp> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4FPKXy1t2Wz4sds X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=m8NRyU1g; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of yaneurabeya@gmail.com designates 2607:f8b0:4864:20::1031 as permitted sender) smtp.mailfrom=yaneurabeya@gmail.com X-Spamd-Result: default: False [-4.60 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RBL_SENDERSCORE_FAIL(0.00)[2607:f8b0:4864:20::1031:server fail]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_ATTACHMENT(0.00)[]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::1031:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RECEIVED_SPAMHAUS_PBL(0.00)[73.19.52.228:received]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::1031:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1031:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Apr 2021 21:22:59 -0000 --Apple-Mail=_3E326F09-A3D5-4F1B-AA50-AC100C313AA5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Apr 19, 2021, at 6:04 AM, Alexander Richardson = wrote: >=20 > On Sat, 17 Apr 2021 at 17:14, Enji Cooper = wrote: >>=20 >>=20 >>> On Apr 14, 2021, at 10:40 PM, Gleb Popov wrote: >>=20 >> =E2=80=A6 >>=20 >>> Take a look at LLVM testing infrastructure and specifically the = llvm-lit >>> tool. >>=20 >> Does their test infrastructure leverage GoogleTest? If so, then the = bulk majority of the work would just be integrating it into Kyua. >> Cheers, >> -Enji >>=20 >=20 > If you are interested in writing tests for llvm/clang/lldb those > should be submitted upstream and not be part of the FreeBSD source > tree. Therefore, Kyua is almost certainly not the right solution. > https://www.llvm.org/docs/TestingGuide.html explains how to write > tests for LLVM, however, it is quite stale. I would suggest looking at > existing tests inside LLVM and adjusting those to match what you want > to test. >=20 > The FreeBSD src contrib/ import does not include the extensive LLVM > testsuite, so you could start by looking at > https://github.com/llvm/llvm-project/tree/main/clang/test for clang > and https://github.com/llvm/llvm-project/tree/main/lldb/test for LLDB. > I have never looked at the LLDB tests, but as far as I know there are > tests that use python to drive the interaction. Hi Alex, My suggestion was more that kyua should be used to run/aggregate = the test information, not be the actual test = framework/infrastructure/payload. I really appreciate the references :). Cheers! -Enji --Apple-Mail=_3E326F09-A3D5-4F1B-AA50-AC100C313AA5 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEtvtxN6kOllEF3nmX5JFNMZeDGN4FAmB99K4ACgkQ5JFNMZeD GN6eHhAAh9+0yoHta9uR6TU7HNqc81khBKHkdTgQzqBO56bqDzB3Lp9uWkreT05G 1eqATq/34x4+SV3pb5gjM+79UjwjtyRJ6UGdjpw5OKST4Qo/tvppPG+iFNbHIFgC rkyFsCNg5zFvk+S1gJ+OtE/8ppC5ytMNOa+DN05BgSdb/NFJHxkj5PMWO5xIbllV 5ut0IeHaksM8EVCPZnzmTgKDs1nyrl7g98unYOkv3k4ED0GmRo0+P9xAX0Twac8+ WE9RDeU3gOZt79EnsZ+rmL6RZTvPHZrgFY+6YQd02/XXU+A369UrE/E9TdAO7GKn UEPNL6LGFUYlF89wAcJjbrSTMnc6OriHtCIKqgw68H6101uvZGA3/86g4fsOmTwk 5LTLV7EKEq5ZCcXjJydKUcSKtHhgcHR6l54lU4H1LRhjciFfoxdH0eqP5ykFF+ij 63hT1cf7/DJlt56pabCiIvZNyH3fR5t/K5vtZ9NATUAxclsmBlB3aguv/yMD96wf cKQ2GA2HMZtv1AEVKE5ax2v2UTr0o+KQ916TzuWdM6J535msKXJMewlvfgzl2riW 3y8dyOZP46mJ5MtuGSFKA4Qf9uXsCxUn5Qct+0yYUUMAOBCNNLf4nJIr/jQaUyZv O2Vie2mN+zlmfX/a4EC94DD3M7fEynjjcGobCiey9JHpRdJk15o= =pnlE -----END PGP SIGNATURE----- --Apple-Mail=_3E326F09-A3D5-4F1B-AA50-AC100C313AA5--