Date: Sat, 2 May 2026 09:31:46 -0600 From: Alan Somers <asomers@freebsd.org> To: Gleb Smirnoff <glebius@freebsd.org> Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 3e845b109056 - main - ping: fix listing test cases when scapy is not installed Message-ID: <CAOtMX2jt=6NWL1YwTT3vGOn%2BCTLP6Oioz6xsz7gV954NfyAbiw@mail.gmail.com> In-Reply-To: <afYXOsKk5jouvCZ2@cell.glebi.us> References: <69f61649.1c022.73b2a773@gitrepo.freebsd.org> <afYXOsKk5jouvCZ2@cell.glebi.us>
index | next in thread | previous in thread | raw e-mail
On Sat, May 2, 2026 at 9:24 AM Gleb Smirnoff <glebius@freebsd.org> wrote: > > On Sat, May 02, 2026 at 03:20:41PM +0000, Alan Somers wrote: > A> commit 3e845b1090565912375c5578cf0399d27b7fa70c > A> Author: Alan Somers <asomers@FreeBSD.org> > A> AuthorDate: 2026-05-01 23:06:35 +0000 > A> Commit: Alan Somers <asomers@FreeBSD.org> > A> CommitDate: 2026-05-02 15:20:16 +0000 > A> > A> ping: fix listing test cases when scapy is not installed > A> > A> The ATF-python test program was attempting to list test cases that > A> require scapy. But it attempted to import the scapy module before the > A> test cases had been listed, resulting in an ImportError that kyua > A> interpreted as a test program crash. > A> > A> Fix this behavior by handling that ImportError well enough to list test > A> cases, but not run them. If scapy isn't present, Kyua will refuse to > A> run the test cases. But it needs to be able to list them in order to > A> know to skip them. > A> > A> Sponsored by: ConnectWise > A> MFC after: 2 weeks > A> Reviewed by: maxim > A> Differential Revision: https://reviews.freebsd.org/D56765 > > Sorry for not reviewing in time. I don't think Kyua expects a test program to > be able to list the test cases, but then fail to run them. > > Isn't there a standard way of dealing with the problem? Many tests have > @pytest.mark.require_progs(["scapy"]) in them. > > -- > Gleb Smirnoff Yes, these tests already include @pytest.mark.require_progs(["scapy"]) correctly. That's how Kyua is supposed to learn that it can't run those test cases. You can tell by running the program manually, bypassing kyua: <after my patch> $ ./test_ping.py - Content-Type: application/X-atf-tp; version="1" ident: TestPing::test_ping[_4_c1_s56_t1_localhost] descr: Test ping has.cleanup: true require.config: unprivileged_user require.user: root ... ident: TestPing::test_pinger[_0_0] descr: Test ping using pinger(), a reply faker has.cleanup: true require.user: root require.progs: scapy ... Notice that the first test case does not require scapy, but the second one does. This is how Kyua learns that it can't execute the second test if scapy is not installed. However, prior to my commit, if scapy was not installed, running test_ping.py -l would fail with exit status 2 and produce no output, and Kyua would interpret that as a broken test program, like this: test_ping.py:__test_cases_list__home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2jt=6NWL1YwTT3vGOn%2BCTLP6Oioz6xsz7gV954NfyAbiw>