From nobody Sat May 17 09:33:00 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZzzJX3lfZz5wJND; Sat, 17 May 2025 09:33:00 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZzzJX18lyz41y4; Sat, 17 May 2025 09:33:00 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1747474380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=beQ8JLnLPUUSUFGFN5LHAAZT6S0p/O/QtWVyDQjCuHA=; b=RbwTYsyA5+I0Fd6PrRr1dRoLSu5XxCtwYvVwtymr/W8xQ14e3PhLyDE4gwRhcXJr+kIicT xUVTdZCQpKP0PNDcV6fuHYSEFBb3FbXUhUk+zgq06k7CaBg7XphhbkvDB+C4sGyKEqhQ9w j7troV/reqMcy0XsgHHtIcVrxZxvVXRvFWGWkcUkBHyySb1yxwtUowQGzS8sAHj94mnfFw 5aDt80fVLvmWyZW5TwKssNvkdCbjm7MeTwACtXITIwdpy6/cNrljJsC8D/a/Mmj4yLkX2c o8xWiocplkJ6Fu34XkzoaCHCclGv8TCVgaLCVvGJHw11UyNoXZ41j2+0Ws2WTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1747474380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=beQ8JLnLPUUSUFGFN5LHAAZT6S0p/O/QtWVyDQjCuHA=; b=oYr+m+b9zBxwE73yjrN/yDWm5gWiHcAjUAwkJ45S5HDTYisSWLrCe5RNi5spZR6p/VW/34 G4Q6xHzaBZUP78wtrXUYuHQ+i3/XYbIDW+nd4J59sAkArdm+f4VrmoIpiQya8LYm9Jt0r0 xcGXH9PvK6CzCoJ/MUgCCpdR6JfMmSqCX71vuOZScjHWZZmGkKnUFQT8sIi93xgS7L6YyD XKRaoxMW7uxXM7emNv/IRpv+yiOjX2XjnF15TOMSszo3cozIo5437GAYtN+kyUNHr+hMsn IbBv7xE0LZ6uQPCD+rhrOrVu5gPKCpl42drBH2Xn4z6O88KzB7Ie025ce0B9Hw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1747474380; a=rsa-sha256; cv=none; b=x2tJZFBZmIMSmHTZXCvzfiTtvc9/E/ZYeP8Yf91jsa+GxCAFfO5SUZFwr+5RQ+eT6JDGCL 16OH6Rzmj5nlhByOPpKY8DmMYm22Bgvq80ngNUHxfuQ99rOru70cFsJaTQGi5Pu7IT1o8O w5YdUMMzCbfKxLCgBmbNjjyd/rCgnsBxSQ/ro1aro3j4j5si2esy7tx/RA3+oP6dG+niGS w0GpivyENs2/Jx+FOECbPtgou++L4loc0ybf6iOS0wSCZtR7odwXHb2FSFmi8NxjuIG46c ZaxXUvi1w2Hdf7mDp2/K9vqR3aZHzTL/e2bP+YnD4o8lK6svzGc3DJ9AwbxXqQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZzzJX0lgzzjyP; Sat, 17 May 2025 09:33:00 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 54H9X0Of016205; Sat, 17 May 2025 09:33:00 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 54H9X0Oh016202; Sat, 17 May 2025 09:33:00 GMT (envelope-from git) Date: Sat, 17 May 2025 09:33:00 GMT Message-Id: <202505170933.54H9X0Oh016202@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Igor Ostapenko Subject: git: 6b8222793fbb - main - kyua: Add "kyua debug -p" option List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: igoro X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6b8222793fbb4c0e162232716bc454dad31b709f Auto-Submitted: auto-generated The branch main has been updated by igoro: URL: https://cgit.FreeBSD.org/src/commit/?id=6b8222793fbb4c0e162232716bc454dad31b709f commit 6b8222793fbb4c0e162232716bc454dad31b709f Author: Igor Ostapenko AuthorDate: 2025-05-17 09:30:55 +0000 Commit: Igor Ostapenko CommitDate: 2025-05-17 09:30:55 +0000 kyua: Add "kyua debug -p" option This allows the test engine to be paused right before the test cleanup routine, simplifying debugging. Reviewed by: ziaee, ngie Differential Revision: https://reviews.freebsd.org/D49463 --- contrib/kyua/cli/cmd_debug.cpp | 70 ++++++++++++++++++++++++++++++++++++ contrib/kyua/doc/kyua-debug.1.in | 21 ++++++++++- contrib/kyua/drivers/debug_test.cpp | 7 +++- contrib/kyua/drivers/debug_test.hpp | 6 +++- contrib/kyua/engine/debugger.hpp | 71 +++++++++++++++++++++++++++++++++++++ contrib/kyua/engine/scheduler.cpp | 8 +++++ contrib/kyua/model/test_case.cpp | 21 +++++++++++ contrib/kyua/model/test_case.hpp | 5 +++ 8 files changed, 206 insertions(+), 3 deletions(-) diff --git a/contrib/kyua/cli/cmd_debug.cpp b/contrib/kyua/cli/cmd_debug.cpp index b7a29b7ab804..c00920c1a3f7 100644 --- a/contrib/kyua/cli/cmd_debug.cpp +++ b/contrib/kyua/cli/cmd_debug.cpp @@ -29,6 +29,7 @@ #include "cli/cmd_debug.hpp" #include +#include #include "cli/common.ipp" #include "drivers/debug_test.hpp" @@ -38,13 +39,76 @@ #include "utils/cmdline/parser.ipp" #include "utils/cmdline/ui.hpp" #include "utils/format/macros.hpp" +#include "utils/process/executor.hpp" namespace cmdline = utils::cmdline; namespace config = utils::config; +namespace executor = utils::process::executor; using cli::cmd_debug; +namespace { + + +const cmdline::bool_option pause_before_cleanup_upon_fail_option( + 'p', + "pause-before-cleanup-upon-fail", + "Pauses right before the test cleanup upon fail"); + + +const cmdline::bool_option pause_before_cleanup_option( + "pause-before-cleanup", + "Pauses right before the test cleanup"); + + +/// The debugger interface implementation. +class dbg : public engine::debugger { + /// Object to interact with the I/O of the program. + cmdline::ui* _ui; + + /// Representation of the command line to the subcommand. + const cmdline::parsed_cmdline& _cmdline; + +public: + /// Constructor. + /// + /// \param ui_ Object to interact with the I/O of the program. + /// \param cmdline Representation of the command line to the subcommand. + dbg(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline) : + _ui(ui), _cmdline(cmdline) + {} + + void before_cleanup( + const model::test_program_ptr&, + const model::test_case&, + optional< model::test_result >& result, + executor::exit_handle& eh) const + { + if (_cmdline.has_option(pause_before_cleanup_upon_fail_option + .long_name())) { + if (result && !result.get().good()) { + _ui->out("The test failed and paused right before its cleanup " + "routine."); + _ui->out(F("Test work dir: %s") % eh.work_directory().str()); + _ui->out("Press any key to continue..."); + (void) std::cin.get(); + } + } else if (_cmdline.has_option(pause_before_cleanup_option + .long_name())) { + _ui->out("The test paused right before its cleanup routine."); + _ui->out(F("Test work dir: %s") % eh.work_directory().str()); + _ui->out("Press any key to continue..."); + (void) std::cin.get(); + } + }; + +}; + + +} // anonymous namespace + + /// Default constructor for cmd_debug. cmd_debug::cmd_debug(void) : cli_command( "debug", "test_case", 1, 1, @@ -53,6 +117,9 @@ cmd_debug::cmd_debug(void) : cli_command( add_option(build_root_option); add_option(kyuafile_option); + add_option(pause_before_cleanup_upon_fail_option); + add_option(pause_before_cleanup_option); + add_option(cmdline::path_option( "stdout", "Where to direct the standard output of the test case", "path", "/dev/stdout")); @@ -82,7 +149,10 @@ cmd_debug::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, const engine::test_filter filter = engine::test_filter::parse( test_case_name); + auto debugger = std::shared_ptr< engine::debugger >(new dbg(ui, cmdline)); + const drivers::debug_test::result result = drivers::debug_test::drive( + debugger, kyuafile_path(cmdline), build_root_path(cmdline), filter, user_config, cmdline.get_option< cmdline::path_option >("stdout"), cmdline.get_option< cmdline::path_option >("stderr")); diff --git a/contrib/kyua/doc/kyua-debug.1.in b/contrib/kyua/doc/kyua-debug.1.in index 9e962a465421..263aef3667a7 100644 --- a/contrib/kyua/doc/kyua-debug.1.in +++ b/contrib/kyua/doc/kyua-debug.1.in @@ -25,7 +25,7 @@ .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE .\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -.Dd October 13, 2014 +.Dd March 25, 2025 .Dt KYUA-DEBUG 1 .Os .Sh NAME @@ -35,6 +35,8 @@ .Nm .Op Fl -build-root Ar path .Op Fl -kyuafile Ar file +.Op Fl -pause-before-cleanup-upon-fail +.Op Fl -pause-before-cleanup .Op Fl -stdout Ar path .Op Fl -stderr Ar path .Ar test_case @@ -84,6 +86,23 @@ Specifies the Kyuafile to process. Defaults to .Pa Kyuafile file in the current directory. +.It Fl -pause-before-cleanup-upon-fail , Fl p +Causes Kyua to pause right before the test cleanup routine is invoked if the +test fails. +When paused, Kyua waits for input on stdin, and any key press resumes +execution. +.sp +This can be useful for debugging system or end-to-end tests +which alter the system by creating artifacts such as files, network +interfaces, routing entries, firewall configuration, containers, etc. +This flag makes it possible to preserve such artifacts immediately after a test +failure, simplifying debugging. +Otherwise, these artifacts would be removed by the test cleanup routine or +by Kyua built-in cleanup mechanism. +.It Fl -pause-before-cleanup +The unconditional variant of the previous option. +This can be helpful for reproducing the infrastructure or fixture of a passing +test for further development or additional analysis. .It Fl -stderr Ar path Specifies the file to which to send the standard error of the test program's body. diff --git a/contrib/kyua/drivers/debug_test.cpp b/contrib/kyua/drivers/debug_test.cpp index 0717a9ad9419..6bd3d4dc0731 100644 --- a/contrib/kyua/drivers/debug_test.cpp +++ b/contrib/kyua/drivers/debug_test.cpp @@ -63,7 +63,8 @@ using utils::optional; /// /// \returns A structure with all results computed by this driver. drivers::debug_test::result -drivers::debug_test::drive(const fs::path& kyuafile_path, +drivers::debug_test::drive(engine::debugger_ptr debugger, + const fs::path& kyuafile_path, const optional< fs::path > build_root, const engine::test_filter& filter, const config::tree& user_config, @@ -92,6 +93,10 @@ drivers::debug_test::drive(const fs::path& kyuafile_path, const model::test_program_ptr test_program = match.get().first; const std::string& test_case_name = match.get().second; + const model::test_case test_case = test_program->find(test_case_name); + if (debugger) + test_case.attach_debugger(debugger); + scheduler::result_handle_ptr result_handle = handle.debug_test( test_program, test_case_name, user_config, stdout_path, stderr_path); diff --git a/contrib/kyua/drivers/debug_test.hpp b/contrib/kyua/drivers/debug_test.hpp index cbaa2f6acea0..0b4abce6f2ea 100644 --- a/contrib/kyua/drivers/debug_test.hpp +++ b/contrib/kyua/drivers/debug_test.hpp @@ -36,12 +36,15 @@ #if !defined(DRIVERS_DEBUG_TEST_HPP) #define DRIVERS_DEBUG_TEST_HPP +#include "engine/debugger.hpp" #include "engine/filters.hpp" #include "model/test_result.hpp" #include "utils/config/tree_fwd.hpp" #include "utils/fs/path_fwd.hpp" #include "utils/optional_fwd.hpp" +using engine::debugger; + namespace drivers { namespace debug_test { @@ -68,7 +71,8 @@ public: }; -result drive(const utils::fs::path&, const utils::optional< utils::fs::path >, +result drive(std::shared_ptr< debugger >, + const utils::fs::path&, const utils::optional< utils::fs::path >, const engine::test_filter&, const utils::config::tree&, const utils::fs::path&, const utils::fs::path&); diff --git a/contrib/kyua/engine/debugger.hpp b/contrib/kyua/engine/debugger.hpp new file mode 100644 index 000000000000..3c4d087f8ad0 --- /dev/null +++ b/contrib/kyua/engine/debugger.hpp @@ -0,0 +1,71 @@ +// Copyright 2025 The Kyua Authors. +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// * Neither the name of Google Inc. nor the names of its contributors +// may be used to endorse or promote products derived from this software +// without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +/// \file engine/debugger.hpp +/// The interface between the engine and the users outside. + +#if !defined(ENGINE_DEBUGGER_HPP) +#define ENGINE_DEBUGGER_HPP + +#include "model/test_case_fwd.hpp" +#include "model/test_program_fwd.hpp" +#include "model/test_result_fwd.hpp" +#include "utils/optional_fwd.hpp" +#include "utils/process/executor_fwd.hpp" + +namespace executor = utils::process::executor; + +using utils::optional; + + +namespace engine { + + +/// Abstract debugger interface. +class debugger { +public: + debugger() {} + virtual ~debugger() {} + + /// Called right before test cleanup. + virtual void before_cleanup( + const model::test_program_ptr&, + const model::test_case&, + optional< model::test_result >&, + executor::exit_handle&) const = 0; +}; + + +/// Pointer to a debugger implementation. +typedef std::shared_ptr< debugger > debugger_ptr; + + +} // namespace engine + + +#endif // !defined(ENGINE_DEBUGGER_HPP) diff --git a/contrib/kyua/engine/scheduler.cpp b/contrib/kyua/engine/scheduler.cpp index 5179436073b8..8b2498514f6d 100644 --- a/contrib/kyua/engine/scheduler.cpp +++ b/contrib/kyua/engine/scheduler.cpp @@ -39,6 +39,7 @@ extern "C" { #include #include "engine/config.hpp" +#include "engine/debugger.hpp" #include "engine/exceptions.hpp" #include "engine/execenv/execenv.hpp" #include "engine/requirements.hpp" @@ -1396,8 +1397,15 @@ scheduler::scheduler_handle::wait_any(void) handle.stderr_file()); } + std::shared_ptr< debugger > debugger = test_case.get_debugger(); + if (debugger) { + debugger->before_cleanup(test_data->test_program, test_case, + result, handle); + } + if (test_data->needs_cleanup) { INV(test_case.get_metadata().has_cleanup()); + // The test body has completed and we have processed it. If there // is a cleanup routine, trigger it now and wait for any other test // completion. The caller never knows about cleanup routines. diff --git a/contrib/kyua/model/test_case.cpp b/contrib/kyua/model/test_case.cpp index f5f6a979eed3..faa4f3cebaf2 100644 --- a/contrib/kyua/model/test_case.cpp +++ b/contrib/kyua/model/test_case.cpp @@ -60,6 +60,9 @@ struct model::test_case::impl : utils::noncopyable { /// Fake result to return instead of running the test case. optional< model::test_result > fake_result; + /// Optional pointer to a debugger attached. + engine::debugger_ptr debugger; + /// Constructor. /// /// \param name_ The name of the test case within the test program. @@ -233,6 +236,24 @@ model::test_case::get_raw_metadata(void) const } +/// Attach a debugger to the test case. +void +model::test_case::attach_debugger(engine::debugger_ptr debugger) const +{ + _pimpl->debugger = debugger; +} + + +/// Gets the optional pointer to a debugger. +/// +/// \return An optional pointer to a debugger. +engine::debugger_ptr +model::test_case::get_debugger() const +{ + return _pimpl->debugger; +} + + /// Gets the fake result pre-stored for this test case. /// /// \return A fake result, or none if not defined. diff --git a/contrib/kyua/model/test_case.hpp b/contrib/kyua/model/test_case.hpp index 3c6fe32c8e62..c92ebacff98d 100644 --- a/contrib/kyua/model/test_case.hpp +++ b/contrib/kyua/model/test_case.hpp @@ -38,11 +38,13 @@ #include #include +#include "engine/debugger.hpp" #include "model/metadata_fwd.hpp" #include "model/test_result_fwd.hpp" #include "utils/noncopyable.hpp" #include "utils/optional_fwd.hpp" + namespace model { @@ -71,6 +73,9 @@ public: const metadata& get_raw_metadata(void) const; utils::optional< test_result > fake_result(void) const; + void attach_debugger(engine::debugger_ptr) const; + engine::debugger_ptr get_debugger() const; + bool operator==(const test_case&) const; bool operator!=(const test_case&) const; };