From owner-svn-src-all@freebsd.org Wed Oct 14 17:39:52 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1060943EE4F; Wed, 14 Oct 2020 17:39:52 +0000 (UTC) (envelope-from arichardson@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBKRq74fvz4HQy; Wed, 14 Oct 2020 17:39:51 +0000 (UTC) (envelope-from arichardson@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D5FFC120AB; Wed, 14 Oct 2020 17:39:51 +0000 (UTC) (envelope-from arichardson@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 09EHdpmT014497; Wed, 14 Oct 2020 17:39:51 GMT (envelope-from arichardson@FreeBSD.org) Received: (from arichardson@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 09EHdpbi014493; Wed, 14 Oct 2020 17:39:51 GMT (envelope-from arichardson@FreeBSD.org) Message-Id: <202010141739.09EHdpbi014493@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: arichardson set sender to arichardson@FreeBSD.org using -f From: Alex Richardson Date: Wed, 14 Oct 2020 17:39:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r366708 - in head/sbin/pfctl/tests: . files X-SVN-Group: head X-SVN-Commit-Author: arichardson X-SVN-Commit-Paths: in head/sbin/pfctl/tests: . files X-SVN-Commit-Revision: 366708 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 17:39:52 -0000 Author: arichardson Date: Wed Oct 14 17:39:50 2020 New Revision: 366708 URL: https://svnweb.freebsd.org/changeset/base/366708 Log: Rewrite pfctl_test in C to reduce testsuite run time The new C test takes 25 seconds on QEMU-RISC-V, wheras the shell version takes 332 seconds. Even with the latest optimizations to atf-sh this test still takes a few seconds to startup in QEMU. Re-writing it in C reduces the runtime for a single test from about 2-3 seconds to less than .5 seconds. Since there are ~80 tests, this adds up to about 3-4 minutes. This may not seem like a big speedup, but before the recent optimizations to avoid atf_get_srcdir, each test took almost 100 seconds on QEMU RISC-V instead of 3. This also significantly reduces the time it takes to list the available test cases, which speeds up running the tests via kyua: ``` root@qemu-riscv64-alex:~ # /usr/bin/time kyua test -k /usr/tests/sbin/pfctl/Kyuafile pfctl_test_old ... 158/158 passed (0 failed) 332.08 real 42.58 user 286.17 sys root@qemu-riscv64-alex:~ # /usr/bin/time kyua test -k /usr/tests/sbin/pfctl/Kyuafile pfctl_test 158/158 passed (0 failed) 24.96 real 9.75 user 14.26 sys root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test pf1001 pfctl_test: WARNING: Running test cases outside of kyua(1) is unsupported pfctl_test: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4) Running pfctl -o none -nvf /usr/tests/sbin/pfctl/./files/pf1001.in --- binat on em0 inet6 from fc00::/64 to any -> fc00:0:0:1::/64 binat on em0 inet6 from any to fc00:0:0:1::/64 -> fc00::/64 --- passed 0.17 real 0.06 user 0.08 sys root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_old pf1001 pfctl_test_old: WARNING: Running test cases outside of kyua(1) is unsupported pfctl_test_old: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4) Id Refs Name 141 1 pf Executing command [ pfctl -o none -nvf - ] passed 1.73 real 0.25 user 1.41 sys root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_old -l > /dev/null 24.36 real 2.26 user 21.86 sys root@qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test -l > /dev/null 0.04 real 0.02 user 0.01 sys ``` The speedups are even more noticeable on CHERI-RISC-V (since QEMU runs slower when emulating CHERI instructions): ``` root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_new -l > /dev/null 0.51 real 0.49 user 0.00 sys root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test -l > /dev/null 34.20 real 32.69 user 0.16 sys root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test pf1001 pfctl_test: WARNING: Running test cases outside of kyua(1) is unsupported pfctl_test: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4) Id Refs Name 147 1 pf Executing command [ pfctl -o none -nvf - ] passed 5.74 real 5.41 user 0.03 sys root@qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_new pf1001 pfctl_test_new: WARNING: Running test cases outside of kyua(1) is unsupported pfctl_test_new: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4) Running pfctl -o none -nvf /usr/tests/sbin/pfctl/./files/pf1001.in --- binat on em0 inet6 from fc00::/64 to any -> fc00:0:0:1::/64 binat on em0 inet6 from any to fc00:0:0:1::/64 -> fc00::/64 --- passed 0.68 real 0.66 user 0.00 sys root@qemu-cheri-alex:/usr/tests/sbin/pfctl # ``` Reviewed By: kp Differential Revision: https://reviews.freebsd.org/D26779 Added: head/sbin/pfctl/tests/pfctl_test.c (contents, props changed) head/sbin/pfctl/tests/pfctl_test_list.inc (contents, props changed) Deleted: head/sbin/pfctl/tests/files/pfctl_test_descr.sh head/sbin/pfctl/tests/pfctl_test.sh Modified: head/sbin/pfctl/tests/Makefile head/sbin/pfctl/tests/files/Makefile Modified: head/sbin/pfctl/tests/Makefile ============================================================================== --- head/sbin/pfctl/tests/Makefile Wed Oct 14 15:50:28 2020 (r366707) +++ head/sbin/pfctl/tests/Makefile Wed Oct 14 17:39:50 2020 (r366708) @@ -2,9 +2,11 @@ PACKAGE= tests -ATF_TESTS_SH= pfctl_test \ - macro +ATF_TESTS_C= pfctl_test +ATF_TESTS_SH= macro +LIBADD+= sbuf SUBDIR+= files +WARNS=6 .include Modified: head/sbin/pfctl/tests/files/Makefile ============================================================================== --- head/sbin/pfctl/tests/files/Makefile Wed Oct 14 15:50:28 2020 (r366707) +++ head/sbin/pfctl/tests/files/Makefile Wed Oct 14 17:39:50 2020 (r366708) @@ -7,6 +7,5 @@ BINDIR= ${TESTSDIR} # We use ${.CURDIR} as workaround so that the glob patterns work. FILES!= echo ${.CURDIR}/pf????.in ${.CURDIR}/pf????.include ${.CURDIR}/pf????.ok -FILES+= ${.CURDIR}/pfctl_test_descr.sh .include Added: head/sbin/pfctl/tests/pfctl_test.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sbin/pfctl/tests/pfctl_test.c Wed Oct 14 17:39:50 2020 (r366708) @@ -0,0 +1,230 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2020 Alex Richardson + * + * This software was developed by SRI International and the University of + * Cambridge Computer Laboratory (Department of Computer Science and + * Technology) under DARPA contract HR0011-18-C-0016 ("ECATS"), as part of the + * DARPA SSITH research programme. + * + * This work was supported by Innovate UK project 105694, "Digital Security by + * Design (DSbD) Technology Platform Prototype". + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +/* + * Tests 0001-0999 are copied from OpenBSD's regress/sbin/pfctl. + * Tests 1001-1999 are ours (FreeBSD's own). + * + * pf: Run pfctl -nv on pfNNNN.in and check that the output matches pfNNNN.ok. + * Copied from OpenBSD. Main differences are some things not working + * in FreeBSD: + * * The action 'match' + * * The command 'set reassemble' + * * The 'from'/'to' options together with 'route-to' + * * The option 'scrub' (it is an action in FreeBSD) + * * Accepting undefined routing tables in actions (??: see pf0093.in) + * * The 'route' option + * * The 'set queue def' option + * selfpf: Feed pfctl output through pfctl again and verify it stays the same. + * Copied from OpenBSD. + */ + +static bool +check_pf_module_available() +{ + int modid; + struct module_stat stat; + + if ((modid = modfind("pf")) < 0) { + warn("pf module not found"); + return false; + } + stat.version = sizeof(struct module_stat); + if (modstat(modid, &stat) < 0) { + warn("can't stat pf module id %d", modid); + return false; + } + return (true); +} + +extern char **environ; + +static struct sbuf * +read_fd(int fd, size_t sizehint) +{ + struct sbuf *sb; + ssize_t count; + char buffer[MAXBSIZE]; + + sb = sbuf_new(NULL, NULL, sizehint, SBUF_AUTOEXTEND); + errno = 0; + while ((count = read(fd, buffer, sizeof(buffer) - 1)) > 0) { + sbuf_bcat(sb, buffer, count); + } + ATF_REQUIRE_ERRNO(0, count == 0 && "Should have reached EOF"); + sbuf_finish(sb); /* Ensure NULL-termination */ + return (sb); +} + +static struct sbuf * +read_file(const char *filename) +{ + struct stat s; + struct sbuf *result; + int fd; + + errno = 0; + ATF_REQUIRE_EQ_MSG(stat(filename, &s), 0, "cannot stat %s", filename); + fd = open(filename, O_RDONLY); + ATF_REQUIRE_ERRNO(0, fd > 0); + result = read_fd(fd, s.st_size); + ATF_REQUIRE_ERRNO(0, close(fd) == 0); + return (result); +} + +static void +run_pfctl_test(const char *input_path, const char *expected_path, + const atf_tc_t *tc) +{ + int status; + pid_t pid; + int pipefds[2]; + char input_files_path[PATH_MAX]; + struct sbuf *expected_output; + struct sbuf *real_output; + posix_spawn_file_actions_t action; + + if (!check_pf_module_available()) + atf_tc_skip("pf(4) is not loaded"); + + /* The test inputs need to be able to use relative includes. */ + snprintf(input_files_path, sizeof(input_files_path), "%s/files", + atf_tc_get_config_var(tc, "srcdir")); + ATF_REQUIRE_ERRNO(0, chdir(input_files_path) == 0); + + ATF_REQUIRE_ERRNO(0, pipe(pipefds) == 0); + expected_output = read_file(expected_path); + + posix_spawn_file_actions_init(&action); + posix_spawn_file_actions_addclose(&action, STDIN_FILENO); + posix_spawn_file_actions_addclose(&action, pipefds[1]); + posix_spawn_file_actions_adddup2(&action, pipefds[0], STDOUT_FILENO); + posix_spawn_file_actions_adddup2(&action, pipefds[0], STDERR_FILENO); + + const char *argv[] = { "pfctl", "-o", "none", "-nvf", input_path, + NULL }; + printf("Running %s %s %s %s %s\n", argv[0], argv[1], argv[2], argv[3], + argv[4]); + status = posix_spawnp( + &pid, "pfctl", &action, NULL, __DECONST(char **, argv), environ); + ATF_REQUIRE_EQ_MSG( + status, 0, "posix_spawn failed: %s", strerror(errno)); + posix_spawn_file_actions_destroy(&action); + close(pipefds[0]); + + real_output = read_fd(pipefds[1], 0); + printf("---\n%s---\n", sbuf_data(real_output)); + ATF_REQUIRE_EQ(waitpid(pid, &status, 0), pid); + ATF_REQUIRE_MSG(WIFEXITED(status), + "pfctl returned non-zero! Output:\n %s", sbuf_data(real_output)); + + ATF_CHECK_STREQ(sbuf_data(expected_output), sbuf_data(real_output)); + sbuf_delete(expected_output); + sbuf_delete(real_output); + close(pipefds[1]); +} + +static void +do_pf_test(const char *number, const atf_tc_t *tc) +{ + char *input_path; + char *expected_path; + asprintf(&input_path, "%s/files/pf%s.in", + atf_tc_get_config_var(tc, "srcdir"), number); + asprintf(&expected_path, "%s/files/pf%s.ok", + atf_tc_get_config_var(tc, "srcdir"), number); + run_pfctl_test(input_path, expected_path, tc); + free(input_path); + free(expected_path); +} + +static void +do_selfpf_test(const char *number, const atf_tc_t *tc) +{ + char *expected_path; + asprintf(&expected_path, "%s/files/pf%s.ok", + atf_tc_get_config_var(tc, "srcdir"), number); + run_pfctl_test(expected_path, expected_path, tc); + free(expected_path); +} + +#define PFCTL_TEST(number, descr) \ + ATF_TC(pf##number); \ + ATF_TC_HEAD(pf##number, tc) \ + { \ + atf_tc_set_md_var(tc, "descr", descr); \ + } \ + ATF_TC_BODY(pf##number, tc) \ + { \ + do_pf_test(#number, tc); \ + } \ + ATF_TC(selfpf##number); \ + ATF_TC_HEAD(selfpf##number, tc) \ + { \ + atf_tc_set_md_var(tc, "descr", "Self " descr); \ + } \ + ATF_TC_BODY(selfpf##number, tc) \ + { \ + do_selfpf_test(#number, tc); \ + } +#include "pfctl_test_list.inc" +#undef PFCTL_TEST + +ATF_TP_ADD_TCS(tp) +{ +#define PFCTL_TEST(number, descr) \ + ATF_TP_ADD_TC(tp, pf##number); \ + ATF_TP_ADD_TC(tp, selfpf##number); +#include "pfctl_test_list.inc" +#undef PFCTL_TEST + + return atf_no_error(); +} Added: head/sbin/pfctl/tests/pfctl_test_list.inc ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sbin/pfctl/tests/pfctl_test_list.inc Wed Oct 14 17:39:50 2020 (r366708) @@ -0,0 +1,118 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2020 Alex Richardson + * + * This software was developed by SRI International and the University of + * Cambridge Computer Laboratory (Department of Computer Science and + * Technology) under DARPA contract HR0011-18-C-0016 ("ECATS"), as part of the + * DARPA SSITH research programme. + * + * This work was supported by Innovate UK project 105694, "Digital Security by + * Design (DSbD) Technology Platform Prototype". + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + * $FreeBSD$ + */ + +/* + * No include guards since this file is included multiple times by pfctl_test + * to avoid duplicating code. + */ +PFCTL_TEST(0001, "Pass with labels") +PFCTL_TEST(0002, "Block/pass") +PFCTL_TEST(0003, "Block/pass with flags") +PFCTL_TEST(0004, "Block") +PFCTL_TEST(0005, "Block with variables") +PFCTL_TEST(0006, "Variables") +PFCTL_TEST(0007, "Block/pass with return") +PFCTL_TEST(0008, "Block with address list") +PFCTL_TEST(0009, "Block with interface list") +PFCTL_TEST(0010, "Block/pass with return") +PFCTL_TEST(0011, "Block/pass ICMP") +PFCTL_TEST(0012, "Pass to subnets") +PFCTL_TEST(0013, "Pass quick") +PFCTL_TEST(0014, "Pass quick IPv6") +PFCTL_TEST(0016, "Pass with no state") +PFCTL_TEST(0018, "Address lists") +PFCTL_TEST(0019, "Lists") +PFCTL_TEST(0020, "Lists") +PFCTL_TEST(0022, "Set options") +PFCTL_TEST(0023, "Block on negated interface") +PFCTL_TEST(0024, "Variable concatenation") +PFCTL_TEST(0025, "Antispoof") +PFCTL_TEST(0026, "Block from negated interface") +PFCTL_TEST(0028, "Block with log and quick") +PFCTL_TEST(0030, "Line continuation") +PFCTL_TEST(0031, "Block policy") +PFCTL_TEST(0032, "Pass to any") +PFCTL_TEST(0034, "Pass with probability") +PFCTL_TEST(0035, "Matching on TOS") +PFCTL_TEST(0038, "Pass with user") +PFCTL_TEST(0039, "Ordered opts") +PFCTL_TEST(0040, "Block/pass") +PFCTL_TEST(0041, "Anchors") +PFCTL_TEST(0047, "Pass with labels") +PFCTL_TEST(0048, "Tables") +PFCTL_TEST(0049, "Broadcast and network modifiers") +PFCTL_TEST(0050, "Double macro set") +PFCTL_TEST(0052, "Set optimization") +PFCTL_TEST(0053, "Pass with labels") +PFCTL_TEST(0055, "Set options") +PFCTL_TEST(0056, "State opts") +PFCTL_TEST(0057, "Variables") +PFCTL_TEST(0060, "Pass from multicast") +PFCTL_TEST(0061, "Dynaddr with netmask") +PFCTL_TEST(0065, "Antispoof with labels") +PFCTL_TEST(0067, "Tags") +PFCTL_TEST(0069, "Tags") +PFCTL_TEST(0070, "Tags") +PFCTL_TEST(0071, "Tags") +PFCTL_TEST(0072, "Tags") +PFCTL_TEST(0074, "Synproxy") +PFCTL_TEST(0075, "Block quick with tags") +PFCTL_TEST(0077, "Dynaddr with netmask") +PFCTL_TEST(0078, "Table with label") +PFCTL_TEST(0079, "No-route with label") +PFCTL_TEST(0081, "Address list and table list with no-route") +PFCTL_TEST(0082, "Pass with interface, table and no-route") +PFCTL_TEST(0084, "Source track") +PFCTL_TEST(0085, "Tag macro expansion") +PFCTL_TEST(0087, "Optimization rule reordering") +PFCTL_TEST(0088, "Optimization duplicate rules handling") +PFCTL_TEST(0089, "TCP connection tracking") +PFCTL_TEST(0090, "Log opts") +PFCTL_TEST(0091, "Nested anchors") +PFCTL_TEST(0092, "Comments") +PFCTL_TEST(0094, "Address ranges") +PFCTL_TEST(0095, "Include") +PFCTL_TEST(0096, "Variables") +PFCTL_TEST(0097, "Divert-to") +PFCTL_TEST(0098, "Pass") +PFCTL_TEST(0100, "Anchor with multiple path components") +PFCTL_TEST(0101, "Prio") +PFCTL_TEST(0102, "Address lists with mixed address family") +PFCTL_TEST(0104, "Divert-to with localhost") +PFCTL_TEST(1001, "Binat") +PFCTL_TEST(1002, "Set timeout interval") +PFCTL_TEST(1003, "ALTQ") +PFCTL_TEST(1004, "ALTQ with Codel") +PFCTL_TEST(1005, "PR 231323")