From nobody Thu Aug 28 14:18:22 2025 X-Original-To: testing@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cCNmH3yMrz665Q7 for ; Thu, 28 Aug 2025 14:18:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cCNmH00Vxz3xtm for ; Thu, 28 Aug 2025 14:18:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1756390703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XJdS+5WBnv2ze08u3NCfrtLm3a+9yynSXeESO1XTJoc=; b=ScLxYzXIferB3tQxZHdHb2tyMVv37rAfd9XvUEKIvzHMjR7LkwrT5II9pqB4ELVe7pvTAa FOLjMKu4HpodBHzOMFToigWotJl6l2xrvkVijkdqhfGqDOwb9CZSddU5yzgtSJNahDI0pf tWxcYUYMG4aIw31qrYZANSK/kQVf4ID3/TZPK7+7TnLrxs+no/wNi2gotPc3Gj5U/TWYgu 1K32AKAQjS4WR0MdyoKvJYENrvaGQ5q6lfMnc14ev2mdtroErM3YvJ71v3L6zKhxmiBW5C tz2Klq+J0bbQtAIS6TbwS2wMQ2OXBUUTE6kNZk6q9MEdHqY5UjytLNwb8oSJpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1756390703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XJdS+5WBnv2ze08u3NCfrtLm3a+9yynSXeESO1XTJoc=; b=RFyarjO6Hk3iE+HCOq0TodWRhsk+d4QXLP5cappDugHb7iNK9lQdAGZbq33EaaxQLH+lqY 6fjOn7VgYWVMHT2EoM6zQdq5uxDAWrP5xsot9o/Q2wz0LVEvQj/UsQY0ann9CeJJoSaiw7 8Fhlq6Y7viW0TmaylzjFKUrJBsZueNKeVByAGU0ooDDjl1bep1Qc1mX6F9B/LMr99bGxUA bg0YUPgJfO3A98vbr1uN80iqR/0I7RaTYvE0W42fBMxoEiSZPVHr6vOevPasR5+3jYxaIv eROYGffvcB/oKjGEkVoNSXC+3IVNdAwkBjtH6xp/X5SssXfYdWOI606ApQqVXg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1756390703; a=rsa-sha256; cv=none; b=bSDipZg7uWH2RngVv7sA7HvB82HgblCWgBIy4z0W3/+U+aZlIof83CAHTlR2sQ3BxAIOJx +3w/172ewnCZKlsiolNBp7hP7UYrU2taDNI3D7n2g0s6Gny58CEsQ8WHUwrXsiG19bz6xN T/LwLr8WEUCHyrwUYa3JuTxj7VXQVO29PSHCNHDTWEmBLOm0U8MRQYvUCLwTeB3+0RV0RN 3kzSJJuzccTCGMpYQiEHSBjAc9zo9r5S+hf4pHHfMLcybkdj5CkevFyB990cfJZTjw30DI hmzJIrr4XNHk+a/OJ+V//qyLhnCCwNSLCRr5etyS6ag34E5uMLBOEhjcu/vlIg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cCNmG676Dz1BWx for ; Thu, 28 Aug 2025 14:18:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 57SEIMvv038820 for ; Thu, 28 Aug 2025 14:18:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 57SEIMSI038819 for testing@FreeBSD.org; Thu, 28 Aug 2025 14:18:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: testing@FreeBSD.org Subject: [Bug 289146] sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action tests failing in CI Date: Thu, 28 Aug 2025 14:18:22 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: tests X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: me@svmhdvn.name X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: melifaro@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status keywords bug_severity priority component assigned_to reporter cc attachments.created Message-ID: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Testing List-Archive: https://lists.freebsd.org/archives/freebsd-testing List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-testing@freebsd.org Sender: owner-freebsd-testing@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D289146 Bug ID: 289146 Summary: sbin/ipfw/test_add_rule.py:TestAddRule::test_add_actio n tests failing in CI Product: Base System Version: CURRENT Hardware: Any OS: Any Status: New Keywords: regression Severity: Affects Only Me Priority: --- Component: tests Assignee: melifaro@FreeBSD.org Reporter: me@svmhdvn.name CC: testing@FreeBSD.org Created attachment 263289 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D263289&action= =3Dedit kyua test report After https://cgit.freebsd.org/src/commit/?id=3Dd879f1c8961d99bdbe6727004bfc6c239= 24526f8 added the HAS_TESTS=3D variable to sbin/ipfw/Makefile (to reveal the hidden testcases that were never being run), the following parameterized testcases inside the 'sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action' testsu= ite are now showing up to fail in CI. sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action[call_420] -> fail= ed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError [0.231s]=20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action[return] -> failed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError [0.272s]=20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action[skipto_42] -> fai= led: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError [0.278s]=20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_check_state] ->= =20 failed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError=20 [0.223s]=20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_eaction_ntp] ->= =20 failed: /usr/tests/sbin/ipfw/test_add_rule.py:119: AssertionError [0.189s]= =20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_eaction_tcp-setm= ss]=20 -> failed: /usr/tests/sbin/ipfw/test_add_rule.py:119: AssertionError [0.1= 92s]=20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_keep_state] ->= =20 failed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError=20 [0.222s]=20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_record_state] -= >=20 failed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError=20 [0.218s]=20 sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_tables] -> fai= led: /usr/tests/sbin/ipfw/test_add_rule.py:119: AssertionError [0.202s] I've attached the full Kyua test report from my test run in this bug, which contains the stderr from these failing cases. I have reproduced this error with two methods and attached a sample test-report.txt: 1. using src in-tree local CI: # make -C tests/ci CITYPE=3Dfull PARALLEL_JOBS=3D2 VM_MEM_SIZE=3D4g KYUA_TEST_FILTERS=3D'sbin/ipfw/test_add_rule.py' 2. using bricoler[0]: $ bricoler run freebsd-src-regression-suite --param freebsd-src:url=3D/usr/= src --param freebsd-src:branch=3D --param freebsd-src-regression-suite:hypervisor=3Dbhyve --param freebsd-src-regression-suite:memory=3D4096 --param freebsd-src-regression-suite:ncpus=3D2 --param freebsd-src-regression-suite:parallelism=3D1 --param freebsd-src-regression-suite:tests=3D'sbin/ipfw/test_add_rule.py' @melifaro I see that you wrote these test cases. Please triage as necessary. [0] https://github.com/markjdb/bricoler --=20 You are receiving this mail because: You are on the CC list for the bug.=