Date: Wed, 08 Oct 2025 15:32:52 -0700 From: Cy Schubert <Cy.Schubert@cschubert.com> To: Rick Macklem <rick.macklem@gmail.com>, "freebsd-net@FreeBSD.org" <freebsd-net@freebsd.org>, cy@FreeBSD.org Subject: Re: FreeBSD-stable-15-amd64-test - Build #30 - Still Failing Message-ID: <20251008223252.34410305@slippy.cwsent.com> In-Reply-To: <20251008142911.D37DE4A0@slippy.cwsent.com> References: <314494105.505.1759897474336@jenkins.ci.freebsd.org> <345944856.515.1759902279808@jenkins.ci.freebsd.org> <CAM5tNy7kS3O=sAsVPev%2BcDmOGoe1C55nPz13Z0511xdPryQKFg@mail.gmail.com> <aOZimMDw7aB26ob2@amaryllis.le-fay.org> <20251008142911.D37DE4A0@slippy.cwsent.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The tests have been run on a couple of my 16-CURRENT machines. No panics were experienced. I need more information. Though, my guess is that pf or the stack may have freed the mbuf before ipfilter managed to finish inspecting it. -- Cheers, Cy Schubert <Cy.Schubert@cschubert.com> FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org NTP: <cy@nwtime.org> Web: https://nwtime.org e**(i*pi)+1=0 In message <20251008142911.D37DE4A0@slippy.cwsent.com>, Cy Schubert writes: > Thanks. I'll try the pf tests on my sandbox machine here. > > > -- > Cheers, > Cy Schubert <Cy.Schubert@cschubert.com> > FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org > NTP: <cy@nwtime.org> Web: https://nwtime.org > > e**(i*pi)+1=0 > > > In message <aOZimMDw7aB26ob2@amaryllis.le-fay.org>, Lexi Winter writes: > > > > > > --5tActHfoHppOuWl4 > > Content-Type: text/plain; charset=utf-8 > > Content-Disposition: inline > > Content-Transfer-Encoding: quoted-printable > > > > Rick Macklem wrote in <CAM5tNy7kS3O=3DsAsVPev+cDmOGoe1C55nPz13Z0511xdPryQKF > = > > g@mail.gmail.com>: > > > Just in case no one in networking is seeing this, > > > I'm reposting it here. > > =20 > > cc: cy for ipfilter. > > > > > On Tue, Oct 7, 2025 at 10:44=E2=80=AFPM <jenkins-admin@freebsd.org> wrote > : > > > > > > > > CAUTION: This email originated from outside of the University of Guelph > = > > =2E Do not click links or open attachments unless you recognize the sender > = > > and know the content is safe. If in doubt, forward suspicious emails to ITh > = > > elp@uoguelph.ca. > > > > > > > > FreeBSD-stable-15-amd64-test - Build #30 (23fbea8cf2f360dd45cb69c98f6f2 > = > > 931fff3721e) - Still Failing > > > > > > > > Build information: https://ci.FreeBSD.org/job/FreeBSD-stable-15-amd64-t > = > > est/30/ > > > > Full change log: https://ci.FreeBSD.org/job/FreeBSD-stable-15-amd64-tes > = > > t/30/changes > > > > Full build log: https://ci.FreeBSD.org/job/FreeBSD-stable-15-amd64-test > = > > /30/console > > > > > > > > Status explanation: > > > > "Failure" - the build is suspected being broken by the following change > s > > > > "Still Failing" - the build has not been fixed by the following changes > = > > and > > > > this is a notification to note that these changes hav > e > > > > not been fully tested by the CI system > > > > > > > > Change summaries: > > > > (Those commits are likely but not certainly responsible) > > > > > > > > 23fbea8cf2f360dd45cb69c98f6f2931fff3721e by rmacklem: > > > > kadmin.8: Document the new dump -f flag > > > > > > > > > > > > > > > > The end of the build log: > > > > > > > > [...truncated 3.96 MiB...] > > > > epair0b: Ethernet address: d2:f4:4c:7c:25:d3 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: c6:fa:71:45:1f:b3 > > > > epair1b: Ethernet address: 3a:02:db:62:eb:5a > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.13 > = > > 6s] > > > > sys/netpfil/pf/nat64:tcp_in_floating -> epair0a: Ethernet address: b2 > = > > :20:57:6c:d4:c5 > > > > epair0b: Ethernet address: 5a:f0:ad:fb:8a:50 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 52:58:0c:ea:9b:f3 > > > > epair1b: Ethernet address: ca:3f:da:2a:4d:86 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.12 > = > > 5s] > > > > sys/netpfil/pf/nat64:tcp_in_if_bound -> epair0a: Ethernet address: 52 > = > > :7d:e8:24:9b:38 > > > > epair0b: Ethernet address: 6e:8e:70:d3:10:60 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 32:25:e3:7f:fe:7c > > > > epair1b: Ethernet address: e6:f1:01:c1:4f:84 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.12 > = > > 6s] > > > > sys/netpfil/pf/nat64:tcp_out_floating -> epair0a: Ethernet address: c > = > > e:73:6d:57:a2:67 > > > > epair0b: Ethernet address: 2a:28:48:af:e2:4a > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 82:12:d3:13:44:bb > > > > epair1b: Ethernet address: c2:16:12:72:31:45 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.12 > = > > 7s] > > > > sys/netpfil/pf/nat64:tcp_out_if_bound -> epair0a: Ethernet address: 7 > = > > 6:e6:ab:94:3a:a3 > > > > epair0b: Ethernet address: 12:ae:f5:b6:b8:19 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 56:3a:12:2a:60:8a > > > > epair1b: Ethernet address: 76:ac:dc:80:e3:72 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.12 > = > > 6s] > > > > sys/netpfil/pf/nat64:tos -> epair0a: Ethernet address: 62:39:ae:b2:2a > = > > :db > > > > epair0b: Ethernet address: 36:9f:e7:31:94:32 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 42:37:24:71:41:ed > > > > epair1b: Ethernet address: d2:ca:da:c1:6c:54 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.12 > = > > 6s] > > > > sys/netpfil/pf/nat64:udp_in -> epair0a: Ethernet address: 16:99:da:0e > = > > :91:0d > > > > epair0b: Ethernet address: 3e:57:7e:63:8d:87 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 6e:95:d1:aa:b3:a9 > > > > epair1b: Ethernet address: ba:59:41:61:6b:19 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.13 > = > > 5s] > > > > sys/netpfil/pf/nat64:udp_out -> epair0a: Ethernet address: 7a:b4:11:1 > = > > 9:87:66 > > > > epair0b: Ethernet address: 5e:70:01:e2:ea:b6 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 52:fd:81:7b:fc:61 > > > > epair1b: Ethernet address: 1e:ba:76:e2:79:36 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair1b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.12 > = > > 7s] > > > > sys/netpfil/pf/nat64:v6_gateway -> epair0a: Ethernet address: ee:55:d > = > > 2:9b:91:07 > > > > epair0b: Ethernet address: fa:df:53:46:63:90 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 1e:a2:5a:f3:ee:cf > > > > epair1b: Ethernet address: 26:8a:99:45:02:97 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair2a: Ethernet address: ae:b7:eb:ed:ec:d8 > > > > epair2b: Ethernet address: 5a:ef:f6:5c:17:bd > > > > epair2a: link state changed to UP > > > > epair2b: link state changed to UP > > > > epair2b: link state changed to DOWN > > > > epair2a: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair1b: link state changed to DOWN > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > failed: atf-check failed; see the output of the test for details [0.19 > = > > 7s] > > > > sys/netpfil/pf/nat64.py:TestNAT64::test_address_unreachable -> epair0 > = > > a: Ethernet address: f6:fd:4b:c1:53:2d > > > > epair0b: Ethernet address: 5e:59:15:d0:79:82 > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 02:1f:0f:68:f7:17 > > > > epair1b: Ethernet address: 92:43:e4:76:55:96 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair2a: Ethernet address: 7a:d1:1e:45:06:b3 > > > > epair2b: Ethernet address: 4a:7d:09:c7:93:73 > > > > epair2a: link state changed to UP > > > > epair2b: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: promiscuous mode enabled > > > > lo0: promiscuous mode disabled > > > > epair0a: link state changed to DOWN > > > > epair0b: link state changed to DOWN > > > > epair2a: link state changed to DOWN > > > > epair2b: link state changed to DOWN > > > > epair1a: link state changed to DOWN > > > > epair1b: link state changed to DOWN > > > > failed: /usr/tests/sys/netpfil/pf/nat64.py:173: AttributeError [4.710s > ] > > > > sys/netpfil/pf/nat64.py:TestNAT64::test_bad_len -> epair0a: Ethernet > = > > address: 2e:b0:05:ae:d6:d8 > > > > epair0b: Ethernet address: 16:e7:ed:1d:32:aa > > > > epair0a: link state changed to UP > > > > epair0b: link state changed to UP > > > > epair1a: Ethernet address: 8e:fb:19:37:fb:8e > > > > epair1b: Ethernet address: 06:53:47:b9:ae:a8 > > > > epair1a: link state changed to UP > > > > epair1b: link state changed to UP > > > > epair2a: Ethernet address: 2a:90:ff:29:ca:a0 > > > > epair2b: Ethernet address: 56:89:04:58:36:84 > > > > epair2a: link state changed to UP > > > > epair2b: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: link state changed to UP > > > > lo0: promiscuous mode enabled > > > > > > > > > > > > Fatal trap 12: page fault while in kernel mode > > > > cpuid =3D 0; apic id =3D 00 > > > > fault virtual address =3D 0x18 > > > > fault code =3D supervisor read data, page not present > > > > instruction pointer =3D 0x20:0xffffffff80c0f132 > > > > stack pointer =3D 0x28:0xfffffe008aef2980 > > > > frame pointer =3D 0x28:0xfffffe008aef29c0 > > > > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > > > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > > > > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > > > > current process =3D 12 (swi1: netisr 0) > > > > rdi: fffffe008aef29d8 rsi: fffff8008be43e96 rdx: 00000000ffffffff > > > > rcx: 0000000000008466 r8: fffffe008aef2900 r9: 0000000000000003 > > > > rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe008aef29c0 > > > > r10: 0000000000000001 r11: 0000000000000000 r12: 0000000000000000 > > > > r13: 000000000000000b r14: 000000000000000b r15: 00000000000001f5 > > > > trap number =3D 12 > > > > panic: page fault > > > > cpuid =3D 0 > > > > time =3D 1759902263 > > > > KDB: stack backtrace: > > > > #0 0xffffffff80bc033d at kdb_backtrace+0x5d > > > > #1 0xffffffff80b73726 at vpanic+0x136 > > > > #2 0xffffffff80b735e3 at panic+0x43 > > > > #3 0xffffffff8107bf69 at trap_pfault+0x3c9 > > > > #4 0xffffffff81051a88 at calltrap+0x8 > > > > #5 0xffffffff80d84b19 at in6_cksum_partial_l2+0x109 > > > > #6 0xffffffff8295cab5 at fr_cksum+0xa5 > > > > #7 0xffffffff829621dd at ipf_checkl4sum+0xbd > > > > #8 0xffffffff829586f6 at ipf_checkv6sum+0x56 > > > > #9 0xffffffff8295a601 at ipf_makefrip+0x1691 > > > > #10 0xffffffff8295b6de at ipf_check+0x15e > > > > #11 0xffffffff82958c28 at ipf_check_wrapper6+0x58 > > > > #12 0xffffffff80cc3438 at pfil_mbuf_in+0x38 > > > > #13 0xffffffff80d92b94 at ip6_input+0x484 > > > > #14 0xffffffff80cc0c98 at swi_net+0x128 > > > > #15 0xffffffff80b2d359 at ithread_loop+0x239 > > > > #16 0xffffffff80b298eb at fork_exit+0x7b > > > > #17 0xffffffff81052aae at fork_trampoline+0xe > > > > Uptime: 1h18m43s > > > > Automatic reboot in 15 seconds - press a key on the console to abort > > > > + rc=3D0 > > > > + echo 'bhyve return code =3D 0' > > > > bhyve return code =3D 0 > > > > + sudo /usr/sbin/bhyvectl '--vm=3Dtestvm-15-amd64-30' --destroy > > > > + sh -ex freebsd-ci/scripts/test/extract-meta.sh > > > > + METAOUTDIR=3Dmeta-out > > > > + rm -fr meta-out > > > > + mkdir meta-out > > > > + tar xvf meta.tar -C meta-out > > > > x ./ > > > > x ./disable-zfs-tests.sh > > > > x ./disable-dtrace-tests.sh > > > > x ./auto-shutdown > > > > x ./run.sh > > > > x ./run-kyua.sh > > > > x ./disable-notyet-tests.sh > > > > + rm -f test-report.txt test-report.xml > > > > + mv 'meta-out/test-report.*' . > > > > mv: rename meta-out/test-report.* to ./test-report.*: No such file or d > = > > irectory > > > > + report=3Dtest-report.xml > > > > + [ -e freebsd-ci/jobs/FreeBSD-stable-15-amd64-test/xfail-list -a -e te > = > > st-report.xml ] > > > > + rm -f disk-cam > > > > + jot 5 > > > > + rm -f disk1 > > > > + rm -f disk2 > > > > + rm -f disk3 > > > > + rm -f disk4 > > > > + rm -f disk5 > > > > + rm -f disk-test.img > > > > [PostBuildScript] - [INFO] Executing post build scripts. > > > > [FreeBSD-stable-15-amd64-test] $ /bin/sh -xe /tmp/jenkins12196992922124 > = > > 583673.sh > > > > + ./freebsd-ci/artifact/post-link.py > > > > Post link: {'job_name': 'FreeBSD-stable-15-amd64-test', 'commit': '23fb > = > > ea8cf2f360dd45cb69c98f6f2931fff3721e', 'branch': 'stable-15', 'target': 'am > = > > d64', 'target_arch': 'amd64', 'link_type': 'latest_tested'} > > > > "Link created: stable-15/latest_tested/amd64/amd64 -> ../../23fbea8cf2f > = > > 360dd45cb69c98f6f2931fff3721e/amd64/amd64\n" > > > > Recording test results > > > > ERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed: > = > > No test report files were found. Configuration error? > > > > Checking for post-build > > > > Performing post-build step > > > > Checking if email needs to be generated > > > > Email was triggered for: Failure - Any > > > > Sending email for trigger: Failure - Any > > > > Sending mail from default account using System Admin e-mail address > > >=20 > > > > --5tActHfoHppOuWl4 > > Content-Type: application/pgp-signature; name=signature.asc > > > > -----BEGIN PGP SIGNATURE----- > > > > iHUEABYKAB0WIQSyjTg96lp3RifySyn1nT63mIK/YAUCaOZilAAKCRD1nT63mIK/ > > YLi5AQD7wS2pcYg+ig9nhZmwuklZlqnpXyi/Hylmv/rYUPccpgD+JCXJLVTFlH02 > > Y4I64B5dilhI+fS+2aiv3A07ZJGXbA4= > > =kxko > > -----END PGP SIGNATURE----- > > > > --5tActHfoHppOuWl4-- >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20251008223252.34410305>