From nobody Thu Dec 5 07:57:03 2024 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Y3mv34ykGz5g5pc for ; Thu, 05 Dec 2024 07:57:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Y3mv33Tj8z4Ppn for ; Thu, 5 Dec 2024 07:57:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1733385423; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ytgaUllOnrDIgHMYuFUgeEpLrXwW15ZbyJF1+F6JCJA=; b=UX0JQ4+m1ugJl9wPv5SvGsHoixrPzieC5KZPorlmCGFnmLGAY0b6F8juvITd7u0//kXfyB T8JJZ27ZSmYMHi0rZFUMAX1PEapz6AYZtZTydfgMnPaWfFteHoAmN+xwskDYG8Y1wINt21 +ydZXOSa9vwKKLg3jBctwRM/eK1GYze0/Nv6RiOd+4VvQk3bDvJAMzfhO5jfMz5oV+UhZS IRtqbapSaEyLtqss/hE4zah68Klj8QjVZN7cXcrusRjQ5Ll3XDyyDOhtgatVBUYHGqPhI8 yFvFzZzjBNxUpU2k49mFT0xbUwNa/54n92mpIfcBpH7JUsRezzMneRLv18MKUQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1733385423; a=rsa-sha256; cv=none; b=VAIK32Qds6USd6Lnx9vHKOQcm2C+ji9UOGDAthetZ1N2Vvgjvzx/8H6+nJqByGoiqtWnPA Y664mP9GwpmfxlDkVKzsU3VqehaMySiExBBZIPW+sVVimyfNJWLOfn4mscL/EKYFW9a3m8 bmX6iWpB8YIv4eJArhs7Q7F3zsXI+6/eXmNPSHxytjxIuOnRxoFmg+8D7kyvBs4rS3ke37 h5lMYUd2jtkqVZdgbepcs8iA2+qRLi+leGABEKhorkGFBmRrYvIGFSfdwzP7lBIrqNZpG1 9TT92uhZ6as2hBPwTs7i6ZRKFpKPFAkUdYgnS3/zzr+fqNaYBdbM6jv/9kH00w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Y3mv32vVHzkVg for ; Thu, 5 Dec 2024 07:57:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4B57v3wE052095 for ; Thu, 5 Dec 2024 07:57:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4B57v3tg052094 for bugs@FreeBSD.org; Thu, 5 Dec 2024 07:57:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 283137] pf: states corruption since 93c80b79ad65c leading to kernel panic Date: Thu, 05 Dec 2024 07:57:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: franco@opnsense.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform bug_file_loc op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D283137 Bug ID: 283137 Summary: pf: states corruption since 93c80b79ad65c leading to kernel panic Product: Base System Version: 14.2-STABLE Hardware: Any URL: https://github.com/opnsense/src/issues/230 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: franco@opnsense.org Hi, OPNsense users report a pf state corruption since the deployment of 93c80b79ad65 which ends up in at least one kernel panic, but due to the nat= ure of the situation it could actually be multiple. The issue seems quite prevalent on production systems and may crash a system after just a couple of minutes of runtime. One user provided a kernel dump. I'm attaching the info for triage here: (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=3Dtextdump@entry=3D0) at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xffffffff8049c36a in db_dump (dummy=3D, dummy2=3D, dummy3=3D, dummy4=3D) at /usr/src/sys/ddb/db_command.c:591 #3 0xffffffff8049c16d in db_command (last_cmdp=3D, cmd_table=3D, dopager=3Dfalse) at /usr/src/sys/ddb/db_comman= d.c:504 #4 0xffffffff8049c2b6 in db_command_script (command=3Dcommand@entry=3D0xffffffff81bbf6d3 "dump")= at /usr/src/sys/ddb/db_command.c:569 #5 0xffffffff804a1528 in db_script_exec (scriptname=3D, warnifnotfound=3Dwarnifnotfound@entry=3D0) at /usr/src/sys/ddb/db_script.c:= 302 #6 0xffffffff804a1435 in db_script_kdbenter (eventname=3D) = at /usr/src/sys/ddb/db_script.c:325 #7 0xffffffff8049f4f1 in db_trap (type=3D, code=3D) at /usr/src/sys/ddb/db_main.c:267 #8 0xffffffff80c09868 in kdb_trap (type=3Dtype@entry=3D3, code=3Dcode@entr= y=3D0, tf=3Dtf@entry=3D0xfffffe00e206e2e0) at /usr/src/sys/kern/subr_kdb.c:790 #9 0xffffffff810e0419 in trap (frame=3D0xfffffe00e206e2e0) at /usr/src/sys/amd64/amd64/trap.c:608 #10 #11 kdb_enter (why=3D, msg=3D) at /usr/src/sys/kern/subr_kdb.c:556 #12 0xffffffff80bb91d2 in vpanic (fmt=3D0xffffffff823f5cbd "Bad link elm %p prev->next !=3D elm", ap=3Dap@entry=3D0xfffffe00e206e510) at /usr/src/sys/kern/kern_shutdown.c:955 #13 0xffffffff80bb9283 in panic (fmt=3D0xffffffff81d82c18 "= ") at /usr/src/sys/kern/kern_shutdown.c:891 #14 0xffffffff823c1dd0 in pf_state_key_detach (s=3Ds@entry=3D0xfffff803cc29= 7b00, idx=3Didx@entry=3D0) at /usr/src/sys/netpfil/pf/pf.c:1456 #15 0xffffffff823ad0ef in pf_detach_state (s=3Ds@entry=3D0xfffff803cc297b00= ) at /usr/src/sys/netpfil/pf/pf.c:1442 #16 0xffffffff823ac6d9 in pf_state_key_attach (skw=3D0xfffff803cc2c4420, sk= s=3D0x0, s=3D0xfffff803cc297b00) at /usr/src/sys/netpfil/pf/pf.c:1355 #17 pf_state_insert (kif=3D, orig_kif=3Dorig_kif@entry=3D0xfffff80002150600, skw=3D0xfffff803cc2c4420, sks=3D, s=3Ds@entry=3D0xfffff803cc297b00) at /usr/src/sys/netpfil/pf/pf.c:1535 #18 0xffffffff823ba740 in pf_create_state (r=3D0xfffff80227b7e000, nr=3D0xfffff80189e7a800, a=3D, pd=3D0xfffffe00e206eb00, nsn= =3D0x0, nk=3D0x12, sk=3D,=20 m=3D0xfffff8001dc64800, off=3D20, sport=3D4843, dport=3D59668, rewrite=3D0xfffffe00e206ea0c, kif=3D0xfffff80002150600, sm=3D0xfffffe00e206= ec18, tag=3D-1, bproto_sum=3D25520,=20 bip_sum=3D979, hdrlen=3D8, match_rules=3D) at /usr/src/sys/netpfil/pf/pf.c:5025 #19 pf_test_rule (rm=3Drm@entry=3D0xfffffe00e206ebf0, sm=3Dsm@entry=3D0xfffffe00e206ec18, kif=3Dkif@entry=3D0xfffff80002150600, m=3D0xfffff8001dc64800, off=3D20,=20 pd=3Dpd@entry=3D0xfffffe00e206eb00, am=3D0xfffffe00e206ebd8, rsm=3D0xfffffe00e206ebc8, inp=3D0x0) at /usr/src/sys/netpfil/pf/pf.c:4800 #20 0xffffffff823b4471 in pf_test (dir=3Ddir@entry=3D1, pflags=3D, ifp=3D0xfffff80001906000, m0=3Dm0@entry=3D0xfffffe00e206ed08, inp=3D,=20 default_actions=3Ddefault_actions@entry=3D0x0) at /usr/src/sys/netpfil/pf/pf.c:8269 #21 0xffffffff823dc177 in pf_check_in (m=3D0xfffffe00e206ed08, ifp=3D0x12, flags=3D-502865312, ruleset=3D, inp=3D0xffffffff80c10af0 ) at /usr/src/sys/netpfil/pf/pf_ioctl.c:6575 #22 0xffffffff80d19e98 in pfil_mbuf_common (pch=3D, m=3D0xfffffe00e206ed08, m@entry=3D0xfffffe00e206ec48, ifp=3D0xfffff80001906= 000, flags=3D65536, inp=3Dinp@entry=3D0x0) at /usr/src/sys/net/pfil.c:212 #23 pfil_mbuf_in (head=3D, m=3Dm@entry=3D0xfffffe00e206ed08, ifp=3D0xfffff80001906000, inp=3Dinp@entry=3D0x0) at /usr/src/sys/net/pfil.c= :230 #24 0xffffffff80d9c59a in ip_tryforward (m=3D0xfffff8001dc64800) at /usr/src/sys/netinet/ip_fastfwd.c:312 #25 0xffffffff80d9fa9c in ip_input (m=3D0xfffff8001dc64800) at /usr/src/sys/netinet/ip_input.c:621 #26 0xffffffff80d1682b in netisr_process_workstream_proto (nwsp=3D0xfffffe003a5eca40, proto=3D1) at /usr/src/sys/net/netisr.c:927 #27 swi_net (arg=3D0xfffffe003a5eca40) at /usr/src/sys/net/netisr.c:974 #28 0xffffffff80b6ffc6 in intr_event_execute_handlers (ie=3D0xfffff80001a59= 100, p=3D) at /usr/src/sys/kern/kern_intr.c:1205 #29 ithread_execute_handlers (ie=3D0xfffff80001a59100, p=3D)= at /usr/src/sys/kern/kern_intr.c:1218 #30 ithread_loop (arg=3Darg@entry=3D0xfffff80001a7a620) at /usr/src/sys/kern/kern_intr.c:1306 #31 0xffffffff80b6c402 in fork_exit (callout=3D0xffffffff80b6fd70 , arg=3D0xfffff80001a7a620, frame=3D0xfffffe00e206ef40) at /usr/src/sys/kern/kern_fork.c:1164 #32 (kgdb) frame 14 #14 0xffffffff823c1dd0 in pf_state_key_detach (s=3Ds@entry=3D0xfffff803cc29= 7b00, idx=3Didx@entry=3D0) at /usr/src/sys/netpfil/pf/pf.c:1456 warning: Source file is more recent than executable. 1456 TAILQ_REMOVE(&sk->states[idx], s, key_list[idx]); (kgdb) list 1451 #ifdef INVARIANTS 1452 struct pf_keyhash *kh =3D &V_pf_keyhash[pf_hashkey(sk)]; 1453=20=20=20=20 1454 PF_HASHROW_ASSERT(kh); 1455 #endif 1456 TAILQ_REMOVE(&sk->states[idx], s, key_list[idx]); 1457 s->key[idx] =3D NULL; 1458=20=20=20=20 1459 if (TAILQ_EMPTY(&sk->states[0]) && TAILQ_EMPTY(&sk->states[= 1])) { 1460 LIST_REMOVE(sk, entry); (kgdb) p *sk $1 =3D {addr =3D {{{v4 =3D {s_addr =3D XXX}, v6 =3D {__u6_addr =3D {__u6_ad= dr8 =3D "XXX", ,=20 __u6_addr16 =3D {XXX, XXX, XXX, XXX, XXX, XXX, XXX, XXX}, __u6_= addr32 =3D {XXX, XXX, XXX, XXX}}},=20 addr8 =3D "XXX", , addr16 =3D {XXX, XXX, = XXX, XXX, XXX, XXX, XXX,=20 XXX}, addr32 =3D {XXX, XXX, XXX, XXX}}}, {{v4 =3D {s_addr =3D XXX= }, v6 =3D {__u6_addr =3D { __u6_addr8 =3D "XXX", , __u6_addr16 = =3D {XXX, XXX, XXX, XXX, XXX,=20 XXX, XXX, XXX}, __u6_addr32 =3D {XXX, XXX, XXX, XXX}}},=20 addr8 =3D "XXX", , addr16 =3D {XXX, XXX, X= XX, XXX, XXX, XXX, XXX,=20 XXX}, addr32 =3D {XXX, XXX, XXX, XXX}}}}, port =3D {49374, 57005}= , af =3D 222 '\336', proto =3D 192 '\300',=20 pad =3D "\255", , entry =3D {le_next =3D 0xdeadc0dedeadc0de, le_prev =3D 0xdeadc0dedeadc0de}, states =3D {{tqh_first= =3D 0xdeadc0dedeadc0de,=20 tqh_last =3D 0xdeadc0dedeadc0de}, {tqh_first =3D 0xdeadc0dedeadc0de, = tqh_last =3D 0xdeadc0dedeadc0de}}} (kgdb) p *sk->states $2 =3D {tqh_first =3D 0xdeadc0dedeadc0de, tqh_last =3D 0xdeadc0dedeadc0de} (kgdb) p *s $3 =3D {id =3D 10415225491559546880, creatorid =3D 1082503010, direction = =3D 1 '\001', pad =3D "\000\000", state_flags =3D 128, timeout =3D 27 '\033', sync_state = =3D 255 '\377',=20 sync_updates =3D 0 '\000', refs =3D 0, lock =3D 0xfffffe0109794688, sync_= list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, key_list =3D {{tqe_next =3D 0x0,=20 tqe_prev =3D 0xfffff803cc2c4458}, {tqe_next =3D 0x0, tqe_prev =3D 0x0= }}, entry =3D {le_next =3D 0x0, le_prev =3D 0x0}, src =3D {scrub =3D 0x0, seqlo =3D 0= , seqhi =3D 0, seqdiff =3D 0,=20 max_win =3D 0, mss =3D 0, state =3D 1 '\001', wscale =3D 0 '\000', tcp_= est =3D 0 '\000', pad =3D ""}, dst =3D {scrub =3D 0x0, seqlo =3D 0, seqhi =3D 0, seqd= iff =3D 0, max_win =3D 0,=20 mss =3D 0, state =3D 0 '\000', wscale =3D 0 '\000', tcp_est =3D 0 '\000= ', pad =3D ""}, match_rules =3D {slh_first =3D 0x0}, rule =3D {ptr =3D 0xfffff80227b7e= 000, nr =3D 666361856},=20 anchor =3D {ptr =3D 0x0, nr =3D 0}, nat_rule =3D {ptr =3D 0xfffff80189e7a= 800, nr =3D 2313660416}, rt_addr =3D {{v4 =3D {s_addr =3D 0}, v6 =3D {__u6_addr =3D { __u6_addr8 =3D '\000' , __u6_addr16 =3D {0, 0, = 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}, addr8 =3D '\000' , addr16 =3D {0,=20 0, 0, 0, 0, 0, 0, 0}, addr32 =3D {0, 0, 0, 0}}}, key =3D {0xfffff803cc2c4420, 0x0}, kif =3D 0xfffff80002150600, orig_kif =3D 0xfffff80002150600, rt_kif =3D 0x0,=20 src_node =3D 0x0, nat_src_node =3D 0x0, packets =3D {0, 0}, bytes =3D {0,= 0}, creation =3D 127, expire =3D 127, pfsync_time =3D 0, act =3D {rtableid =3D = -1, qid =3D 0, pqid =3D 0,=20 max_mss =3D 0, log =3D 0 '\000', set_tos =3D 0 '\000', min_ttl =3D 0 '\= 000', dnpipe =3D 0, dnrpipe =3D 0, flags =3D 128, set_prio =3D "\000"}, tag =3D 0, rt = =3D 0 '\000'} Cheers, Franco --=20 You are receiving this mail because: You are the assignee for the bug.=