Date: Thu, 05 Dec 2024 07:57:03 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 283137] pf: states corruption since 93c80b79ad65c leading to kernel panic Message-ID: <bug-283137-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D283137 Bug ID: 283137 Summary: pf: states corruption since 93c80b79ad65c leading to kernel panic Product: Base System Version: 14.2-STABLE Hardware: Any URL: https://github.com/opnsense/src/issues/230 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: franco@opnsense.org Hi, OPNsense users report a pf state corruption since the deployment of 93c80b79ad65 which ends up in at least one kernel panic, but due to the nat= ure of the situation it could actually be multiple. The issue seems quite prevalent on production systems and may crash a system after just a couple of minutes of runtime. One user provided a kernel dump. I'm attaching the info for triage here: (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=3Dtextdump@entry=3D0) at /usr/src/sys/kern/kern_shutdown.c:405 #2 0xffffffff8049c36a in db_dump (dummy=3D<optimized out>, dummy2=3D<optim= ized out>, dummy3=3D<optimized out>, dummy4=3D<optimized out>) at /usr/src/sys/ddb/db_command.c:591 #3 0xffffffff8049c16d in db_command (last_cmdp=3D<optimized out>, cmd_table=3D<optimized out>, dopager=3Dfalse) at /usr/src/sys/ddb/db_comman= d.c:504 #4 0xffffffff8049c2b6 in db_command_script (command=3Dcommand@entry=3D0xffffffff81bbf6d3 <db_recursion_data+3> "dump")= at /usr/src/sys/ddb/db_command.c:569 #5 0xffffffff804a1528 in db_script_exec (scriptname=3D<optimized out>, warnifnotfound=3Dwarnifnotfound@entry=3D0) at /usr/src/sys/ddb/db_script.c:= 302 #6 0xffffffff804a1435 in db_script_kdbenter (eventname=3D<optimized out>) = at /usr/src/sys/ddb/db_script.c:325 #7 0xffffffff8049f4f1 in db_trap (type=3D<optimized out>, code=3D<optimize= d out>) at /usr/src/sys/ddb/db_main.c:267 #8 0xffffffff80c09868 in kdb_trap (type=3Dtype@entry=3D3, code=3Dcode@entr= y=3D0, tf=3Dtf@entry=3D0xfffffe00e206e2e0) at /usr/src/sys/kern/subr_kdb.c:790 #9 0xffffffff810e0419 in trap (frame=3D0xfffffe00e206e2e0) at /usr/src/sys/amd64/amd64/trap.c:608 #10 <signal handler called> #11 kdb_enter (why=3D<optimized out>, msg=3D<optimized out>) at /usr/src/sys/kern/subr_kdb.c:556 #12 0xffffffff80bb91d2 in vpanic (fmt=3D0xffffffff823f5cbd "Bad link elm %p prev->next !=3D elm", ap=3Dap@entry=3D0xfffffe00e206e510) at /usr/src/sys/kern/kern_shutdown.c:955 #13 0xffffffff80bb9283 in panic (fmt=3D0xffffffff81d82c18 <cnputs_mtx+24> "= ") at /usr/src/sys/kern/kern_shutdown.c:891 #14 0xffffffff823c1dd0 in pf_state_key_detach (s=3Ds@entry=3D0xfffff803cc29= 7b00, idx=3Didx@entry=3D0) at /usr/src/sys/netpfil/pf/pf.c:1456 #15 0xffffffff823ad0ef in pf_detach_state (s=3Ds@entry=3D0xfffff803cc297b00= ) at /usr/src/sys/netpfil/pf/pf.c:1442 #16 0xffffffff823ac6d9 in pf_state_key_attach (skw=3D0xfffff803cc2c4420, sk= s=3D0x0, s=3D0xfffff803cc297b00) at /usr/src/sys/netpfil/pf/pf.c:1355 #17 pf_state_insert (kif=3D<optimized out>, orig_kif=3Dorig_kif@entry=3D0xfffff80002150600, skw=3D0xfffff803cc2c4420, sks=3D<optimized out>, s=3Ds@entry=3D0xfffff803cc297b00) at /usr/src/sys/netpfil/pf/pf.c:1535 #18 0xffffffff823ba740 in pf_create_state (r=3D0xfffff80227b7e000, nr=3D0xfffff80189e7a800, a=3D<optimized out>, pd=3D0xfffffe00e206eb00, nsn= =3D0x0, nk=3D0x12, sk=3D<optimized out>,=20 m=3D0xfffff8001dc64800, off=3D20, sport=3D4843, dport=3D59668, rewrite=3D0xfffffe00e206ea0c, kif=3D0xfffff80002150600, sm=3D0xfffffe00e206= ec18, tag=3D-1, bproto_sum=3D25520,=20 bip_sum=3D979, hdrlen=3D8, match_rules=3D<optimized out>) at /usr/src/sys/netpfil/pf/pf.c:5025 #19 pf_test_rule (rm=3Drm@entry=3D0xfffffe00e206ebf0, sm=3Dsm@entry=3D0xfffffe00e206ec18, kif=3Dkif@entry=3D0xfffff80002150600, m=3D0xfffff8001dc64800, off=3D20,=20 pd=3Dpd@entry=3D0xfffffe00e206eb00, am=3D0xfffffe00e206ebd8, rsm=3D0xfffffe00e206ebc8, inp=3D0x0) at /usr/src/sys/netpfil/pf/pf.c:4800 #20 0xffffffff823b4471 in pf_test (dir=3Ddir@entry=3D1, pflags=3D<optimized= out>, ifp=3D0xfffff80001906000, m0=3Dm0@entry=3D0xfffffe00e206ed08, inp=3D<optimi= zed out>,=20 default_actions=3Ddefault_actions@entry=3D0x0) at /usr/src/sys/netpfil/pf/pf.c:8269 #21 0xffffffff823dc177 in pf_check_in (m=3D0xfffffe00e206ed08, ifp=3D0x12, flags=3D-502865312, ruleset=3D<optimized out>, inp=3D0xffffffff80c10af0 <pu= tchar>) at /usr/src/sys/netpfil/pf/pf_ioctl.c:6575 #22 0xffffffff80d19e98 in pfil_mbuf_common (pch=3D<optimized out>, m=3D0xfffffe00e206ed08, m@entry=3D0xfffffe00e206ec48, ifp=3D0xfffff80001906= 000, flags=3D65536, inp=3Dinp@entry=3D0x0) at /usr/src/sys/net/pfil.c:212 #23 pfil_mbuf_in (head=3D<optimized out>, m=3Dm@entry=3D0xfffffe00e206ed08, ifp=3D0xfffff80001906000, inp=3Dinp@entry=3D0x0) at /usr/src/sys/net/pfil.c= :230 #24 0xffffffff80d9c59a in ip_tryforward (m=3D0xfffff8001dc64800) at /usr/src/sys/netinet/ip_fastfwd.c:312 #25 0xffffffff80d9fa9c in ip_input (m=3D0xfffff8001dc64800) at /usr/src/sys/netinet/ip_input.c:621 #26 0xffffffff80d1682b in netisr_process_workstream_proto (nwsp=3D0xfffffe003a5eca40, proto=3D1) at /usr/src/sys/net/netisr.c:927 #27 swi_net (arg=3D0xfffffe003a5eca40) at /usr/src/sys/net/netisr.c:974 #28 0xffffffff80b6ffc6 in intr_event_execute_handlers (ie=3D0xfffff80001a59= 100, p=3D<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205 #29 ithread_execute_handlers (ie=3D0xfffff80001a59100, p=3D<optimized out>)= at /usr/src/sys/kern/kern_intr.c:1218 #30 ithread_loop (arg=3Darg@entry=3D0xfffff80001a7a620) at /usr/src/sys/kern/kern_intr.c:1306 #31 0xffffffff80b6c402 in fork_exit (callout=3D0xffffffff80b6fd70 <ithread_= loop>, arg=3D0xfffff80001a7a620, frame=3D0xfffffe00e206ef40) at /usr/src/sys/kern/kern_fork.c:1164 #32 <signal handler called> (kgdb) frame 14 #14 0xffffffff823c1dd0 in pf_state_key_detach (s=3Ds@entry=3D0xfffff803cc29= 7b00, idx=3Didx@entry=3D0) at /usr/src/sys/netpfil/pf/pf.c:1456 warning: Source file is more recent than executable. 1456 TAILQ_REMOVE(&sk->states[idx], s, key_list[idx]); (kgdb) list 1451 #ifdef INVARIANTS 1452 struct pf_keyhash *kh =3D &V_pf_keyhash[pf_hashkey(sk)]; 1453=20=20=20=20 1454 PF_HASHROW_ASSERT(kh); 1455 #endif 1456 TAILQ_REMOVE(&sk->states[idx], s, key_list[idx]); 1457 s->key[idx] =3D NULL; 1458=20=20=20=20 1459 if (TAILQ_EMPTY(&sk->states[0]) && TAILQ_EMPTY(&sk->states[= 1])) { 1460 LIST_REMOVE(sk, entry); (kgdb) p *sk $1 =3D {addr =3D {{{v4 =3D {s_addr =3D XXX}, v6 =3D {__u6_addr =3D {__u6_ad= dr8 =3D "XXX", <incomplete sequence XXX>,=20 __u6_addr16 =3D {XXX, XXX, XXX, XXX, XXX, XXX, XXX, XXX}, __u6_= addr32 =3D {XXX, XXX, XXX, XXX}}},=20 addr8 =3D "XXX", <incomplete sequence \XXX>, addr16 =3D {XXX, XXX, = XXX, XXX, XXX, XXX, XXX,=20 XXX}, addr32 =3D {XXX, XXX, XXX, XXX}}}, {{v4 =3D {s_addr =3D XXX= }, v6 =3D {__u6_addr =3D { __u6_addr8 =3D "XXX", <incomplete sequence XXX>, __u6_addr16 = =3D {XXX, XXX, XXX, XXX, XXX,=20 XXX, XXX, XXX}, __u6_addr32 =3D {XXX, XXX, XXX, XXX}}},=20 addr8 =3D "XXX", <incomplete sequence XXX>, addr16 =3D {XXX, XXX, X= XX, XXX, XXX, XXX, XXX,=20 XXX}, addr32 =3D {XXX, XXX, XXX, XXX}}}}, port =3D {49374, 57005}= , af =3D 222 '\336', proto =3D 192 '\300',=20 pad =3D "\255", <incomplete sequence \336>, entry =3D {le_next =3D 0xdeadc0dedeadc0de, le_prev =3D 0xdeadc0dedeadc0de}, states =3D {{tqh_first= =3D 0xdeadc0dedeadc0de,=20 tqh_last =3D 0xdeadc0dedeadc0de}, {tqh_first =3D 0xdeadc0dedeadc0de, = tqh_last =3D 0xdeadc0dedeadc0de}}} (kgdb) p *sk->states $2 =3D {tqh_first =3D 0xdeadc0dedeadc0de, tqh_last =3D 0xdeadc0dedeadc0de} (kgdb) p *s $3 =3D {id =3D 10415225491559546880, creatorid =3D 1082503010, direction = =3D 1 '\001', pad =3D "\000\000", state_flags =3D 128, timeout =3D 27 '\033', sync_state = =3D 255 '\377',=20 sync_updates =3D 0 '\000', refs =3D 0, lock =3D 0xfffffe0109794688, sync_= list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, key_list =3D {{tqe_next =3D 0x0,=20 tqe_prev =3D 0xfffff803cc2c4458}, {tqe_next =3D 0x0, tqe_prev =3D 0x0= }}, entry =3D {le_next =3D 0x0, le_prev =3D 0x0}, src =3D {scrub =3D 0x0, seqlo =3D 0= , seqhi =3D 0, seqdiff =3D 0,=20 max_win =3D 0, mss =3D 0, state =3D 1 '\001', wscale =3D 0 '\000', tcp_= est =3D 0 '\000', pad =3D ""}, dst =3D {scrub =3D 0x0, seqlo =3D 0, seqhi =3D 0, seqd= iff =3D 0, max_win =3D 0,=20 mss =3D 0, state =3D 0 '\000', wscale =3D 0 '\000', tcp_est =3D 0 '\000= ', pad =3D ""}, match_rules =3D {slh_first =3D 0x0}, rule =3D {ptr =3D 0xfffff80227b7e= 000, nr =3D 666361856},=20 anchor =3D {ptr =3D 0x0, nr =3D 0}, nat_rule =3D {ptr =3D 0xfffff80189e7a= 800, nr =3D 2313660416}, rt_addr =3D {{v4 =3D {s_addr =3D 0}, v6 =3D {__u6_addr =3D { __u6_addr8 =3D '\000' <repeats 15 times>, __u6_addr16 =3D {0, 0, = 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}, addr8 =3D '\000' <repeats 15 time= s>, addr16 =3D {0,=20 0, 0, 0, 0, 0, 0, 0}, addr32 =3D {0, 0, 0, 0}}}, key =3D {0xfffff803cc2c4420, 0x0}, kif =3D 0xfffff80002150600, orig_kif =3D 0xfffff80002150600, rt_kif =3D 0x0,=20 src_node =3D 0x0, nat_src_node =3D 0x0, packets =3D {0, 0}, bytes =3D {0,= 0}, creation =3D 127, expire =3D 127, pfsync_time =3D 0, act =3D {rtableid =3D = -1, qid =3D 0, pqid =3D 0,=20 max_mss =3D 0, log =3D 0 '\000', set_tos =3D 0 '\000', min_ttl =3D 0 '\= 000', dnpipe =3D 0, dnrpipe =3D 0, flags =3D 128, set_prio =3D "\000"}, tag =3D 0, rt = =3D 0 '\000'} Cheers, Franco --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-283137-227>