Date: Thu, 26 Jan 2012 16:38:26 +0200 From: David =?iso-8859-1?q?Sieb=F6rger?= <d.sieborger@ru.ac.za> To: freebsd-pf@freebsd.org Subject: pf crashes in pfr_update_stats() Message-ID: <201201261638.26599.d.sieborger@ru.ac.za>
next in thread | raw e-mail | index | archive | help
--Boundary-00=_iVWIP9/oBMJI11C
Content-Type: Text/Plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi,
I have a pair of FreeBSD 9.0-RELEASE firewalls which are crashing=20
repeatedly. I've been able to connect to one of them with remote kgdb=20
after it crashed (see kgdb session attached), but I haven't been able to=20
get to the bottom of what's wrong. Is anyone able to shed more light on=20
this?
The first problem that I see is that the kt argument to=20
pfr_update_stats() is null, so the kernel panics as soon as that's=20
dereferenced.
Where pfr_update_stats() is called from pf_test(), kgdb tells me that=20
"Variable "tr" is not available." (Is that because of a gcc=20
optimisation?) But, tr ought to equal r in this instance, and r is=20
available, so I looked at r. r->dst.addr.p.tbl is indeed null.
Does anyone have any theories about why that could be the case, or=20
anything else that I could do to debug this? I can provide more=20
configuration information if needed.
=2D-=20
David Sieb=F6rger
System Administrator, IT Division, Rhodes University
--Boundary-00=_iVWIP9/oBMJI11C
Content-Type: text/plain;
charset="ISO-8859-1";
name="kgdb2.txt"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="kgdb2.txt"
# kgdb -r /dev/cuau0 /usr/obj/usr/src/sys/FIREWALL/kernel.debug
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain condition=
s.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Switching to remote protocol
pfr_update_stats (kt=3D0x0, a=3D0xfffffe000e0a4c90, af=3D2 '\002', len=3D48,
dir_out=3D0, op_pass=3D0, notrule=3D0)
at /usr/src/sys/contrib/pf/net/pf_table.c:2242
2242 if (!(kt->pfrkt_flags & PFR_TFLAG_ACTIVE) && kt->pfrkt_root=
!=3D NULL)
(kgdb) where
#0 pfr_update_stats (kt=3D0x0, a=3D0xfffffe000e0a4c90, af=3D2 '\002', len=
=3D48,
dir_out=3D0, op_pass=3D0, notrule=3D0)
at /usr/src/sys/contrib/pf/net/pf_table.c:2242
#1 0xffffffff8031140c in pf_test (dir=3D1, ifp=3DVariable "ifp" is not ava=
ilable.
)
at /usr/src/sys/contrib/pf/net/pf.c:7064
#2 0xffffffff80316b5b in pf_check_in (arg=3DVariable "arg" is not availabl=
e.
)
at /usr/src/sys/contrib/pf/net/pf_ioctl.c:4139
#3 0xffffffff8093965e in pfil_run_hooks (ph=3DVariable "ph" is not availab=
le.
) at /usr/src/sys/net/pfil.c:82
#4 0xffffffff809a0907 in ip_input (m=3D0xfffffe000e0a4c00)
at /usr/src/sys/netinet/ip_input.c:510
#5 0xffffffff8093892b in netisr_dispatch_src (proto=3D1, source=3DVariable=
"source" is not available.
)
at /usr/src/sys/net/netisr.c:1013
#6 0xffffffff8092dd6d in ether_demux (ifp=3D0xfffffe0003d91000, m=3Ddwarf2=
_read_address: Corrupted DWARF expression.
)
at /usr/src/sys/net/if_ethersubr.c:937
#7 0xffffffff8092e044 in ether_nh_input (m=3DVariable "m" is not available.
)
at /usr/src/sys/net/if_ethersubr.c:756
#8 0xffffffff8093892b in netisr_dispatch_src (proto=3D9, source=3DVariable=
"source" is not available.
)
at /usr/src/sys/net/netisr.c:1013
#9 0xffffffff8092dc8f in ether_demux (ifp=3D0xfffffe0002acb000, m=3Ddwarf2=
_read_address: Corrupted DWARF expression.
)
at /usr/src/sys/net/if_ethersubr.c:846
#10 0xffffffff8092e044 in ether_nh_input (m=3DVariable "m" is not available.
)
at /usr/src/sys/net/if_ethersubr.c:756
#11 0xffffffff8093892b in netisr_dispatch_src (proto=3D9, source=3DVariable=
"source" is not available.
)
at /usr/src/sys/net/netisr.c:1013
#12 0xffffffff8043f88a in bce_intr (xsc=3DVariable "xsc" is not available.
) at /usr/src/sys/dev/bce/if_bce.c:6600
#13 0xffffffff80849c74 in intr_event_execute_handlers (p=3DVariable "p" is =
not available.
)
at /usr/src/sys/kern/kern_intr.c:1257
#14 0xffffffff8084b434 in ithread_loop (arg=3D0xfffffe0002b0dc00)
at /usr/src/sys/kern/kern_intr.c:1270
#15 0xffffffff808468cf in fork_exit (
callout=3D0xffffffff8084b390 <ithread_loop>, arg=3D0xfffffe0002b0dc00,
frame=3D0xffffff80f6d19c50) at /usr/src/sys/kern/kern_fork.c:995
#16 0xffffffff80b5fd6e in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:602
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000001 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
=2D--Type <return> to continue, or q <return> to quit---
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0xffffffff81192900 in affinity ()
#42 0xfffffe0002ac7000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0xfffffe0002ac7000 in ?? ()
#45 0xffffff80f6d19b40 in ?? ()
#46 0xffffff80f6d19ae8 in ?? ()
#47 0xfffffe0002907460 in ?? ()
#48 0xffffffff8089c3d2 in sched_switch (td=3D0xffffffff8084b390,
newtd=3D0xfffffe0002b0dc00, flags=3Ddwarf2_read_address: Corrupted DWAR=
=46 expression.
) at /usr/src/sys/kern/sched_ule.c:1848
Previous frame inner to this frame (corrupt stack?)
(kgdb) info args
kt =3D (struct pfr_ktable *) 0x0
a =3D (struct pf_addr *) 0xfffffe000e0a4c90
af =3D 2 '\002'
len =3D 48
dir_out =3D 0
op_pass =3D 0
notrule =3D 0
(kgdb) p *a
$1 =3D {pfa =3D {v4 =3D {s_addr =3D 3414615954}, v6 =3D {__u6_addr =3D {
__u6_addr8 =3D "\222=E7\206=CB=BC&\000\031=C9U=FF=EF\000\000\000", =
__u6_addr16 =3D {
59282, 52102, 9916, 6400, 21961, 61439, 0, 0}, __u6_addr32 =3D {
3414615954, 419440316, 4026488265, 0}}},
addr8 =3D "\222=E7\206=CB=BC&\000\031=C9U=FF=EF\000\000\000", addr16 =
=3D {59282, 52102,
9916, 6400, 21961, 61439, 0, 0}, addr32 =3D {3414615954, 419440316,
4026488265, 0}}}
(kgdb) up
#1 0xffffffff8031140c in pf_test (dir=3D1, ifp=3DVariable "ifp" is not ava=
ilable.
)
at /usr/src/sys/contrib/pf/net/pf.c:7064
7064 pfr_update_stats(tr->dst.addr.p.tbl,
(kgdb) info args
dir =3D 1
ifp =3D Variable "ifp" is not available.
(kgdb) p tr
Variable "tr" is not available.
(kgdb) p nr
Variable "nr" is not available.
(kgdb) p s
$2 =3D (struct pf_state *) 0x0
(kgdb) p pd.nat_rule
$3 =3D (struct pf_rule *) 0x0
(kgdb) p r
$4 =3D (struct pf_rule *) 0xfffffe000f2593a8
(kgdb) p *r
$5 =3D {src =3D {addr =3D {v =3D {a =3D {addr =3D {pfa =3D {v4 =3D {s_addr =
=3D 0}, v6 =3D {
__u6_addr =3D {__u6_addr8 =3D '\0' <repeats 15 times>,
__u6_addr16 =3D {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D=
{0, 0,
0, 0}}}, addr8 =3D '\0' <repeats 15 times>, addr16 =3D =
{0, 0,
0, 0, 0, 0, 0, 0}, addr32 =3D {0, 0, 0, 0}}}, mask =3D {pfa=
=3D {
v4 =3D {s_addr =3D 0}, v6 =3D {__u6_addr =3D {
__u6_addr8 =3D '\0' <repeats 15 times>, __u6_addr16 =3D {=
0, 0,
0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}},
addr8 =3D '\0' <repeats 15 times>, addr16 =3D {0, 0, 0, 0, 0,=
0, 0,
0}, addr32 =3D {0, 0, 0, 0}}}},
ifname =3D '\0' <repeats 15 times>, tblname =3D '\0' <repeats 31 ti=
mes>,
rtlabelname =3D '\0' <repeats 31 times>, rtlabel =3D 0}, p =3D {dyn=
=3D 0x0,
tbl =3D 0x0, dyncnt =3D 0, tblcnt =3D 0}, type =3D 0 '\0', iflags =
=3D 0 '\0'},
port =3D {0, 0}, neg =3D 0 '\0', port_op =3D 0 '\0'}, dst =3D {addr =3D=
{v =3D {a =3D {
addr =3D {pfa =3D {v4 =3D {s_addr =3D 1970168173}, v6 =3D {__u6_a=
ddr =3D {
__u6_addr8 =3D "manualblock\000\000\000\000", __u6_addr16=
=3D {
24941, 30062, 27745, 27746, 25455, 107, 0, 0},
__u6_addr32 =3D {1970168173, 1818389601, 7037807, 0}}},
addr8 =3D "manualblock\000\000\000\000", addr16 =3D {24941, 3=
0062,
27745, 27746, 25455, 107, 0, 0}, addr32 =3D {1970168173,
1818389601, 7037807, 0}}}, mask =3D {pfa =3D {v4 =3D {s_add=
r =3D 0},
v6 =3D {__u6_addr =3D {__u6_addr8 =3D '\0' <repeats 15 times>,
__u6_addr16 =3D {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D=
{0, 0,
0, 0}}}, addr8 =3D '\0' <repeats 15 times>, addr16 =3D =
{0, 0,
0, 0, 0, 0, 0, 0}, addr32 =3D {0, 0, 0, 0}}}},
ifname =3D "manualblock\000\000\000\000",
tblname =3D "manualblock", '\0' <repeats 20 times>,
rtlabelname =3D "manualblock", '\0' <repeats 20 times>,
rtlabel =3D 1970168173}, p =3D {dyn =3D 0x0, tbl =3D 0x0, dyncnt =
=3D 0,
tblcnt =3D 0}, type =3D 3 '\003', iflags =3D 0 '\0'}, port =3D {0, =
0},
neg =3D 0 '\0', port_op =3D 0 '\0'}, skip =3D {{ptr =3D 0xfffffe000f105=
000,
nr =3D 252727296}, {ptr =3D 0xfffffe000e9573a8, nr =3D 244675496}, {
ptr =3D 0xfffffe000e988af8, nr =3D 244878072}, {ptr =3D 0xfffffe000e9=
88af8,
nr =3D 244878072}, {ptr =3D 0xfffffe000e988af8, nr =3D 244878072}, {
ptr =3D 0xfffffe0098924750, nr =3D 2559723344}, {ptr =3D 0xfffffe000e=
988af8,
nr =3D 244878072}, {ptr =3D 0xfffffe000e988af8, nr =3D 244878072}},
label =3D '\0' <repeats 63 times>,
ifname =3D "tenet0\000\000\000\000\000\000\000\000\000",
qname =3D '\0' <repeats 63 times>, pqname =3D '\0' <repeats 63 times>,
tagname =3D '\0' <repeats 63 times>, match_tagname =3D '\0' <repeats 63 t=
imes>,
overload_tblname =3D '\0' <repeats 31 times>, entries =3D {
tqe_next =3D 0xfffffe000f105000, tqe_prev =3D 0x0}, rpool =3D {list =3D=
{
tqh_first =3D 0x0, tqh_last =3D 0xfffffe000f2595d8}, cur =3D 0x0, key=
=3D {
pfk =3D {key8 =3D '\0' <repeats 15 times>, key16 =3D {0, 0, 0, 0, 0, =
0, 0, 0},
key32 =3D {0, 0, 0, 0}}}, counter =3D {pfa =3D {v4 =3D {s_addr =3D =
0}, v6 =3D {
__u6_addr =3D {__u6_addr8 =3D '\0' <repeats 15 times>, __u6_addr1=
6 =3D {0,
0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}},
addr8 =3D '\0' <repeats 15 times>, addr16 =3D {0, 0, 0, 0, 0, 0, 0,=
0},
=2D--Type <return> to continue, or q <return> to quit---
addr32 =3D {0, 0, 0, 0}}}, tblidx =3D 0, proxy_port =3D {0, 0},
port_op =3D 0 '\0', opts =3D 0 '\0'}, evaluations =3D 123840, packets =
=3D {363,
0}, bytes =3D {29033, 0}, kif =3D 0xfffffe000e0b3e00, anchor =3D 0x0,
overload_tbl =3D 0x0, os_fingerprint =3D 0, rtableid =3D -1, timeout =3D {
0 <repeats 20 times>}, states_cur =3D 0, states_tot =3D 0, max_states =
=3D 0,
src_nodes =3D 0, max_src_nodes =3D 0, max_src_states =3D 0, spare1 =3D 0,
max_src_conn =3D 0, max_src_conn_rate =3D {limit =3D 0, seconds =3D 0}, q=
id =3D 0,
pqid =3D 0, rt_listid =3D 0, nr =3D 4294967295, prob =3D 0, cuid =3D 0, c=
pid =3D 38081,
return_icmp =3D 771, return_icmp6 =3D 260, max_mss =3D 0, tag =3D 0, matc=
h_tag =3D 0,
spare2 =3D 0, uid =3D {uid =3D {0, 0}, op =3D 0 '\0'}, gid =3D {gid =3D {=
0, 0},
op =3D 0 '\0'}, rule_flag =3D 8, action =3D 1 '\001', direction =3D 1 '=
\001',
log =3D 1 '\001', logif =3D 0 '\0', quick =3D 1 '\001', ifnot =3D 0 '\0',
match_tag_not =3D 0 '\0', natpass =3D 0 '\0', keep_state =3D 0 '\0', af =
=3D 0 '\0',
proto =3D 0 '\0', type =3D 0 '\0', code =3D 0 '\0', flags =3D 0 '\0',
flagset =3D 0 '\0', min_ttl =3D 0 '\0', allow_opts =3D 0 '\0', rt =3D 0 '=
\0',
return_ttl =3D 0 '\0', tos =3D 0 '\0', set_tos =3D 0 '\0',
anchor_relative =3D 0 '\0', anchor_wildcard =3D 0 '\0', flush =3D 0 '\0',
divert =3D {addr =3D {pfa =3D {v4 =3D {s_addr =3D 0}, v6 =3D {__u6_addr =
=3D {
__u6_addr8 =3D '\0' <repeats 15 times>, __u6_addr16 =3D {0, 0, =
0, 0,
0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}},
addr8 =3D '\0' <repeats 15 times>, addr16 =3D {0, 0, 0, 0, 0, 0, 0,=
0},
addr32 =3D {0, 0, 0, 0}}}, port =3D 0}}
(kgdb) p r->dst.addr.p.tbl
$6 =3D (struct pfr_ktable *) 0x0
(kgdb) p r->dst.addr.p
$7 =3D {dyn =3D 0x0, tbl =3D 0x0, dyncnt =3D 0, tblcnt =3D 0}
(kgdb) p r->dst.addr
$8 =3D {v =3D {a =3D {addr =3D {pfa =3D {v4 =3D {s_addr =3D 1970168173}, v6=
=3D {__u6_addr =3D {
__u6_addr8 =3D "manualblock\000\000\000\000", __u6_addr16 =3D=
{
24941, 30062, 27745, 27746, 25455, 107, 0, 0}, __u6_addr32 =
=3D {
1970168173, 1818389601, 7037807, 0}}},
addr8 =3D "manualblock\000\000\000\000", addr16 =3D {24941, 30062,
27745, 27746, 25455, 107, 0, 0}, addr32 =3D {1970168173, 181838=
9601,
7037807, 0}}}, mask =3D {pfa =3D {v4 =3D {s_addr =3D 0}, v6 =3D=
{
__u6_addr =3D {__u6_addr8 =3D '\0' <repeats 15 times>, __u6_add=
r16 =3D {
0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}},
addr8 =3D '\0' <repeats 15 times>, addr16 =3D {0, 0, 0, 0, 0, 0, =
0, 0},
addr32 =3D {0, 0, 0, 0}}}}, ifname =3D "manualblock\000\000\000\0=
00",
tblname =3D "manualblock", '\0' <repeats 20 times>,
rtlabelname =3D "manualblock", '\0' <repeats 20 times>,
rtlabel =3D 1970168173}, p =3D {dyn =3D 0x0, tbl =3D 0x0, dyncnt =3D 0,=
tblcnt =3D 0},
type =3D 3 '\003', iflags =3D 0 '\0'}
--Boundary-00=_iVWIP9/oBMJI11C--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201201261638.26599.d.sieborger>