From owner-freebsd-pf@FreeBSD.ORG Thu Jan 26 14:38:33 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57CFD1065672 for ; Thu, 26 Jan 2012 14:38:33 +0000 (UTC) (envelope-from d.sieborger@ru.ac.za) Received: from mail.ru.ac.za (mail.ru.ac.za [IPv6:2001:4200:1010:0:250:56ff:fe8d:5]) by mx1.freebsd.org (Postfix) with ESMTP id EA2008FC08 for ; Thu, 26 Jan 2012 14:38:30 +0000 (UTC) Received: from damar.ru.ac.za ([2001:4200:1010:1058:219:d1ff:fe9f:726a]:47429) by mail.ru.ac.za with esmtp (Exim 4.76 (FreeBSD)) (envelope-from ) id 1RqQT5-000Pr0-Ht for freebsd-pf@freebsd.org; Thu, 26 Jan 2012 16:38:27 +0200 Received: by damar.ru.ac.za (Postfix, from userid 1001) id 698C839EE; Thu, 26 Jan 2012 16:38:27 +0200 (SAST) From: David =?iso-8859-1?q?Sieb=F6rger?= To: freebsd-pf@freebsd.org Date: Thu, 26 Jan 2012 16:38:26 +0200 User-Agent: KMail/1.13.7 (FreeBSD/8.2-RELEASE-p3; KDE/4.7.3; amd64; ; ) Organization: Rhodes University MIME-Version: 1.0 Content-Type: Multipart/Mixed; boundary="Boundary-00=_iVWIP9/oBMJI11C" Message-Id: <201201261638.26599.d.sieborger@ru.ac.za> X-Virus-Scanned: mail.ru.ac.za (2001:4200:1010:0:250:56ff:fe8d:5) Subject: pf crashes in pfr_update_stats() X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jan 2012 14:38:33 -0000 --Boundary-00=_iVWIP9/oBMJI11C Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, I have a pair of FreeBSD 9.0-RELEASE firewalls which are crashing=20 repeatedly. I've been able to connect to one of them with remote kgdb=20 after it crashed (see kgdb session attached), but I haven't been able to=20 get to the bottom of what's wrong. Is anyone able to shed more light on=20 this? The first problem that I see is that the kt argument to=20 pfr_update_stats() is null, so the kernel panics as soon as that's=20 dereferenced. Where pfr_update_stats() is called from pf_test(), kgdb tells me that=20 "Variable "tr" is not available." (Is that because of a gcc=20 optimisation?) But, tr ought to equal r in this instance, and r is=20 available, so I looked at r. r->dst.addr.p.tbl is indeed null. Does anyone have any theories about why that could be the case, or=20 anything else that I could do to debug this? I can provide more=20 configuration information if needed. =2D-=20 David Sieb=F6rger System Administrator, IT Division, Rhodes University --Boundary-00=_iVWIP9/oBMJI11C Content-Type: text/plain; charset="ISO-8859-1"; name="kgdb2.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="kgdb2.txt" # kgdb -r /dev/cuau0 /usr/obj/usr/src/sys/FIREWALL/kernel.debug GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Switching to remote protocol pfr_update_stats (kt=3D0x0, a=3D0xfffffe000e0a4c90, af=3D2 '\002', len=3D48, dir_out=3D0, op_pass=3D0, notrule=3D0) at /usr/src/sys/contrib/pf/net/pf_table.c:2242 2242 if (!(kt->pfrkt_flags & PFR_TFLAG_ACTIVE) && kt->pfrkt_root= !=3D NULL) (kgdb) where #0 pfr_update_stats (kt=3D0x0, a=3D0xfffffe000e0a4c90, af=3D2 '\002', len= =3D48, dir_out=3D0, op_pass=3D0, notrule=3D0) at /usr/src/sys/contrib/pf/net/pf_table.c:2242 #1 0xffffffff8031140c in pf_test (dir=3D1, ifp=3DVariable "ifp" is not ava= ilable. ) at /usr/src/sys/contrib/pf/net/pf.c:7064 #2 0xffffffff80316b5b in pf_check_in (arg=3DVariable "arg" is not availabl= e. ) at /usr/src/sys/contrib/pf/net/pf_ioctl.c:4139 #3 0xffffffff8093965e in pfil_run_hooks (ph=3DVariable "ph" is not availab= le. ) at /usr/src/sys/net/pfil.c:82 #4 0xffffffff809a0907 in ip_input (m=3D0xfffffe000e0a4c00) at /usr/src/sys/netinet/ip_input.c:510 #5 0xffffffff8093892b in netisr_dispatch_src (proto=3D1, source=3DVariable= "source" is not available. ) at /usr/src/sys/net/netisr.c:1013 #6 0xffffffff8092dd6d in ether_demux (ifp=3D0xfffffe0003d91000, m=3Ddwarf2= _read_address: Corrupted DWARF expression. ) at /usr/src/sys/net/if_ethersubr.c:937 #7 0xffffffff8092e044 in ether_nh_input (m=3DVariable "m" is not available. ) at /usr/src/sys/net/if_ethersubr.c:756 #8 0xffffffff8093892b in netisr_dispatch_src (proto=3D9, source=3DVariable= "source" is not available. ) at /usr/src/sys/net/netisr.c:1013 #9 0xffffffff8092dc8f in ether_demux (ifp=3D0xfffffe0002acb000, m=3Ddwarf2= _read_address: Corrupted DWARF expression. ) at /usr/src/sys/net/if_ethersubr.c:846 #10 0xffffffff8092e044 in ether_nh_input (m=3DVariable "m" is not available. ) at /usr/src/sys/net/if_ethersubr.c:756 #11 0xffffffff8093892b in netisr_dispatch_src (proto=3D9, source=3DVariable= "source" is not available. ) at /usr/src/sys/net/netisr.c:1013 #12 0xffffffff8043f88a in bce_intr (xsc=3DVariable "xsc" is not available. ) at /usr/src/sys/dev/bce/if_bce.c:6600 #13 0xffffffff80849c74 in intr_event_execute_handlers (p=3DVariable "p" is = not available. ) at /usr/src/sys/kern/kern_intr.c:1257 #14 0xffffffff8084b434 in ithread_loop (arg=3D0xfffffe0002b0dc00) at /usr/src/sys/kern/kern_intr.c:1270 #15 0xffffffff808468cf in fork_exit ( callout=3D0xffffffff8084b390 , arg=3D0xfffffe0002b0dc00, frame=3D0xffffff80f6d19c50) at /usr/src/sys/kern/kern_fork.c:995 #16 0xffffffff80b5fd6e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602 #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000001 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () =2D--Type to continue, or q to quit--- #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0xffffffff81192900 in affinity () #42 0xfffffe0002ac7000 in ?? () #43 0x0000000000000000 in ?? () #44 0xfffffe0002ac7000 in ?? () #45 0xffffff80f6d19b40 in ?? () #46 0xffffff80f6d19ae8 in ?? () #47 0xfffffe0002907460 in ?? () #48 0xffffffff8089c3d2 in sched_switch (td=3D0xffffffff8084b390, newtd=3D0xfffffe0002b0dc00, flags=3Ddwarf2_read_address: Corrupted DWAR= =46 expression. ) at /usr/src/sys/kern/sched_ule.c:1848 Previous frame inner to this frame (corrupt stack?) (kgdb) info args kt =3D (struct pfr_ktable *) 0x0 a =3D (struct pf_addr *) 0xfffffe000e0a4c90 af =3D 2 '\002' len =3D 48 dir_out =3D 0 op_pass =3D 0 notrule =3D 0 (kgdb) p *a $1 =3D {pfa =3D {v4 =3D {s_addr =3D 3414615954}, v6 =3D {__u6_addr =3D { __u6_addr8 =3D "\222=E7\206=CB=BC&\000\031=C9U=FF=EF\000\000\000", = __u6_addr16 =3D { 59282, 52102, 9916, 6400, 21961, 61439, 0, 0}, __u6_addr32 =3D { 3414615954, 419440316, 4026488265, 0}}}, addr8 =3D "\222=E7\206=CB=BC&\000\031=C9U=FF=EF\000\000\000", addr16 = =3D {59282, 52102, 9916, 6400, 21961, 61439, 0, 0}, addr32 =3D {3414615954, 419440316, 4026488265, 0}}} (kgdb) up #1 0xffffffff8031140c in pf_test (dir=3D1, ifp=3DVariable "ifp" is not ava= ilable. ) at /usr/src/sys/contrib/pf/net/pf.c:7064 7064 pfr_update_stats(tr->dst.addr.p.tbl, (kgdb) info args dir =3D 1 ifp =3D Variable "ifp" is not available. (kgdb) p tr Variable "tr" is not available. (kgdb) p nr Variable "nr" is not available. (kgdb) p s $2 =3D (struct pf_state *) 0x0 (kgdb) p pd.nat_rule $3 =3D (struct pf_rule *) 0x0 (kgdb) p r $4 =3D (struct pf_rule *) 0xfffffe000f2593a8 (kgdb) p *r $5 =3D {src =3D {addr =3D {v =3D {a =3D {addr =3D {pfa =3D {v4 =3D {s_addr = =3D 0}, v6 =3D { __u6_addr =3D {__u6_addr8 =3D '\0' , __u6_addr16 =3D {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D= {0, 0, 0, 0}}}, addr8 =3D '\0' , addr16 =3D = {0, 0, 0, 0, 0, 0, 0, 0}, addr32 =3D {0, 0, 0, 0}}}, mask =3D {pfa= =3D { v4 =3D {s_addr =3D 0}, v6 =3D {__u6_addr =3D { __u6_addr8 =3D '\0' , __u6_addr16 =3D {= 0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}, addr8 =3D '\0' , addr16 =3D {0, 0, 0, 0, 0,= 0, 0, 0}, addr32 =3D {0, 0, 0, 0}}}}, ifname =3D '\0' , tblname =3D '\0' , rtlabelname =3D '\0' , rtlabel =3D 0}, p =3D {dyn= =3D 0x0, tbl =3D 0x0, dyncnt =3D 0, tblcnt =3D 0}, type =3D 0 '\0', iflags = =3D 0 '\0'}, port =3D {0, 0}, neg =3D 0 '\0', port_op =3D 0 '\0'}, dst =3D {addr =3D= {v =3D {a =3D { addr =3D {pfa =3D {v4 =3D {s_addr =3D 1970168173}, v6 =3D {__u6_a= ddr =3D { __u6_addr8 =3D "manualblock\000\000\000\000", __u6_addr16= =3D { 24941, 30062, 27745, 27746, 25455, 107, 0, 0}, __u6_addr32 =3D {1970168173, 1818389601, 7037807, 0}}}, addr8 =3D "manualblock\000\000\000\000", addr16 =3D {24941, 3= 0062, 27745, 27746, 25455, 107, 0, 0}, addr32 =3D {1970168173, 1818389601, 7037807, 0}}}, mask =3D {pfa =3D {v4 =3D {s_add= r =3D 0}, v6 =3D {__u6_addr =3D {__u6_addr8 =3D '\0' , __u6_addr16 =3D {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D= {0, 0, 0, 0}}}, addr8 =3D '\0' , addr16 =3D = {0, 0, 0, 0, 0, 0, 0, 0}, addr32 =3D {0, 0, 0, 0}}}}, ifname =3D "manualblock\000\000\000\000", tblname =3D "manualblock", '\0' , rtlabelname =3D "manualblock", '\0' , rtlabel =3D 1970168173}, p =3D {dyn =3D 0x0, tbl =3D 0x0, dyncnt = =3D 0, tblcnt =3D 0}, type =3D 3 '\003', iflags =3D 0 '\0'}, port =3D {0, = 0}, neg =3D 0 '\0', port_op =3D 0 '\0'}, skip =3D {{ptr =3D 0xfffffe000f105= 000, nr =3D 252727296}, {ptr =3D 0xfffffe000e9573a8, nr =3D 244675496}, { ptr =3D 0xfffffe000e988af8, nr =3D 244878072}, {ptr =3D 0xfffffe000e9= 88af8, nr =3D 244878072}, {ptr =3D 0xfffffe000e988af8, nr =3D 244878072}, { ptr =3D 0xfffffe0098924750, nr =3D 2559723344}, {ptr =3D 0xfffffe000e= 988af8, nr =3D 244878072}, {ptr =3D 0xfffffe000e988af8, nr =3D 244878072}}, label =3D '\0' , ifname =3D "tenet0\000\000\000\000\000\000\000\000\000", qname =3D '\0' , pqname =3D '\0' , tagname =3D '\0' , match_tagname =3D '\0' , overload_tblname =3D '\0' , entries =3D { tqe_next =3D 0xfffffe000f105000, tqe_prev =3D 0x0}, rpool =3D {list =3D= { tqh_first =3D 0x0, tqh_last =3D 0xfffffe000f2595d8}, cur =3D 0x0, key= =3D { pfk =3D {key8 =3D '\0' , key16 =3D {0, 0, 0, 0, 0, = 0, 0, 0}, key32 =3D {0, 0, 0, 0}}}, counter =3D {pfa =3D {v4 =3D {s_addr =3D = 0}, v6 =3D { __u6_addr =3D {__u6_addr8 =3D '\0' , __u6_addr1= 6 =3D {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}, addr8 =3D '\0' , addr16 =3D {0, 0, 0, 0, 0, 0, 0,= 0}, =2D--Type to continue, or q to quit--- addr32 =3D {0, 0, 0, 0}}}, tblidx =3D 0, proxy_port =3D {0, 0}, port_op =3D 0 '\0', opts =3D 0 '\0'}, evaluations =3D 123840, packets = =3D {363, 0}, bytes =3D {29033, 0}, kif =3D 0xfffffe000e0b3e00, anchor =3D 0x0, overload_tbl =3D 0x0, os_fingerprint =3D 0, rtableid =3D -1, timeout =3D { 0 }, states_cur =3D 0, states_tot =3D 0, max_states = =3D 0, src_nodes =3D 0, max_src_nodes =3D 0, max_src_states =3D 0, spare1 =3D 0, max_src_conn =3D 0, max_src_conn_rate =3D {limit =3D 0, seconds =3D 0}, q= id =3D 0, pqid =3D 0, rt_listid =3D 0, nr =3D 4294967295, prob =3D 0, cuid =3D 0, c= pid =3D 38081, return_icmp =3D 771, return_icmp6 =3D 260, max_mss =3D 0, tag =3D 0, matc= h_tag =3D 0, spare2 =3D 0, uid =3D {uid =3D {0, 0}, op =3D 0 '\0'}, gid =3D {gid =3D {= 0, 0}, op =3D 0 '\0'}, rule_flag =3D 8, action =3D 1 '\001', direction =3D 1 '= \001', log =3D 1 '\001', logif =3D 0 '\0', quick =3D 1 '\001', ifnot =3D 0 '\0', match_tag_not =3D 0 '\0', natpass =3D 0 '\0', keep_state =3D 0 '\0', af = =3D 0 '\0', proto =3D 0 '\0', type =3D 0 '\0', code =3D 0 '\0', flags =3D 0 '\0', flagset =3D 0 '\0', min_ttl =3D 0 '\0', allow_opts =3D 0 '\0', rt =3D 0 '= \0', return_ttl =3D 0 '\0', tos =3D 0 '\0', set_tos =3D 0 '\0', anchor_relative =3D 0 '\0', anchor_wildcard =3D 0 '\0', flush =3D 0 '\0', divert =3D {addr =3D {pfa =3D {v4 =3D {s_addr =3D 0}, v6 =3D {__u6_addr = =3D { __u6_addr8 =3D '\0' , __u6_addr16 =3D {0, 0, = 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}, addr8 =3D '\0' , addr16 =3D {0, 0, 0, 0, 0, 0, 0,= 0}, addr32 =3D {0, 0, 0, 0}}}, port =3D 0}} (kgdb) p r->dst.addr.p.tbl $6 =3D (struct pfr_ktable *) 0x0 (kgdb) p r->dst.addr.p $7 =3D {dyn =3D 0x0, tbl =3D 0x0, dyncnt =3D 0, tblcnt =3D 0} (kgdb) p r->dst.addr $8 =3D {v =3D {a =3D {addr =3D {pfa =3D {v4 =3D {s_addr =3D 1970168173}, v6= =3D {__u6_addr =3D { __u6_addr8 =3D "manualblock\000\000\000\000", __u6_addr16 =3D= { 24941, 30062, 27745, 27746, 25455, 107, 0, 0}, __u6_addr32 = =3D { 1970168173, 1818389601, 7037807, 0}}}, addr8 =3D "manualblock\000\000\000\000", addr16 =3D {24941, 30062, 27745, 27746, 25455, 107, 0, 0}, addr32 =3D {1970168173, 181838= 9601, 7037807, 0}}}, mask =3D {pfa =3D {v4 =3D {s_addr =3D 0}, v6 =3D= { __u6_addr =3D {__u6_addr8 =3D '\0' , __u6_add= r16 =3D { 0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}, addr8 =3D '\0' , addr16 =3D {0, 0, 0, 0, 0, 0, = 0, 0}, addr32 =3D {0, 0, 0, 0}}}}, ifname =3D "manualblock\000\000\000\0= 00", tblname =3D "manualblock", '\0' , rtlabelname =3D "manualblock", '\0' , rtlabel =3D 1970168173}, p =3D {dyn =3D 0x0, tbl =3D 0x0, dyncnt =3D 0,= tblcnt =3D 0}, type =3D 3 '\003', iflags =3D 0 '\0'} --Boundary-00=_iVWIP9/oBMJI11C--