Date: Tue, 6 Sep 2011 01:45:56 GMT From: Nikos Vassiliadis <nvass@gmx.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/160496: [patch] kernel panic with pf + VIMAGE Message-ID: <201109060145.p861juxO096933@red.freebsd.org> Resent-Message-ID: <201109060150.p861o14A083645@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 160496
>Category: kern
>Synopsis: [patch] kernel panic with pf + VIMAGE
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Sep 06 01:50:01 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator: Nikos Vassiliadis
>Release: 9.0-CURRENT
>Organization:
>Environment:
FreeBSD lab.local 9.0-BETA2 FreeBSD 9.0-BETA2 #77 r225405: Tue Sep 6 01:25:42 EEST 2011 root@lab.local:/usr/obj/usr/src/sys/LAB i386
>Description:
When multiple instances of pf are used on an "options VIMAGE" kernel,
a kernel panic is quickly triggered. It happens when pf is trying to
remove a state from its state table. Two indicative backtraces:
#9 0xc0a127a4 in panic (fmt=0xc404ec26 "Bad link elm %p next->prev != elm")
at /usr/src/sys/kern/kern_shutdown.c:587
#10 0xc4027099 in pf_free_state (cur=Variable "cur" is not available.
)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:1655
#11 0xc4027110 in pf_purge_expired_states (maxcheck=1, waslocked=0)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:1727
#12 0xc40285d0 in pf_purge_thread (v=0xc35a85a0)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:1370
#13 0xc09e5818 in fork_exit (callout=0xc4028460 <pf_purge_thread>,
arg=0xc35a85a0, frame=0xcd69ed28) at /usr/src/sys/kern/kern_fork.c:1025
#14 0xc0d72574 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:275
(kgdb)
and
#8 0xc0d724fc in calltrap () at /usr/src/sys/i386/i386/exception.s:168
#9 0xc40e776a in pf_state_tree_id_RB_REMOVE_COLOR (head=0xc4081148,
parent=0x0, elm=0xc4183198)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:474
#10 0xc40e7aa0 in pf_state_tree_id_RB_REMOVE (head=0xc4081148, elm=0xc417ce58)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:474
#11 0xc40ec83e in pf_unlink_state (cur=0xc417ce58)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:1592
#12 0xc40ed15a in pf_purge_expired_states (maxcheck=429496730, waslocked=0)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:1717
#13 0xc40ee5d0 in pf_purge_thread (v=0xc35a8a00)
at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:1370
#14 0xc09e5818 in fork_exit (callout=0xc40ee460 <pf_purge_thread>,
arg=0xc35a8a00, frame=0xcd6ddd28) at /usr/src/sys/kern/kern_fork.c:1025
#15 0xc0d72574 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:275
(kgdb)
Sometimes the state table corruption is observable using
"vmstat -z":
lab# vmstat -z | grep pfstate
pfstatepl: 204, 10013,18446744073709551615, 1, 0, 0, 0
pfstatekeypl: 204, 0,18446744073709551615, 1, 0, 0, 0
pfstateitempl: 204, 0,18446744073709551615, 1, 0, 0, 0
pfstatescrub: 28, 0, 0, 0, 0, 0, 0
pfstatepl: 204, 10013,18446744073709551615, 1, 0, 0, 0
pfstatekeypl: 204, 0,18446744073709551615, 1, 0, 0, 0
pfstateitempl: 204, 0,18446744073709551615, 1, 0, 0, 0
pfstatescrub: 28, 0, 0, 0, 0, 0, 0
pfstatepl: 204, 10013, 4, 34, 4, 0, 0
pfstatekeypl: 204, 0, 4, 34, 4, 0, 0
pfstateitempl: 204, 0, 4, 34, 4, 0, 0
pfstatescrub: 28, 0, 0, 0, 0, 0, 0
pfstatepl: 204, 10013, 8, 30, 8, 0, 0
pfstatekeypl: 204, 0, 8, 30, 8, 0, 0
pfstateitempl: 204, 0, 8, 30, 8, 0, 0
pfstatescrub: 28, 0, 0, 0, 0, 0, 0
lab#
>How-To-Repeat:
build a kernel with the VIMAGE option
create a few vnet jails
kldload pf
enable pf on all jails
create a very basic ruleset like "pass out all\npass in all" on all jails
create some IP traffic that will create pf states
the kernel will soon panic on a state expiration
>Fix:
There is a static non virtualized variable that causes the problem
in pf_purge_expired_states():/sys/contrib/pf/net/pf.c. This should be
virtualized in an "option VIMAGE" kernel. See the attached patch.
Patch attached with submission follows:
Index: sys/contrib/pf/net/pf.c
===================================================================
--- sys/contrib/pf/net/pf.c (revision 225405)
+++ sys/contrib/pf/net/pf.c (working copy)
@@ -1677,30 +1677,41 @@
pf_purge_expired_states(u_int32_t maxcheck)
#endif
{
- static struct pf_state *cur = NULL;
struct pf_state *next;
#ifdef __FreeBSD__
+ static VNET_DEFINE(struct pf_state *, cur) = NULL;
int locked = waslocked;
#else
+ static struct pf_state *cur = NULL;
int locked = 0;
#endif
while (maxcheck--) {
/* wrap to start of list when we hit the end */
- if (cur == NULL) {
#ifdef __FreeBSD__
- cur = TAILQ_FIRST(&V_state_list);
+ if (VNET(cur) == NULL) {
+ VNET(cur) = TAILQ_FIRST(&V_state_list);
+ if (VNET(cur) == NULL)
#else
+ if (cur == NULL) {
cur = TAILQ_FIRST(&state_list);
+ if (cur == NULL)
#endif
- if (cur == NULL)
break; /* list empty */
}
/* get next state, as cur may get deleted */
+#ifdef __FreeBSD__
+ next = TAILQ_NEXT(VNET(cur), entry_list);
+#else
next = TAILQ_NEXT(cur, entry_list);
+#endif
+#ifdef __FreeBSD__
+ if (VNET(cur)->timeout == PFTM_UNLINKED) {
+#else
if (cur->timeout == PFTM_UNLINKED) {
+#endif
/* free unlinked state */
if (! locked) {
#ifdef __FreeBSD__
@@ -1711,10 +1722,17 @@
#endif
locked = 1;
}
+#ifdef __FreeBSD__
+ pf_free_state(VNET(cur));
+ } else if (pf_state_expires(VNET(cur)) <= time_second) {
+ /* unlink and free expired state */
+ pf_unlink_state(VNET(cur));
+#else
pf_free_state(cur);
} else if (pf_state_expires(cur) <= time_second) {
/* unlink and free expired state */
pf_unlink_state(cur);
+#endif
if (! locked) {
#ifdef __FreeBSD__
if (!sx_try_upgrade(&V_pf_consistency_lock))
@@ -1724,9 +1742,15 @@
#endif
locked = 1;
}
+#ifdef __FreeBSD__
+ pf_free_state(VNET(cur));
+ }
+ VNET(cur) = next;
+#else
pf_free_state(cur);
}
cur = next;
+#endif
}
#ifdef __FreeBSD__
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201109060145.p861juxO096933>