Date: Wed, 14 Feb 2018 12:47:02 +0000 From: Joe Jones <joe@stream-technologies.com> To: freebsd-pf@freebsd.org Subject: Kernel Panic Message-ID: <5A842FC6.7020806@stream-technologies.com>
next in thread | raw e-mail | index | archive | help
Hi,
we are running test traffic through our system, after between 1 and 12
hours we get a kernel panic, always in the pfr_pool_get function in
/usr/src/sys/netpfil/pf/pf_table.c line 2140. After a bit of
investigation I confirmed that ke2 is set to null on line 2122.
We have tried a kernel compiled from /base/releng/11.1 and the
equivalent branch of pfsense and get the same thing.
The network card is a 4 port Intel I350 (igb). The test uses 2 of the 4
ports, msix with 2 queues per port.
net.link.ifqmaxlen="2048"
hw.igb.rx_process_limit="-1"
hw.igb.num_queues="2"
Next I'm going to start stripping parts out of our setup to try and get
to a system that is stable under the test load.
Can anyone shed any light on what is happening here.
Thanks
Joe Jones
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
current process = 12 (irq272: igb1:que 1)
trap number = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff80aadb97 at kdb_backtrace+0x67
#1 0xffffffff80a6bbf6 at vpanic+0x186
#2 0xffffffff80a6ba63 at panic+0x43
#3 0xffffffff80ee0832 at trap_fatal+0x322
#4 0xffffffff80ee0889 at trap_pfault+0x49
#5 0xffffffff80ee00c6 at trap+0x286
#6 0xffffffff80ec4d41 at calltrap+0x8
#7 0xffffffff824c65df at pf_map_addr+0x58f
#8 0xffffffff824c6a11 at pf_get_translation+0x1e1
#9 0xffffffff824bade1 at pf_test_rule+0x241
#10 0xffffffff824b78bb at pf_test+0x174b
#11 0xffffffff824c961d at pf_check_out+0x1d
#12 0xffffffff80b71f4b at pfil_run_hooks+0x7b
#13 0xffffffff80bcdb71 at ip_tryforward+0x1f1
#14 0xffffffff80bcfb88 at ip_input+0x368
#15 0xffffffff80b71140 at netisr_dispatch_src+0xa0
#16 0xffffffff80b5ac0d at ether_demux+0x16d
#17 0xffffffff80b5b89b at ether_nh_input+0x31b
Uptime: 4h17m34s
Dumping 2778 out of 65374
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/carp.ko...Reading symbols from
/usr/lib/debug//boot/kernel/carp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/carp.ko
Reading symbols from /boot/kernel/pfsync.ko...Reading symbols from
/usr/lib/debug//boot/kernel/pfsync.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pfsync.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from
/usr/lib/debug//boot/kernel/pf.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/modules/mellynat.ko...done.
Loaded symbols for /boot/modules/mellynat.ko
Reading symbols from /boot/modules/mellycount.ko...done.
Loaded symbols for /boot/modules/mellycount.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from
/usr/lib/debug//boot/kernel/pflog.ko.debug...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
/usr/lib/debug//boot/kernel/uhid.ko.debug...done.
done.
Loaded symbols for /boot/kernel/uhid.ko
#0 doadump (textdump=<value optimized out>) at pcpu.h:222
222 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) backtrace
#0 doadump (textdump=<value optimized out>) at pcpu.h:222
#1 0xffffffff80a6b771 in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80a6bc30 in vpanic (fmt=<value optimized out>, ap=<value
optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80a6ba63 in panic (fmt=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff80ee0832 in trap_fatal (frame=0xfffffe000031ac40, eva=153)
at /usr/src/sys/amd64/amd64/trap.c:801
#5 0xffffffff80ee0889 in trap_pfault (frame=0xfffffe000031ac40,
usermode=0) at pcpu.h:222
#6 0xffffffff80ee00c6 in trap (frame=0xfffffe000031ac40) at
/usr/src/sys/amd64/amd64/trap.c:421
#7 0xffffffff80ec4d41 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
#8 0xffffffff824dba80 in pfr_pool_get (kt=0xfffff80010e17000,
pidx=<value optimized out>, counter=0xfffff8000e925298, af=<value
optimized out>) at /usr/src/sys/netpfil/pf/pf_table.c:2141
#9 0xffffffff824c65df in pf_map_addr (af=<value optimized out>,
r=<value optimized out>, saddr=<value optimized out>, naddr=<value
optimized out>, init_addr=<value optimized out>, sn=<value optimized
out>) at uma.h:363
#10 0xffffffff824c6a11 in pf_get_translation (pd=<value optimized out>,
m=<value optimized out>, off=<value optimized out>, direction=<value
optimized out>, kif=<value optimized out>, sn=0xfffffe000031b248,
skp=<value optimized out>,
nkp=<value optimized out>, saddr=<value optimized out>,
daddr=<value optimized out>, sport=<value optimized out>, dport=<value
optimized out>) at /usr/src/sys/netpfil/pf/pf_lb.c:223
#11 0xffffffff824bade1 in pf_test_rule () at
/usr/src/sys/netpfil/pf/pf.c:3310
#12 0xffffffff824b78bb in pf_test (dir=<value optimized out>, ifp=<value
optimized out>, m0=<value optimized out>, inp=<value optimized out>) at
/usr/src/sys/netpfil/pf/pf.c:6280
#13 0xffffffff824c961d in pf_check_out (arg=<value optimized out>,
m=0xfffffe000031b6a0, ifp=<value optimized out>, dir=<value optimized
out>, inp=<value optimized out>) at /usr/src/sys/netpfil/pf/pf_ioctl.c:3627
#14 0xffffffff80b71f4b in pfil_run_hooks (ph=0xffffffff81dca688,
mp=0xfffffe000031b700, ifp=0xfffff8000e7fe000, dir=2, inp=0x0) at
/usr/src/sys/net/pfil.c:109
#15 0xffffffff80bcdb71 in ip_tryforward (m=0xfffff805f1da2800) at
/usr/src/sys/netinet/ip_fastfwd.c:306
#16 0xffffffff80bcfb88 in ip_input (m=0xfffff805f1da2800) at
/usr/src/sys/netinet/ip_input.c:573
#17 0xffffffff80b71140 in netisr_dispatch_src (proto=1, source=<value
optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1120
#18 0xffffffff80b5ac0d in ether_demux (ifp=0xfffff8000e2f8800, m=<value
optimized out>) at /usr/src/sys/net/if_ethersubr.c:860
#19 0xffffffff80b5b89b in ether_nh_input (m=<value optimized out>) at
/usr/src/sys/net/if_ethersubr.c:641
#20 0xffffffff80b71140 in netisr_dispatch_src (proto=5, source=<value
optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1120
#21 0xffffffff80b5aea6 in ether_input (ifp=<value optimized out>, m=0x0)
at /usr/src/sys/net/if_ethersubr.c:764
#22 0xffffffff80541558 in igb_rxeof (count=-29544262) at
/usr/src/sys/dev/e1000/if_igb.c:4957
#23 0xffffffff8054091f in igb_msix_que (arg=0xfffff8000e6b2c68) at
/usr/src/sys/dev/e1000/if_igb.c:1612
#24 0xffffffff80a3223c in intr_event_execute_handlers (p=<value
optimized out>, ie=0xfffff8000e6b2e00) at /usr/src/sys/kern/kern_intr.c:1262
#25 0xffffffff80a32526 in ithread_loop (arg=0xfffff8000e7881c0) at
/usr/src/sys/kern/kern_intr.c:1275
#26 0xffffffff80a2f895 in fork_exit (callout=0xffffffff80a32450
<ithread_loop>, arg=0xfffff8000e7881c0, frame=0xfffffe000031bac0) at
/usr/src/sys/kern/kern_fork.c:1042
#27 0xffffffff80ec527e in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:611
#28 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5A842FC6.7020806>