Date: Mon, 21 May 2012 09:26:06 +0200 (CEST) From: Joerg Pulz <Joerg.Pulz@frm2.tum.de> To: FreeBSD-gnats-submit@freebsd.org Cc: freebsd-pf@freebsd.org Subject: panic when using pf and route-to (maybe: bad fragment handling?) Message-ID: <201205210726.q4L7Q6m9064258@hades.admin.frm2>
next in thread | raw e-mail | index | archive | help
>Submitter-Id: current-users
>Originator: Joerg Pulz
>Organization: TU Muenchen / FRM II
>Confidential: no
>Synopsis: panic when using pf and route-to (maybe: bad fragment handling?)
>Severity: critical
>Priority: high
>Category: kern
>Class: sw-bug
>Release: FreeBSD 9.0-RELEASE-p1
>Environment:
System: FreeBSD charon 9.0-RELEASE-p1 FreeBSD 9.0-RELEASE-p1 #3: Sun May 20 08:42:19 CEST 2012 root@charon:/usr/obj/usr/src/sys/IPSEC amd64
>Description:
We have a dual home VPN (IPSec) gateway running ipsec-tools.
All packets coming from VPN clients have to hit the main router to
generate and evaluate states. Therefor we use pf(4) and the route-to
feature.
Unfortunately the system panics (and it paniced with FreeBSD-8.x too)
at irregular intervals with:
"panic: m_copym, offset > size of mbuf chain"
I'm unable to reproduce the network traffic to force the problem to
appear.
I recompiled the Kernel and configured the system to keep dumps to find
the find the relevant place in the code for this issue.
As i'm not that deep in network processing and pf code i can only guess
that handling fragmented packets with pf(4) and route-to is buggy
somewhere.
Below is the kgdb(1) output for the last dump with some more
information.
If someone can give me some advise i can produce more and detailed
output.
If necessary i can upload all requiered stuff somewhere.
>How-To-Repeat:
>Fix:
--- kgdb.out begins here ---
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: m_copym, offset > size of mbuf chain
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x182
m_copym() at m_copym+0x280
ip_fragment() at ip_fragment+0x1e5
pf_route() at pf_route+0x75c
pf_test() at pf_test+0xc29
pf_route() at pf_route+0x30a
pf_test() at pf_test+0xc29
pf_check_out() at pf_check_out+0x3a
pfil_run_hooks() at pfil_run_hooks+0xd2
ip_output() at ip_output+0x655
ip_forward() at ip_forward+0x175
ip_input() at ip_input+0x5fd
swi_net() at swi_net+0x15a
intr_event_execute_handlers() at intr_event_execute_handlers+0x66
ithread_loop() at ithread_loop+0xaf
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
KDB: enter: panic
Dumping 649 out of 4077 MB:..3%..13%..23%..33%..42%..52%..62%..72%..82%..92%
Reading symbols from /boot/kernel.old/geom_mirror.ko...Reading symbols from /boot/kernel.old/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel.old/geom_mirror.ko
Reading symbols from /boot/kernel.old/ipmi.ko...Reading symbols from /boot/kernel.old/ipmi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel.old/ipmi.ko
#0 doadump (textdump=0) at pcpu.h:224
224 __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) up 10
#10 0xffffffff806e9079 in m_copym (m=0x0, off0=1500, len=1480, wait=1)
at /usr/src/sys/kern/uipc_mbuf.c:541
541 KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
(kgdb) list
536 KASSERT(len >= 0, ("m_copym, negative len %d", len));
537 MBUF_CHECKSLEEP(wait);
538 if (off == 0 && m->m_flags & M_PKTHDR)
539 copyhdr = 1;
540 while (off > 0) {
541 KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
542 if (off < m->m_len)
543 break;
544 off -= m->m_len;
545 m = m->m_next;
(kgdb) p off
$1 = 1166
(kgdb) up
#11 0xffffffff8077fe1f in ip_fragment (ip=0xfffffe0005b7a974,
m_frag=0xffffff8000241428, mtu=) at /usr/src/sys/netinet/ip_output.c:816
816 m->m_next = m_copym(m0, off, len, M_DONTWAIT);
(kgdb) list
811 len = ip->ip_len - off;
812 m->m_flags |= M_LASTFRAG;
813 } else
814 mhip->ip_off |= IP_MF;
815 mhip->ip_len = htons((u_short)(len + mhlen));
816 m->m_next = m_copym(m0, off, len, M_DONTWAIT);
817 if (m->m_next == NULL) { /* copy failed */
818 m_free(m);
819 error = ENOBUFS; /* ??? */
820 IPSTAT_INC(ips_odropped);
(kgdb) p *m0
$2 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0,
mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
pad = "ÞÞÀÞ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
header = 0x0, len = 334, flowid = 0, csum_flags = 1,
csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = {
ext_buf = 0x493a017c0045 <Address 0x493a017c0045 out of bounds>,
ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437,
ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586,
ref_cnt = 0xc6be758202079b0a, ext_type = 89195267},
MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"}},
M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000ÞÀÞÀ\220\n\005\000þÿÿE\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"...}}
(kgdb) p off
$3 = 1500
(kgdb) p len
$4 = 1480
(kgdb) p hlen
$5 = 20
(kgdb) up
#12 0xffffffff8032842a in pf_route (m=0xffffff8000241658,
r=0xfffffe0005dc8af8, dir=) at /usr/src/sys/contrib/pf/net/pf.c:6138
6138 error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
(kgdb) list
6133 /*
6134 * XXX: is cheaper + less error prone than own function
6135 */
6136 NTOHS(ip->ip_len);
6137 NTOHS(ip->ip_off);
6138 error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
6139 #else
6140 error = ip_fragment(m0, ifp, ifp->if_mtu);
6141 #endif
6142 if (error) {
(kgdb) p *ip
$6 = {ip_hl = 5 '\005', ip_v = 4 '\004', ip_tos = 0 '\0', ip_len = 19969,
ip_id = 3586, ip_off = 0, ip_ttl = 63 '?', ip_p = 1 '\001', ip_sum = 51492,
ip_src = {s_addr = 34052874}, ip_dst = {s_addr = 3334370690}}
(kgdb) p *m0
$7 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0,
mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
pad = "ÞÞÀÞ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
header = 0x0, len = 334, flowid = 0, csum_flags = 1,
csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = {
ext_buf = 0x493a017c0045 <Address 0x493a017c0045 out of bounds>,
ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437,
ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586,
ref_cnt = 0xc6be758202079b0a, ext_type = 89195267},
MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"}},
M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000ÞÀÞÀ\220\n\005\000þÿÿE\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"...}}
(kgdb) p *m1
$8 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0,
mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
pad = "ÞÞÀÞ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
header = 0x0, len = 334, flowid = 0, csum_flags = 1,
csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = {
ext_buf = 0x493a017c0045 <Address 0x493a017c0045 out of bounds>,
ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437,
ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586,
ref_cnt = 0xc6be758202079b0a, ext_type = 89195267},
MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"}},
M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000ÞÀÞÀ\220\n\005\000þÿÿE\000|\001:I\000\000E\000N\001:I\000\0007\004|(OËGI\201»³WE\000\001N\002\016\000\000?\001$É\n\233\a\002\202u¾Æ\003\003Q\005\000\000\000\000E\000\0012ûS\000\0000\021;\217\202u¾Æ\n\233\a\002\aÑùÃ\001\036éQKS\000\000\001B\000\000\000\001\v\001ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"...}}
(kgdb)
--- kgdb.out ends here ---
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201205210726.q4L7Q6m9064258>