Date: Thu, 12 Dec 2002 00:32:46 +0100 From: Alexander Langer <alex@big.endian.de> To: Ian Dowse <iedowse@maths.tcd.ie> Cc: Patrick Soltani <psoltani@ultradns.com>, freebsd-hackers@FreeBSD.ORG Subject: Re: panic: icmp_error: bad length Message-ID: <20021211233246.GR64335@fump.kawo2.rwth-aachen.de> In-Reply-To: <200212112304.aa30909@salmon.maths.tcd.ie> References: <3DBB075EEB95944492E127F2B9A96FAF5397C3@ultra-exchange.ultradns.com> <200212112304.aa30909@salmon.maths.tcd.ie>
next in thread | previous in thread | raw e-mail | index | archive | help
Thus spake Ian Dowse (iedowse@maths.tcd.ie):
> >In the last couple of months, upgraded to 4.6 and 4.7 using RELENG_4 =
> >with again no errors, however, now under a light smurf attack, I get:
> >panic: icmp_error: bad length
> >Hardware: Dell PowerEdge 350, 2 built-in Intel nic cards, 256 meg of ram =
> >and only doing ipfw.=20
> >The kernel is built with options BRIDGE. Don't know what other info you =
> >might be interested.
Yeah, same situation here. 4.6 used to work w/o problem, 4.7 doesn't.
It was able to track it down: It's always the same Windows 95 box on
our LAN that sends out NetBIOS-Broadcast packets that produce icmp error
replies, which then result in that panic.
That did NOT occur with 4.6 from ... I believe May/June, and started, when I
upgraded the kernel in the first week of October.
I dropped a private Mail to Luigi about this, but he couldn't help me.
I still have the mail here, where I'm taking the following trace from.
Unfortunately, I have deleted the crashdumps.
But first some more info about this:
- It does happen with both IPFW1 and IPFW2 on -STABLE
- I could track it down to a "unreach" rule in my IPFW setting.
The firewall is a bridging firewall, and this rule produced the
panic once the said Win95 box sends out its NetBIOS broadcasts here on
the LAN. We are using the rule to prevent people from using the wrong
netmask.
ipfw add 210 unreach host ip from 134.130.180.0/22 to 134.130.180.0/22
removing this rule gave us a fairly stable firewall (well, it panic'ed
once last week, but /usr/crash was full, so I got now dump).
I'm currently rather unhappy with the state of RELENG_4. The same
firewall had an uptime of 100 or so days with FreeBSD 4.4-STABLE (and
older)
Here's the complete trace, from an IPFW1 kernel:
root@cherub ...src/sys/CHERUB # gdb -k
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd".
(kgdb) symbol-file kernel.debug
Reading symbols from kernel.debug...Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 933 in fill_symbuf
done.
(kgdb) exec-file /usr/crash/kernel.4
(kgdb) core-file /usr/crash/vmcore.4
IdlePTD at phsyical address 0x002ca000
initial pcb at physical address 0x00249fc0
panicstr: page fault
panic messages:
---
panic: icmp_error: bad length
syncing disks...
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x30
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc01b1820
stack pointer = 0x10:0xc022b838
frame pointer = 0x10:0xc022b840
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = Idle
interrupt mask = net bio cam
trap number = 12
panic: page fault
Uptime: 6m21s
dumping to dev #ad/0x20021, offset 262304
dump ata2: resetting devices .. done
127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110
109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89
88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65
64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41
40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17
16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) where
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc014a613 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:316
#2 0xc014aa38 in poweroff_wait (junk=0xc02232ec, howto=-1071501809) at
/usr/src/sys/kern/kern_shutdown.c:595
#3 0xc01f2e9a in trap_fatal (frame=0xc022b7f8, eva=48) at
/usr/src/sys/i386/i386/trap.c:974
#4 0xc01f2b6d in trap_pfault (frame=0xc022b7f8, usermode=0, eva=48) at
/usr/src/sys/i386/i386/trap.c:867
#5 0xc01f2757 in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16,
tf_edi = -1071255392, tf_esi = 0,
tf_ebp = -1071466432, tf_isp = -1071466460, tf_ebx = -1071399332,
tf_edx = 6865984, tf_ecx = 2, tf_eax = 0,
tf_trapno = 12, tf_err = 0, tf_eip = -1071966176, tf_cs = 8,
tf_eflags = 66050, tf_esp = 0, tf_ss = 0})
at /usr/src/sys/i386/i386/trap.c:466
#6 0xc01b1820 in acquire_lock (lk=0xc023be5c) at
/usr/src/sys/ufs/ffs/ffs_softdep.c:266
#7 0xc01b5e42 in softdep_fsync_mountdev (vp=0xc86a6b40) at
/usr/src/sys/ufs/ffs/ffs_softdep.c:4024
#8 0xc01ba072 in ffs_fsync (ap=0xc022b8b4) at
/usr/src/sys/ufs/ffs/ffs_vnops.c:134
#9 0xc01b8d03 in ffs_sync (mp=0xc0c01200, waitfor=2, cred=0xc0741600,
p=0xc025f0a0) at vnode_if.h:558
#10 0xc017a403 in sync (p=0xc025f0a0, uap=0x0) at
/usr/src/sys/kern/vfs_syscalls.c:576
#11 0xc014a3ae in boot (howto=256) at
/usr/src/sys/kern/kern_shutdown.c:235
#12 0xc014aa38 in poweroff_wait (junk=0xc0214240, howto=-1059622400) at
/usr/src/sys/kern/kern_shutdown.c:595
#13 0xc01969e1 in icmp_error (n=0xc077a800, type=3, code=1, dest=0,
destifp=0x0) at /usr/src/sys/netinet/ip_icmp.c:176
#14 0xc0195dcd in ip_fw_chk (args=0xc022ba48) at
/usr/src/sys/netinet/ip_fw.c:1564
#15 0xc0185be2 in bdg_forward (m0=0xc0768700, eh=0xc0819012, dst=0x1) at
/usr/src/sys/net/bridge.c:932
#16 0xc0188668 in ether_input (ifp=0xc0bad800, eh=0x0, m=0xc0768700) at
/usr/src/sys/net/if_ethersubr.c:603
#17 0xc012edca in fxp_intr_body (sc=0xc0bad800, statack=64 '@',
count=-1) at /usr/src/sys/dev/fxp/if_fxp.c:1345
#18 0xc012ec9d in fxp_intr (xsc=0xc0bad800) at
/usr/src/sys/dev/fxp/if_fxp.c:1228
#19 0xc01fb165 in intr_mux (arg=0xc072a900) at
/usr/src/sys/i386/isa/intr_machdep.c:582
#20 0xc01ecbd2 in cpu_idle () at /usr/src/sys/i386/i386/machdep.c:1024
(kgdb) up 13
#13 0xc01969e1 in icmp_error (n=0xc077a800, type=3, code=1, dest=0,
destifp=0x0) at /usr/src/sys/netinet/ip_icmp.c:176
176 panic("icmp_error: bad length");
(kgdb) list
171 m = m_gethdr(M_DONTWAIT, MT_HEADER);
172 if (m == NULL)
173 goto freeit;
174 icmplen = min(oiplen + 8, oip->ip_len);
175 if (icmplen < sizeof(struct ip))
176 panic("icmp_error: bad length");
177 m->m_len = icmplen + ICMP_MINLEN;
178 MH_ALIGN(m, m->m_len);
179 icp = mtod(m, struct icmp *);
180 if ((u_int)type > ICMP_MAXTYPE)
(kgdb) print icmplen
$1 = 0
(kgdb) print oiplen
$2 = 3223405120
(kgdb) print oip->ip_len
$3 = 1
(kgdb) up
#14 0xc0195dcd in ip_fw_chk (args=0xc022ba48) at
/usr/src/sys/netinet/ip_fw.c:1564
1564 icmp_error(*m, ICMP_UNREACH,
(kgdb) list
1559 }
1560 *m = NULL;
1561 break;
1562 }
1563 default: /* Send an ICMP unreachable
using code */
1564 icmp_error(*m, ICMP_UNREACH,
1565 f->fw_reject_code, 0L, 0);
1566 *m = NULL;
1567 break;
1568 }
(kgdb) up
#15 0xc0185be2 in bdg_forward (m0=0xc0768700, eh=0xc0819012, dst=0x1) at
/usr/src/sys/net/bridge.c:932
932 i = ip_fw_chk_ptr(&args);
(kgdb) print args
$4 = {m = 0xc077a800, oif = 0x0, next_hop = 0x0, rule = 0x0, eh =
0xc022ba38, ro = 0xc0819012, dst = 0xc0c3d4b0,
flags = -1072146350, f_id = {dst_ip = 2256713727, src_ip = 2256713317,
dst_port = 138, src_port = 138,
proto = 17 '\021', flags = 0 '\000'}, divert_rule = 0, retval =
3233470464}
(kgdb)
dst_ip is our broadcast IP, 134.130.183.255. src_ip is a valid IP here,
the IP of the Win95 box I mentioned.
Here is more of the trace, where I tried to check the values in the
mbuf.
(kgdb) up
#13 0xc01969e1 in icmp_error (n=0xc07d5000, type=3, code=1, dest=0,
destifp=0x0) at /usr/src/sys/netinet/ip_icmp.c:176
176 panic("icmp_error: bad length");
(kgdb) print *n
$27 = {m_hdr = {mh_next = 0xc07b7b00, mh_nextpkt = 0x0,
mh_data = 0xc07d502c "E", mh_len = 40, mh_type = 1, mh_flags = 2},
M_dat = {MH = {MH_pkthdr = {rcvif = 0xc0bad800, len = 256,
header = 0x5ddb0b4f, csum_flags = 0, csum_data = 16, aux = 0x0},
MH_dat = {MH_ext = {ext_buf = 0x10045cannot read proc at 0
(kgdb) up
#14 0xc0195dcd in ip_fw_chk (args=0xc022ba48)
at /usr/src/sys/netinet/ip_fw.c:1564
1564 icmp_error(*m, ICMP_UNREACH,
(kgdb) print *args
$28 = {m = 0xc07d5000, oif = 0x0, next_hop = 0x0, rule = 0x0, eh =
0xc022ba38,
ro = 0xc07b8812, dst = 0xc0c3c4b0, flags = -1072146350, f_id = {
dst_ip = 2256713727, src_ip = 2256713317, dst_port = 138, src_port =
138,
proto = 17 '\021', flags = 0 '\000'}, divert_rule = 0, retval =
3233470464}
(kgdb) up
#15 0xc0185be2 in bdg_forward (m0=0xc07b7b00, eh=0xc07b8812, dst=0x1)
at /usr/src/sys/net/bridge.c:932
932 i = ip_fw_chk_ptr(&args);
(kgdb) print m0
$20 = (struct mbuf *) 0xc07d5000
(kgdb) print *m0
$21 = {m_hdr = {mh_next = 0xc07b7b00, mh_nextpkt = 0x0,
mh_data = 0xc07d502c "E", mh_len = 40, mh_type = 1, mh_flags = 2},
M_dat = {MH = {MH_pkthdr = {rcvif = 0xc0bad800, len = 256,
header = 0x5ddb0b4f, csum_flags = 0, csum_data = 16, aux = 0x0},
MH_dat = {MH_ext = {ext_buf = 0x10045cannot read proc at 0
(kgdb) print *eh
$22 = {ether_dhost = "ÿÿÿÿÿÿ", ether_shost = "\000à)0¹T", ether_type =
8}
(kgdb) up
#16 0xc0188668 in ether_input (ifp=0xc0bad800, eh=0x0, m=0xc07b7b00)
at /usr/src/sys/net/if_ethersubr.c:603
603 m = bdg_forward_ptr(m, eh, bif); /*
needs forwarding */
(kgdb) print *ifp
$23 = {if_softc = 0xc0bad800, if_name = 0xc02095c0 "fxp", if_link = {
tqe_next = 0xc0bad400, tqe_prev = 0xc025ffd4}, if_addrhead = {
tqh_first = 0xc0bb7500, tqh_last = 0xc0c2f260}, if_pcount = 2,
if_bpf = 0xc072a800, if_index = 1, if_unit = 0, if_timer = 0,
if_flags = -30461, if_ipending = 0, if_linkmib = 0x0, if_linkmiblen =
0,
if_data = {ifi_type = 6 '\006', ifi_physical = 0 '\000',
ifi_addrlen = 6 '\006', ifi_hdrlen = 18 '\022', ifi_recvquota = 0
'\000',
ifi_xmitquota = 0 '\000', ifi_mtu = 1500, ifi_metric = 0,
ifi_baudrate = 100000000, ifi_ipackets = 26306, ifi_ierrors = 0,
ifi_opackets = 11683, ifi_oerrors = 0, ifi_collisions = 0,
ifi_ibytes = 3561366, ifi_obytes = 1605251, ifi_imcasts = 14459,
ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 0,
ifi_unused = 0, ifi_lastchange = {tv_sec = 1034607648, tv_usec =
220022}},
if_multiaddrs = {lh_first = 0xc0c2de40}, if_amcount = 0,
if_output = 0xc0187cc4 <ether_output>, if_start = 0xc012e684
<fxp_start>,
_u1 = {if_done = 0, uif_capabilities = 0},
if_ioctl = 0xc0130384 <fxp_ioctl>, if_watchdog = 0xc012f1f0
<fxp_watchdog>,
_u2 = {if_poll_recv = 0, uif_capenable = 0}, if_poll_xmit = 0,
if_poll_intren = 0, if_poll_slowinput = 0, if_init = 0xc012f21c
<fxp_init>,
if_resolvemulti = 0xc0188aa0 <ether_resolvemulti>, if_snd = {ifq_head
= 0x0,
ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 127, ifq_drops = 0},
if_poll_slowq = 0x0, if_prefixhead = {tqh_first = 0x0,
tqh_last = 0xc0bad8d0}}
(kgdb) print *m
$24 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc07b8848
"",
mh_len = 216, mh_type = 1, mh_flags = 1}, M_dat = {MH = {MH_pkthdr =
{
rcvif = 0xc0bad800, len = 256, header = 0x5ddb0b4f, csum_flags =
0,
csum_data = 16, aux = 0x0}, MH_dat = {MH_ext = {
ext_buf = 0xc07b8800 "", ext_free = 0, ext_size = 2048,
ext_ref = 0},
MH_databuf =
"\000\210{À\000\000\000\000\000\b\000\000\000\000\000\000µ$\000\000\000\000\000\000\206\202´-",
' ' <repeats 12 times>,
' ' <repeats 12 times>,
"3SvÁÍâW©\200\020\000à\000\000\000\000\001\001\b\n\000\004§4\000\004§*\026\005Â\f\210y\001\013\211\a\002\"ï½Mò\r]ô1C»ä*\230üB½P\002ûÕ\031\200l\024à¥Å©\t|
'3Õ\236\031ÅiÔ«\215úÐÎ9t\027)5\000\231s[8I\231Ë&z\220f\222m\000@\2253\207X\000\002³\211\005\020\b\000E\000\00083¢\000\000@\001Ñî\206\202´\002\206\202´-\003\001$\025\000\000\000\000E\000N\000þ\211\000\000\200\021Âã\206\202´-\206\202"...}},
M_databuf =
"\000غÀ\000\001\000\000O\013Û]\000\000\000\000\020\000\000\000\000\000\000\000\000\210{À\000\000\000\000\000\b\000\000\000\000\000\000µ$\000\000\000\000\000\000\206\202´-",
' ' <repeats 12 times>,
"3SvÁÍâW©\200\020\000à\000\000\000\000\001\001\b\n\000\004§4\000\004§*\026\005Â\f\210y\001\013\211\a\002\"ï½Mò\r]ô1C»ä*\230üB½P\002ûÕ\031\200l\024à¥Å©\t|
'3Õ\236\031ÅiÔ«\215úÐÎ9t\027)5\000\231s[8I\231Ë&z\220f\222m\000@\2253\207X\000\002³\211\005\020\b\000E\000\00083¢\000\000@\001Ñî\206\202´\002\206\202´-\003\001"...}}
(kgdb) up
#17 0xc012edca in fxp_intr_body (sc=0xc0bad800, statack=64 '@',
count=-1)
at /usr/src/sys/dev/fxp/if_fxp.c:1345
1345 ether_input(ifp, NULL, m);
...
I have no idea what's going wrong. Help would be very much appreciated.
Maybe it shouldn't send out the icmp unreach reply to the broadcast
address, but I'm not quite sure. It's interesting enough that only
packets fromt hat win95 box crash the firewall (I had 6 identical
crashdumps here, both IPFW1 and IPFW2 - always with src_ip set to this IP).
I also phoned the user, and he said he hasn't changed anything at his
box for months, so it must be a change int he networking code from
within May/June 2002 - October 2002.
I can usually reproduce the panic within 10 minutes, if the user has his
Win95 box up, so I can also test patches.
Thanks
Alex
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021211233246.GR64335>