From owner-freebsd-hackers Wed Dec 11 15:32:58 2002 Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 68A4437B401 for ; Wed, 11 Dec 2002 15:32:50 -0800 (PST) Received: from fump.kawo2.rwth-aachen.de (fump.kawo2.RWTH-Aachen.DE [134.130.181.148]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5383643E4A for ; Wed, 11 Dec 2002 15:32:49 -0800 (PST) (envelope-from alex@fump.kawo2.rwth-aachen.de) Received: from fump.kawo2.rwth-aachen.de (localhost [127.0.0.1]) by fump.kawo2.rwth-aachen.de (8.12.5/8.12.5) with ESMTP id gBBNWlUf071426; Thu, 12 Dec 2002 00:32:47 +0100 (CET) (envelope-from alex@fump.kawo2.rwth-aachen.de) Received: (from alex@localhost) by fump.kawo2.rwth-aachen.de (8.12.5/8.12.5/Submit) id gBBNWlbP071425; Thu, 12 Dec 2002 00:32:47 +0100 (CET) Date: Thu, 12 Dec 2002 00:32:46 +0100 From: Alexander Langer To: Ian Dowse Cc: Patrick Soltani , freebsd-hackers@FreeBSD.ORG Subject: Re: panic: icmp_error: bad length Message-ID: <20021211233246.GR64335@fump.kawo2.rwth-aachen.de> References: <3DBB075EEB95944492E127F2B9A96FAF5397C3@ultra-exchange.ultradns.com> <200212112304.aa30909@salmon.maths.tcd.ie> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <200212112304.aa30909@salmon.maths.tcd.ie> X-PGP-at: finger alex@big.endian.de X-Verwirrung: Dieser Header dient der allgemeinen Verwirrung. User-Agent: Mutt/1.5.1i Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Thus spake Ian Dowse (iedowse@maths.tcd.ie): > >In the last couple of months, upgraded to 4.6 and 4.7 using RELENG_4 = > >with again no errors, however, now under a light smurf attack, I get: > >panic: icmp_error: bad length > >Hardware: Dell PowerEdge 350, 2 built-in Intel nic cards, 256 meg of ram = > >and only doing ipfw.=20 > >The kernel is built with options BRIDGE. Don't know what other info you = > >might be interested. Yeah, same situation here. 4.6 used to work w/o problem, 4.7 doesn't. It was able to track it down: It's always the same Windows 95 box on our LAN that sends out NetBIOS-Broadcast packets that produce icmp error replies, which then result in that panic. That did NOT occur with 4.6 from ... I believe May/June, and started, when I upgraded the kernel in the first week of October. I dropped a private Mail to Luigi about this, but he couldn't help me. I still have the mail here, where I'm taking the following trace from. Unfortunately, I have deleted the crashdumps. But first some more info about this: - It does happen with both IPFW1 and IPFW2 on -STABLE - I could track it down to a "unreach" rule in my IPFW setting. The firewall is a bridging firewall, and this rule produced the panic once the said Win95 box sends out its NetBIOS broadcasts here on the LAN. We are using the rule to prevent people from using the wrong netmask. ipfw add 210 unreach host ip from 134.130.180.0/22 to 134.130.180.0/22 removing this rule gave us a fairly stable firewall (well, it panic'ed once last week, but /usr/crash was full, so I got now dump). I'm currently rather unhappy with the state of RELENG_4. The same firewall had an uptime of 100 or so days with FreeBSD 4.4-STABLE (and older) Here's the complete trace, from an IPFW1 kernel: root@cherub ...src/sys/CHERUB # gdb -k GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd". (kgdb) symbol-file kernel.debug Reading symbols from kernel.debug...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf done. (kgdb) exec-file /usr/crash/kernel.4 (kgdb) core-file /usr/crash/vmcore.4 IdlePTD at phsyical address 0x002ca000 initial pcb at physical address 0x00249fc0 panicstr: page fault panic messages: --- panic: icmp_error: bad length syncing disks... Fatal trap 12: page fault while in kernel mode fault virtual address = 0x30 fault code = supervisor read, page not present instruction pointer = 0x8:0xc01b1820 stack pointer = 0x10:0xc022b838 frame pointer = 0x10:0xc022b840 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = Idle interrupt mask = net bio cam trap number = 12 panic: page fault Uptime: 6m21s dumping to dev #ad/0x20021, offset 262304 dump ata2: resetting devices .. done 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 --- #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) where #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc014a613 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc014aa38 in poweroff_wait (junk=0xc02232ec, howto=-1071501809) at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc01f2e9a in trap_fatal (frame=0xc022b7f8, eva=48) at /usr/src/sys/i386/i386/trap.c:974 #4 0xc01f2b6d in trap_pfault (frame=0xc022b7f8, usermode=0, eva=48) at /usr/src/sys/i386/i386/trap.c:867 #5 0xc01f2757 in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = -1071255392, tf_esi = 0, tf_ebp = -1071466432, tf_isp = -1071466460, tf_ebx = -1071399332, tf_edx = 6865984, tf_ecx = 2, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1071966176, tf_cs = 8, tf_eflags = 66050, tf_esp = 0, tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:466 #6 0xc01b1820 in acquire_lock (lk=0xc023be5c) at /usr/src/sys/ufs/ffs/ffs_softdep.c:266 #7 0xc01b5e42 in softdep_fsync_mountdev (vp=0xc86a6b40) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4024 #8 0xc01ba072 in ffs_fsync (ap=0xc022b8b4) at /usr/src/sys/ufs/ffs/ffs_vnops.c:134 #9 0xc01b8d03 in ffs_sync (mp=0xc0c01200, waitfor=2, cred=0xc0741600, p=0xc025f0a0) at vnode_if.h:558 #10 0xc017a403 in sync (p=0xc025f0a0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576 #11 0xc014a3ae in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235 #12 0xc014aa38 in poweroff_wait (junk=0xc0214240, howto=-1059622400) at /usr/src/sys/kern/kern_shutdown.c:595 #13 0xc01969e1 in icmp_error (n=0xc077a800, type=3, code=1, dest=0, destifp=0x0) at /usr/src/sys/netinet/ip_icmp.c:176 #14 0xc0195dcd in ip_fw_chk (args=0xc022ba48) at /usr/src/sys/netinet/ip_fw.c:1564 #15 0xc0185be2 in bdg_forward (m0=0xc0768700, eh=0xc0819012, dst=0x1) at /usr/src/sys/net/bridge.c:932 #16 0xc0188668 in ether_input (ifp=0xc0bad800, eh=0x0, m=0xc0768700) at /usr/src/sys/net/if_ethersubr.c:603 #17 0xc012edca in fxp_intr_body (sc=0xc0bad800, statack=64 '@', count=-1) at /usr/src/sys/dev/fxp/if_fxp.c:1345 #18 0xc012ec9d in fxp_intr (xsc=0xc0bad800) at /usr/src/sys/dev/fxp/if_fxp.c:1228 #19 0xc01fb165 in intr_mux (arg=0xc072a900) at /usr/src/sys/i386/isa/intr_machdep.c:582 #20 0xc01ecbd2 in cpu_idle () at /usr/src/sys/i386/i386/machdep.c:1024 (kgdb) up 13 #13 0xc01969e1 in icmp_error (n=0xc077a800, type=3, code=1, dest=0, destifp=0x0) at /usr/src/sys/netinet/ip_icmp.c:176 176 panic("icmp_error: bad length"); (kgdb) list 171 m = m_gethdr(M_DONTWAIT, MT_HEADER); 172 if (m == NULL) 173 goto freeit; 174 icmplen = min(oiplen + 8, oip->ip_len); 175 if (icmplen < sizeof(struct ip)) 176 panic("icmp_error: bad length"); 177 m->m_len = icmplen + ICMP_MINLEN; 178 MH_ALIGN(m, m->m_len); 179 icp = mtod(m, struct icmp *); 180 if ((u_int)type > ICMP_MAXTYPE) (kgdb) print icmplen $1 = 0 (kgdb) print oiplen $2 = 3223405120 (kgdb) print oip->ip_len $3 = 1 (kgdb) up #14 0xc0195dcd in ip_fw_chk (args=0xc022ba48) at /usr/src/sys/netinet/ip_fw.c:1564 1564 icmp_error(*m, ICMP_UNREACH, (kgdb) list 1559 } 1560 *m = NULL; 1561 break; 1562 } 1563 default: /* Send an ICMP unreachable using code */ 1564 icmp_error(*m, ICMP_UNREACH, 1565 f->fw_reject_code, 0L, 0); 1566 *m = NULL; 1567 break; 1568 } (kgdb) up #15 0xc0185be2 in bdg_forward (m0=0xc0768700, eh=0xc0819012, dst=0x1) at /usr/src/sys/net/bridge.c:932 932 i = ip_fw_chk_ptr(&args); (kgdb) print args $4 = {m = 0xc077a800, oif = 0x0, next_hop = 0x0, rule = 0x0, eh = 0xc022ba38, ro = 0xc0819012, dst = 0xc0c3d4b0, flags = -1072146350, f_id = {dst_ip = 2256713727, src_ip = 2256713317, dst_port = 138, src_port = 138, proto = 17 '\021', flags = 0 '\000'}, divert_rule = 0, retval = 3233470464} (kgdb) dst_ip is our broadcast IP, 134.130.183.255. src_ip is a valid IP here, the IP of the Win95 box I mentioned. Here is more of the trace, where I tried to check the values in the mbuf. (kgdb) up #13 0xc01969e1 in icmp_error (n=0xc07d5000, type=3, code=1, dest=0, destifp=0x0) at /usr/src/sys/netinet/ip_icmp.c:176 176 panic("icmp_error: bad length"); (kgdb) print *n $27 = {m_hdr = {mh_next = 0xc07b7b00, mh_nextpkt = 0x0, mh_data = 0xc07d502c "E", mh_len = 40, mh_type = 1, mh_flags = 2}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xc0bad800, len = 256, header = 0x5ddb0b4f, csum_flags = 0, csum_data = 16, aux = 0x0}, MH_dat = {MH_ext = {ext_buf = 0x10045cannot read proc at 0 (kgdb) up #14 0xc0195dcd in ip_fw_chk (args=0xc022ba48) at /usr/src/sys/netinet/ip_fw.c:1564 1564 icmp_error(*m, ICMP_UNREACH, (kgdb) print *args $28 = {m = 0xc07d5000, oif = 0x0, next_hop = 0x0, rule = 0x0, eh = 0xc022ba38, ro = 0xc07b8812, dst = 0xc0c3c4b0, flags = -1072146350, f_id = { dst_ip = 2256713727, src_ip = 2256713317, dst_port = 138, src_port = 138, proto = 17 '\021', flags = 0 '\000'}, divert_rule = 0, retval = 3233470464} (kgdb) up #15 0xc0185be2 in bdg_forward (m0=0xc07b7b00, eh=0xc07b8812, dst=0x1) at /usr/src/sys/net/bridge.c:932 932 i = ip_fw_chk_ptr(&args); (kgdb) print m0 $20 = (struct mbuf *) 0xc07d5000 (kgdb) print *m0 $21 = {m_hdr = {mh_next = 0xc07b7b00, mh_nextpkt = 0x0, mh_data = 0xc07d502c "E", mh_len = 40, mh_type = 1, mh_flags = 2}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xc0bad800, len = 256, header = 0x5ddb0b4f, csum_flags = 0, csum_data = 16, aux = 0x0}, MH_dat = {MH_ext = {ext_buf = 0x10045cannot read proc at 0 (kgdb) print *eh $22 = {ether_dhost = "ÿÿÿÿÿÿ", ether_shost = "\000à)0¹T", ether_type = 8} (kgdb) up #16 0xc0188668 in ether_input (ifp=0xc0bad800, eh=0x0, m=0xc07b7b00) at /usr/src/sys/net/if_ethersubr.c:603 603 m = bdg_forward_ptr(m, eh, bif); /* needs forwarding */ (kgdb) print *ifp $23 = {if_softc = 0xc0bad800, if_name = 0xc02095c0 "fxp", if_link = { tqe_next = 0xc0bad400, tqe_prev = 0xc025ffd4}, if_addrhead = { tqh_first = 0xc0bb7500, tqh_last = 0xc0c2f260}, if_pcount = 2, if_bpf = 0xc072a800, if_index = 1, if_unit = 0, if_timer = 0, if_flags = -30461, if_ipending = 0, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {ifi_type = 6 '\006', ifi_physical = 0 '\000', ifi_addrlen = 6 '\006', ifi_hdrlen = 18 '\022', ifi_recvquota = 0 '\000', ifi_xmitquota = 0 '\000', ifi_mtu = 1500, ifi_metric = 0, ifi_baudrate = 100000000, ifi_ipackets = 26306, ifi_ierrors = 0, ifi_opackets = 11683, ifi_oerrors = 0, ifi_collisions = 0, ifi_ibytes = 3561366, ifi_obytes = 1605251, ifi_imcasts = 14459, ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 0, ifi_unused = 0, ifi_lastchange = {tv_sec = 1034607648, tv_usec = 220022}}, if_multiaddrs = {lh_first = 0xc0c2de40}, if_amcount = 0, if_output = 0xc0187cc4 , if_start = 0xc012e684 , _u1 = {if_done = 0, uif_capabilities = 0}, if_ioctl = 0xc0130384 , if_watchdog = 0xc012f1f0 , _u2 = {if_poll_recv = 0, uif_capenable = 0}, if_poll_xmit = 0, if_poll_intren = 0, if_poll_slowinput = 0, if_init = 0xc012f21c , if_resolvemulti = 0xc0188aa0 , if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 127, ifq_drops = 0}, if_poll_slowq = 0x0, if_prefixhead = {tqh_first = 0x0, tqh_last = 0xc0bad8d0}} (kgdb) print *m $24 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc07b8848 "", mh_len = 216, mh_type = 1, mh_flags = 1}, M_dat = {MH = {MH_pkthdr = { rcvif = 0xc0bad800, len = 256, header = 0x5ddb0b4f, csum_flags = 0, csum_data = 16, aux = 0x0}, MH_dat = {MH_ext = { ext_buf = 0xc07b8800 "", ext_free = 0, ext_size = 2048, ext_ref = 0}, MH_databuf = "\000\210{À\000\000\000\000\000\b\000\000\000\000\000\000µ$\000\000\000\000\000\000\206\202´-", ' ' , ' ' , "3SvÁÍâW©\200\020\000à\000\000\000\000\001\001\b\n\000\004§4\000\004§*\026\005Â\f\210y\001\013\211\a\002\"ï½Mò\r]ô1C»ä*\230üB½P\002ûÕ\031\200l\024à¥Å©\t| '3Õ\236\031ÅiÔ«\215úÐÎ9t\027)5\000\231s[8I\231Ë&z\220f\222m\000@\2253\207X\000\002³\211\005\020\b\000E\000\00083¢\000\000@\001Ñî\206\202´\002\206\202´-\003\001$\025\000\000\000\000E\000N\000þ\211\000\000\200\021Âã\206\202´-\206\202"...}}, M_databuf = "\000غÀ\000\001\000\000O\013Û]\000\000\000\000\020\000\000\000\000\000\000\000\000\210{À\000\000\000\000\000\b\000\000\000\000\000\000µ$\000\000\000\000\000\000\206\202´-", ' ' , "3SvÁÍâW©\200\020\000à\000\000\000\000\001\001\b\n\000\004§4\000\004§*\026\005Â\f\210y\001\013\211\a\002\"ï½Mò\r]ô1C»ä*\230üB½P\002ûÕ\031\200l\024à¥Å©\t| '3Õ\236\031ÅiÔ«\215úÐÎ9t\027)5\000\231s[8I\231Ë&z\220f\222m\000@\2253\207X\000\002³\211\005\020\b\000E\000\00083¢\000\000@\001Ñî\206\202´\002\206\202´-\003\001"...}} (kgdb) up #17 0xc012edca in fxp_intr_body (sc=0xc0bad800, statack=64 '@', count=-1) at /usr/src/sys/dev/fxp/if_fxp.c:1345 1345 ether_input(ifp, NULL, m); ... I have no idea what's going wrong. Help would be very much appreciated. Maybe it shouldn't send out the icmp unreach reply to the broadcast address, but I'm not quite sure. It's interesting enough that only packets fromt hat win95 box crash the firewall (I had 6 identical crashdumps here, both IPFW1 and IPFW2 - always with src_ip set to this IP). I also phoned the user, and he said he hasn't changed anything at his box for months, so it must be a change int he networking code from within May/June 2002 - October 2002. I can usually reproduce the panic within 10 minutes, if the user has his Win95 box up, so I can also test patches. Thanks Alex To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message