Date: Fri, 4 Sep 2009 01:39:07 -0800 (AKDT) From: Mel Flynn <mel@rachie.is-a-geek.net> To: FreeBSD-gnats-submit@freebsd.org Cc: freebsd-net@FreeBSD.org Subject: [panic] Kernel corruption of pppoe lists Message-ID: <20090904093907.B61227E854@mailhub.rachie.is-a-geek.net>
next in thread | raw e-mail | index | archive | help
>Submitter-Id: current-users
>Originator: Mel Flynn
>Confidential: no
>Synopsis: [panic] Kernel corruption of pppoe lists
>Severity: critical
>Priority: low
>Category: kern
>Class: sw-bug
>Release: FreeBSD 7.2-STABLE i386
>Environment:
System: FreeBSD gate.rachie.is-a-geek.net 7.2-STABLE FreeBSD 7.2-STABLE #0: Sun Jun 28 00:01:59 AKDT 2009 mdev@squish.rachie.is-a-geek.net:/data/obj/data/RELENG_7/src/sys/GATE i386
>Description:
I realize the kernel is a bit old, but it also very hard to reproduce. Kernel
was up 56 days and this crash happened shortly after a very long connect time,
hangup by ISP and some renegotiation issues. I can provide the ppp.log of the
incident if needed.
What bothers me is the contents of the session list element, preceding the element
cannot be accessed. Clearly, there is random kernel memory present there, judging
from ether_dhost and ether_shost.
# kgdb /boot/kernel/kernel /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x2d465459
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc06cd0a0
stack pointer = 0x28:0xc3b86a98
frame pointer = 0x28:0xc3b86ac0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 35 (irq22: xl0)
trap number = 12
panic: page fault
Uptime: 56d6h29m38s
Physical memory: 1007 MB
Dumping 174 MB: 159 143 127 111 95 79 63 47 31 15
Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_journal.ko
Reading symbols from /boot/kernel/wlan_xauth.ko...Reading symbols from /boot/kernel/wlan_xauth.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_xauth.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from /boot/kernel/daemon_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0 doadump () at pcpu.h:196
196 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc05ef5d3 in boot (howto=260) at /data/RELENG_7/src/sys/kern/kern_shutdown.c:418
#2 0xc05ef7de in panic (fmt=Variable "fmt" is not available.
) at /data/RELENG_7/src/sys/kern/kern_shutdown.c:574
#3 0xc085c72c in trap_fatal (frame=0xc3b86a58, eva=759583833)
at /data/RELENG_7/src/sys/i386/i386/trap.c:938
#4 0xc085c9b0 in trap_pfault (frame=0xc3b86a58, usermode=0, eva=759583833)
at /data/RELENG_7/src/sys/i386/i386/trap.c:851
#5 0xc085d339 in trap (frame=0xc3b86a58) at /data/RELENG_7/src/sys/i386/i386/trap.c:529
#6 0xc0844a4b in calltrap () at /data/RELENG_7/src/sys/i386/i386/exception.s:166
#7 0xc06cd0a0 in pppoe_findsession (privp=0xc4258000, wh=Variable "wh" is not available.
)
at /data/RELENG_7/src/sys/netgraph/ng_pppoe.c:567
#8 0xc06ce1a0 in ng_pppoe_rcvdata_ether (hook=0xc41b6380, item=0xc4256120)
at /data/RELENG_7/src/sys/netgraph/ng_pppoe.c:1612
#9 0xc06c566f in ng_apply_item (node=0xc4111e80, item=0xc4256120, rw=0)
at /data/RELENG_7/src/sys/netgraph/ng_base.c:2336
#10 0xc06c47e0 in ng_snd_item (item=0xc4256120, flags=Variable "flags" is not available.
)
at /data/RELENG_7/src/sys/netgraph/ng_base.c:2254
#11 0xc068de5f in ether_demux (ifp=0xc3dbb400, m=0xc8024d00)
at /data/RELENG_7/src/sys/net/if_ethersubr.c:851
#12 0xc068e1b3 in ether_input (ifp=0xc3dbb400, m=0xc8024d00)
at /data/RELENG_7/src/sys/net/if_ethersubr.c:692
#13 0xc07b5348 in xl_rxeof (sc=0xc3dbc000) at /data/RELENG_7/src/sys/pci/if_xl.c:2022
#14 0xc07b7834 in xl_intr (arg=0xc3dbc000) at /data/RELENG_7/src/sys/pci/if_xl.c:2257
#15 0xc05cd10b in ithread_loop (arg=0xc3dc10c0)
at /data/RELENG_7/src/sys/kern/kern_intr.c:1127
#16 0xc05c9ae6 in fork_exit (callout=0xc05ccf60 <ithread_loop>, arg=0xc3dc10c0,
frame=0xc3b86d38) at /data/RELENG_7/src/sys/kern/kern_fork.c:811
#17 0xc0844ac0 in fork_trampoline () at /data/RELENG_7/src/sys/i386/i386/exception.s:271
(kgdb) frame 7
#7 0xc06cd0a0 in pppoe_findsession (privp=0xc4258000, wh=Variable "wh" is not available.
)
at /data/RELENG_7/src/sys/netgraph/ng_pppoe.c:567
567 if (sp->Session_ID == session &&
(kgdb) print sp
$1 = 0x2d465455
(kgdb) print *sp
Cannot access memory at address 0x2d465455
(kgdb) print *privp
$2 = {node = 0xc4111e80, ethernet_hook = 0xc41b6380, debug_hook = 0x0,
packets_in = 126728356, packets_out = 69301432, flags = 0, eh = {ether_dhost = "ÿÿÿÿÿÿ",
ether_shost = "\000\001\002Çù6", ether_type = 25480}, listeners = {lh_first = 0x0},
sesshash = {{mtx = {lock_object = {lo_name = 0xc08b8dc2 "PPPoE hash mutex",
lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = {
lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 3285460096,
mtx_recurse = 0}, head = {lh_first = 0xc5093780}}, {mtx = {lock_object = {
lo_name = 0xc08b8dc2 "PPPoE hash mutex",
lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = {
lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4,
mtx_recurse = 0}, head = {lh_first = 0x0}} <repeats 89 times>, {mtx = {
lock_object = {lo_name = 0xc08b8dc2 "PPPoE hash mutex",
lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = {
lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4,
mtx_recurse = 0}, head = {lh_first = 0xc4dc5bc0}}, {mtx = {lock_object = {
lo_name = 0xc08b8dc2 "PPPoE hash mutex",
lo_type = 0xc08b8dc2 "PPPoE hash mutex", lo_flags = 16973824, lo_witness_data = {
lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, mtx_lock = 4,
mtx_recurse = 0}, head = {lh_first = 0x0}} <repeats 165 times>}}
(kgdb) print *privp->sesshash.head.lh_first->sessions.le_next->sessions.le_next
$4 = {hook = 0x1, Session_ID = 52, state = 1920169263, creator = 1852400175, pkt_hdr = {
eh = {ether_dhost = "/zcat", ether_shost = "/usr/s", ether_type = 24936}, ph = {
ver = 2 '\002', type = 7 '\a', code = 101 'e', sid = 27951, length = 28257,
tag = 0xc4e68524}}, neg = 0x2e6e652f, sessions = {le_next = 0x2d465455,
le_prev = 0x61632f38}}
>How-To-Repeat:
Unknown at this time.
>Fix:
I don't expect this to be fixed, without a reproduction scenario, so I'm mainly
reporting this to see if others have experienced a similar crash.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090904093907.B61227E854>