Date: Fri, 16 May 2008 20:16:29 GMT From: C Fan <mimielliot@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/123741: [netgraph] [panic] kernel panic due to netgraph mpd Message-ID: <200805162016.m4GKGTdI040471@www.freebsd.org> Resent-Message-ID: <200805162020.m4GKK09C086668@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 123741 >Category: kern >Synopsis: [netgraph] [panic] kernel panic due to netgraph mpd >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri May 16 20:20:00 UTC 2008 >Closed-Date: >Last-Modified: >Originator: C Fan >Release: FreeBSD 7.0 stable >Organization: >Environment: FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #6: Sat May 10 05:27:18 PDT 2008 root@xxx:/usr/obj/usr/src/sys/XXX i386 >Description: I'm having a system, console, and keyboard locked up problems on both of my production server and home server after upgrading to Freebsd 7.0. There is a mpd VPN setup between these 2 servers. A PR has been filed regarding to the problem. http://www.freebsd.org/cgi/query-pr.cgi?pr=123729 After the upgrade, my home server always got locked up after few hours. I compiled my home server with DDB and KDB options. The problem seemed to be gone. However, after few days, I got a kernel panic and core dump. Fatal trap 12: page fault while i kernel mode cpuid = 0; apic id = 00 fault virtua address = 0x8 fault code = supervisor write, page not present instruction pointer = 0x20:0xc2f62a9b stack pointer = 0x28:0xd615ebc4 frame pointer = 0x28:0xd615ebe8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 99236 (ngctl) [thread pid 99236 tid 100214 ] Stopped at ng_address_path+0x4d: movl %eax,0x8(%ebx) db> show msgbuf .. lot of these messages <7>TCP: [x.x.x.x]:443 to [y.y.y.y]:56747 tcpflags 0x19<FIN,PUSH,ACK>;tcp_do_segment: FIN_WAIT_1: Received 23 bytes of data after socket was closed, sending RST and removing tcpcb db> bt Tracing pid 99236 tid 100214 td 0xc383aaa0 ng_address_path(0,0,c332cbe2,0,c293a000,...) at ng_address_path+0x4d ngd_connect(c334c000,c332cbe0,c383aaa0,25,d615ec60,...) at ngd_connect+0x65 soconnect(c334c000,c332cbe0,c383aaa0,c065c456,bfbfed2c,...) at soconnect+0x52 kern_connect(c383aaa0,4,c332cbe0,c332cbe0,0,...) at kern_connect+0x56 syscall(d615ed38) at syscall+0x32e Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (98, FreeBSD ELF32, connect), eip = 0x281a2783, esp = 0xbfbfecdc, ebp=0xbfbfecf8 kgdb -c vmcore.0 kernel.debug [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". There is no member named pathname. Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done. done. Loaded symbols for /boot/kernel/acpi.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_socket.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /boot/kernel/ng_iface.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_iface.ko Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ppp.ko Reading symbols from /boot/kernel/ng_bpf.ko...Reading symbols from /boot/kernel/ng_bpf.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_bpf.ko Reading symbols from /boot/kernel/ng_vjc.ko...Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_vjc.ko Reading symbols from /boot/kernel/ng_pptpgre.ko...Reading symbols from /boot/kernel/ng_pptpgre.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_pptpgre.ko Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /boot/kernel/ng_ksocket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ksocket.ko Reading symbols from /boot/kernel/fire_saver.ko...Reading symbols from /boot/kernel/fire_saver.ko.symbols...done. done. Loaded symbols for /boot/kernel/fire_saver.ko Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_mppc.ko Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done. done. Loaded symbols for /boot/kernel/rc4.ko Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x8 fault code = supervisor write, page not present instruction pointer = 0x20:0xc2f62a9b stack pointer = 0x28:0xd615ebc4 frame pointer = 0x28:0xd615ebe8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 99236 (ngctl) panic: from debugger cpuid = 0 Uptime: 4d22h59m14s Physical memory: 501 MB Dumping 88 MB: 73 57 41 25 9 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:195 #1 0xc06057d6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc0605ac9 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:572 #3 0xc04b2be9 in db_panic (addr=Could not find the frame base for "db_panic". ) at /usr/src/sys/ddb/db_command.c:446 #4 0xc04b3307 in db_command (last_cmdp=0xc0929614, cmd_table=0x0, dopager=1) at /usr/src/sys/ddb/db_command.c:413 #5 0xc04b340a in db_command_loop () at /usr/src/sys/ddb/db_command.c:466 #6 0xc04b4d7e in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_main.c:228 #7 0xc062e15c in kdb_trap (type=12, code=0, tf=0xd615eb84) at /usr/src/sys/kern/subr_kdb.c:524 #8 0xc0860055 in trap_fatal (frame=0xd615eb84, eva=8) at /usr/src/sys/i386/i386/trap.c:890 #9 0xc08602e9 in trap_pfault (frame=0xd615eb84, usermode=0, eva=8) at /usr/src/sys/i386/i386/trap.c:812 #10 0xc0860c76 in trap (frame=0xd615eb84) at /usr/src/sys/i386/i386/trap.c:490 #11 0xc084869b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #12 0xc2f62a9b in ng_address_path (here=0x0, item=0x0, address=0xc332cbe2 "ngctl99236:", retaddr=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3599 ---Type <return> to continue, or q <return> to quit--- #13 0xc2f5cb25 in ngd_connect (so=0xc334c000, nam=0xc332cbe0, td=0xc383aaa0) at /usr/src/sys/modules/netgraph/socket/../../../netgraph/ng_socket.c:737 #14 0xc06568f7 in soconnect (so=0xc334c000, nam=0xc332cbe0, td=0xc383aaa0) at /usr/src/sys/kern/uipc_socket.c:765 #15 0xc065ce62 in kern_connect (td=0xc383aaa0, fd=4, sa=0xc332cbe0) at /usr/src/sys/kern/uipc_syscalls.c:558 #16 0xc065cfe7 in connect (td=0xc383aaa0, uap=0xd615ecfc) at /usr/src/sys/kern/uipc_syscalls.c:526 #17 0xc0860635 in syscall (frame=0xd615ed38) at /usr/src/sys/i386/i386/trap.c:1035 #18 0xc0848700 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 12 #12 0xc2f62a9b in ng_address_path (here=0x0, item=0x0, address=0xc332cbe2 "ngctl99236:", retaddr=0) at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3599 3599 NGI_SET_NODE(item, dest); (kgdb) list 3594 error = ng_path2noderef(here, address, &dest, &hook); 3595 if (error) { 3596 NG_FREE_ITEM(item); 3597 return (error); 3598 } 3599 NGI_SET_NODE(item, dest); 3600 if ( hook) { 3601 NG_HOOK_REF(hook); /* don't let it go while on the queue */ 3602 NGI_SET_HOOK(item, hook); 3603 } Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #6: Sat May 10 05:27:18 PDT 2008 root@xxx:/usr/obj/usr/src/sys/XXX Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2394.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Stepping = 5 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4400<CNXT-ID,xTPR> Logical CPUs per core: 2 real memory = 534970368 (510 MB) avail memory = 513785856 (489 MB) ACPI APIC Table: <INTEL D875BZLK> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 [chifung@marx /var/log]$ dmesg | less Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-STABLE #6: Sat May 10 05:27:18 PDT 2008 root@xxx:/usr/obj/usr/src/sys/XXX Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2394.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Stepping = 5 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4400<CNXT-ID,xTPR> Logical CPUs per core: 2 real memory = 534970368 (510 MB) avail memory = 513785856 (489 MB) ACPI APIC Table: <INTEL D875BZLK> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 <Version 2.0> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <INTEL D875BZLK> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 1ff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: <ACPI CPU> on acpi0 p4tcc0: <CPU Frequency Thermal Control> on cpu0 cpu1: <ACPI CPU> on acpi0 p4tcc1: <CPU Frequency Thermal Control> on cpu1 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 agp0: <Intel 82875P host to AGP bridge> on hostb0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 vgapci0: <VGA-compatible display> mem 0xf5000000-0xf5ffffff,0xf8000000-0xfbffffff irq 16 at device 0.0 on pci1 pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0 pci2: <ACPI PCI bus> on pcib2 em0: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xac00-0xac1f mem 0xf7000000-0xf701ffff irq 18 at device 1.0 on pci2 em0: [FILTER] em0: Ethernet address: 00:07:e9:3b:bc:37 uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0xcc00-0xcc1f irq 16 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0 usb0: USB revision 1.0 uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0xd000-0xd01f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1 usb1: USB revision 1.0 uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0xd400-0xd41f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2 usb2: USB revision 1.0 uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: <Intel 82801EB (ICH5) USB controller USB-D> port 0xd800-0xd81f irq 16 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: <Intel 82801EB (ICH5) USB controller USB-D> on uhci3 usb3: USB revision 1.0 uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xf7200000-0xf72003ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0 usb4: USB revision 2.0 uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4 uhub4: 8 ports with 8 removable, self powered pcib3: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci3: <ACPI PCI bus> on pcib3 rl0: <RealTek 8139 10/100BaseTX> port 0xbc00-0xbc7f mem 0xf7100000-0xf710007f irq 22 at device 1.0 on pci3 miibus0: <MII bus> on rl0 rlphy0: <RealTek internal media interface> PHY 0 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto rl0: Ethernet address: 00:a0:4b:04:91:b4 rl0: [ITHREAD] isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel ICH5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] atapci1: <Intel ICH5 SATA150 controller> port 0xec00-0xec07,0xe800-0xe803,0xe400-0xe407,0xe000-0xe003,0xdc00-0xdc0f irq 18 at device 31.2 on pci0 atapci1: [ITHREAD] ata2: <ATA channel 0> on atapci1 ata2: [ITHREAD] ata3: <ATA channel 1> on atapci1 ata3: [ITHREAD] pci0: <serial bus, SMBus> at device 31.3 (no driver attached) acpi_button0: <Sleep Button> on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model Generic PS/2 mouse, device ID 0 fdc0: <floppy drive controller> port 0x3f0-0x3f1,0x3f2-0x3f3,0x3f4-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] cryptosoft0: <software crypto> on motherboard pmtimer0 on isa0 ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode ppbus0: <Parallel port bus> on ppc0 ppbus0: [ITHREAD] plip0: <PLIP network interface> on ppbus0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. ad0: 57241MB <WDC WD600LB-55DNA0 77.07W77> at ata0-master UDMA100 SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/ad0s1a WARNING: /var/backups was not properly dismounted WARNING: attempt to net_add_domain(netgraph) after domainfinalize() WARNING: /crypt was not properly dismounted >How-To-Repeat: Setup VPN using mpd between 2 servers. >Fix: >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200805162016.m4GKGTdI040471>