From owner-freebsd-bugs@FreeBSD.ORG Sat Mar 22 10:10:01 2008 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99E72106566C for ; Sat, 22 Mar 2008 10:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 96B648FC26 for ; Sat, 22 Mar 2008 10:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m2MAA1Q5096007 for ; Sat, 22 Mar 2008 10:10:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m2MAA1eS096006; Sat, 22 Mar 2008 10:10:01 GMT (envelope-from gnats) Resent-Date: Sat, 22 Mar 2008 10:10:01 GMT Resent-Message-Id: <200803221010.m2MAA1eS096006@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, "Alexander V. Shulikov" Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E4A71065672 for ; Sat, 22 Mar 2008 10:00:10 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 1CEB28FC16 for ; Sat, 22 Mar 2008 10:00:10 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m2MA04JQ054789 for ; Sat, 22 Mar 2008 10:00:04 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.2/8.14.1/Submit) id m2MA048T054788; Sat, 22 Mar 2008 10:00:04 GMT (envelope-from nobody) Message-Id: <200803221000.m2MA048T054788@www.freebsd.org> Date: Sat, 22 Mar 2008 10:00:04 GMT From: "Alexander V. Shulikov" To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/121955: freebsd 7.0 panic with mpd X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Mar 2008 10:10:01 -0000 >Number: 121955 >Category: kern >Synopsis: freebsd 7.0 panic with mpd >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Mar 22 10:10:01 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Alexander V. Shulikov >Release: 7.0-RELEASE >Organization: ISP DonEC >Environment: FreeBSD hostname 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sat Mar 22 10:47:30 EET 2008 hostname:/usr/obj/usr/src/sys/kernconf i386 >Description: I use FreeBSD server as pptp server with mpd-5.0. Before 7.0 it work under FreeBSD 6.2+mpd-4.4. When I try update to FreeBSD 6.3 I received systems reboot in different intervals of time and no crash dumps. Then I try configure new server on FreeBSD 7.0. Kernel was build with GENERIC config with turned of options COMPAT_FREEBSD5 and 6 and added IPFW, DUMMYNET, option HZ=1000 and some drivers with nodevice. options IPFIREWALL #firewall options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default options IPFIREWALL_FORWARD #packet destination changes options IPDIVERT #divert sockets options DUMMYNET options HZ=1000 With this kernel system works. But when I turned on mpd-5.0 and clients connecting to them after some time (3 min - 30 min) system or freeze with message: Fault double fault: eip = .. esp = .. or panic with crash dump. # cat info.0 Dump header from device /dev/ad4s1b Architecture: i386 Architecture Version: 2 Dump Length: 65822720B (62 MB) Blocksize: 512 Dumptime: Sat Mar 22 08:13:28 2008 Hostname: uzbek.matrixhome.net Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.0-RELEASE #0: Fri Mar 21 22:08:21 EET 2008 hostname:/usr/obj/usr/src/sys/kernconf Panic String: double fault Dump Parity: 3717215795 Bounds: 0 Dump Status: good # kgdb /root/debug/kernel.debug /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal double fault: eip = 0xc40914f5 esp = 0xe3ff4000 ebp = 0xe3ff40b4 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 Uptime: 3m43s Physical memory: 1015 MB Dumping 62 MB: 47 31 15 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0xc05591c7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0559489 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc079007b in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:928 #4 0xc40914f5 in ?? () #5 0xc4238980 in ?? () #6 0xc40bc7e0 in ?? () .....there is many addresses, that I can't find in *.debug with addr2line #439 0x00000000 in ?? () #440 0xe3ff4a14 in ?? () #441 0xe3ff49fc in ?? () #442 0xc061495e in ipfw_chk (args=0xc4238ad4) at /usr/src/sys/netinet/ip_fw2.c:2659 Previous frame inner to this frame (corrupt stack?) Then I rebuild kernel added: options NETGRAPH # netgraph(4) system options NETGRAPH_DEBUG # enable extra debugging, this # affects netgraph(4) and nodes # Node types options NETGRAPH_CAR options NETGRAPH_IFACE options NETGRAPH_KSOCKET options NETGRAPH_NETFLOW options NETGRAPH_PPP options NETGRAPH_PPTPGRE options NETGRAPH_SOCKET options NETGRAPH_TCPMSS options NETGRAPH_TEE options NETGRAPH_VJC After this kldstat: # kldstat Id Refs Address Size Name 1 10 0xc0400000 533f50 kernel 2 1 0xc0934000 6a32c acpi.ko 3 1 0xc3fcd000 3000 pflog.ko 4 1 0xc3fd0000 33000 pf.ko 5 1 0xc40dc000 4000 ng_mppc.ko 6 1 0xc40e0000 2000 rc4.ko 7 1 0xc415d000 2000 blank_saver.ko And panic was: # cat /var/crash/info.1 Dump header from device /dev/ad4s1b Architecture: i386 Architecture Version: 2 Dump Length: 61734912B (58 MB) Blocksize: 512 Dumptime: Sat Mar 22 11:17:16 2008 Hostname: uzbek.matrixhome.net Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.0-RELEASE #0: Sat Mar 22 10:47:30 EET 2008 hostname:/usr/obj/usr/src/sys/kernconf Panic String: double fault Dump Parity: 3313853732 Bounds: 1 Dump Status: good # kgdb /usr/obj/usr/src/sys/uzbek/kernel.debug /var/crash/vmcore.1 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal double fault: eip = 0xc42bbdfb esp = 0xe3fedff4 ebp = 0xe3fee030 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 Uptime: 7m16s Physical memory: 1015 MB Dumping 58 MB: (CTRL-C to abort) 43 (CTRL-C to abort) 27 (CTRL-C to abort) 11 #0 doadump () at pcpu.h:195 195 pcpu.h: No such file or directory. in pcpu.h (kgdb) backtrace #0 doadump () at pcpu.h:195 #1 0xc055a0b7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc055a379 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc07ac2ab in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:928 #4 0xc42bbdfb in ?? () Cannot access memory at address 0xe3fedff4 (kgdb) Some notes: Server used for connecting 100-150 tunnels at one time. In system used ipfw and pf. ipfw for dummynet and count with net.inet.ip.fw.one_pass=0 pf for filtering, nat and scrub The same system on FreeBSD 6.2-RELEASE-p11 (mpd-5.0, ipfw. pf) don't panic after 3 minutes. I can't update my server from 6.2, because in this configuration I have troubles in 6.3 and in 7.0. System or freezes or panic and reboot. >How-To-Repeat: Read full description. >Fix: >Release-Note: >Audit-Trail: >Unformatted: