From owner-freebsd-net@FreeBSD.ORG Mon Jan 31 06:32:08 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 786A4106566C; Mon, 31 Jan 2011 06:32:08 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id E27CE8FC17; Mon, 31 Jan 2011 06:32:07 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id p0V6VxfC048545; Mon, 31 Jan 2011 12:31:59 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D46575A.802@rdtc.ru> Date: Mon, 31 Jan 2011 12:31:54 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: John Baldwin References: <4D3011DB.9050900@frasunek.com> <4D30458D.30007@sentex.net> <4D309983.70709@rdtc.ru> <201101141437.55421.jhb@freebsd.org> In-Reply-To: <201101141437.55421.jhb@freebsd.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Przemyslaw Frasunek , Mike Tancsa Subject: Re: panic: bufwrite: buffer is not busy??? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Jan 2011 06:32:08 -0000 On 15.01.2011 01:37, John Baldwin wrote: > On Friday, January 14, 2011 1:44:19 pm Eugene Grosbein wrote: >> On 14.01.2011 18:46, Mike Tancsa wrote: >> >>>> I'm using mpd 5.5 on three PPPoE routers, each servicing about 300 PPPoE >>>> concurrent sessions. Routers are based on Intel SR1630GP hardware platforms and >>>> runs FreeBSD 7.3-RELEASE. >>>> >>>> I'm experiencing stability issues related to Netgraph. None of above routers can >>>> survive more than 20-30 days of uptime under typical load. There are different >>>> flavors of kernel panics, but all are somehow related to netgraph. Typical >>>> backtraces follow >>> >>> I also have stability issues on RELENG_8. >>> >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=153497 >> >> And for one of my servers (8.2-PRERELEASE/amd64 with 4GB RAM) I just cannot obtain crashdump, >> it cannot finish to write it. For example, it happened an hour ago: >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 2; apic id = 04 >> fault virtual address = 0x200000040 >> fault code = supervisor read data, page not present >> instruction pointer = 0x20:0xffffffff803cc979 > > Assuming your kernel is built with debug symbols (which is the default), one > thing you can do to aid in debugging is this: > > gdb /boot/kernel/kernel > (gdb) l *0xffffffff803cc979 > > Where the 0xfff bit is the part of the 'instruction pointer' value > above after the colon (:) and then send the output of that in your e-mail to > the list. This allows us to the source line at which the fault occurred. > Yesterday I've got another kernel panic of this kind with RELENG_8 updated 20 January and it still could not finish writing of crashdump: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x200000030 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803c1315 stack pointer = 0x28:0xffffff8000130780 frame pointer = 0x28:0xffffff80001307a0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (irq259: em1:rx 0) trap number = 12 panic: page fault cpuid = 1 Uptime: 19h41m8s Dumping 4087 MB (3 chunks) chunk 0: 1MB (150 pages) ... ok chunk 1: 3575MB (915088 pages) 3559 3543panic: bufwrite: buffer is not busy??? cpuid = 1 Uptime: 19h41m9s Automatic reboot in 15 seconds - press a key on the console to abort This time I have all debug symbols handy: # gdb kernel GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... (gdb) l *0xffffffff803c1315 0xffffffff803c1315 is in ng_address_hook (/home/src/sys/netgraph/ng_base.c:3504). 3499 * Quick sanity check.. 3500 * Since a hook holds a reference on it's node, once we know 3501 * that the peer is still connected (even if invalid,) we know 3502 * that the peer node is present, though maybe invalid. 3503 */ 3504 if ((hook == NULL) || 3505 NG_HOOK_NOT_VALID(hook) || 3506 NG_HOOK_NOT_VALID(peer = NG_HOOK_PEER(hook)) || 3507 NG_NODE_NOT_VALID(peernode = NG_PEER_NODE(hook))) { 3508 NG_FREE_ITEM(item);