From owner-freebsd-net@FreeBSD.ORG Sat Feb 5 13:49:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F35EF1065670; Sat, 5 Feb 2011 13:49:09 +0000 (UTC) (envelope-from egrosbein@rdtc.ru) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [62.231.161.221]) by mx1.freebsd.org (Postfix) with ESMTP id A6B6F8FC13; Sat, 5 Feb 2011 13:49:08 +0000 (UTC) Received: from eg.sd.rdtc.ru (localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.14.4/8.14.4) with ESMTP id p15Dn4xM092068; Sat, 5 Feb 2011 19:49:05 +0600 (NOVT) (envelope-from egrosbein@rdtc.ru) Message-ID: <4D4D554B.4050407@rdtc.ru> Date: Sat, 05 Feb 2011 19:48:59 +0600 From: Eugene Grosbein User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.2.13) Gecko/20110112 Thunderbird/3.1.7 MIME-Version: 1.0 To: Gleb Smirnoff References: <4D3011DB.9050900@frasunek.com> <4D30458D.30007@sentex.net> <4D309983.70709@rdtc.ru> <201101141437.55421.jhb@freebsd.org> <4D46575A.802@rdtc.ru> <4D4670C2.4050500@freebsd.org> <4D48513C.40503@rdtc.ru> <20110201185026.GB62007@glebius.int.ru> In-Reply-To: <20110201185026.GB62007@glebius.int.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Alexander Motin , John Baldwin Subject: Re: panic: bufwrite: buffer is not busy??? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Feb 2011 13:49:10 -0000 On 02.02.2011 00:50, Gleb Smirnoff wrote: > On Wed, Feb 02, 2011 at 12:30:20AM +0600, Eugene Grosbein wrote: > E> On 31.01.2011 14:20, Julian Elischer wrote: > E> > E> > replace with: > E> > > E> > 3504 if ((hook == NULL) || > E> > 3505 NG_HOOK_NOT_VALID(hook) || > E> > ((peer = NG_HOOK_PEER(hook)) == NULL) || > E> > 3506 NG_HOOK_NOT_VALID(peer) || > E> > ((peernode = NG_PEER_NODE(hook)) == NULL) || > E> > 3507 NG_NODE_NOT_VALID(peernode)) { > E> > if (peer) > E> > kassert((peernode != NULL), ("peer node NULL wile peer hook exists")); > E> > 3508 NG_FREE_ITEM(item); > E> > E> This day I have updated panicing router to RELENG_8 and combined changes supposed > E> by Julian and Gleb. After 8 hours it has just paniced again and could not finish > E> to write crashdump again: > E> > E> Fatal trap 12: page fault while in kernel mode > E> cpuid = 3; apic id = 06 > E> fault virtual address = 0x63 > E> fault code = supervisor read data, page not present > E> instruction pointer = 0x20:0xffffffff803d4ccd > E> stack pointer = 0x28:0xffffff80ebffc600 > E> frame pointer = 0x28:0xffffff80ebffc680 > E> code segment = base 0x0, limit 0xfffff, type 0x1b > E> = DPL 0, pres 1, long 1, def32 0, gran 1 > E> processor eflags = interrupt enabled, resume, IOPL = 0 > E> current process = 2390 (mpd5) > E> trap number = 12 > E> panic: page fault > E> cpuid = 3 > E> Uptime: 8h3m51s > E> Dumping 4087 MB (3 chunks) > E> chunk 0: 1MB (150 pages) ... ok > E> chunk 1: 3575MB (915088 pages) 3559 3543panic: bufwrite: buffer is not busy??? > E> cpuid = 3 > E> Uptime: 8h3m52s > E> Automatic reboot in 15 seconds - press a key on the console to abort > E> > E> # gdb kernel > E> GNU gdb 6.1.1 [FreeBSD] > E> Copyright 2004 Free Software Foundation, Inc. > E> GDB is free software, covered by the GNU General Public License, and you are > E> welcome to change it and/or distribute copies of it under certain conditions. > E> Type "show copying" to see the conditions. > E> There is absolutely no warranty for GDB. Type "show warranty" for details. > E> This GDB was configured as "amd64-marcel-freebsd"... > E> (gdb) l *0xffffffff803d4ccd > E> 0xffffffff803d4ccd is in ng_pppoe_disconnect (netgraph.h:191). > E> 186 int line); > E> 187 > E> 188 static __inline void > E> 189 _chkhook(hook_p hook, char *file, int line) > E> 190 { > E> 191 if (hook->hk_magic != HK_MAGIC) { > E> 192 printf("Accessing freed hook "); > E> 193 dumphook(hook, file, line); > E> 194 } > E> 195 hook->lastline = line; > E> (gdb) x/i 0xffffffff803d4ccd > E> 0xffffffff803d4ccd : cmpl $0x78573011,0x64(%rbx) > > This looks like ng_pppoe_disconnect() was called with NULL argument. > > Can you add KDB_TRACE option to kernel? Your boxes for some reason can't > dump core, but with this option we will have at least trace. Same box, more panics with KDB_TRACE, NETGRAPGH_DEBUG and your patch and Julian's. First: again, no dump (not even started to dump, and no "Uptime:" written to console): Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 06 fault virtual address = 0x20000006c fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803e5a6d stack pointer = 0x28:0xffffff80ec03d600 frame pointer = 0x28:0xffffff80ec03d680 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2390 (mpd5) trap number = 12 panic: page fault cpuid = 3 KDB: stack backtrace: X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37 panic() at 0xffffffff8030b567 = panic+0x187 dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330 dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f trap() at 0xffffffff804c155f = trap+0x3df calltrap() at 0xffffffff804a8de4 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff803e5a6d, rsp = 0xffffff80ec03d600, rbp = 0xffffff80ec03d680 --- ng_parse_get_token() at 0xffffffff803e5a6d = ng_parse_get_token+0x70cd ng_destroy_hook() at 0xffffffff803d53b2 = ng_destroy_hook+0x222 ng_rmnode() at 0xffffffff803d69bb = ng_rmnode+0x12ab ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0 ng_parse_get_token() at 0xffffffff803e97fa = ng_parse_get_token+0xae5a sosend_generic() at 0xffffffff80373df6 = sosend_generic+0x436 kern_sendit() at 0xffffffff803776d5 = kern_sendit+0x1a5 kern_sendit() at 0xffffffff8037790c = kern_sendit+0x3dc sendto() at 0xffffffff803779fd = sendto+0x4d syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5 syscall() at 0xffffffff804c10fb = syscall+0x4b Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2 --- syscall (133, FreeBSD ELF64, sendto), rip = 0x8018c971c, rsp = 0x7fffffbfe838, rbp = 0x8020f3d00 --- Then IPMI watchdog rebooted this box, after 5 minutes. (gdb) l *0xffffffff803e5a6d 0xffffffff803e5a6d is in ng_pppoe_disconnect (netgraph.h:191). 186 int line); 187 188 static __inline void 189 _chkhook(hook_p hook, char *file, int line) 190 { 191 if (hook->hk_magic != HK_MAGIC) { 192 printf("Accessing freed hook "); 193 dumphook(hook, file, line); 194 } 195 hook->lastline = line; (gdb) x/i 0xffffffff803e5a6d 0xffffffff803e5a6d : cmpl $0x78573011,0x64(%rbx) Second: after 3 hours and half, another panic (started to dump, not finished). Note: instruction pointer is the same, fault address differs. Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 00 fault virtual address = 0x63 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff803e5a6d stack pointer = 0x28:0xffffff80ec06f600 frame pointer = 0x28:0xffffff80ec06f680 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2390 (mpd5) trap number = 12 panic: page fault cpuid = 1 KDB: stack backtrace: X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37 panic() at 0xffffffff8030b567 = panic+0x187 dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330 dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f trap() at 0xffffffff804c155f = trap+0x3df calltrap() at 0xffffffff804a8de4 = calltrap+0x8 --- trap 0xc, rip = 0xffffffff803e5a6d, rsp = 0xffffff80ec06f600, rbp = 0xffffff80ec06f680 --- ng_parse_get_token() at 0xffffffff803e5a6d = ng_parse_get_token+0x70cd ng_destroy_hook() at 0xffffffff803d53b2 = ng_destroy_hook+0x222 ng_rmnode() at 0xffffffff803d69bb = ng_rmnode+0x12ab ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0 ng_parse_get_token() at 0xffffffff803e97fa = ng_parse_get_token+0xae5a sosend_generic() at 0xffffffff80373df6 = sosend_generic+0x436 kern_sendit() at 0xffffffff803776d5 = kern_sendit+0x1a5 kern_sendit() at 0xffffffff8037790c = kern_sendit+0x3dc sendto() at 0xffffffff803779fd = sendto+0x4d syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5 syscall() at 0xffffffff804c10fb = syscall+0x4b Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2 --- syscall (133, FreeBSD ELF64, sendto), rip = 0x8018c971c, rsp = 0x7fffffbfe838, rbp = 0x802a867c0 --- Uptime: 3h32m11s Dumping 4087 MB (3 chunks) chunk 0: 1MB (150 pages) ... ok chunk 1: 3575MB (915088 pages)panic: bufwrite: buffer is not busy??? cpuid = 1 Uptime: 3h32m11s Automatic reboot in 15 seconds - press a key on the console to abort