Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 05 Feb 2011 19:48:59 +0600
From:      Eugene Grosbein <egrosbein@rdtc.ru>
To:        Gleb Smirnoff <glebius@freebsd.org>
Cc:        freebsd-net@freebsd.org, Alexander Motin <mav@freebsd.org>, John Baldwin <jhb@freebsd.org>
Subject:   Re: panic: bufwrite: buffer is not busy???
Message-ID:  <4D4D554B.4050407@rdtc.ru>
In-Reply-To: <20110201185026.GB62007@glebius.int.ru>
References:  <4D3011DB.9050900@frasunek.com> <4D30458D.30007@sentex.net>	<4D309983.70709@rdtc.ru> <201101141437.55421.jhb@freebsd.org>	<4D46575A.802@rdtc.ru> <4D4670C2.4050500@freebsd.org>	<4D48513C.40503@rdtc.ru> <20110201185026.GB62007@glebius.int.ru>

next in thread | previous in thread | raw e-mail | index | archive | help
On 02.02.2011 00:50, Gleb Smirnoff wrote:
> On Wed, Feb 02, 2011 at 12:30:20AM +0600, Eugene Grosbein wrote:
> E> On 31.01.2011 14:20, Julian Elischer wrote:
> E> 
> E> > replace with:
> E> > 
> E> > 3504            if ((hook == NULL) ||
> E> > 3505                NG_HOOK_NOT_VALID(hook) ||
> E> >                      ((peer = NG_HOOK_PEER(hook)) == NULL) ||
> E> > 3506                NG_HOOK_NOT_VALID(peer) ||
> E> >                      ((peernode = NG_PEER_NODE(hook)) == NULL) ||
> E> > 3507                NG_NODE_NOT_VALID(peernode)) {
> E> >                          if (peer)
> E> >                                kassert((peernode != NULL), ("peer node NULL wile peer hook exists"));
> E> > 3508                    NG_FREE_ITEM(item);
> E> 
> E> This day I have updated panicing router to RELENG_8 and combined changes supposed
> E> by Julian and Gleb. After 8 hours it has just paniced again and could not finish
> E> to write crashdump again:
> E> 
> E> Fatal trap 12: page fault while in kernel mode
> E> cpuid = 3; apic id = 06
> E> fault virtual address   = 0x63
> E> fault code              = supervisor read data, page not present
> E> instruction pointer     = 0x20:0xffffffff803d4ccd
> E> stack pointer           = 0x28:0xffffff80ebffc600
> E> frame pointer           = 0x28:0xffffff80ebffc680
> E> code segment            = base 0x0, limit 0xfffff, type 0x1b
> E>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> E> processor eflags        = interrupt enabled, resume, IOPL = 0
> E> current process         = 2390 (mpd5)
> E> trap number             = 12
> E> panic: page fault
> E> cpuid = 3
> E> Uptime: 8h3m51s
> E> Dumping 4087 MB (3 chunks)
> E>   chunk 0: 1MB (150 pages) ... ok
> E>   chunk 1: 3575MB (915088 pages) 3559 3543panic: bufwrite: buffer is not busy???
> E> cpuid = 3
> E> Uptime: 8h3m52s
> E> Automatic reboot in 15 seconds - press a key on the console to abort
> E> 
> E> # gdb kernel
> E> GNU gdb 6.1.1 [FreeBSD]
> E> Copyright 2004 Free Software Foundation, Inc.
> E> GDB is free software, covered by the GNU General Public License, and you are
> E> welcome to change it and/or distribute copies of it under certain conditions.
> E> Type "show copying" to see the conditions.
> E> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> E> This GDB was configured as "amd64-marcel-freebsd"...
> E> (gdb) l *0xffffffff803d4ccd
> E> 0xffffffff803d4ccd is in ng_pppoe_disconnect (netgraph.h:191).
> E> 186                                     int line);
> E> 187
> E> 188     static __inline void
> E> 189     _chkhook(hook_p hook, char *file, int line)
> E> 190     {
> E> 191             if (hook->hk_magic != HK_MAGIC) {
> E> 192                     printf("Accessing freed hook ");
> E> 193                     dumphook(hook, file, line);
> E> 194             }
> E> 195             hook->lastline = line;
> E> (gdb) x/i 0xffffffff803d4ccd
> E> 0xffffffff803d4ccd <ng_pppoe_disconnect+301>:   cmpl   $0x78573011,0x64(%rbx)
> 
> This looks like ng_pppoe_disconnect() was called with NULL argument.
> 
> Can you add KDB_TRACE option to kernel? Your boxes for some reason can't
> dump core, but with this option we will have at least trace.

Same box, more panics with KDB_TRACE, NETGRAPGH_DEBUG and your patch and Julian's.

First: again, no dump (not even started to dump, and no "Uptime:" written to console):

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x20000006c
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff803e5a6d
stack pointer           = 0x28:0xffffff80ec03d600
frame pointer           = 0x28:0xffffff80ec03d680
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 2390 (mpd5)
trap number             = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a
kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37
panic() at 0xffffffff8030b567 = panic+0x187
dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330
dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f
trap() at 0xffffffff804c155f = trap+0x3df
calltrap() at 0xffffffff804a8de4 = calltrap+0x8
--- trap 0xc, rip = 0xffffffff803e5a6d, rsp = 0xffffff80ec03d600, rbp = 0xffffff80ec03d680 ---
ng_parse_get_token() at 0xffffffff803e5a6d = ng_parse_get_token+0x70cd
ng_destroy_hook() at 0xffffffff803d53b2 = ng_destroy_hook+0x222
ng_rmnode() at 0xffffffff803d69bb = ng_rmnode+0x12ab
ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0
ng_parse_get_token() at 0xffffffff803e97fa = ng_parse_get_token+0xae5a
sosend_generic() at 0xffffffff80373df6 = sosend_generic+0x436
kern_sendit() at 0xffffffff803776d5 = kern_sendit+0x1a5
kern_sendit() at 0xffffffff8037790c = kern_sendit+0x3dc
sendto() at 0xffffffff803779fd = sendto+0x4d
syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5
syscall() at 0xffffffff804c10fb = syscall+0x4b
Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x8018c971c, rsp = 0x7fffffbfe838, rbp = 0x8020f3d00 ---

Then IPMI watchdog rebooted this box, after 5 minutes.

(gdb) l *0xffffffff803e5a6d
0xffffffff803e5a6d is in ng_pppoe_disconnect (netgraph.h:191).
186                                     int line);
187
188     static __inline void
189     _chkhook(hook_p hook, char *file, int line)
190     {
191             if (hook->hk_magic != HK_MAGIC) {
192                     printf("Accessing freed hook ");
193                     dumphook(hook, file, line);
194             }
195             hook->lastline = line;
(gdb) x/i 0xffffffff803e5a6d
0xffffffff803e5a6d <ng_pppoe_disconnect+301>:   cmpl   $0x78573011,0x64(%rbx)

Second: after 3 hours and half, another panic (started to dump, not finished).
Note: instruction pointer is the same, fault address differs.

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 00
fault virtual address   = 0x63
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff803e5a6d
stack pointer           = 0x28:0xffffff80ec06f600
frame pointer           = 0x28:0xffffff80ec06f680
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 2390 (mpd5)
trap number             = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a
kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37
panic() at 0xffffffff8030b567 = panic+0x187
dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330
dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f
trap() at 0xffffffff804c155f = trap+0x3df
calltrap() at 0xffffffff804a8de4 = calltrap+0x8
--- trap 0xc, rip = 0xffffffff803e5a6d, rsp = 0xffffff80ec06f600, rbp = 0xffffff80ec06f680 ---
ng_parse_get_token() at 0xffffffff803e5a6d = ng_parse_get_token+0x70cd
ng_destroy_hook() at 0xffffffff803d53b2 = ng_destroy_hook+0x222
ng_rmnode() at 0xffffffff803d69bb = ng_rmnode+0x12ab
ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0
ng_parse_get_token() at 0xffffffff803e97fa = ng_parse_get_token+0xae5a
sosend_generic() at 0xffffffff80373df6 = sosend_generic+0x436
kern_sendit() at 0xffffffff803776d5 = kern_sendit+0x1a5
kern_sendit() at 0xffffffff8037790c = kern_sendit+0x3dc
sendto() at 0xffffffff803779fd = sendto+0x4d
syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5
syscall() at 0xffffffff804c10fb = syscall+0x4b
Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x8018c971c, rsp = 0x7fffffbfe838, rbp = 0x802a867c0 ---
Uptime: 3h32m11s
Dumping 4087 MB (3 chunks)
  chunk 0: 1MB (150 pages) ... ok
  chunk 1: 3575MB (915088 pages)panic: bufwrite: buffer is not busy???
cpuid = 1
Uptime: 3h32m11s
Automatic reboot in 15 seconds - press a key on the console to abort




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D4D554B.4050407>