Date: Sat, 05 Feb 2011 19:48:59 +0600 From: Eugene Grosbein <egrosbein@rdtc.ru> To: Gleb Smirnoff <glebius@freebsd.org> Cc: freebsd-net@freebsd.org, Alexander Motin <mav@freebsd.org>, John Baldwin <jhb@freebsd.org> Subject: Re: panic: bufwrite: buffer is not busy??? Message-ID: <4D4D554B.4050407@rdtc.ru> In-Reply-To: <20110201185026.GB62007@glebius.int.ru> References: <4D3011DB.9050900@frasunek.com> <4D30458D.30007@sentex.net> <4D309983.70709@rdtc.ru> <201101141437.55421.jhb@freebsd.org> <4D46575A.802@rdtc.ru> <4D4670C2.4050500@freebsd.org> <4D48513C.40503@rdtc.ru> <20110201185026.GB62007@glebius.int.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On 02.02.2011 00:50, Gleb Smirnoff wrote:
> On Wed, Feb 02, 2011 at 12:30:20AM +0600, Eugene Grosbein wrote:
> E> On 31.01.2011 14:20, Julian Elischer wrote:
> E>
> E> > replace with:
> E> >
> E> > 3504 if ((hook == NULL) ||
> E> > 3505 NG_HOOK_NOT_VALID(hook) ||
> E> > ((peer = NG_HOOK_PEER(hook)) == NULL) ||
> E> > 3506 NG_HOOK_NOT_VALID(peer) ||
> E> > ((peernode = NG_PEER_NODE(hook)) == NULL) ||
> E> > 3507 NG_NODE_NOT_VALID(peernode)) {
> E> > if (peer)
> E> > kassert((peernode != NULL), ("peer node NULL wile peer hook exists"));
> E> > 3508 NG_FREE_ITEM(item);
> E>
> E> This day I have updated panicing router to RELENG_8 and combined changes supposed
> E> by Julian and Gleb. After 8 hours it has just paniced again and could not finish
> E> to write crashdump again:
> E>
> E> Fatal trap 12: page fault while in kernel mode
> E> cpuid = 3; apic id = 06
> E> fault virtual address = 0x63
> E> fault code = supervisor read data, page not present
> E> instruction pointer = 0x20:0xffffffff803d4ccd
> E> stack pointer = 0x28:0xffffff80ebffc600
> E> frame pointer = 0x28:0xffffff80ebffc680
> E> code segment = base 0x0, limit 0xfffff, type 0x1b
> E> = DPL 0, pres 1, long 1, def32 0, gran 1
> E> processor eflags = interrupt enabled, resume, IOPL = 0
> E> current process = 2390 (mpd5)
> E> trap number = 12
> E> panic: page fault
> E> cpuid = 3
> E> Uptime: 8h3m51s
> E> Dumping 4087 MB (3 chunks)
> E> chunk 0: 1MB (150 pages) ... ok
> E> chunk 1: 3575MB (915088 pages) 3559 3543panic: bufwrite: buffer is not busy???
> E> cpuid = 3
> E> Uptime: 8h3m52s
> E> Automatic reboot in 15 seconds - press a key on the console to abort
> E>
> E> # gdb kernel
> E> GNU gdb 6.1.1 [FreeBSD]
> E> Copyright 2004 Free Software Foundation, Inc.
> E> GDB is free software, covered by the GNU General Public License, and you are
> E> welcome to change it and/or distribute copies of it under certain conditions.
> E> Type "show copying" to see the conditions.
> E> There is absolutely no warranty for GDB. Type "show warranty" for details.
> E> This GDB was configured as "amd64-marcel-freebsd"...
> E> (gdb) l *0xffffffff803d4ccd
> E> 0xffffffff803d4ccd is in ng_pppoe_disconnect (netgraph.h:191).
> E> 186 int line);
> E> 187
> E> 188 static __inline void
> E> 189 _chkhook(hook_p hook, char *file, int line)
> E> 190 {
> E> 191 if (hook->hk_magic != HK_MAGIC) {
> E> 192 printf("Accessing freed hook ");
> E> 193 dumphook(hook, file, line);
> E> 194 }
> E> 195 hook->lastline = line;
> E> (gdb) x/i 0xffffffff803d4ccd
> E> 0xffffffff803d4ccd <ng_pppoe_disconnect+301>: cmpl $0x78573011,0x64(%rbx)
>
> This looks like ng_pppoe_disconnect() was called with NULL argument.
>
> Can you add KDB_TRACE option to kernel? Your boxes for some reason can't
> dump core, but with this option we will have at least trace.
Same box, more panics with KDB_TRACE, NETGRAPGH_DEBUG and your patch and Julian's.
First: again, no dump (not even started to dump, and no "Uptime:" written to console):
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address = 0x20000006c
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803e5a6d
stack pointer = 0x28:0xffffff80ec03d600
frame pointer = 0x28:0xffffff80ec03d680
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 2390 (mpd5)
trap number = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a
kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37
panic() at 0xffffffff8030b567 = panic+0x187
dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330
dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f
trap() at 0xffffffff804c155f = trap+0x3df
calltrap() at 0xffffffff804a8de4 = calltrap+0x8
--- trap 0xc, rip = 0xffffffff803e5a6d, rsp = 0xffffff80ec03d600, rbp = 0xffffff80ec03d680 ---
ng_parse_get_token() at 0xffffffff803e5a6d = ng_parse_get_token+0x70cd
ng_destroy_hook() at 0xffffffff803d53b2 = ng_destroy_hook+0x222
ng_rmnode() at 0xffffffff803d69bb = ng_rmnode+0x12ab
ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0
ng_parse_get_token() at 0xffffffff803e97fa = ng_parse_get_token+0xae5a
sosend_generic() at 0xffffffff80373df6 = sosend_generic+0x436
kern_sendit() at 0xffffffff803776d5 = kern_sendit+0x1a5
kern_sendit() at 0xffffffff8037790c = kern_sendit+0x3dc
sendto() at 0xffffffff803779fd = sendto+0x4d
syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5
syscall() at 0xffffffff804c10fb = syscall+0x4b
Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x8018c971c, rsp = 0x7fffffbfe838, rbp = 0x8020f3d00 ---
Then IPMI watchdog rebooted this box, after 5 minutes.
(gdb) l *0xffffffff803e5a6d
0xffffffff803e5a6d is in ng_pppoe_disconnect (netgraph.h:191).
186 int line);
187
188 static __inline void
189 _chkhook(hook_p hook, char *file, int line)
190 {
191 if (hook->hk_magic != HK_MAGIC) {
192 printf("Accessing freed hook ");
193 dumphook(hook, file, line);
194 }
195 hook->lastline = line;
(gdb) x/i 0xffffffff803e5a6d
0xffffffff803e5a6d <ng_pppoe_disconnect+301>: cmpl $0x78573011,0x64(%rbx)
Second: after 3 hours and half, another panic (started to dump, not finished).
Note: instruction pointer is the same, fault address differs.
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 00
fault virtual address = 0x63
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803e5a6d
stack pointer = 0x28:0xffffff80ec06f600
frame pointer = 0x28:0xffffff80ec06f680
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 2390 (mpd5)
trap number = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
X_db_sym_numargs() at 0xffffffff801a227a = X_db_sym_numargs+0x15a
kdb_backtrace() at 0xffffffff8033d547 = kdb_backtrace+0x37
panic() at 0xffffffff8030b567 = panic+0x187
dblfault_handler() at 0xffffffff804c0ca0 = dblfault_handler+0x330
dblfault_handler() at 0xffffffff804c107f = dblfault_handler+0x70f
trap() at 0xffffffff804c155f = trap+0x3df
calltrap() at 0xffffffff804a8de4 = calltrap+0x8
--- trap 0xc, rip = 0xffffffff803e5a6d, rsp = 0xffffff80ec06f600, rbp = 0xffffff80ec06f680 ---
ng_parse_get_token() at 0xffffffff803e5a6d = ng_parse_get_token+0x70cd
ng_destroy_hook() at 0xffffffff803d53b2 = ng_destroy_hook+0x222
ng_rmnode() at 0xffffffff803d69bb = ng_rmnode+0x12ab
ng_snd_item() at 0xffffffff803d8520 = ng_snd_item+0x3f0
ng_parse_get_token() at 0xffffffff803e97fa = ng_parse_get_token+0xae5a
sosend_generic() at 0xffffffff80373df6 = sosend_generic+0x436
kern_sendit() at 0xffffffff803776d5 = kern_sendit+0x1a5
kern_sendit() at 0xffffffff8037790c = kern_sendit+0x3dc
sendto() at 0xffffffff803779fd = sendto+0x4d
syscallenter() at 0xffffffff8034a015 = syscallenter+0x1e5
syscall() at 0xffffffff804c10fb = syscall+0x4b
Xfast_syscall() at 0xffffffff804a90c2 = Xfast_syscall+0xe2
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x8018c971c, rsp = 0x7fffffbfe838, rbp = 0x802a867c0 ---
Uptime: 3h32m11s
Dumping 4087 MB (3 chunks)
chunk 0: 1MB (150 pages) ... ok
chunk 1: 3575MB (915088 pages)panic: bufwrite: buffer is not busy???
cpuid = 1
Uptime: 3h32m11s
Automatic reboot in 15 seconds - press a key on the console to abort
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D4D554B.4050407>