Date: Tue, 16 Jul 2013 11:07:37 -0400 From: John Baldwin <jhb@freebsd.org> To: freebsd-hackers@freebsd.org Cc: Yuri <yuri@rawbw.com> Subject: Re: Kernel crashes after sleep: how to debug? Message-ID: <201307161107.37460.jhb@freebsd.org> In-Reply-To: <51E3A334.8020203@rawbw.com> References: <51E3A334.8020203@rawbw.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday, July 15, 2013 3:22:28 am Yuri wrote:
>
> After sleep/wakeup cycle my 9.1-STABLE r253105 amd64 system has a
> tendency to sometimes randomly crash after a while. It doesn't happen
> every time.
> See kgdb log below. I am not sure there is enough information to lead to
> the cause of the issue.
>
> It looks like it crashes near the line:
> #7 0xffffffff8091a181 in _mtx_trylock (m=0x100000000, opts=0,
> file=<value optimized out>, line=0) at /usr/src/sys/kern/kern_mutex.c:295
> 295 if (SCHEDULER_STOPPED())
> Current language: auto; currently c
> (kgdb) l
> 290 uint64_t waittime = 0;
> 291 int contested = 0;
> 292 #endif
> 293 int rval;
> 294
> 295 if (SCHEDULER_STOPPED())
> 296 return (1);
> 297
> 298 KASSERT(m->mtx_lock != MTX_DESTROYED,
> 299 ("mtx_trylock() of destroyed mutex @ %s:%d", file,
> line));
>
> Current thread was:
> * 67 Thread 100064 (PID=5: pagedaemon) doadump (textdump=<value
> optimized out>) at pcpu.h:234
>
> How to find the cause of the crash?
>
> Yuri
>
>
> --- kgdb log ---
> # kgdb /boot/kernel/kernel vmcore.0
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x100000018
> fault code = supervisor read data, page not present
> instruction pointer = 0x20:0xffffffff8091a181
> stack pointer = 0x28:0xffffff80d51c6ab0
> frame pointer = 0x28:0xffffff80d51c6ad0
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 5 (pagedaemon)
> trap number = 12
> panic: page fault
> cpuid = 0
> KDB: stack backtrace:
> #0 0xffffffff80968416 at kdb_backtrace+0x66
> #1 0xffffffff8092e43e at panic+0x1ce
> #2 0xffffffff80d12940 at trap_fatal+0x290
> #3 0xffffffff80d12ca1 at trap_pfault+0x211
> #4 0xffffffff80d13254 at trap+0x344
> #5 0xffffffff80cfc583 at calltrap+0x8
> #6 0xffffffff80baea78 at vm_pageout+0x998
> #7 0xffffffff808fc10f at fork_exit+0x11f
> #8 0xffffffff80cfcaae at fork_trampoline+0xe
> Uptime: 2h21m27s
> Dumping 407 out of 2919 MB:..4%..12%..24%..32%..44%..52%..63%..71%..83%..91%
>
> Reading symbols from /boot/modules/cuse4bsd.ko...done.
> Loaded symbols for /boot/modules/cuse4bsd.ko
> Reading symbols from /boot/kernel/linux.ko...Reading symbols from
> /boot/kernel/linux.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/linux.ko
> Reading symbols from /usr/local/libexec/linux_adobe/linux_adobe.ko...done.
> Loaded symbols for /usr/local/libexec/linux_adobe/linux_adobe.ko
> Reading symbols from /boot/kernel/radeon.ko...Reading symbols from
> /boot/kernel/radeon.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/radeon.ko
> Reading symbols from /boot/kernel/drm.ko...Reading symbols from
> /boot/kernel/drm.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/drm.ko
> #0 doadump (textdump=<value optimized out>) at pcpu.h:234
> 234 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) bt
> #0 doadump (textdump=<value optimized out>) at pcpu.h:234
> #1 0xffffffff8092df16 in kern_reboot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:449
> #2 0xffffffff8092e417 in panic (fmt=0x1 <Address 0x1 out of bounds>) at
> /usr/src/sys/kern/kern_shutdown.c:637
> #3 0xffffffff80d12940 in trap_fatal (frame=0xc, eva=<value optimized
> out>) at /usr/src/sys/amd64/amd64/trap.c:879
> #4 0xffffffff80d12ca1 in trap_pfault (frame=0xffffff80d51c6a00,
> usermode=0) at /usr/src/sys/amd64/amd64/trap.c:795
> #5 0xffffffff80d13254 in trap (frame=0xffffff80d51c6a00) at
> /usr/src/sys/amd64/amd64/trap.c:463
> #6 0xffffffff80cfc583 in calltrap () at
> /usr/src/sys/amd64/amd64/exception.S:232
> #7 0xffffffff8091a181 in _mtx_trylock (m=0x100000000, opts=0,
> file=<value optimized out>, line=0) at /usr/src/sys/kern/kern_mutex.c:295
> #8 0xffffffff80baea78 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:829
> #9 0xffffffff808fc10f in fork_exit (callout=0xffffffff80bae0e0
> <vm_pageout>, arg=0x0, frame=0xffffff80d51c6c40)
> at /usr/src/sys/kern/kern_fork.c:988
> #10 0xffffffff80cfcaae in fork_trampoline () at
> /usr/src/sys/amd64/amd64/exception.S:606
> #11 0x0000000000000000 in ?? ()
Can you go to frame 8 and do 'l' in kgdb?
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201307161107.37460.jhb>