Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 24 Jun 2012 20:05:27 +0200
From:      Mateusz Guzik <mjguzik@gmail.com>
To:        Wojciech Puchar <wojtek@wojtek.tensor.gdynia.pl>
Cc:        freebsd-hackers@freebsd.org
Subject:   Re: reason for "magic" crashes.
Message-ID:  <20120624180526.GA15899@dft-labs.eu>
In-Reply-To: <alpine.BSF.2.00.1206241856090.73001@wojtek.tensor.gdynia.pl>
References:  <20120623162415.303430@gmx.com> <alpine.BSF.2.00.1206231947160.40095@wojtek.tensor.gdynia.pl> <alpine.BSF.2.00.1206241856090.73001@wojtek.tensor.gdynia.pl>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Jun 24, 2012 at 07:05:35PM +0200, Wojciech Puchar wrote:
> i've got third crash third week in a row.
> 
> Every time in sunday after 18:00, every time with rsync process
> (which means rsync based backup that is done every day, not just in
> sunday!),
> 
> you may see a crash (viewed from KVM) at
> 
> http://www.tensor.gdynia.pl/~wojtek/crash.png
> 
> what is important - syncing disk doesn't go on, system hangs here.
> 
> For 99% system is not overheating at sunday, but i will be 100% sure
> as i added ipmitool sensor logged from cron every 5 minutes.
> 
> Please give me an idea what to check.
> 
> 
> There is nothing in cron that is done at sunday.
> 
> i don't run "periodic" stuff in /etc/crontab
> 

Compile the kernel with the following:

makeoptions     DEBUG="-O0 -g"

options         KDB                     # Enable kernel debugger support.
options         DDB                     # Support DDB.
options         GDB                     # Support remote GDB.
options         DEADLKRES               # Enable the deadlock resolver
options         INVARIANTS              # Enable calls of extra sanity checking
options         INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
options         WITNESS                 # Enable checks to detect deadlocks and cycles
options         WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed
options         DIAGNOSTIC

After kernel panic ddb prompt will be waiting for you. Type in:
dump <enter>
reset <enter>

Make sure you have swap that can handle crashdumps.

See this for more details:
http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html

You can check if everything works correctly by issuing panic manually:
sysctl debug.kdb.panic=1

then typing aforementioned ddb commands. After reboot you should get
core in /var/crash.

Also provide the following:
- system version
- filesystems involved in rsync with mount details (e.g. UFS with SU+J)
- dmesg

Hopefully this will be enough for someone to help.

-- 
Mateusz Guzik <mjguzik gmail.com>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120624180526.GA15899>