Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 04 Nov 2013 04:49:28 -0800
From:      Alfred Perlstein <alfred@freebsd.org>
To:        Colin Percival <cperciva@freebsd.org>,  FreeBSD Hackers <freebsd-hackers@freebsd.org>, FreeBSD current <freebsd-current@freebsd.org>,  Jordan Hubbard <jkh@ixsystems.com>
Subject:   Re: Automated submission of kernel panic reports: sysutils/panicmail
Message-ID:  <527797D8.5040404@freebsd.org>
In-Reply-To: <527779ED.9040303@freebsd.org>
References:  <527779ED.9040303@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help

On 11/4/13, 2:41 AM, Colin Percival wrote:
> Hi all,
>
> After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have
> now added sysutils/panicmail to the FreeBSD ports tree.  If you install this
> and add
> 	panicmail_enable="YES"
> to your /etc/rc.conf, a panic report will be generated and sent to root@ for
> you to review and submit (via email).  You can skip the reviewing step and
> submit panics automatically by setting panicmail_autosubmit="YES".
>
> The panics submitted are encrypted to an RSA key which I hold in order to keep
> them secure in transit; and I intend to keep the raw panic reports confidential
> except to the minimum extent necessary for other developers to help me process
> the incoming reports.
>
> If I receive enough panic reports to be useful, I hope to provide developers
> with aggregate statistics.  This may include:
>
> * regular email reports listing the "top panics", to help guide developers
> towards the most fertile areas for stability improvements;
>
> * email to specific developers alerting them to recurring panics in code they
> maintain (especially if it becomes clear that the panic has been recently
> introduced); and
>
> * guidance to re@ and secteam@ about how often a particular panic occurs if
> an errata notice is being considered
>
> as well as other yet-to-be-imagined reports of a similarly aggregate and
> anonymized nature.
>
> So please install the sysutils/panicmail port and enable it in rc.conf!  This
> all depends on getting useful data, and I can't do that without your help.
>
Colin, have you had a few minutes to check out the crash reporting 
facilities in FreeNAS?

The reason I ask is that:

1) we would like to share code.
2) we have this running for a few months now and have a huge corpus of 
information.
3) we are building a nice UI (screenshots attached) over it, we have a 
couple of thousands of lines of code we can share for this.

Our scripts can be found here:

1) A startup script that sends us the crashes on system start:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/etc/rc.d/ix_textdump 

2) A script to submit data at boot OR from command line that sends more 
comprehensive system information "ixdiagnose":
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/ixdiagnose 

3) A very simple script to upload that report:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/crashuploader 


We send a minimal set of information: kernel stack trace, ddb buffer and 
hardware.  Just enough to get some very, very handy stuff.

I can share with you offline the crash server code, it's django and 
relatively straight forward.

The screenshots can also be seen at:
http://people.freebsd.org/~alfred/crashreporter/

We could modify our framework for FreeBSD to do so by checking for a 
sentinel file depending on the host type and only auto-sending if we see 
that.


-Alfred



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?527797D8.5040404>