From owner-freebsd-current@FreeBSD.ORG Fri May 29 18:23:03 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F231F10656B8; Fri, 29 May 2009 18:23:03 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [192.147.25.65]) by mx1.freebsd.org (Postfix) with ESMTP id AD74D8FC0C; Fri, 29 May 2009 18:23:03 +0000 (UTC) (envelope-from ler@lerctr.org) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=lerami; d=lerctr.org; h=Received:Date:From:To:cc:Subject:In-Reply-To:Message-ID:References:User-Agent:MIME-Version:Content-Type:X-Spam-Score:X-LERCTR-Spam-Score:X-Spam-Report:X-LERCTR-Spam-Report:DomainKey-Status; b=P0LWLWwxM1R6cdIjnx8HVNwCcYeh8LKQHIjTF+YGT1U3Ll25F39ciR1JNJFiRsmkO86vsAnVfFpZNwZslE9z6J5pw9zp0b3vaUgN50fMOT2xJJU5wYB1wxoT3027rMh5jx4itWGEtkEfoVcngY4OD11U3bDFSwQK9WSRDvfcmgI=; Received: from thebighonker.lerctr.org ([192.147.25.65]:57126) by thebighonker.lerctr.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1MA6jN-000KjK-KN; Fri, 29 May 2009 13:23:02 -0500 Date: Fri, 29 May 2009 13:22:59 -0500 (CDT) From: Larry Rosenman To: Kip Macy In-Reply-To: Message-ID: References: <3c1674c90905242253n544c3f0cqb10952f349391ce7@mail.gmail.com> <454b8cc37c60ab7af2663ba70ddbfd59.squirrel@webmail.lerctr.org> <5a9a181a12e9e4ef864d23ae063f7277.squirrel@webmail.lerctr.org> <3c1674c90905280055h740bce23p33b18fefacf31196@mail.gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -4.2 (----) X-LERCTR-Spam-Score: -4.2 (----) X-Spam-Report: SpamScore (-4.2/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TW_WU=0.077 X-LERCTR-Spam-Report: SpamScore (-4.2/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, SARE_SUB_OBFU_OTHER=0.135, TW_WU=0.077 DomainKey-Status: no signature Cc: freebsd-current@freebsd.org Subject: Re: ZFS Crash X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 May 2009 18:23:04 -0000 On Fri, 29 May 2009, Larry Rosenman wrote: > On Fri, 29 May 2009, Larry Rosenman wrote: >> >> Ok, it just crashed. Unfortunately, I'm at work and the box is at home. >> >> I did have my script running every minute of that entire boot. >> >> What I saw was a full backup running, and then we started paging, and then >> the backup jobs got pager errors, and were killed. >> >> I'm not sure what else went on, so I restarted the bacula daemons that >> got killed, and was in the bacula console when it died. >> >> I'll see if I can get a cell-phone camera shot of the console. >> >> I'll also tar up the vmstat outputs and put them on my web server. >> >> What other forensics should I get? Bear in mind the system is probably >> locked up with no dump taken :( > One other "interesting" thing is the IPMI card seems to also be locked up. > I.E. if I try to login to it, it just hangs after giving id/pw. > Ok, I let the IPMI sit, and it eventually showed me the console. I took a screenshot, and then reset the box. I did get a textdump but it didn't run my scripts for the ddb stuff. Here is links to what I do have: http://www.lerctr.org/~ler/ZFS_CRASH/ $ ls -l total 5017 -rw-r--r-- 1 ler ler 5051465 May 29 13:14 crash.stats.tar.gz -rw-r--r-- 1 ler ler 253 May 29 13:18 index.html -rw-r--r-- 1 ler ler 77004 May 29 13:16 ipmiconsole.png -rw-r--r-- 1 ler ler 70656 May 29 13:14 textdump.tar.6 $ What else can I supply? The crash.stats.tar.gz contains the minute by minute output of the followuing script for the entire boot: ----- #!/bin/sh DATE=`date +%Y%m%d.%H%M%S` (echo "Uptime:";uptime;echo "vmstat -m:";vmstat -m echo "vmstat -z:";vmstat -z) >/home/ler/stats/${DATE}.stats ----- -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893