From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 15:25:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46ED8106566C for ; Fri, 10 Sep 2010 15:25:47 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id E66F18FC15 for ; Fri, 10 Sep 2010 15:25:46 +0000 (UTC) Received: from omta20.westchester.pa.mail.comcast.net ([76.96.62.71]) by QMTA11.westchester.pa.mail.comcast.net with comcast id 50RX1f00C1YDfWL5B3RnJY; Fri, 10 Sep 2010 15:25:47 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta20.westchester.pa.mail.comcast.net with comcast id 53Rl1f00V3LrwQ23g3Rmpq; Fri, 10 Sep 2010 15:25:47 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7A2D59B423; Fri, 10 Sep 2010 08:25:44 -0700 (PDT) Date: Fri, 10 Sep 2010 08:25:44 -0700 From: Jeremy Chadwick To: a.smith@ukgrid.net Message-ID: <20100910152544.GA14636@icarus.home.lan> References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 15:25:47 -0000 On Fri, Sep 10, 2010 at 03:55:10PM +0100, a.smith@ukgrid.net wrote: > Quoting Jeremy Chadwick : > > >What we're looking for is something that resembles what's shown here > >(using this as an example): > > > >http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00413.html > > > >The KDB and KDB_TRACE options should result in a backtrace automatically > >being shown when the system panics. Are you absolutely certain your > >kernel config has the options Andriy provided, and you rebuilt + > >reinstalled the kernel? > > > > Hi Jeremy, > > well Ive not had to built a custom kernel before, but Ive done the > make buildkernel and make installkernel and all seems to have gone > thro as expected. Here is the evidence from after a reboot: > > > # pwd > /usr/src/sys/amd64/conf > # grep KDB ZFSDEBUG > options KDB > options KDB_TRACE > [root@kappa /usr/src/sys/amd64/conf]# uname -a > FreeBSD kappa.ukgrid.net 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Fri Sep > 10 12:30:24 BST 2010 > root@kappa.ukgrid.net:/usr/obj/usr/src/sys/ZFSDEBUG amd64 > > Any ideas? Not particularly. Sure looks like you should be seeing a backtrace to me. I'd recommend following Andriy's advice of adding "options DDB" to your kernel configuration, rebuild + install kernel + reboot. When the panic happens, you will be dropped to a db> prompt at the console. You should type "trace" to get a backtrace at that point. Something you might also try/do before the kernel rebuild/reboot: If you have a swap slice defined (not on ZFS!), I would would advocate setting dumpdev="auto" in your /etc/rc.conf prior to the reboot. This will allow you to do a post-crash analysis. You'll need to make sure that your /var filesystem has enough space to hold an entire memory dump (e.g. if your system RAM is 8GB, ensure you have at least 8GB free on /var). Next, depending on the state of the system (particularly the controller driver and ATA subsystem), the following may or may not work for you. After the above db> commands, type "call doadump", which should dump all the contents of kernel memory to swap. The next time the system starts up, savecore(8) should automatically extract the crash contents from swap and save them in /var/crash -- hopefully without the system crashing in the middle of writing the contents. :-) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |