From owner-freebsd-stable@FreeBSD.ORG Thu Jun 10 18:17:05 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C7121065673 for ; Thu, 10 Jun 2010 18:17:05 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id 750F08FC1D for ; Thu, 10 Jun 2010 18:17:04 +0000 (UTC) Received: from omta07.emeryville.ca.mail.comcast.net ([76.96.30.59]) by qmta13.emeryville.ca.mail.comcast.net with comcast id UCPl1e0041GXsucADJH40s; Thu, 10 Jun 2010 18:17:04 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta07.emeryville.ca.mail.comcast.net with comcast id UJH31e00G3S48mS8UJH3Eo; Thu, 10 Jun 2010 18:17:04 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 2EE9F9B418; Thu, 10 Jun 2010 11:17:03 -0700 (PDT) Date: Thu, 10 Jun 2010 11:17:03 -0700 From: Jeremy Chadwick To: Robin Sommer Message-ID: <20100610181703.GA80162@icarus.home.lan> References: <20100610162918.GA23022@icir.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100610162918.GA23022@icir.org> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-stable@freebsd.org Subject: Re: File system trouble with ICH9 controller X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jun 2010 18:17:05 -0000 On Thu, Jun 10, 2010 at 09:29:19AM -0700, Robin Sommer wrote: > I'm running 8.0-RELEASE-p2 (amd64) on a larger number of Supermicro > SBI-7425C-T3 blades. Each of the blades has 2 x 500GB disks striped > into a single volume via the on-board ICH9 RAID controller. > > However, after running fine for a while (days), the blades crash > eventually with file system problems such as the one below. > Initially I thought that must be a bad disk, but by now 5 different > blades have shown similar problems so I'm suspecting some OS issue. > > Has anybody seen something similar before? Could this be an > incompatibility with the RAID controller (I haven't found much > recent on Google but there are a number of older threads indicating > that it might not be well supported. Not sure though whether those > still apply). > > Jun 9 10:00:02 blade19 kernel: ar0s1a[WRITE(offset=704187858944, length=114688)]error = 5 > Jun 9 10:00:02 blade19 kernel: g_vfs_done():ar0s1a[WRITE(offset=704188219392, length=131072)]error = 5 > Jun 9 10:00:02 blade19 kernel: g_vfs_done():ar0s1a[WRITE(offset=704188891136, length=114688)]error = 5 > Jun 9 10:00:02 blade19 kernel: g_vfs_done():ar0s1a[WRITE(offset=704189382656, length=114688)]error = 5 > Jun 9 10:00:02 blade19 kernel: g_vfs_done():ar0s1a[WRITE(offset=704189743104, length=131072)] > Jun 9 10:00:02 blade19 kernel: error = 5 You're using Intel MatrixRAID. Please stop[1]; you're living dangerously. The messages your kernel is spitting out could indicate a lot of different things. Tracking it down will take time. So let's start wit this: 1) Provide output from "gpart show ar0s1". I'm curious about something (likely a red herring, but I want to see). 2) Install sysutils/smartmontools and run "smartctl -a /dev/adXX" on each of the disks which make up the RAID array. I believe FreeBSD can see the disks associated with the array (meaning you should have a few adXX disks, in addition to an ar0 entry). I can help you decode the output, to see if any of the disks have actual problems that indicate they could be going bad. 3) Remove use of MatrixRAID. Alternatives include ccd, gstripe, gvinum, or ZFS. I would recommend ZFS if you ran RELENG_8 instead of -RELEASE, system was amd64, and has at least 4GB RAM. Remove use of MatrixRAID first, then see if the problem goes away. 4) If the problem still happens after this, there should be developers who can help diagnose the problem. Keeping MatrixRAID out of the picture helps greatly. More details: you might consider these opinions, but they're based on personal experience (I've dealt many a time with MatrixRAID). The problem is not with the ICH9, given that most of our systems are Supermicro (not blades but that doesn't matter) and use ICH9 with AHCI (both with and without ahci.ko). Intel ICHxx and ESBx controllers are heavily tested on FreeBSD, both by users and developers. [1]: http://en.wikipedia.org/wiki/Intel_Matrix_RAID -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |