From owner-freebsd-stable@FreeBSD.ORG Mon May 30 10:33:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87CE0106564A for ; Mon, 30 May 2011 10:33:51 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.emeryville.ca.mail.comcast.net (qmta08.emeryville.ca.mail.comcast.net [76.96.30.80]) by mx1.freebsd.org (Postfix) with ESMTP id 70C4C8FC08 for ; Mon, 30 May 2011 10:33:51 +0000 (UTC) Received: from omta21.emeryville.ca.mail.comcast.net ([76.96.30.88]) by qmta08.emeryville.ca.mail.comcast.net with comcast id pmZ51g0041u4NiLA8mZqud; Mon, 30 May 2011 10:33:50 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta21.emeryville.ca.mail.comcast.net with comcast id pmZq1g0041t3BNj8hmZqUt; Mon, 30 May 2011 10:33:51 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5BA7A102C19; Mon, 30 May 2011 03:33:49 -0700 (PDT) Date: Mon, 30 May 2011 03:33:49 -0700 From: Jeremy Chadwick To: Olaf Seibert Message-ID: <20110530103349.GA73825@icarus.home.lan> References: <20110530093546.GX6733@twoquid.cs.ru.nl> <20110530101051.GA49825@twoquid.cs.ru.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110530101051.GA49825@twoquid.cs.ru.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: ZFS I/O errors X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2011 10:33:51 -0000 On Mon, May 30, 2011 at 12:10:51PM +0200, Olaf Seibert wrote: > On Mon 30 May 2011 at 11:35:46 +0200, Olaf Seibert wrote: > > How do I identify which files it is listing here? On Solaris anyway, "zpool status -v" is supposed to show this. Occasionally it shows something like <0xXX>:, rather than a path/filename, and on FreeBSD it appears to behave the same. Per your own output: tank/vol-fourquid-1:<0x0> tank/vol-fourquid-1:<0xc8190da> I'm not sure why this didn't actually map to a filename on the system however. I've never quite understood what the hexadecimal values shown represent (I have ideas but it'd be useful to know what they meant). > The nighly 'find' has found the files for me. It is actually a bunch of > directories, that were likely in use when the crash occurred. > > They give an "interesting" error when you try to ls them: > > find: /tank/vol-fourquid-1/evadh/CLEF-IP11/PARSED_CORPUS/EP/000000/45/97: Illegal byte sequence Yes, and this is what error 86 was in the very first line of your kernel output: May 30 10:38:28 fourquid root: ZFS: zpool I/O failure, zpool=tank error=86 $ perror 86 Illegal byte sequence > The file system is compressed, that may be the reason it can identify > "illegal byte sequence"s. > > It isn't even possible to rm -r the directories, or even to mv them... > > (fortunately the standard trick works: move the parent directory > instead, create a new one in its old place, and move the old, good, > contents back). > > but now I seem to be left with some directories (elsewhere) that I still > can't remove... I sincerely hope your "zpool scrub" addresses these problems. Otherwise I hope you have backup and can recreate the pool (zpool destroy, etc.). Try running without compression and see if that improves things. It's important to note that the I/O errors shown happened on "random disks" (meaning more than just one device). What you didn't disclose is what the disks are attached to. "camcontrol devlist -v" would have been a good start, followed by any details of controller/driver/etc. you're using. Possibly it's an underlying (silent) storage driver bug. Finally, and leaving the most important point for last: you didn't state what FreeBSD version you're using, and ceased to provide uname -a output (to see kernel build date, etc.). It matters greatly. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |