From owner-freebsd-stable@FreeBSD.ORG Mon May 30 11:09:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7EC781065680 for ; Mon, 30 May 2011 11:09:50 +0000 (UTC) (envelope-from O.Seibert@cs.ru.nl) Received: from kookpunt.science.ru.nl (kookpunt.science.ru.nl [131.174.30.61]) by mx1.freebsd.org (Postfix) with ESMTP id 211A58FC19 for ; Mon, 30 May 2011 11:09:49 +0000 (UTC) Received: from twoquid.cs.ru.nl (twoquid.cs.ru.nl [131.174.142.38]) by kookpunt.science.ru.nl (8.13.7/5.31) with ESMTP id p4UB9kg6010961; Mon, 30 May 2011 13:09:46 +0200 (MEST) Received: by twoquid.cs.ru.nl (Postfix, from userid 4100) id EA4D02E03C; Mon, 30 May 2011 13:09:46 +0200 (CEST) Date: Mon, 30 May 2011 13:09:46 +0200 From: Olaf Seibert To: Jeremy Chadwick Message-ID: <20110530110946.GC6733@twoquid.cs.ru.nl> References: <20110530093546.GX6733@twoquid.cs.ru.nl> <20110530101051.GA49825@twoquid.cs.ru.nl> <20110530103349.GA73825@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110530103349.GA73825@icarus.home.lan> User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam-Score: -1.799 () ALL_TRUSTED,BAYES_50 X-Scanned-By: MIMEDefang 2.63 on 131.174.30.61 Cc: freebsd-stable@freebsd.org, Olaf Seibert Subject: Re: ZFS I/O errors X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2011 11:09:50 -0000 On Mon 30 May 2011 at 03:33:49 -0700, Jeremy Chadwick wrote: > On Mon, May 30, 2011 at 12:10:51PM +0200, Olaf Seibert wrote: > I'm not sure why this didn't actually map to a filename on the system > however. I've never quite understood what the hexadecimal values shown > represent (I have ideas but it'd be useful to know what they meant). The scrub is starting to add some filenames to the list. So far they are two filenames in snapshots (where current versions of the file have been modified since then). > Try running without compression and see if that improves things. That sounds like a good idea. My theory so far is that it ran out of memory while compressing, with incorrect compressed data written to the disk. > It's important to note that the I/O errors shown happened on "random > disks" (meaning more than just one device). What you didn't disclose is > what the disks are attached to. "camcontrol devlist -v" would have been > a good start, followed by any details of controller/driver/etc. you're > using. Possibly it's an underlying (silent) storage driver bug. They are connected to an Areca host adapter: arcmsr0: mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 17 at device 14.0 on pci7 ARECA RAID ADAPTER0: Driver Version 1.20.00.19 2010-11-11 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06 arcmsr0: [ITHREAD] da0 at arcmsr0 bus 0 scbus0 target 0 lun 0 da1 at arcmsr0 bus 0 scbus0 target 0 lun 1 da2 at arcmsr0 bus 0 scbus0 target 0 lun 2 da3 at arcmsr0 bus 0 scbus0 target 0 lun 3 da4 at arcmsr0 bus 0 scbus0 target 0 lun 4 da5 at arcmsr0 bus 0 scbus0 target 0 lun 5 pass6 at arcmsr0 bus 0 scbus0 target 16 lun 0 camcontrol devlist -v: scbus0 on arcmsr0 bus 0: at scbus0 target 0 lun 0 (pass0,da0) at scbus0 target 0 lun 1 (pass1,da1) at scbus0 target 0 lun 2 (pass2,da2) at scbus0 target 0 lun 3 (pass3,da3) at scbus0 target 0 lun 4 (pass4,da4) at scbus0 target 0 lun 5 (pass5,da5) at scbus0 target 16 lun 0 (pass6) <> at scbus0 target -1 lun -1 () scbus-1 on xpt0 bus 0: <> at scbus-1 target -1 lun -1 (xpt0) > Finally, and leaving the most important point for last: you didn't state > what FreeBSD version you're using, and ceased to provide uname -a output > (to see kernel build date, etc.). It matters greatly. Sorry. Here is it: FreeBSD fourquid.cs.ru.nl 8.2-RELEASE FreeBSD 8.2-RELEASE #3: Tue Apr 19 13:02:11 CEST 2011 root@fourquid.cs.ru.nl:/usr/obj/usr/src/sys/FOURQUID amd64 The kernel is built from unmodified 8.2-RELEASE sources, but I adapted the configuration a bit to throw out irrelevant hardware support (this does make a small difference): -r-xr-xr-x 1 root wheel 12612347 Apr 19 12:27 /boot/GENERIC/kernel* -r-xr-xr-x 1 root wheel 7031915 Apr 19 13:02 /boot/kernel/kernel* The real original reason to make a custom kernel was to add the SCSI passthrough device (needed by smartctl) , but istr that that is included in GENERIC now. Thanks so far, -Olaf. -- Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe));