From owner-freebsd-fs@FreeBSD.ORG Mon Jan 7 19:22:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7861D16A46D; Mon, 7 Jan 2008 19:22:05 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 1078413C448; Mon, 7 Jan 2008 19:22:04 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id m07JLwpd044625; Mon, 7 Jan 2008 20:21:58 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (cicely12.cicely.de [10.1.1.14]) by cicely5.cicely.de (8.13.4/8.13.4) with ESMTP id m07JLi1W078037 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 7 Jan 2008 20:21:45 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (localhost [127.0.0.1]) by cicely12.cicely.de (8.13.4/8.13.3) with ESMTP id m07JLiUg076746; Mon, 7 Jan 2008 20:21:44 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: (from ticso@localhost) by cicely12.cicely.de (8.13.4/8.13.3/Submit) id m07JLibM076745; Mon, 7 Jan 2008 20:21:44 +0100 (CET) (envelope-from ticso) Date: Mon, 7 Jan 2008 20:21:44 +0100 From: Bernd Walter To: Tz-Huan Huang Message-ID: <20080107192143.GC76422@cicely12.cicely.de> References: <477B16BB.8070104@freebsd.org> <20080102070146.GH49874@cicely12.cicely.de> <477B8440.1020501@freebsd.org> <200801031750.31035.peter.schuller@infidyne.com> <477D16EE.6070804@freebsd.org> <20080103171825.GA28361@lor.one-eyed-alien.net> <6a7033710801061844m59f8c62dvdd3eea80f6c239c1@mail.gmail.com> <20080107135925.GF65134@cicely12.cicely.de> <6a7033710801070917w4b453f10l7115bd9fe3a53a1b@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6a7033710801070917w4b453f10l7115bd9fe3a53a1b@mail.gmail.com> X-Operating-System: FreeBSD cicely12.cicely.de 5.4-STABLE alpha User-Agent: Mutt/1.5.9i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, BAYES_00=-2.599 autolearn=ham version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on cicely12.cicely.de Cc: freebsd-fs@freebsd.org, Brooks Davis , ticso@cicely.de Subject: Re: ZFS i/o errors - which disk is the problem? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2008 19:22:05 -0000 On Tue, Jan 08, 2008 at 01:17:03AM +0800, Tz-Huan Huang wrote: > 2008/1/7, Bernd Walter : > > The data is corrupted by controller and/or disk subsystem. > > You have no other data sources for the broken data, so it is lost. > > The only garantied way is to get it back from backup. > > Maybe older snapshots/clones are still readable - I don't know. > > Nevertheless data is corrupted and that's the purpose for alternative > > data sources such as raidz/mirror and at last backup. > > You shouldn't have ignored those errors at first, because you are > > running with faulty hardware. > > Without ZFS checksumming the system would just process the broken > > data with unpredictable results. > > If all those errors are fresh then you likely used a broken RAID > > controller below ZFS, which silently corrupted syncronity and then > > blow when disk state changed. > > Unfortunately many RAID controllers are broken and therefor useless. > > Hi, > > Thank you very much for your answer. > > We have run the self-test for all raid controllers and they all reported ok. > Do you mean that many raid controllers are broken (buggy?) even if the > self-test is passed? If all the disks are pass-through to the zfs, is > it the safe > way to use the buggy controllers? If the controller is that buggy that even their own self test fails it would be even worse. But they can't test if they corrupted data - they just can test the current state and the syncronisation. They could do massive read/write tests, but this would mean overwriting the current data. If you export single disks ZFS can handle this using the redundancy, which means if it encounters an error it can use the other disks to recover the data. In your case ZFS doesn't know about redundancy and your controller returns faulty data, so there is no try to recover. You RAID controller can't help either because it isn't aware of it's own mess, since it is not using CRC itself and even then it could also be a case were the data gets corrupted while transmitting into the host or from the host, or even a driver problem. But relying on ZFS is not a safe way either, just a bit less critical. Safe is only to not use buggy controller at all. The good point with ZFS CRC is that you are aware of the problem even in case of corrupted file data. In your case unfortunately it seem to have broken too much. -- B.Walter http://www.bwct.de http://www.fizon.de bernd@bwct.de info@bwct.de support@fizon.de