From owner-freebsd-current@FreeBSD.ORG Sun Jul 19 19:04:07 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C8AE106564A; Sun, 19 Jul 2009 19:04:07 +0000 (UTC) (envelope-from ianjhart@ntlworld.com) Received: from mtaout01-winn.ispmail.ntl.com (mtaout01-winn.ispmail.ntl.com [81.103.221.47]) by mx1.freebsd.org (Postfix) with ESMTP id 0C9388FC0A; Sun, 19 Jul 2009 19:03:34 +0000 (UTC) (envelope-from ianjhart@ntlworld.com) Received: from aamtaout04-winn.ispmail.ntl.com ([81.103.221.35]) by mtaout01-winn.ispmail.ntl.com (InterMail vM.7.08.04.00 201-2186-134-20080326) with ESMTP id <20090719190333.DERV6742.mtaout01-winn.ispmail.ntl.com@aamtaout04-winn.ispmail.ntl.com>; Sun, 19 Jul 2009 20:03:33 +0100 Received: from cpc1-cove3-0-0-cust909.sol2.cable.ntl.com ([86.20.31.142]) by aamtaout04-winn.ispmail.ntl.com (InterMail vG.2.02.00.01 201-2161-120-102-20060912) with ESMTP id <20090719190333.OUTN22934.aamtaout04-winn.ispmail.ntl.com@cpc1-cove3-0-0-cust909.sol2.cable.ntl.com>; Sun, 19 Jul 2009 20:03:33 +0100 X-Virus-Scanned: amavisd-new at omega.private.lan Received: from localhost (localhost [127.0.0.1]) by cpc1-cove3-0-0-cust909.sol2.cable.ntl.com (8.14.3/8.14.3) with ESMTP id n6JJ3RRr017922; Sun, 19 Jul 2009 20:03:27 +0100 (BST) (envelope-from ianjhart@cpc1-cove3-0-0-cust909.sol2.cable.ntl.com) Received: from gamma.private.lan (gamma.private.lan [192.168.0.12]) by webmail.private.lan (Horde Framework) with HTTP; Sun, 19 Jul 2009 20:03:27 +0100 Message-ID: <20090719200327.848399266qvz2jgg@webmail.private.lan> Date: Sun, 19 Jul 2009 20:03:27 +0100 From: Ian J Hart To: Kip Macy References: <20090624153442.137934uzyotkb5og@10.248.192.16> <20090707210345.13681mi2dwvan78k@webmail.private.lan> <3c1674c90907071412t346b1591rfecfae22bb60a8f5@mail.gmail.com> <20090708122417.14619w86w7wfu4ms@10.248.192.16> <3c1674c90907081313q31c48953u8c4bde5d0e156acf@mail.gmail.com> In-Reply-To: <3c1674c90907081313q31c48953u8c4bde5d0e156acf@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) 4.3.3 / FreeBSD-7.2 X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on cpc1-cove3-0-0-cust909.sol2.cable.ntl.com X-Cloudmark-Analysis: v=1.0 c=1 a=ERehf_AEJYYA:10 a=6I5d2MoRAAAA:8 a=szcsTo1WmaNtUgNiKykA:9 a=gND7lsdQPnmvpj85HeoA:7 a=qQD79z1B9CsrlDau0RXBtGhtLS0A:4 a=SV7veod9ZcQA:10 Cc: freebsd-current@freebsd.org, Ian J Hart Subject: Re: zpool scrub errors on 3ware 9550SXU X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2009 19:04:07 -0000 Quoting Kip Macy : > The reason why the results on 7-STABLE are so important is that it is > now running the exact same version of ZFS as HEAD. So if you aren't > seeing checksum errors on 7-STABLE then the problem lies elsewhere in > the storage stack. There are so many things that could cause this that > I won't even speculate. > > Thanks for the information. I'll try to figure out how to narrow it down. > > -Kip > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > I realise this is not going anywhere unless someone else gets the same error (preferably on different hardware) but FWIW I still get the same error on 8.0-BETA2. I only did one run so far but it took 3 hours rather than 40 minutes. Offset is roughly twice the typical value, although again, this may not be significant. Any commits in the storage stack? I have a new mobo/cpu/ram/raid card on order which I'll be able to test the week after next. The old board etc will go into a system with a set of 500GB disks, so I'll get two sets of tests for the price of one. Very soon after that I have to put both boxes into production. Jul 19 19:12:54 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da14 offset=624129919488 size=22016 Jul 19 19:12:54 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da11 offset=624129591808 size=21504 Jul 19 19:12:54 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da9 offset=624130029568 size=22016 Jul 19 19:13:06 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da4 offset=624318266368 size=21504 Jul 19 19:13:06 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da15 offset=624317789696 size=22016 Jul 19 19:13:08 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da14 offset=624339560448 size=22016 Jul 19 19:13:10 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da10 offset=624369130496 size=22016 Jul 19 19:13:10 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da11 offset=624369171456 size=22016 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da0 offset=624426701824 size=22016 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da1 offset=624426701824 size=22016 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da2 offset=624426701824 size=22016 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da3 offset=624426701824 size=22016 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da4 offset=624426701824 size=22016 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da5 offset=624426701824 size=22016 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da6 offset=624426701824 size=21504 Jul 19 19:13:14 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da7 offset=624426701824 size=21504 Jul 19 19:13:16 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da10 offset=624469054464 size=21504 Jul 19 19:13:16 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da2 offset=624469552640 size=21504 Jul 19 19:13:16 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da3 offset=624469552640 size=21504 Jul 19 19:13:37 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da5 offset=626105954816 size=22016 Jul 19 19:13:53 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da2 offset=626296076800 size=22016 Jul 19 19:13:55 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da3 offset=626306811904 size=22016 Jul 19 19:13:55 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status bit(s): status reg = 0xd00000 Unexpected bits: [PCI_ABRT,Q_ERR,PCI_PERR] Jul 19 19:13:55 data kernel: twa0: parity error: clearing... Re-seat/move/replace card: (0x49435000: 0x1303): PCI parity error: clearing... Re-seat/move/replace card: status reg = 0x30db64a7 [MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INTR,HOS T_INTR,PCI_ABRT,Q_ERR,PCI_PERR] Jul 19 19:13:55 data kernel: twa0: abort: clearing... : (0x49435000: 0x1304): PCI abort: clearing... : status reg = 0x30db64a7 [MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INTR,HOST_INTR,PCI_ABRT,Q_ERR,PCI_PERR] Jul 19 19:13:55 data kernel: twa0: troller queue error: clearing... : (0x6E6F4300: 0x1305): Controller queue error: clearing... : status reg = 0x30db64a7 [MC_RDY,RESP_Q_EMPTY,RESP_INTR,CMD_INTR,HOST_INTR,PCI_ABRT,Q_ERR,PCI_PERR] Jul 19 19:13:55 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da6 offset=626306920960 size=22016 Jul 19 19:13:55 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da7 offset=626306920960 size=22016 Jul 19 19:13:56 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da7 offset=626316084224 size=16384 Jul 19 19:13:56 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da6 offset=626317622784 size=22016 Jul 19 19:13:56 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da13 offset=626316797440 size=22016 Jul 19 19:14:12 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da15 offset=626486498304 size=21504 Jul 19 19:14:12 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da2 offset=626486997504 size=21504 Jul 19 19:14:12 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da3 offset=626486756352 size=22016 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da2 offset=626830958592 size=21504 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da3 offset=626830980096 size=17920 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da12 offset=626830480384 size=22016 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da13 offset=626830480384 size=22016 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da14 offset=626830480384 size=22016 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da15 offset=626830480384 size=22016 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da8 offset=626830480896 size=22016 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da9 offset=626830480896 size=22016 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da10 offset=626830480896 size=21504 Jul 19 19:14:43 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da11 offset=626830480896 size=21504 Jul 19 19:14:44 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da7 offset=626839974400 size=17920 Jul 19 19:14:44 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da9 offset=626839497216 size=17920 Jul 19 19:14:44 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da8 offset=626839429120 size=18944 Jul 19 19:14:47 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da12 offset=626870235648 size=21504 Jul 19 19:14:47 data kernel: twa0: ERROR: (0x16: 0x1302): Unexpected status bit(s): status reg = 0x200000 Unexpected bits: [MC_ERR,] Jul 19 19:14:47 data kernel: twa0: or! : (0x72726520: 0x1307): Micro-controller error! : status reg = 0x97273372 [CMD_Q_EMPTY,MC_RDY,RESP_INTR,CMD_INTR,ATTN_INTR,MC_ERR,] Jul 19 19:14:56 data root: ZFS: checksum mismatch, zpool=tank path=/dev/da4 offset=626964436992 size=21504 Cheers -- ian j hart ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.